FoolFuuka/Install/Debian: Difference between revisions
Antonizoon (talk | contribs) |
|||
(19 intermediate revisions by 3 users not shown) | |||
Line 51: | Line 51: | ||
== Set required php.ini vars. == | == Set required php.ini vars. == | ||
{{hc|sudo nano /etc/php5/fpm/php.ini|<nowiki> | {{hc|sudo nano /etc/php5/fpm/php.ini|<nowiki> | ||
date.timezone = UTC | |||
upload_max_filesize = 32M | |||
post_max_size = 32M | |||
</nowiki>}} | </nowiki>}} | ||
Line 101: | Line 101: | ||
index index.php index.html index.htm; | index index.php index.html index.htm; | ||
client_max_body_size 32M; # set to the same as max PHP upload size in HHVM | |||
location / { | location / { | ||
try_files $uri $uri/ /index.php$is_args$args; | try_files $uri $uri/ /index.php$is_args$args; | ||
} | } | ||
location ~ \.php$ { | location ~ \.php$ { | ||
fastcgi_pass unix:/var/run/php5-fpm.sock; | fastcgi_pass unix:/var/run/php5-fpm.sock; | ||
Line 109: | Line 111: | ||
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; | fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; | ||
include fastcgi_params; | include fastcgi_params; | ||
} | } | ||
Line 118: | Line 119: | ||
<pre>sudo service nginx restart</pre> | <pre>sudo service nginx restart</pre> | ||
== Create the database and install Foolfuuka user accounts. == | == Create the database and install Foolfuuka user accounts. == | ||
<pre>sudo mysql_secure_installation</pre> | <pre>sudo mysql_secure_installation</pre> | ||
Line 136: | Line 135: | ||
{{hc|sudo nano /etc/mysql/my.cnf|<nowiki> | {{hc|sudo nano /etc/mysql/my.cnf|<nowiki> | ||
[client] | |||
default-character-set = utf8mb4 | |||
[mysql] | |||
default-character-set = utf8mb4 | |||
[mysqld] | |||
character-set-client-handshake = FALSE | |||
character-set-server = utf8mb4 | |||
collation-server = utf8mb4_general_ci | |||
</nowiki>}} | </nowiki>}} | ||
== Restart MariaDB == | == Restart MariaDB == | ||
Line 167: | Line 167: | ||
{{hc|sudo nano /etc/security/limits.conf| | {{hc|sudo nano /etc/security/limits.conf| | ||
* soft nofile 63545 | |||
* hard nofile 63545 | |||
* soft nproc 63545 | |||
* hard nproc 63545 | |||
}} | }} | ||
Line 176: | Line 176: | ||
{{hc|sudo nano /etc/mysql/my.cnf|<nowiki> | {{hc|sudo nano /etc/mysql/my.cnf|<nowiki> | ||
# Put this after [mysqld] | |||
open-files-limit=40000 | |||
</nowiki>}} | </nowiki>}} | ||
Line 183: | Line 183: | ||
sudo reboot</pre> | sudo reboot</pre> | ||
<pre>mysql -u root -p | <pre>mysql -u root -p | ||
set global default_storage_engine=TokuDB; | |||
SET GLOBAL log_bin_trust_function_creators = 1;</pre> | |||
Now, Install FoolFuuka at the website using foolfuuka credentials | Now, Install FoolFuuka at the website using foolfuuka credentials | ||
Line 201: | Line 201: | ||
Insert the following JSON into that file: | Insert the following JSON into that file: | ||
<pre> | <pre> | ||
{"settings": { | {"settings": { | ||
"dumperEngine": "DumperJSON", | "dumperEngine": "DumperJSON", | ||
Line 306: | Line 306: | ||
} | } | ||
}} | }} | ||
</pre> | |||
== Start asagi (There is a better way to do this). == | == Start asagi (There is a better way to do this). == | ||
Line 328: | Line 329: | ||
The installer is quite self explanatory, just make sure to remove the Boards prefix. | The installer is quite self explanatory, just make sure to remove the Boards prefix. | ||
Also make sure to install imagemagick and set the path in the config to it. | |||
{{Note|In the installer one should have FoolFuuka table prefix set (default is fine ff_). But after installing one needs to set BOARD PREFIX to blank since asagi doesn't have any. FoolFuuka table prefix != board table prefix. Here's [https://github.com/p0wer0n/FoolFrame/commit/3368e24dc9d89b881460f8c876ddb91c78b5b955 a code change that does it.]}} | |||
== Add boards, Configure Sphinx == | == Add boards, Configure Sphinx == | ||
http://IP/admin | To configure Sphinx Search, take the following links: | ||
http://IP/admin -> Search -> Generate Config. | |||
http://IP/admin -> Boards -> Edit -> Check Use SphinxSearch as search engine. | |||
On the Search panel, change the working directory (on Ubuntu) to the following: | |||
<pre> | |||
/var/lib/sphinxsearch | |||
</pre> | |||
=== Supplement Sphinxsearch === | |||
<pre> | |||
cd /var/run | |||
sudo mkdir sphinxsearch | |||
sudo chmod -Rf 777 sphinxsearch | |||
</pre> | |||
=== Start Sphinxsearch === | |||
<pre> | |||
sudo nano /etc/default/sphinxsearch | |||
</pre> | |||
Change the following line from | |||
<pre>START=no</pre> | |||
to | |||
<pre>START=yes</pre> | |||
Save and run the following command | |||
<pre>sudo /etc/init.d/sphinxsearch start</pre> | |||
Right-click and copy this config out of Foolfuuka and add your MySQL database information for asagi. | |||
Change the following line from | |||
<pre>pid_file = /var/lib/sphinxsearch/searchd.pid</pre> | |||
to | |||
<pre>pid_file = /var/run/sphinxsearch/searchd.pid </pre> | |||
Then, paste this to <code>/etc/sphinxsearch/sphinx.conf</code> | |||
If anything errors, kill any running instance of asagi and restart it | |||
=== Build the indexes === | === Build the indexes === | ||
Line 341: | Line 381: | ||
=== Configure Crontab === | === Configure Crontab === | ||
<pre> | |||
sudo nano /etc/crontab | |||
</pre> | |||
<pre> | <pre> | ||
@hourly sudo -u sphinxsearch /usr/bin/indexer --rotate *_delta > /dev/null | @hourly sudo -u sphinxsearch /usr/bin/indexer --rotate *_delta > /dev/null | ||
Line 347: | Line 389: | ||
@monthly sudo -u sphinxsearch /usr/bin/indexer --rotate *_ancient > /dev/null | @monthly sudo -u sphinxsearch /usr/bin/indexer --rotate *_ancient > /dev/null | ||
</pre> | </pre> | ||
=== Adding Archives === | === Adding Archives === | ||
Line 357: | Line 400: | ||
== Extra configuration == | == Extra configuration == | ||
Depending on which distro you use, some configs might need to be changed. | Depending on which distro you use or how many posts you have, some configs might need to be changed. | ||
=== Fixing thumbnail creation === | === Fixing thumbnail creation === | ||
Line 366: | Line 409: | ||
</pre> | </pre> | ||
and make sure it is the same value in Admin panel -> general -> Imagemagick Convert | and make sure it is the same value in Admin panel -> general -> Imagemagick Convert | ||
=== For Huge Boards: Problem with random dropped posts/threads due to stats triggers === | |||
When Desuarchive was configured with a new server using the 4plebs fork of FoolFuuka and Asagi, it was a mystery why /a/ was behind in archiving. Normal posts were being bled somehow. | |||
But everything was running fine configuration-wise. No asagi errors. Nothing. And all other boards were great. | |||
However, ''just get rid of stats triggers/procedures'' and it should be fine. | |||
Asagi has triggers that update {{ic|_daily}} and {{ic|_users}} tables. FFuuka stats plugin reads those tables and make stats. Probably the {{ic|_users}} table was messed up/too large for /a/, so it slows down inserts because the update runs before it. | |||
4plebs is working on stats plugins that does it without them, since stats calc on insert isn't a smart idea. | |||
==== Procedure ==== | |||
I just removed the calls to stats procedures. Should work but check it out first. But should leave the procedures there (commented out?) so that you can go back. | |||
# Stop Asagi. | |||
# Run the code below: | |||
# Start Asagi. | |||
{{lc|<nowiki> | |||
DROP TRIGGER IF EXISTS "after_ins_a"; | |||
delimiter // | |||
CREATE TRIGGER "after_ins_a" AFTER INSERT ON "a" | |||
FOR EACH ROW | |||
BEGIN | |||
IF NEW.op = 1 THEN | |||
CALL create_thread_a(NEW.num, NEW.timestamp); | |||
END IF; | |||
CALL update_thread_a(NEW.thread_num); | |||
END// | |||
DROP TRIGGER IF EXISTS "after_del_a"// | |||
CREATE TRIGGER "after_del_a" AFTER DELETE ON "a" | |||
FOR EACH ROW | |||
BEGIN | |||
CALL update_thread_a(OLD.thread_num); | |||
IF OLD.op = 1 THEN | |||
CALL delete_thread_a(OLD.num); | |||
END IF; | |||
IF OLD.media_hash IS NOT NULL THEN | |||
CALL delete_image_a(OLD.media_id); | |||
END IF; | |||
END// | |||
delimiter ; | |||
</nowiki>|lang=sql}} |
Latest revision as of 00:53, 19 March 2017
This guide was written by ATC of desuarchive, and is designed for Debian/Ubuntu Linux servers.
Prepare things[edit]
apt-get update apt-get upgrade
Add the keys for HipHop Virtual Machine[edit]
sudo apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0x5a16e7281be7a449 sudo add-apt-repository "deb http://dl.hhvm.com/ubuntu $(lsb_release -sc) main"
Add repo for MariaDB 10.1[edit]
sudo apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xcbcb082a1bb943db sudo add-apt-repository 'deb [arch=amd64,i386] http://nyc2.mirrors.digitalocean.com/mariadb/repo/10.1/ubuntu trusty main'
Add repo for sphinxsearch[edit]
sudo add-apt-repository ppa:builds/sphinxsearch-stable
Then, Comment out repo in /etc/apt/sources.list.d/*
and add to end:
deb http://ppa.launchpad.net/builds/sphinxsearch-stable/ubuntu precise main deb-src http://ppa.launchpad.net/builds/sphinxsearch-stable/ubuntu precise main
sudo apt-get update
Install all the needed packages.[edit]
Nginx, HHVM, PHP5 Cli, PHP5 Curl, MariaDB, OpenJDK w/ compilier, Maven, Sphinxsearch
sudo apt-get install git nginx hhvm php5-fpm php5-cli php5-curl php5-mysqlnd mariadb-server libmariadbclient-dev mariadb-client imagemagick openjdk-7-jre openjdk-7-jdk openjdk-7-jre-lib maven sphinxsearch
Install composer for downloading FoolFuuka dependencies.[edit]
curl -sS https://getcomposer.org/installer | php
Move composer to bin[edit]
sudo mv composer.phar /usr/local/bin/composer
Install hhvm as a service so it launches at boot.[edit]
sudo update-rc.d hhvm defaults
Install hhvm with nginx[edit]
sudo /usr/share/hhvm/install_fastcgi.sh
Edit HHVM config to use a unix socket.[edit]
sudo nano /etc/hhvm/server.ini
Then remove the line starting with hhvm.server.port, and in its place add the following one:
hhvm.server.file_socket=/var/run/hhvm/hhvm.sock
Set required php.ini vars.[edit]
sudo nano /etc/php5/fpm/php.ini
date.timezone = UTC upload_max_filesize = 32M post_max_size = 32M
Restart HHVM[edit]
sudo service hhvm restart
Configure NGINX to use hhvm's new unix socket.[edit]
sudo nano /etc/nginx/hhvm.conf
In this file make sure that the fastcgi_pass directive points to the HHVM socket and looks like this:
fastcgi_pass unix:/var/run/hhvm/hhvm.sock;
Download FoolFuuka in home folder and install dependencies.[edit]
cd ~ composer create-project foolz/foolfuuka foolfuuka --prefer-source cd foolfuuka composer dump-autoload --optimize composer install
Install Foolfuuka Plugins[edit]
You may need to find the repo on their github if their site doesn't work: https://github.com/FoolCode
cd ~/foolfuuka/app/foolz/foolframe/plugins mkdir foolz cd foolz git clone https://github.com/FoolCode/foolframe-plugin-articles.git
cd ~/foolfuuka/app/foolz/foolfuuka/plugins mkdir foolz cd foolz git clone https://github.com/FoolCode/foolfuuka-plugin-board-statistics.git git clone https://github.com/FoolCode/foolfuuka-plugin-dice-roll.git git clone https://github.com/FoolCode/foolfuuka-plugin-quests.git
Set Permissions and Group for FoolFuuka Directory[edit]
sudo chmod -Rf 775 ~/foolfuuka sudo chown -Rf www-data:www-data ~/foolfuuka
Configure Nginx to host foolfuuka[edit]
sudo nano /etc/nginx/sites-enabled/default
root /home/ubuntu/foolfuuka/public; index index.php index.html index.htm; client_max_body_size 32M; # set to the same as max PHP upload size in HHVM location / { try_files $uri $uri/ /index.php$is_args$args; } location ~ \.php$ { fastcgi_pass unix:/var/run/php5-fpm.sock; fastcgi_index index.php; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; include fastcgi_params; }
Restart Nginx[edit]
sudo service nginx restart
Create the database and install Foolfuuka user accounts.[edit]
sudo mysql_secure_installation
mysql -uroot -p CREATE USER 'asagi'@'localhost' IDENTIFIED BY 'pass'; CREATE USER 'foolfuuka'@'localhost' IDENTIFIED BY 'pass'; CREATE DATABASE asagi DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_general_ci; CREATE DATABASE foolfuuka DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_general_ci; GRANT ALL PRIVILEGES ON `asagi`. * TO 'asagi'@'localhost'; GRANT ALL PRIVILEGES ON `asagi`. * TO 'foolfuuka'@'localhost'; GRANT ALL PRIVILEGES ON `foolfuuka`. * TO 'foolfuuka'@'localhost'; FLUSH PRIVILEGES; exit
edit MariaDB to use utf8mb4 for correct unicode processing[edit]
sudo nano /etc/mysql/my.cnf
[client] default-character-set = utf8mb4 [mysql] default-character-set = utf8mb4 [mysqld] character-set-client-handshake = FALSE character-set-server = utf8mb4 collation-server = utf8mb4_general_ci
Restart MariaDB[edit]
sudo service mysql restart
Disable transparent_hugepage
su -c 'echo never > /sys/kernel/mm/transparent_hugepage/enabled'
sudo nano /etc/default/grub
append transparent_hugepage=never
to GRUB_CMDLINE_LINUX_DEFAULT
, then save the file
sudo update-grub
Enable toku (Find and Uncomment the load module)
sudo nano /etc/mysql/conf.d/tokudb.cnf
Raise file limit
sudo nano /etc/security/limits.conf
* soft nofile 63545 * hard nofile 63545 * soft nproc 63545 * hard nproc 63545
Increase mysql file limit:
sudo nano /etc/mysql/my.cnf
# Put this after [mysqld] open-files-limit=40000
sudo service mysql stop sudo reboot
mysql -u root -p set global default_storage_engine=TokuDB; SET GLOBAL log_bin_trust_function_creators = 1;
Now, Install FoolFuuka at the website using foolfuuka credentials
Install and compile Asagi from source.[edit]
cd ~ git clone https://github.com/FoolCode/asagi.git cd asagi mvn package assembly:single mv target/asagi-0.4.0-SNAPSHOT-full.jar ./asagi.jar
Configure Asagi[edit]
cp asagi.json.example asagi.json nano asagi.json
Insert the following JSON into that file:
{"settings": { "dumperEngine": "DumperJSON", "sourceEngine": "YotsubaJSON", "boardSettings": { "default": { "engine": "Mysql", "database": "asagi", "host": "localhost", "username": "asagi", "password": "afj8a428dauwj", "charset": "utf8mb4", "path": "/home/ubuntu/foolfuuka/public/foolfuuka/boards/", "useOldDirectoryStructure": false, "webserverGroup": "www-data", "thumbThreads": 3, "mediaThreads": 0, "newThreadsThreads": 3, "deletedThreadsThresholdPage": 8, "refreshDelay": 30, "throttleAPI": true, "throttleURL": "api.4cdn.org", "throttleMillisec": 11100, "threadRefreshRate": 10, "pageSettings": [ {"delay": 30, "pages": [0, 1]}, {"delay": 500, "pages": [2, 3, 4, 5, 6, 7, 8]}, {"delay": 30, "pages": [9, 10]} ] }, "3": {}, "a": {}, "aco": {}, "adv": {}, "an": {}, "asp": {}, "b": {}, "biz": {}, "c": {}, "cgl": {}, "ck": {}, "cm": {}, "co": {}, "d": {}, "diy": {}, "e": {}, "f": {}, "fa": {}, "fit": {}, "g": {}, "gd": {}, "gif": {}, "h": {}, "hc": {}, "hm": {}, "his": {}, "hr": {}, "i": {}, "ic": {}, "int": {}, "jp": {}, "k": {}, "lgbt": {}, "lit": {}, "m": {}, "mlp": {}, "mu": {}, "n": {}, "news": {}, "o": {}, "out": {}, "p": {}, "po": {}, "pol": {}, "qa": {}, "qst": {}, "r": {}, "r9k": {}, "s": {}, "s4s": {}, "sci": {}, "soc": {}, "sp": {}, "t": {}, "tg": {}, "toy": {}, "trash": {}, "trv": {}, "tv": {}, "u": {}, "v": {}, "vg": {}, "vp": {}, "vr": {}, "w": {}, "wg": {}, "wsg": {}, "wsr": {}, "x": {}, "y": {} } }}
Start asagi (There is a better way to do this).[edit]
Maybe we should use systemd for it...
UseParallelGC adds Parallelization to the garbage collector, which really helps for Java.
screen java -XX:+UseParallelGC -XX:+UseParallelOldGC -verbose:gc -jar asagi.jar
ATC of Desuarchive prefers these parameters, which have two customizations.
- For the first parameter, Java by default will use 1/4th your system memory, so you can set your own hard memory roof to reduce usage or increase it. He used
6144m
.- If you aren't archiving more than 8 boards, this probably won't be an issue that you need to deal with.
- The permsize is about the permanent generation objects: depends on your heap size.
- This setting is probably related to how many boards are on Desuarchive, but Sunako doesn't had that sort of issue.
screen java -Xmx6144m -XX:+UseParallelGC -XX:+UseParallelOldGC -verbose:gc -XX:MaxPermSize=512m -jar asagi.jar
Now, Configure asagi as the database and remove the table prefix from FoolFuuka.
Install FoolFuuka[edit]
The installer is quite self explanatory, just make sure to remove the Boards prefix.
Also make sure to install imagemagick and set the path in the config to it.
Add boards, Configure Sphinx[edit]
To configure Sphinx Search, take the following links:
http://IP/admin -> Search -> Generate Config. http://IP/admin -> Boards -> Edit -> Check Use SphinxSearch as search engine. On the Search panel, change the working directory (on Ubuntu) to the following:
/var/lib/sphinxsearch
Supplement Sphinxsearch[edit]
cd /var/run sudo mkdir sphinxsearch sudo chmod -Rf 777 sphinxsearch
Start Sphinxsearch[edit]
sudo nano /etc/default/sphinxsearch
Change the following line from
START=no
to
START=yes
Save and run the following command
sudo /etc/init.d/sphinxsearch start
Right-click and copy this config out of Foolfuuka and add your MySQL database information for asagi.
Change the following line from
pid_file = /var/lib/sphinxsearch/searchd.pid
to
pid_file = /var/run/sphinxsearch/searchd.pid
Then, paste this to /etc/sphinxsearch/sphinx.conf
If anything errors, kill any running instance of asagi and restart it
Build the indexes[edit]
sudo indexer *_ancient --rotate
sudo indexer *_main --rotate
Configure Crontab[edit]
sudo nano /etc/crontab
@hourly sudo -u sphinxsearch /usr/bin/indexer --rotate *_delta > /dev/null @daily sudo -u sphinxsearch /usr/bin/indexer --rotate *_main > /dev/null @monthly sudo -u sphinxsearch /usr/bin/indexer --rotate *_ancient > /dev/null
Adding Archives[edit]
Inject your sql file into the asagi database (assuming that your sql file is called "dump.sql"):
mysql -u root -p asagi < dump.sql
Then, in the admin panel, simply add the board that corresponds to the board you injected.
Extra configuration[edit]
Depending on which distro you use or how many posts you have, some configs might need to be changed.
Fixing thumbnail creation[edit]
Foolfuuka might not be able to generate thumbnails for images to be posted. If so, check:
$ whereis convert convert: /usr/bin/convert /usr/share/man/man1/convert.1.gz
and make sure it is the same value in Admin panel -> general -> Imagemagick Convert
For Huge Boards: Problem with random dropped posts/threads due to stats triggers[edit]
When Desuarchive was configured with a new server using the 4plebs fork of FoolFuuka and Asagi, it was a mystery why /a/ was behind in archiving. Normal posts were being bled somehow.
But everything was running fine configuration-wise. No asagi errors. Nothing. And all other boards were great.
However, just get rid of stats triggers/procedures and it should be fine.
Asagi has triggers that update _daily
and _users
tables. FFuuka stats plugin reads those tables and make stats. Probably the _users
table was messed up/too large for /a/, so it slows down inserts because the update runs before it.
4plebs is working on stats plugins that does it without them, since stats calc on insert isn't a smart idea.
Procedure[edit]
I just removed the calls to stats procedures. Should work but check it out first. But should leave the procedures there (commented out?) so that you can go back.
- Stop Asagi.
- Run the code below:
- Start Asagi.
DROP TRIGGER IF EXISTS "after_ins_a";
delimiter //
CREATE TRIGGER "after_ins_a" AFTER INSERT ON "a"
FOR EACH ROW
BEGIN
IF NEW.op = 1 THEN
CALL create_thread_a(NEW.num, NEW.timestamp);
END IF;
CALL update_thread_a(NEW.thread_num);
END//
DROP TRIGGER IF EXISTS "after_del_a"//
CREATE TRIGGER "after_del_a" AFTER DELETE ON "a"
FOR EACH ROW
BEGIN
CALL update_thread_a(OLD.thread_num);
IF OLD.op = 1 THEN
CALL delete_thread_a(OLD.num);
END IF;
IF OLD.media_hash IS NOT NULL THEN
CALL delete_image_a(OLD.media_id);
END IF;
END//
delimiter ;