Editing FoolFuuka/Install/Debian

This guide was written by ATC of desuarchive, and is designed for Debian/Ubuntu Linux servers.
__NOTOC__
== Prepare things ==

<pre>apt-get update
apt-get upgrade</pre>
== Add the keys for HipHop Virtual Machine ==

<pre>sudo apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0x5a16e7281be7a449
sudo add-apt-repository &quot;deb http://dl.hhvm.com/ubuntu $(lsb_release -sc) main&quot;</pre>
== Add repo for MariaDB 10.1 ==

<pre>sudo apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xcbcb082a1bb943db
sudo add-apt-repository 'deb [arch=amd64,i386] http://nyc2.mirrors.digitalocean.com/mariadb/repo/10.1/ubuntu trusty main'</pre>
== Add repo for sphinxsearch ==

{{Note|Weird shit happens here}}

<pre>sudo add-apt-repository ppa:builds/sphinxsearch-stable</pre>
Then, Comment out repo in {{ic|/etc/apt/sources.list.d/*}} and add to end:

<pre>deb http://ppa.launchpad.net/builds/sphinxsearch-stable/ubuntu precise main 
deb-src http://ppa.launchpad.net/builds/sphinxsearch-stable/ubuntu precise main </pre>
<pre>sudo apt-get update</pre>

== Install all the needed packages. ==

Nginx, HHVM, PHP5 Cli, PHP5 Curl, MariaDB, OpenJDK w/ compilier, Maven, Sphinxsearch

<pre>sudo apt-get install git nginx hhvm php5-fpm php5-cli php5-curl php5-mysqlnd mariadb-server libmariadbclient-dev mariadb-client imagemagick openjdk-7-jre openjdk-7-jdk openjdk-7-jre-lib maven sphinxsearch</pre>

== Install composer for downloading FoolFuuka dependencies. ==

<pre>curl -sS https://getcomposer.org/installer | php</pre>
== Move composer to bin ==

<pre>sudo mv composer.phar /usr/local/bin/composer</pre>
== Install hhvm as a service so it launches at boot. ==

<pre>sudo update-rc.d hhvm defaults</pre>
== Install hhvm with nginx ==

<pre>sudo /usr/share/hhvm/install_fastcgi.sh</pre>
== Edit HHVM config to use a unix socket. ==

<pre>sudo nano /etc/hhvm/server.ini</pre>
Then remove the line starting with hhvm.server.port, and in its place add the following one:

<pre>hhvm.server.file_socket=/var/run/hhvm/hhvm.sock</pre>

== Set required php.ini vars. ==
{{hc|sudo nano /etc/php5/fpm/php.ini|<nowiki>
date.timezone = UTC
upload_max_filesize = 32M
post_max_size = 32M
</nowiki>}}

== Restart HHVM ==

<pre>sudo service hhvm restart</pre>
== Configure NGINX to use hhvm's new unix socket. ==

<pre>sudo nano /etc/nginx/hhvm.conf</pre>
In this file make sure that the fastcgi_pass directive points to the HHVM socket and looks like this:

<pre>  fastcgi_pass unix:/var/run/hhvm/hhvm.sock;</pre>
== Download FoolFuuka in home folder and install dependencies. ==

<pre>cd ~
composer create-project foolz/foolfuuka foolfuuka --prefer-source
cd foolfuuka
composer dump-autoload --optimize
composer install </pre>
== Install Foolfuuka Plugins ==

You may need to find the repo on their github if their site doesn't work: https://github.com/FoolCode

<pre>cd ~/foolfuuka/app/foolz/foolframe/plugins
mkdir foolz
cd foolz
git clone https://github.com/FoolCode/foolframe-plugin-articles.git</pre>

<!-- Or use <https://github.com/FoolCode/foolframe-plugin-articles> if that's not available -->

<pre>cd ~/foolfuuka/app/foolz/foolfuuka/plugins
mkdir foolz
cd foolz
git clone https://github.com/FoolCode/foolfuuka-plugin-board-statistics.git
git clone https://github.com/FoolCode/foolfuuka-plugin-dice-roll.git
git clone https://github.com/FoolCode/foolfuuka-plugin-quests.git</pre>

== Set Permissions and Group for FoolFuuka Directory ==

<pre>sudo chmod -Rf 775 ~/foolfuuka
sudo chown -Rf www-data:www-data ~/foolfuuka</pre>

== Configure Nginx to host foolfuuka ==

{{hc|sudo nano /etc/nginx/sites-enabled/default|<nowiki>
        root /home/ubuntu/foolfuuka/public;
        index index.php index.html index.htm;

        client_max_body_size 32M; # set to the same as max PHP upload size in HHVM
        location / {
                try_files $uri $uri/ /index.php$is_args$args;
        }

        location ~ \.php$ {
                 fastcgi_pass unix:/var/run/php5-fpm.sock;
                 fastcgi_index index.php;
                 fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                 include fastcgi_params;
        }

</nowiki>}}

== Restart Nginx ==

<pre>sudo service nginx restart</pre>
== Create the database and install Foolfuuka user accounts. ==

<pre>sudo mysql_secure_installation</pre>
<pre>mysql -uroot -p
  CREATE USER 'asagi'@'localhost' IDENTIFIED BY 'pass';  
  CREATE USER 'foolfuuka'@'localhost' IDENTIFIED BY 'pass';
  CREATE DATABASE asagi DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_general_ci;
  CREATE DATABASE foolfuuka DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_general_ci;
  GRANT ALL PRIVILEGES ON `asagi`. * TO 'asagi'@'localhost';
  GRANT ALL PRIVILEGES ON `asagi`. * TO 'foolfuuka'@'localhost';
  GRANT ALL PRIVILEGES ON `foolfuuka`. * TO 'foolfuuka'@'localhost';
  FLUSH PRIVILEGES;
  exit</pre>

== edit MariaDB to use utf8mb4 for correct unicode processing ==

{{hc|sudo nano /etc/mysql/my.cnf|<nowiki>
[client]
default-character-set = utf8mb4

[mysql]
default-character-set = utf8mb4

[mysqld]
character-set-client-handshake = FALSE
character-set-server = utf8mb4
collation-server = utf8mb4_general_ci
</nowiki>}}

== Restart MariaDB ==

<pre>sudo service mysql restart</pre>

Disable transparent_hugepage

<pre>su -c 'echo never > /sys/kernel/mm/transparent_hugepage/enabled'</pre>

<pre>sudo nano /etc/default/grub</pre>

append {{ic|<nowiki>transparent_hugepage=never</nowiki>}} to {{ic|GRUB_CMDLINE_LINUX_DEFAULT}}, then save the file

<pre>sudo update-grub</pre>

Enable toku (Find and Uncomment the load module)

<pre>sudo nano /etc/mysql/conf.d/tokudb.cnf </pre>
Raise file limit

{{hc|sudo nano /etc/security/limits.conf|
* soft nofile 63545
* hard nofile 63545
* soft nproc 63545
* hard nproc 63545
}}

Increase mysql file limit:

{{hc|sudo nano /etc/mysql/my.cnf|<nowiki>
# Put this after [mysqld]
open-files-limit=40000
</nowiki>}}

<pre>sudo service mysql stop
sudo reboot</pre>
<pre>mysql -u root -p
set global default_storage_engine=TokuDB;
SET GLOBAL log_bin_trust_function_creators = 1;</pre>
Now, Install FoolFuuka at the website using foolfuuka credentials

== Install and compile Asagi from source. ==

<pre>cd ~
git clone https://github.com/FoolCode/asagi.git
cd asagi
mvn package assembly:single
mv target/asagi-0.4.0-SNAPSHOT-full.jar ./asagi.jar</pre>
== Configure Asagi ==

<pre>cp asagi.json.example asagi.json
nano asagi.json</pre>

Insert the following JSON into that file:

<pre>
{"settings": {
  "dumperEngine": "DumperJSON",
  "sourceEngine": "YotsubaJSON",

  "boardSettings": {
    "default": {
      "engine": "Mysql",
      "database": "asagi",
      "host": "localhost",
      "username": "asagi",
      "password": "afj8a428dauwj",
      "charset": "utf8mb4",
      "path": "/home/ubuntu/foolfuuka/public/foolfuuka/boards/",
      "useOldDirectoryStructure": false,
      "webserverGroup": "www-data",
      "thumbThreads": 3,
      "mediaThreads": 0,
      "newThreadsThreads": 3,
      "deletedThreadsThresholdPage": 8,
      "refreshDelay": 30,
      "throttleAPI": true,
      "throttleURL": "api.4cdn.org",
      "throttleMillisec": 11100,

      "threadRefreshRate": 10,
      "pageSettings": [
        {"delay": 30, "pages": [0, 1]},
        {"delay": 500, "pages": [2, 3, 4, 5, 6, 7, 8]},
        {"delay": 30, "pages": [9, 10]}
      ]
    },

    "3": {},
    "a": {},
    "aco": {},
    "adv": {},
    "an": {},
    "asp": {},
    "b": {},
    "biz": {},
    "c": {},
    "cgl": {},
    "ck": {},
    "cm": {},
    "co": {},
    "d": {},
    "diy": {},
    "e": {},
    "f": {},
    "fa": {},
    "fit": {},
    "g": {},
    "gd": {},
    "gif": {},
    "h": {},
    "hc": {},
    "hm": {},
    "his": {},
    "hr": {},
    "i": {},
    "ic": {},
    "int": {},
    "jp": {},
    "k": {},
    "lgbt": {},
    "lit": {},
    "m": {},
    "mlp": {},
    "mu": {},
    "n": {},
    "news": {},
    "o": {},
    "out": {},
    "p": {},
    "po": {},
    "pol": {},
    "qa": {},
    "qst": {},
    "r": {},
    "r9k": {},
    "s": {},
    "s4s": {},
    "sci": {},
    "soc": {},
    "sp": {},
    "t": {},
    "tg": {},
    "toy": {},
    "trash": {},
    "trv": {},
    "tv": {},
    "u": {},
    "v": {},
    "vg": {},
    "vp": {},
    "vr": {},
    "w": {},
    "wg": {},
    "wsg": {},
    "wsr": {},
    "x": {},
    "y": {}
  }
}}
</pre>

== Start asagi (There is a better way to do this). ==

Maybe we should use systemd for it...

[http://www.fasterj.com/articles/oraclecollectors1.shtml UseParallelGC] adds Parallelization to the garbage collector, which really helps for Java.

<pre>screen java -XX:+UseParallelGC -XX:+UseParallelOldGC -verbose:gc -jar asagi.jar</pre>

ATC of Desuarchive prefers these parameters, which have two customizations. 

# For the first parameter, Java by default will use 1/4th your system memory, so you can set your own hard memory roof to reduce usage or increase it. He used {{ic|6144m}}.
#* If you aren't archiving more than 8 boards, this probably won't be an issue that you need to deal with.
# The permsize is about the permanent generation objects: depends on your heap size.
#* This setting is probably related to how many boards are on Desuarchive, but Sunako doesn't had that sort of issue.

<pre>screen java -Xmx6144m -XX:+UseParallelGC -XX:+UseParallelOldGC -verbose:gc -XX:MaxPermSize=512m -jar asagi.jar</pre>

Now, Configure asagi as the database and remove the table prefix from FoolFuuka.
== Install FoolFuuka ==

The installer is quite self explanatory, just make sure to remove the Boards prefix.

Also make sure to install imagemagick and set the path in the config to it.

{{Note|In the installer one should have FoolFuuka table prefix set (default is fine ff_). But after installing one needs to set BOARD PREFIX to blank since asagi doesn't have any. FoolFuuka table prefix != board table prefix. Here's [https://github.com/p0wer0n/FoolFrame/commit/3368e24dc9d89b881460f8c876ddb91c78b5b955 a code change that does it.]}}

== Add boards, Configure Sphinx ==

To configure Sphinx Search, take the following links:

http://IP/admin -> Search -> Generate Config.

Right-click and copy this config out of FF and add your MySQL database information. Then, paste this to <code>/etc/sphinxsearch/sphinx.conf</code>

=== Build the indexes ===

<pre>sudo indexer *_ancient --rotate</pre>
<pre>sudo indexer *_main --rotate</pre>

=== Configure Crontab ===

<pre>
@hourly sudo -u sphinxsearch /usr/bin/indexer --rotate *_delta > /dev/null
@daily sudo -u sphinxsearch /usr/bin/indexer --rotate *_main > /dev/null
@monthly sudo -u sphinxsearch /usr/bin/indexer --rotate *_ancient > /dev/null
</pre>

=== Adding Archives ===

Inject your sql file into the asagi database (assuming that your sql file is called "dump.sql"):
<pre>mysql -u root -p asagi < dump.sql</pre>

Then, in the admin panel, simply add the board that corresponds to the board you injected.

== Extra configuration ==

Depending on which distro you use or how many posts you have, some configs might need to be changed.

=== Fixing thumbnail creation ===

Foolfuuka might not be able to generate thumbnails for images to be posted. If so, check:
<pre>$ whereis convert
convert: /usr/bin/convert /usr/share/man/man1/convert.1.gz
</pre>
and make sure it is the same value in Admin panel -> general -> Imagemagick Convert

=== For Huge Boards: Problem with random dropped posts/threads due to stats triggers ===

When Desuarchive was configured with a new server using the 4plebs fork of FoolFuuka and Asagi, it was a mystery why /a/ was behind in archiving. Normal posts were being bled somehow.

But everything was running fine configuration-wise. No asagi errors. Nothing. And all other boards were great.

However, ''just get rid of stats triggers/procedures'' and it should be fine.

Asagi has triggers that update {{ic|_daily}} and {{ic|_users}} tables. FFuuka stats plugin reads those tables and make stats. Probably the {{ic|_users}} table was messed up/too large for /a/, so it slows down inserts because the update runs before it.

4plebs is working on stats plugins that does it without them, since stats calc on insert isn't a smart idea.

==== Procedure ====

I just removed the calls to stats procedures. Should work but check it out first. But should leave the procedures there (commented out?) so that you can go back.

# Stop Asagi. 
# Run the code below:
# Start Asagi.

{{lc|<nowiki>
DROP TRIGGER IF EXISTS "after_ins_a";

delimiter //

CREATE TRIGGER "after_ins_a" AFTER INSERT ON "a"
FOR EACH ROW
BEGIN
  IF NEW.op = 1 THEN
    CALL create_thread_a(NEW.num, NEW.timestamp);
  END IF;
  CALL update_thread_a(NEW.thread_num);
END//

DROP TRIGGER IF EXISTS "after_del_a"//

CREATE TRIGGER "after_del_a" AFTER DELETE ON "a"
FOR EACH ROW
BEGIN
  CALL update_thread_a(OLD.thread_num);
  IF OLD.op = 1 THEN
    CALL delete_thread_a(OLD.num);
  END IF;
  IF OLD.media_hash IS NOT NULL THEN
    CALL delete_image_a(OLD.media_id);
  END IF;
END//


delimiter ;
</nowiki>|lang=sql}}
@@ Line 339: / Line 339: @@
 http://IP/admin -> Search -> Generate Config.
-http://IP/admin -> Boards -> Edit -> Check Use SphinxSearch as search engine.
-On the Search panel, change the working directory (on Ubuntu) to the following:
-<pre>
-/var/lib/sphinxsearch
-</pre>
-=== Supplement Sphinxsearch ===
-<pre>
-cd /var/run
-sudo mkdir sphinxsearch
-sudo chmod -Rf 777 sphinxsearch
-</pre>
-=== Start Sphinxsearch ===
-<pre>
-sudo nano /etc/default/sphinxsearch
-</pre>
-Change the following line from
-<pre>START=no</pre>
-to
-<pre>START=yes</pre>
-Save and run the following command
-<pre>sudo /etc/init.d/sphinxsearch start</pre>
-Right-click and copy this config out of Foolfuuka and add your MySQL database information for asagi.
-Change the following line from
-<pre>pid_file = /var/lib/sphinxsearch/searchd.pid</pre>
-to
-<pre>pid_file = /var/run/sphinxsearch/searchd.pid </pre>
-Then, paste this to <code>/etc/sphinxsearch/sphinx.conf</code>
-If anything errors, kill any running instance of asagi and restart it
+Right-click and copy this config out of FF and add your MySQL database information. Then, paste this to <code>/etc/sphinxsearch/sphinx.conf</code>
 === Build the indexes ===
@@ Line 381: / Line 348: @@
 === Configure Crontab ===
-<pre>
-sudo nano /etc/crontab
-</pre>
 <pre>
 @hourly sudo -u sphinxsearch /usr/bin/indexer --rotate *_delta > /dev/null
@@ Line 389: / Line 354: @@
 @monthly sudo -u sphinxsearch /usr/bin/indexer --rotate *_ancient > /dev/null
 </pre>
 === Adding Archives ===