Moebooru: Difference between revisions

Revision as of 08:03, 30 January 2017

Note: This guide is up to date and tested as of 2017-01-28. Moebooru does evolve gradually, so installation steps may change with time. If you find any divergences, please create an account and fix them.

Moebooru is a well maintained Danbooru-style image tag system, used by Yande.re, Konachan, and many others. It is written in Ruby, and can be a bit of a challenge to install. Since many other installation guides are insufficient, we've noted down our entire installation and configuration process for Eikonos.

Assumptions

In this guide, we make certain assumptions about your configuration. Your situation may vary.

Nginx - The webserver used in this guide. Follow the instructions in that page to install Nginx. Then, you can find some sample Nginx configurations for Moebooru in sections below.

 * www-data - A group used by all webservers. Make sure to add the nginx user to this group!

moe user - The unprivileged user that runs this app. After configuration, this will be converted to a nonlogin user.
/home/moe/booru/ - The home folder of the moebooru user, where the application data will be stored from a git clone.
- /home/moe/booru/shared/ - The location of the UNIX sockets that puma will use. It is probably safer to put it into /var/run/moebooru instead, but it is what it is and we have SELinux policies to account for it.
/var/www/moebooru/public/ - The public directory will be moved to /var/www/, and symlinked to /home/moe/booru/public/. This way, Nginx will not have control over the entire application directory.
- Make sure to restore this symlink every time an upgrade is run with git pull.

Create Moebooru User

Create a specific non-login daemon user just for moebooru (Though it will have bash shell for setup purposes temporarily).

sudo adduser moe      # create whatever password
sudo passwd -d moe    # delete password to prevent normal login
sudo -i -u moe        # become the moe user
git clone https://github.com/moebooru/moebooru.git /home/moe/booru
chown -R moe:moe /home/moe/booru

As root, we also need to move all publicly served assets to /var/www/.

sudo mkdir -p /var/www/moebooru
sudo mv /home/moe/booru/public /var/www/moebooru/
sudo ln -s /var/www/moebooru/public /home/moe/booru/public
sudo chown moe:moe /home/moe/booru/public
sudo chmod 755 /var/www/moebooru/public
sudo chown root:www-data -R /var/www/moebooru

Setup Postgresql

The examples below use PostgreSQL 9.6, but as of 2017-01-28, Moebooru just needs greater than 9.4.

Debian: (Alternatively, you can get it from jessie-backports and newer)

/etc/apt/sources.list.d/pgdg.list

deb http://apt.postgresql.org/pub/repos/apt/ YOUR_DEBIAN_VERSION_HERE-pgdg main

wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc

RHEL/CentOS:

# yum install http://yum.postgresql.org/9.6/redhat/rhel-7-x86_64/pgdg-redhat95-9.5-2.noarch.rpm 
# yum install postgresql96 postgresql96-devel postgresql96-server libpqxx libpqxx-devel

libpqxx is required for libpq-ruby to build.

Now log in and create the moebooru database and user with createdb permissions: (make sure to give it a very good password!)

$ sudo su     # becoming the postgres user doesn't really work with sudo for some reason
# su - postgres
$ psql
postgres # CREATE USER moebooru_user WITH PASSWORD 'your_password' CREATEDB;

Finally, edit pg_hba.conf from ident to md5 (except for the UNIX socket line) to allow users to log in using a password (required by moebooru's config), it should look like the following:

RHEL/CentOS: /var/lib/pgsql/9.6/data/pg_hba.conf     Debian: /etc/postgresql/9.6/main/pg_hba.conf

# "local" is for Unix domain socket connections only
local   all             all                                     md5
# IPv4 local connections:
host    all             all             127.0.0.1/32            md5
# IPv6 local connections:
host    all             all             ::1/128                 md5
# Allow replication connections from localhost, by a user with the
# replication privilege.
#local   replication     postgres                                peer
#host    replication     postgres        127.0.0.1/32            trust
#host    replication     postgres        ::1/128                 trust

Once you have this file edited as seen above, restart postgresql.

sudo systemctl restart postgresql-9.6

Warning: If this postgresql server is on the same machine, make sure the firewall is configured to prevent remote access to postgresql ports. If this postgresql server is on another machine in the network/internet, make sure moebooru is connecting via SSL.

Install Nodejs

NodeJS is necessary for the frontend. You should obtain the latest version, 7.x: by following these guides for your distro.

Setup Ruby

For most small sites, Ruby 2.3 and higher with the Unicorn server is sufficient.

A site with heavier traffic may find it helpful to use Rubinius with the Puma server make better use of concurrent multithreading, but this may require compilation.

Method 1: Set up Normal Ruby 2.3 or newer

Most distros have an outdated version of Ruby, so you must set up repositories from the official developers:

Debian: Brightbox PPA - Ruby 2.3
RHEL/CentOS: RHSCL - Ruby 2.3

Method 2: Installing Rubinius Binaries with chruby

Debian/Ubuntu

sudo apt-get install openssl libreadline6 libreadline6-dev \
curl zlib1g zlib1g-dev libssl-dev libyaml-dev libsqlite3-dev sqlite3 \
libxml2-dev libxslt-dev autoconf libc6-dev ncurses-dev libtool  \
pkg-config libgdbm-dev ncurses-dev libffi-dev

Red Hat/CentOS

sudo yum install ruby
sudo yum install -y patch autoconf libtool sqlite-devel

Rubinius is particularly tough to compile, and as of 2017-01-28 we have not been able to compile it with Debian Jessie. Instead, Rubinius provides Ubuntu 14.04 binaries designed for use with Travis-Cl.

First, download and install chruby as a user with sudo privileges.

wget -O chruby-0.3.9.tar.gz https://github.com/postmodern/chruby/archive/v0.3.9.tar.gz
tar -xzvf chruby-0.3.9.tar.gz
cd chruby-0.3.9/
sudo make install

Next, log in as the moe user, download the latest rubinius binaries, and extract them to /home/moe/:

sudo -i -u moe
wget -O rubinius-3.70.tar.bz2 https://rubinius-binaries-rubinius-com.s3.amazonaws.com/ubuntu/14.04/x86_64/rubinius-3.70.tar.bz2
tar -xjvf rubinius-3.70.tar.bz2

Then, activate chruby with rubinius by adding it to the bashrc.

/home/moe/.bashrc

source /usr/local/share/chruby/chruby.sh

RUBIES+=(
  "$HOME/rubinius/3.70"
)

Finally, log out and log back into the moe user, and set rubinius 3.70 as the default ruby version.

chruby 3.70

Method 3: Install Rubinius with RVM

Note: Unfortunately, until the ActiveRecord-JDBC-Adapter required by JRuby is ported to Rails 5, we can't use it with Moebooru.

For RVM, just install any ol' ruby 2.x, we won't be using it after the compilation stage. Also install all the build dependencies.

Installation with RVM (RHEL/CentOS)

Note: Frustratingly, Rubinius refuses to build with Debian Jessie for some darn reason, so this is for RHEL/CentOS only.

sudo yum install ruby
sudo yum install -y patch autoconf patch automake libtool bison sqlite-devel

One special build dependency is llvm-3.6+. Unfortunately, CentOS 7's EPEL repo only has 3.4, so we will need to get the latest build of LLVM from Fedora COPR.

/etc/yum.repos.d/daveisfera-llvm_3.7-epel-7.repo

[daveisfera-llvm_3.7]
name=Copr repo for llvm_3.7 owned by daveisfera
baseurl=https://copr-be.cloud.fedoraproject.org/results/daveisfera/llvm_3.7/epel-7-$basearch/
type=rpm-md
skip_if_unavailable=True
gpgcheck=1
gpgkey=https://copr-be.cloud.fedoraproject.org/results/daveisfera/llvm_3.7/pubkey.gpg
repo_gpgcheck=0
enabled=1
enabled_metadata=1

Then just install llvm as normal:

sudo yum install clang llvm llvm-devel llvm-static libedit libedit-devel

Note: Make sure to run all these steps as the moe user.

Follow the RVM installation instructions, as replicated below:

sudo -i -u moe
gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
curl -sSL https://get.rvm.io | bash -s stable

Then, log out and log back in to reload bash and make the RVM command work:

exit
sudo -i -u moe

Refresh the rvm repos. And since we have already installed dependencies in the sections above, disable the built-in dependency resolver:

rvm get head
rvm autolibs disable

Note: If there are errors while compiling rubinius, check the compilation logs for any missing libraries, then install them using your distro's package manager.

Install rubinius. Compilation will take a while.

rvm install rbx

After installation, choose Rubinius (rbx) as the default ruby version to use.

rvm list

This will list out various ruby versions.

rvm alias create default rbx

Finally, log out and log back in, and check your current ruby version to ensure that Rubinius is default.

ruby -v

Now that your ruby type is set up, install bundler.

gem install bundler

Note: If you change ruby editions later on after installing gems, make sure to run gem pristine --all to delete the previous ones. After that, reinstall all gems from scratch.

Mandatory Access Control

AppArmor (Debian/Ubuntu)

anyone have apparmor profiles?

We may have to create our own through learning mode: http://www.howtogeek.com/118328/how-to-create-apparmor-profiles-to-lock-down-programs-on-ubuntu/

SELinux Permissions (RHEL/CentOS)

If using SELinux (which we highly recommend), you will need the following policies, assuming that everything is installed to /home/moe/booru/:

needed for proxy pass

sudo chcon -Rt httpd_sys_content_t /var/www/ # allow nginx to access folders
sudo setsebool httpd_can_network_connect 1 -P # allows reverse proxy
sudo setsebool -P httpd_can_network_memcache 1 # allows memcache

needed to serve from moebooru user's directory, but only shared/ and public folders

setsebool -P httpd_enable_homedirs 1
sudo semanage fcontext -a -t httpd_sys_content_t '/home/moe/booru/shared(/.*)?'
sudo restorecon -R -v /home/moe/booru/shared
sudo semanage fcontext -a -t httpd_sys_content_t '/var/www/moebooru/public(/.*)?'
sudo restorecon -R -v /var/www/moebooru/public

https://www.pckr.co.uk/selinux-nginx-and-reverse-proxying-2/

https://www.digitalocean.com/community/tutorials/an-introduction-to-selinux-on-centos-7-part-2-files-and-processes

The last step is to run the final allows.

sudo grep nginx /var/log/audit/audit.log | audit2allow -M nginx > nginx.te

Open up the nginx.te file and see that it is correct (such that no suspicious rule allows are inside). Then run:

sudo grep nginx /var/log/audit/audit.log | audit2allow -M nginx 
sudo semodule -i nginx.pp

http://axilleas.me/en/blog/2013/selinux-policy-for-nginx-and-gitlab-unix-socket-in-fedora-19/

Setup Moebooru

Now, conduct the setup and get dependencies.

Dependencies (Debian/Ubuntu)

sudo apt-get install build-essential libxml2-dev libxslt1-dev libpq-dev git jhead libgd2-noxpm-dev imagemagick

build-essential - a meta-package that installs various tools and libraries necessary for program (mostly C++ and Perl) compilation and interpretation.
libxml2 - library that is required for XML support. It is necessary for moebooru’s external API to work.
libxslt1 - extension of libxml that allows better xml parsing and converting.
libpq-dev - library that is necessary to build postgres connection module for ruby.
git - version control system - we will use it for moebooru installation and updating.
jhead - image processing tool that extracts EXIF information from image files.
libgd2-noxpm - an image processing library.
libgs2-noxpm-dev - an extension for compiling GD2 support for applications.
imagemagick - collection of tools for image processing

Dependencies (RHEL/CentOS)

sudo yum install gcc gcc-c++ ImageMagick jhead libxslt-devel git libyaml-devel openssl-devel pcre-devel readline-devel

Configure Moebooru

All commands in this section must be run as the moe user:

sudo -i -u moe

First, do a bundle config for pg since we're using a specific postgresql version:

bundle config build.pg --with-pg-config=/usr/pgsql-9.6/bin/pg_config    # Red Hat/CentOS Only
bundle config build.pg --with-pg-config=/usr/bin/pg_config              # Debian Only

Install the ruby packages for the moebooru user only (under the directory ./vendor/bundle):

bundle install --path vendor/bundle

Obtain config/database.yml and config/local_config.rb from the .example files, and configure them accordingly. Then set chmod 700 so only the moebooru user can read the database password.

cp config/database.yml.example config/database.yml
cp config/local_config.rb.example config/local_config.rb
chmod 700 config/database.yml
chmod 700 config/local_config.rb

Generate your secret key, which is used for salts and such.

bundle exec rake secret

Edit config/database.yml using your favorite editor (such as nano), and replace imouto with the password of your database user:

config/database.yml

login: &login
  adapter: postgresql
  username: moe
  password: imouto
  host: 127.0.0.1

development:
  database: moebooru_dev
  <<: *login

test:
  database: moebooru_test
  <<: *login

production:
  database: moebooru
  <<: *login

chmod 600 /home/moe/booru/config/database.yml

Initialize database with this command (there will be some errors reported with ERROR: must be owner of extension plpgsql which is normal, proceed)

bundle exec rake db:reset

Note: Whenever you update moebooru using git pull, make sure to migrate the database table schema with the command below.

Then, migrate the database tables.

bundle exec rake db:migrate

Now, you need to provide the correct permissions to the public folder:

chmod 755 /home/moe/booru/public

Start the server: bundle exec unicorn or bundle exec puma if using Rubinius. Note that this will start the server in development mode, which is somewhat slower. See the Production Mode section once you are ready to serve the site.

Customize Header Image and Branding

Note: You will have to restore these each time Moebooru is updated with git pull, so keep them in a safe place.

By default, Moebooru comes with the Yande.re header image and branding, as the site developed the moebooru engine. You should definitely consider removing the original branding unless your site is private.

app/assets/images

public/favicon.ico

Enable Memcached

Memcached is a high performance caching solution and is needed to have Moebooru enumerate posts and create the /posts pagination bar. Follow these instructions to install Memcached. You need at least 2GB free RAM to provide.

Installation (Debian/Ubuntu)

sudo apt-get install memcached

https://kyup.com/tutorials/install-use-memcache/

Installation (RHEL/CentOS)

sudo yum install memcached

http://www.liquidweb.com/kb/how-to-install-memcached-on-centos-7/

Configuration

Edit /etc/sysconfig/memcached and set CACHESIZE=2048 (2GB RAM) if possible.

Then set memcached to start at every boot:

sudo systemctl restart memcached

Config-based Activation

Add these options to the following file. When you start moebooru again, memcached will be active.

config/local_config.rb

# The server and port where the memcache client can be accessed. Only relevant if you enable caching.
CONFIG["memcache_servers"] = ["localhost:11211"]
# This enables various caching mechanisms. You must have memcache (and the memcache-client ruby gem) installed in order for caching to work.
CONFIG["enable_caching"] = true

ENV-based Memcached Activation

Note: We're not really sure whether the below method of enabling caching works. Instead, we use config-based activation as stated above.

Activate it by appending a bash variable to the puma command: MB_MEMCACHE_SERVERS="<url-to-memcached>" . Here are some examples.

TCP: MB_MEMCACHED_SERVERS='127.0.0.1:11211' RAILS_ENV=production bundle exec puma -e production

UNIX Socket: MB_MEMCACHED_SERVERS='127.0.0.1:11211' RAILS_ENV=production bundle exec puma -C shared/puma.rb

SELinux Permissions

You will probably need to allow it through SELinux:

https://major.io/2011/09/07/getting-apache-php-and-memcached-working-with-selinux/

Production Mode

By default, Moebooru runs in development mode, which can be slow (since it's designed to allow debug and automatic recompilation). Here are the steps to set up Production mode.

Preparation

First, you need to create the database, and pregenerate the javascript/css (do this every time you update):

RAILS_ENV=production bundle exec rake db:reset
RAILS_ENV=production bundle exec rake assets:precompile

Then, you need to provide the correct permissions to the public folder:

chmod 755 /var/www/moebooru/public

Serve static files with Nginx

Create an Nginx config under /etc/nginx/conf.d/. Make sure to change the server_name.

Note: If you are using a different port for puma, (by adding -p 3000 to the serving command), also change the port accordingly below.

server {
    listen 80;
    server_name booru.eikonos.org;
    
    # directory of static assets, first generate with the command:
    # RAILS_ENV=production bundle exec rake assets:precompile
    root /var/www/moebooru/public;

    try_files $uri/index.html $uri @app;

    location @app {
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      # Fix the "It appears that your reverse proxy set up is broken" error.
      proxy_pass http://127.0.0.1:9292;
      proxy_read_timeout 90;
      proxy_redirect http://127.0.0.1:9292 http://$server_name;
    }
    
    error_page 500 502 503 504 /500.html;
    client_max_body_size 4G;
    keepalive_timeout 10;
}

Run the Server

Finally, to run the server at, for example, port 9292, run one of the following commands:

Ruby:

bundle exec unicorn -p 9292 -E production

Rubinius:

RAILS_ENV=production bundle exec puma -e production -p 9292

Source: StackOverflow - Rails 4: assets not loading in production

Serve with UNIX Socks in Production Mode

for even more effectiveness, use a UNIX sock: https://www.digitalocean.com/community/tutorials/how-to-deploy-a-rails-app-with-puma-and-nginx-on-ubuntu-14-04

Figure out the amount of CPU cores you have:

grep -c processor /proc/cpuinfo

Create a folder to store your UNIX sockets and PIDs in:

sudo mkdir /var/run/moebooru
sudo chown -R moe:moe /var/run/moebooru

Create the following folders in your application directory to store logs:

mkdir -p shared/log

Place the following into <app_dir>/shared/puma.rb:

shared/puma.rb

# Change to match your CPU core count
workers 8

# Min and Max threads per worker
threads 1, 6

app_dir = File.expand_path("../..", __FILE__)
shared_dir = "#{app_dir}/shared"
socket_dir = "/var/run/moebooru"

# Default to production
rails_env = ENV['RAILS_ENV'] || "production"
environment rails_env

# Set up socket location
bind "unix://#{socket_dir}/puma.sock"

# Logging
stdout_redirect "#{shared_dir}/log/puma.stdout.log", "#{shared_dir}/log/puma.stderr.log", true

# Set master PID and state locations
pidfile "#{socket_dir}/puma.pid"
state_path "#{socket_dir}/puma.state"
activate_control_app

on_worker_boot do
  require "active_record"
  ActiveRecord::Base.connection.disconnect! rescue ActiveRecord::ConnectionNotEstablished
  ActiveRecord::Base.establish_connection(YAML.load_file("#{app_dir}/config/database.yml")[rails_env])
end

Change the Nginx server config to the following:

/etc/nginx/conf.d/booru.eikonos.org.conf

upstream app {
    # Path to Puma SOCK file, as defined previously
    server unix:/var/run/moebooru/puma.sock fail_timeout=0;
}

server {
    listen 80;
    server_name booru.eikonos.org;
    
    # directory of static assets, first generate with the command:
    # RAILS_ENV=production bundle exec rake assets:precompile
    root /var/www/moebooru/public;

    try_files $uri/index.html $uri @app;

    location @app {
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      # Fix the "It appears that your reverse proxy set up is broken" error.
      proxy_pass http://app;
      proxy_read_timeout 90;
      proxy_redirect http://app http://$server_name;
    }
    
    error_page 500 502 503 504 /500.html;
    client_max_body_size 4G;
    keepalive_timeout 10;
}

Finally, to run the server, use the following:

bundle exec puma -C shared/puma.rb

Systemd Service

https://github.com/puma/puma/blob/master/docs/systemd.md

Save this to /usr/systemd/system/moebooru.service. There are two versions, one for TCP and one for unix socket. Change the WorkingDirectory accordingly.

TCP

/usr/systemd/system/moebooru.service

[Unit]
Description=Moebooru's Puma HTTP Server
Requires=postgresql-9.6.service
Wants=postgresql-9.6.service memcached.service
After=network.target postgresql-9.6.service

# Uncomment for socket activation (see below)
# Requires=puma.socket

[Service]
# Foreground process (do not use --daemon in ExecStart or config.rb)
Type=simple

# Preferably configure a non-privileged user
User=moe

# Specify the path to your puma application root
WorkingDirectory=/home/moe/booru

# Helpful for debugging socket activation, etc.
# Environment=PUMA_DEBUG=1

# The command to start Puma
# Here we are using a binstub generated via:
# `bundle binstubs puma --path ./sbin`
# in the WorkingDirectory (replace <WD> below)
# You can alternatively use `bundle exec --keep-file-descriptors puma`
# ExecStart=<WD>/sbin/puma -b tcp://0.0.0.0:9292 -b ssl://0.0.0.0:9293?key=key.pem&cert=cert.pem

# Alternatively with a config file (in WorkingDirectory) and
# comparable `bind` directives
ExecStart=/bin/bash -c 'RAILS_ENV=production /usr/bin/bundle exec puma -e production'

Restart=always

[Install]
WantedBy=multi-user.target

UNIX Socket

Two Systemd services are needed: one for the socket and one for the application.

/usr/systemd/system/moebooru.service

[Unit]
Description=Puma HTTP Server
Requires=postgresql-9.6.service
Wants=postgresql-9.6.service memcached.service
After=network.target postgresql-9.6.service

# Uncomment for socket activation (see below)
# Requires=puma.socket

[Service]
# Foreground process (do not use --daemon in ExecStart or config.rb)
Type=simple

# Preferably configure a non-privileged user
# User=

# Specify the path to your puma application root
# WorkingDirectory=

# Helpful for debugging socket activation, etc.
# Environment=PUMA_DEBUG=1

# The command to start Puma
# Here we are using a binstub generated via:
# `bundle binstubs puma --path ./sbin`
# in the WorkingDirectory (replace <WD> below)
# You can alternatively use `bundle exec --keep-file-descriptors puma`
# ExecStart=<WD>/sbin/puma -b tcp://0.0.0.0:9292 -b ssl://0.0.0.0:9293?key=key.pem&cert=cert.pem

# Alternatively with a config file (in WorkingDirectory) and
# comparable `bind` directives
# ExecStart=<WD>/sbin/puma -C config.rb

Restart=always

[Install]
WantedBy=multi-user.target

Grab some code from here?

https://github.com/puma/puma/issues/976

SSL Certificates

While this is beyond the scope of this guide, you should strongly consider using SSL certificates, which are now free with Let's Encrypt.

https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-centos-7

Full Backup

PostgreSQL Database (run as moebooru user!):

pg_dump -U moebooru -W moebooru > eikonos.org-20170128.pg.sql # will ask for password

Images and thumbs and other generated items:

tar -cvzf eikonos.org-20170128_data.tar.gz /var/www/moebooru/public/data/

Code only, no binaries or image data:

tar --exclude='/home/moe/booru/tmp' --exclude='/home/moe/booru/public' --exclude='/home/moe/booru/vendor' -cvjf eikonos.org-20170128_var_www_booru-eikonos-org.tar.gz /home/moe/booru

Restore/Move

PostgreSQL database

First, create the user with CreateDB permissions:

# su - postgres
$ psql
postgres # CREATE USER moebooru_user CREATEDB;

then import the sql dump with that user.

Sources

Get configuration tips from here: http://wiki.douglasqsantos.com.br/doku.php/deploying_a_rails_app_on_debian_jessie_with_capistrano_nginx_and_puma_en

Anonymous

Search