Hayden Guide: Difference between revisions

From Bibliotheca Anonoma
(Formatting and more sections)
(start proxies section)
 
(3 intermediate revisions by the same user not shown)
Line 21: Line 21:
</syntaxhighlight>
</syntaxhighlight>


=== Install dotnet ===
# Install Microsoft dotnet runtime
# Install Microsoft dotnet runtime
https://docs.microsoft.com/en-us/dotnet/core/install/linux-centos
https://docs.microsoft.com/en-us/dotnet/core/install/linux-centos
Line 47: Line 48:
# Verify docker works
# Verify docker works
#: <syntaxhighlight lang="bash">$ sudo docker run hello-world</syntaxhighlight>
#: <syntaxhighlight lang="bash">$ sudo docker run hello-world</syntaxhighlight>


==== Docker+Mariadb - Install docker-compose ====
==== Docker+Mariadb - Install docker-compose ====
Line 58: Line 58:
# Symlink into $PATH:
# Symlink into $PATH:
#: <syntaxhighlight lang="bash">$ sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose</syntaxhighlight>
#: <syntaxhighlight lang="bash">$ sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose</syntaxhighlight>
#* This is so that you do not need to type the absolute path to run the commands.
# Test installation:
# Test installation:
#: <syntaxhighlight lang="bash">$ docker-compose --version</syntaxhighlight>
#: <syntaxhighlight lang="bash">$ docker-compose --version</syntaxhighlight>


==== Docker+Mariadb - Setup Container ====
==== Docker+Mariadb - Setup Container ====
* TODO
#: Prepare container definition
 
* Example docker-compose definition file:
==== Docker+Mariadb - Setup DB ====
*: <syntaxhighlight lang="bash">
* TODO
# mariadb.yml
# https://hub.docker.com/_/mariadb/
version: '3.1'
services:
  db: # SQL DB server
    image: mariadb
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: "root_super_secret_code"
      MYSQL_DATABASE: hayden_db
      MYSQL_USER: hayden_un
      MYSQL_PASSWORD: "hayden_super_secret_cod"
    ports:
      - "10306:3306" # HOST:CONTAINER
    volumes:
      - "/home/USER/hayden-2020/mariadb_docker:/var/lib/mysql" # HOST:CONTAINER
      - "/home/USER/hayden-2020/mariadb_docker_dump:/dump" # HOST:CONTAINER
  adminer: # WebGUI for SQL DB server
    image: adminer
    restart: always
    ports:
      - 10088:8080
</syntaxhighlight>
#* Ports in the 10XXX range are used in this example on the host side in an attempt to avoid conflicting with other programs wanting specific ports.
#* In practice, any port above 1024 which is not being used by another program should work.
# Bring up defined containers
#* Bring up defined containers in the foreground of your terminal:
#*: Not reccomended outside of testing and development, as it is easy to close by mistake.
#*: <syntaxhighlight lang="bash">sudo docker-compose -f "/home/USER/hayden-2020/mariadb.yml" up</syntaxhighlight>
#* Bring up defined containers in the background ("-d" CLI flag):
#*: Reccomended, as this is harder to close by mistake.
#*: <syntaxhighlight lang="bash">sudo docker-compose -f "/home/USER/hayden-2020/mariadb.yml" -d up</syntaxhighlight>


==== Docker+Mariadb - Usage ====
==== Docker+Mariadb - Usage ====
* TODO
* TODO  
* To connect to the DB through adminer:
*: TODO
* To make a DB dump:
*# Find the name or ID of the DB container
*#:
*# Run the dump command targeting that container
*#: <syntaxhighlight lang="bash">$ sudo docker exec hayden-2020_db_1 sh -c 'exec mysqldump --all-databases -u"root" -p"YOUR_PASSWORD_HERE"' > "/home/USER/hayden-2020/mariadb-container-all-databases.ts`date -u +%s`.sql" </syntaxhighlight>
*#* Note that the dump command in the container outputs the DB dump SQL as text on its STDOUT; which is sent back from the container to the host; and that is redirected into a file on the host.
*#* Explaination:
*#** <code>$ sudo docker exec hayden-2020_db_1 sh -c</code> Runs a command inside a container.
*#** <code>'exec mysqldump --all-databases -u"root" -p"YOUR_PASSWORD_HERE"'</code> Dumps a copy of the DB to STDOUT
*#** <code> > "/home/USER/hayden-2020/mariadb-container-all-databases.ts`date -u +%s`.sql"</code> Redirects the output into a file with a timestamp in the name.
*#** <code>date -u +%s</code> Produces a timestamp in seconds since the epoch. e.g. <code>1597472305</code>.
*#* This is done using the DB root account as it always has full access to the entire database.


== Download and Build ==
== Download and Build Hayden ==
# Prepare location:
# Prepare location:
#: <syntaxhighlight lang="bash">
#: <syntaxhighlight lang="bash">
Line 90: Line 136:
* Some options are implimented but not shown in the example JSON file.
* Some options are implimented but not shown in the example JSON file.
** The option to scan archive.json to save threads that have recently fallen off the board: <code"ReadArchive": true</code>
** The option to scan archive.json to save threads that have recently fallen off the board: <code"ReadArchive": true</code>
**: This value is a child of  "source", as shown here:
**: <syntaxhighlight lang="json">
{
"source" : {
"type" : "4chan",
"boards" : [
"vmg"
],
"apiDelay" : 0.8,
"boardDelay" : 30,
"ReadArchive": true
},
</syntaxhighlight>
* Some databses will not let you connect as the DB root account from a system user other than root.
* Some databses will not let you connect as the DB root account from a system user other than root.
** As a result of this, you should create a DB user specifically for Hayden to use.
** As a result of this, you should create a DB user specifically for Hayden to use.
* The database connection is defined by this string value:
* The database connection is defined by this string value:
<syntaxhighlight lang="json">"connectionString" : "Server=DB_NETWORK_ADDRESS;Port=DB_PORT;Database=DATABASE;Uid=DB_USERNAME;Pwd=DB_PASSWORD;CharSet=utf8mb4;IgnorePrepare=false",</syntaxhighlight>
*: <syntaxhighlight lang="json">"connectionString" : "Server=DB_NETWORK_ADDRESS;Port=DB_PORT;Database=DATABASE;Uid=DB_USERNAME;Pwd=DB_PASSWORD;CharSet=utf8mb4;IgnorePrepare=false",</syntaxhighlight>
<br>e.g. <syntaxhighlight lang="json">"connectionString" : "Server=localhost;Port=3306;Database=asagi;Uid=hayden;Pwd=some_secret_value;CharSet=utf8mb4;IgnorePrepare=false",</syntaxhighlight>
** e.g. <syntaxhighlight lang="json">"connectionString" : "Server=localhost;Port=3306;Database=asagi;Uid=hayden;Pwd=some_secret_value;CharSet=utf8mb4;IgnorePrepare=false",</syntaxhighlight>
* Image downloading is controlled by two options:
* Image downloading is controlled by two options:
<br><code>"fullImagesEnabled" : true,</code>
*: <code>"fullImagesEnabled" : true,</code>
<br><code>"thumbnailsEnabled" : false,</code>
*: <br><code>"thumbnailsEnabled" : false,</code>
** These values must either be <code>false</code> or<code>true</code>.
** These values must either be <code>false</code> or<code>true</code>.
== Running ==
# Ensure DB is running
#* Hayden requires the database to be available to function.
#* See the database sections of this document for information on setting up and running a database in a container.
# Start Hayden
#* It is STRONGLY RECCOMENDED to run Hayden inside a Ccreen / Tmux / Byobu session so Hayden persists after your terminal disconnects.
#*: e.g. One of the following three commands:
#*: <syntaxhighlight lang="bash">$ screen</syntaxhighlight>
#*: <syntaxhighlight lang="bash">$ tmux</syntaxhighlight>
#*: <syntaxhighlight lang="bash">$ byobu</syntaxhighlight>
#* Start Hayden itself:
#** <syntaxhighlight lang="bash">$ "/home/USER/hayden_2020/Hayden/Hayden/bin/Debug/netcoreapp3.1/Hayden" "/home/USER/hayden_2020/hayden-config.json"</syntaxhighlight>
== Proxies ==
* UNTESTED BY AUTHOR
* Hayden supports proxies and these can be configured through the config file.
* The most common SSH server programs used by linux support acting as a proxy out of the box. You just have to tell it to start the proxy connection.
** To test if your machine is rigged for automated SSH access to some remote host, run this command:
**: <syntaxhighlight lang="bash">$ ssh username@host</syntaxhighlight>
** If it connects you to the remote host without needing to type things in, then your SSH key configuration is probably fine.
* To start a SOCKS proxy over SSH, giving you a proxy accessible at the address localhost:10000:
*: <syntaxhighlight lang="bash">$ ssh -D 10000 user@<remote_host_address></syntaxhighlight>
* To tell hayden to use a proxy:
*: <syntaxhighlight lang="json">
"proxies" : [
{
"type" : "socks",
"url" : "http://au393.nordvpn.com:1080",
"username" : "[email protected]",
"password" : "bigguy4u"
}
],
</syntaxhighlight>
* To tell hayden to use a proxy:
*: <syntaxhighlight lang="json">
"proxies" : [
{
"type" : "socks",
"url" : "http://localhost:10000",
"username" : "TODO",
"password" : "TODO"
}
],
</syntaxhighlight>
== Tips ==
* Use absolute filepaths instead of relative filepaths whenever you can, as it makes it harder to screw up as bad.
*: e.g. <code>/home/dir/subdir/file</code> instead of <code>./subdir/file</code> instead of <code>subdir/file</code>
* Back up your database frequently and regularly (i.e. mysqldump or similar)
** Consider scheduling this via cron to occur automatically.
* Run your programs inside screen/tmux/byobu so they dont get closed from you disconnecting your terminal.
* Use a good password. <code>test</code> is a bad password.
** Spaces need extra consideration regarding escaping, so try to avoid them.
** Alphanumerical gibberish is hard to remember and to type in. <code>a2Ggkp5xx1</code>
** Passphrases are easy to remember and to type in. <code>White_horse_lion_wave</code>
** Write down your usernames and passwords so you dont lose them.
* Thumbnails can probably be generated from full images and thus can be turned off if you dont plan on running a public archive.
* Save any error messages to a plain text file via copy/paste otherwise you won't be able to get real help from any experts.
== Example files ==
* These are what the configuration files should broardly look like.
=== Example config.json ===
* The following configuration file code instructs Hayden to download the 4chan /vmg/ board, including the old threads temporarily retained by 4chan's archive.json API endpoint.
*: <syntaxhighlight lang="json">
{
"source" : {
"type" : "4chan",
"boards" : [
"vmg"
],
"apiDelay" : 0.8,
"boardDelay" : 30,
"ReadArchive": true
},
"proxies" : [
],
"backend" : {
"type" : "Asagi",
"connectionString" : "Server=localhost;Port=10306;Database=hayden_db;Uid=hayden_un;Pwd=hayden_pw;CharSet=utf8mb4;IgnorePrepare=false",
"downloadLocation" : "/home/USER/hayden_2020/dl/",
"fullImagesEnabled" : true,
"thumbnailsEnabled" : true,
"sqlConnectionPoolSize" : 12
}
}
</syntaxhighlight>
=== Example docker-compose.yml ===
* The following configuration file defines two containers for docker-compose .
** A MariaDB database container <code>db</code>
** A adminer DB webUI container <code>adminer</code>
*: <syntaxhighlight lang="bash">
# https://hub.docker.com/_/mariadb/
version: '3.1'
services:
  db: # SQL DB server
    image: mariadb
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: "root_super_secret_code"
      MYSQL_DATABASE: hayden_db
      MYSQL_USER: hayden_un
      MYSQL_PASSWORD: "hayden_super_secret_cod"
    ports:
      - "10306:3306" # HOST:CONTAINER
    volumes:
      - "/home/USER/hayden-2020/mariadb_docker:/var/lib/mysql" # HOST:CONTAINER
      - "/home/USER/hayden-2020/mariadb_docker_dump:/dump" # HOST:CONTAINER
  adminer: # WebGUI for SQL DB server
    image: adminer
    restart: always
    ports:
      - 10088:8080
</syntaxhighlight>

Latest revision as of 05:46, 16 August 2020

Guide on how to setup and run hayden on a linux machine.


Warnings[edit]

  • Blame Ctrl-S for this document. (WIP)
  • Hayden is not really reliably stable on linux right now and may decide to enter a state where it will reliably crash on startup.
  • Hayden cares about the JSON config file syntax. If you missplace a comma it will probably reject the config file and crash.


Prerequisites[edit]

How to set up a system so hayden will download, build, and run.

  1. Prep system for updates
    • Debian-based systems (ubuntu, debian, etc)
    $ sudo apt update && sudo apt upgrade -y
    $ sudo apt install -y git curl wget
    
    • RHEL-based systems (redhat, fedora, centos)
    $ sudo yum update -y
    

Install dotnet[edit]

  1. Install Microsoft dotnet runtime

https://docs.microsoft.com/en-us/dotnet/core/install/linux-centos
https://docs.microsoft.com/en-us/dotnet/core/install/linux-ubuntu

    • RHEL-based systems (redhat, fedora, centos)
    $ sudo dnf install dotnet-sdk-3.1
    
    • Debian-based systems (ubuntu, debian, etc)
    $ sudo apt-get install -y dotnet-sdk-3.1
    

Database install[edit]

Hayden works with MariaDB and I dont know if it works with others.
You could use any MariaDB installation if you wanted to, however that is left to advanced users who know how and is outside the scope of this guide.

Docker+Mariadb - Install Docker[edit]

https://docs.docker.com/engine/install/#server
https://docs.docker.com/engine/install/centos/

  1. Setup repository:
    $ sudo yum install -y yum-utils
    $ sudo yum-config-manager  --add-repo  https://download.docker.com/linux/centos/docker-ce.repo
    
  2. Install docker engine:
    $ sudo yum install docker-ce docker-ce-cli containerd.io
    
  3. Start docker service:
    $ sudo systemctl start docker
    
  4. Verify docker works
    $ sudo docker run hello-world
    

Docker+Mariadb - Install docker-compose[edit]

https://docs.docker.com/compose/

  1. Install docker-compose:
    $ sudo curl -L "https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
    $ sudo chmod +x /usr/local/bin/docker-compose
    
  2. Symlink into $PATH:
    $ sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
    
    • This is so that you do not need to type the absolute path to run the commands.
  3. Test installation:
    $ docker-compose --version
    

Docker+Mariadb - Setup Container[edit]

  1. Prepare container definition
  • Example docker-compose definition file:
    # mariadb.yml
    # https://hub.docker.com/_/mariadb/
    version: '3.1'
    services:
      db: # SQL DB server
        image: mariadb
        restart: always
        environment:
          MYSQL_ROOT_PASSWORD: "root_super_secret_code"
          MYSQL_DATABASE: hayden_db
          MYSQL_USER: hayden_un
          MYSQL_PASSWORD: "hayden_super_secret_cod"
        ports:
          - "10306:3306" # HOST:CONTAINER
        volumes:
          - "/home/USER/hayden-2020/mariadb_docker:/var/lib/mysql" # HOST:CONTAINER
          - "/home/USER/hayden-2020/mariadb_docker_dump:/dump" # HOST:CONTAINER
      adminer: # WebGUI for SQL DB server
        image: adminer
        restart: always
        ports:
          - 10088:8080
    
    • Ports in the 10XXX range are used in this example on the host side in an attempt to avoid conflicting with other programs wanting specific ports.
    • In practice, any port above 1024 which is not being used by another program should work.
  1. Bring up defined containers
    • Bring up defined containers in the foreground of your terminal:
      Not reccomended outside of testing and development, as it is easy to close by mistake.
      sudo docker-compose -f "/home/USER/hayden-2020/mariadb.yml" up
      
    • Bring up defined containers in the background ("-d" CLI flag):
      Reccomended, as this is harder to close by mistake.
      sudo docker-compose -f "/home/USER/hayden-2020/mariadb.yml" -d up
      

Docker+Mariadb - Usage[edit]

  • TODO
  • To connect to the DB through adminer:
    TODO
  • To make a DB dump:
    1. Find the name or ID of the DB container
    2. Run the dump command targeting that container
      $ sudo docker exec hayden-2020_db_1 sh -c 'exec mysqldump --all-databases -u"root" -p"YOUR_PASSWORD_HERE"' > "/home/USER/hayden-2020/mariadb-container-all-databases.ts`date -u +%s`.sql"
      
      • Note that the dump command in the container outputs the DB dump SQL as text on its STDOUT; which is sent back from the container to the host; and that is redirected into a file on the host.
      • Explaination:
        • $ sudo docker exec hayden-2020_db_1 sh -c Runs a command inside a container.
        • 'exec mysqldump --all-databases -u"root" -p"YOUR_PASSWORD_HERE"' Dumps a copy of the DB to STDOUT
        • > "/home/USER/hayden-2020/mariadb-container-all-databases.ts`date -u +%s`.sql" Redirects the output into a file with a timestamp in the name.
        • date -u +%s Produces a timestamp in seconds since the epoch. e.g. 1597472305.
      • This is done using the DB root account as it always has full access to the entire database.

Download and Build Hayden[edit]

  1. Prepare location:
    $ mkdir -vp /home/USER/hayden_2020/
    $ cd /home/USER/hayden_2020/
    
  2. Clone (download) Hayden source code from Github:
    $ git clone https://github.com/bbepis/Hayden.git
    
  3. Build the Hayden solution file:
    $ dotnet build "/home/USER/hayden_2020/Hayden/Hayden.sln"
    
  4. Run with config file:
    $ "/home/USER/hayden_2020/Hayden/Hayden/bin/Debug/netcoreapp3.1/Hayden" "/home/USER/hayden_2020/hayden-config.json"
    

Configuration[edit]

  • Configuration for Hayden is done through a JSON file that is specified as a command-line argument when starting Hayden.
$ "/home/USER/hayden_2020/Hayden/Hayden/bin/Debug/netcoreapp3.1/Hayden" "/home/USER/hayden_2020/hayden-config.json"
  • Hayden is picy about the syntax of this JSON file.
    • Commas only go between key:value pars. Trailing commas are not tolerated by Hayden.
  • Some options are implimented but not shown in the example JSON file.
    • The option to scan archive.json to save threads that have recently fallen off the board: <code"ReadArchive": true
      This value is a child of "source", as shown here:
      {
      	"source" : {
      		"type" : "4chan",
      		"boards" : [
      			"vmg"
      		],
      		"apiDelay" : 0.8,
      		"boardDelay" : 30,
      		"ReadArchive": true
      	},
      
  • Some databses will not let you connect as the DB root account from a system user other than root.
    • As a result of this, you should create a DB user specifically for Hayden to use.
  • The database connection is defined by this string value:
    "connectionString" : "Server=DB_NETWORK_ADDRESS;Port=DB_PORT;Database=DATABASE;Uid=DB_USERNAME;Pwd=DB_PASSWORD;CharSet=utf8mb4;IgnorePrepare=false",
    
    • e.g.
      "connectionString" : "Server=localhost;Port=3306;Database=asagi;Uid=hayden;Pwd=some_secret_value;CharSet=utf8mb4;IgnorePrepare=false",
      
  • Image downloading is controlled by two options:
    "fullImagesEnabled" : true,

    "thumbnailsEnabled" : false,
    • These values must either be false ortrue.


Running[edit]

  1. Ensure DB is running
    • Hayden requires the database to be available to function.
    • See the database sections of this document for information on setting up and running a database in a container.
  2. Start Hayden
    • It is STRONGLY RECCOMENDED to run Hayden inside a Ccreen / Tmux / Byobu session so Hayden persists after your terminal disconnects.
      e.g. One of the following three commands:
      $ screen
      
      $ tmux
      
      $ byobu
      
    • Start Hayden itself:
      • $ "/home/USER/hayden_2020/Hayden/Hayden/bin/Debug/netcoreapp3.1/Hayden" "/home/USER/hayden_2020/hayden-config.json"
        

Proxies[edit]

  • UNTESTED BY AUTHOR
  • Hayden supports proxies and these can be configured through the config file.
  • The most common SSH server programs used by linux support acting as a proxy out of the box. You just have to tell it to start the proxy connection.
    • To test if your machine is rigged for automated SSH access to some remote host, run this command:
      $ ssh username@host
      
    • If it connects you to the remote host without needing to type things in, then your SSH key configuration is probably fine.
  • To start a SOCKS proxy over SSH, giving you a proxy accessible at the address localhost:10000:
    $ ssh -D 10000 user@<remote_host_address>
    
  • To tell hayden to use a proxy:
    	"proxies" : [
    		{
    			"type" : "socks",
    			"url" : "http://au393.nordvpn.com:1080",
    			"username" : "[email protected]",
    			"password" : "bigguy4u"
    		}
    	],
    
  • To tell hayden to use a proxy:
    	"proxies" : [
    		{
    			"type" : "socks",
    			"url" : "http://localhost:10000",
    			"username" : "TODO",
    			"password" : "TODO"
    		}
    	],
    


Tips[edit]

  • Use absolute filepaths instead of relative filepaths whenever you can, as it makes it harder to screw up as bad.
    e.g. /home/dir/subdir/file instead of ./subdir/file instead of subdir/file
  • Back up your database frequently and regularly (i.e. mysqldump or similar)
    • Consider scheduling this via cron to occur automatically.
  • Run your programs inside screen/tmux/byobu so they dont get closed from you disconnecting your terminal.
  • Use a good password. test is a bad password.
    • Spaces need extra consideration regarding escaping, so try to avoid them.
    • Alphanumerical gibberish is hard to remember and to type in. a2Ggkp5xx1
    • Passphrases are easy to remember and to type in. White_horse_lion_wave
    • Write down your usernames and passwords so you dont lose them.
  • Thumbnails can probably be generated from full images and thus can be turned off if you dont plan on running a public archive.
  • Save any error messages to a plain text file via copy/paste otherwise you won't be able to get real help from any experts.

Example files[edit]

  • These are what the configuration files should broardly look like.

Example config.json[edit]

  • The following configuration file code instructs Hayden to download the 4chan /vmg/ board, including the old threads temporarily retained by 4chan's archive.json API endpoint.
    {
    	"source" : {
    		"type" : "4chan",
    		"boards" : [
    			"vmg"
    		],
    		"apiDelay" : 0.8,
    		"boardDelay" : 30,
    		"ReadArchive": true
    	},
    
    	"proxies" : [
    	],
    	
    	"backend" : {
    		"type" : "Asagi",
    		
    		"connectionString" : "Server=localhost;Port=10306;Database=hayden_db;Uid=hayden_un;Pwd=hayden_pw;CharSet=utf8mb4;IgnorePrepare=false",
    		
    		"downloadLocation" : "/home/USER/hayden_2020/dl/",
    		
    		"fullImagesEnabled" : true,
    		"thumbnailsEnabled" : true,
    		
    		"sqlConnectionPoolSize" : 12
    	}
    }
    

Example docker-compose.yml[edit]

  • The following configuration file defines two containers for docker-compose .
    • A MariaDB database container db
    • A adminer DB webUI container adminer
    # https://hub.docker.com/_/mariadb/
    version: '3.1'
    services:
      db: # SQL DB server
        image: mariadb
        restart: always
        environment:
          MYSQL_ROOT_PASSWORD: "root_super_secret_code"
          MYSQL_DATABASE: hayden_db
          MYSQL_USER: hayden_un
          MYSQL_PASSWORD: "hayden_super_secret_cod"
        ports:
          - "10306:3306" # HOST:CONTAINER
        volumes:
          - "/home/USER/hayden-2020/mariadb_docker:/var/lib/mysql" # HOST:CONTAINER
          - "/home/USER/hayden-2020/mariadb_docker_dump:/dump" # HOST:CONTAINER
      adminer: # WebGUI for SQL DB server
        image: adminer
        restart: always
        ports:
          - 10088:8080