Latest revision |
Your text |
Line 21: |
Line 21: |
| </syntaxhighlight> | | </syntaxhighlight> |
|
| |
|
| === Install dotnet ===
| |
| # Install Microsoft dotnet runtime | | # Install Microsoft dotnet runtime |
| https://docs.microsoft.com/en-us/dotnet/core/install/linux-centos | | https://docs.microsoft.com/en-us/dotnet/core/install/linux-centos |
Line 48: |
Line 47: |
| # Verify docker works | | # Verify docker works |
| #: <syntaxhighlight lang="bash">$ sudo docker run hello-world</syntaxhighlight> | | #: <syntaxhighlight lang="bash">$ sudo docker run hello-world</syntaxhighlight> |
| | |
|
| |
|
| ==== Docker+Mariadb - Install docker-compose ==== | | ==== Docker+Mariadb - Install docker-compose ==== |
Line 58: |
Line 58: |
| # Symlink into $PATH: | | # Symlink into $PATH: |
| #: <syntaxhighlight lang="bash">$ sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose</syntaxhighlight> | | #: <syntaxhighlight lang="bash">$ sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose</syntaxhighlight> |
| #* This is so that you do not need to type the absolute path to run the commands.
| |
| # Test installation: | | # Test installation: |
| #: <syntaxhighlight lang="bash">$ docker-compose --version</syntaxhighlight> | | #: <syntaxhighlight lang="bash">$ docker-compose --version</syntaxhighlight> |
|
| |
|
| ==== Docker+Mariadb - Setup Container ==== | | ==== Docker+Mariadb - Setup Container ==== |
| #: Prepare container definition
| | * TODO |
| * Example docker-compose definition file:
| | |
| *: <syntaxhighlight lang="bash">
| | ==== Docker+Mariadb - Setup DB ==== |
| # mariadb.yml
| | * TODO |
| # https://hub.docker.com/_/mariadb/
| |
| version: '3.1'
| |
| services:
| |
| db: # SQL DB server
| |
| image: mariadb
| |
| restart: always
| |
| environment:
| |
| MYSQL_ROOT_PASSWORD: "root_super_secret_code"
| |
| MYSQL_DATABASE: hayden_db
| |
| MYSQL_USER: hayden_un
| |
| MYSQL_PASSWORD: "hayden_super_secret_cod"
| |
| ports:
| |
| - "10306:3306" # HOST:CONTAINER
| |
| volumes:
| |
| - "/home/USER/hayden-2020/mariadb_docker:/var/lib/mysql" # HOST:CONTAINER
| |
| - "/home/USER/hayden-2020/mariadb_docker_dump:/dump" # HOST:CONTAINER
| |
| adminer: # WebGUI for SQL DB server
| |
| image: adminer
| |
| restart: always
| |
| ports:
| |
| - 10088:8080
| |
| </syntaxhighlight>
| |
| #* Ports in the 10XXX range are used in this example on the host side in an attempt to avoid conflicting with other programs wanting specific ports.
| |
| #* In practice, any port above 1024 which is not being used by another program should work.
| |
| # Bring up defined containers
| |
| #* Bring up defined containers in the foreground of your terminal:
| |
| #*: Not reccomended outside of testing and development, as it is easy to close by mistake.
| |
| #*: <syntaxhighlight lang="bash">sudo docker-compose -f "/home/USER/hayden-2020/mariadb.yml" up</syntaxhighlight>
| |
| #* Bring up defined containers in the background ("-d" CLI flag):
| |
| #*: Reccomended, as this is harder to close by mistake.
| |
| #*: <syntaxhighlight lang="bash">sudo docker-compose -f "/home/USER/hayden-2020/mariadb.yml" -d up</syntaxhighlight>
| |
|
| |
|
| ==== Docker+Mariadb - Usage ==== | | ==== Docker+Mariadb - Usage ==== |
| * TODO | | * TODO |
| * To connect to the DB through adminer:
| |
| *: TODO
| |
| * To make a DB dump:
| |
| *# Find the name or ID of the DB container
| |
| *#:
| |
| *# Run the dump command targeting that container
| |
| *#: <syntaxhighlight lang="bash">$ sudo docker exec hayden-2020_db_1 sh -c 'exec mysqldump --all-databases -u"root" -p"YOUR_PASSWORD_HERE"' > "/home/USER/hayden-2020/mariadb-container-all-databases.ts`date -u +%s`.sql" </syntaxhighlight>
| |
| *#* Note that the dump command in the container outputs the DB dump SQL as text on its STDOUT; which is sent back from the container to the host; and that is redirected into a file on the host.
| |
| *#* Explaination:
| |
| *#** <code>$ sudo docker exec hayden-2020_db_1 sh -c</code> Runs a command inside a container.
| |
| *#** <code>'exec mysqldump --all-databases -u"root" -p"YOUR_PASSWORD_HERE"'</code> Dumps a copy of the DB to STDOUT
| |
| *#** <code> > "/home/USER/hayden-2020/mariadb-container-all-databases.ts`date -u +%s`.sql"</code> Redirects the output into a file with a timestamp in the name.
| |
| *#** <code>date -u +%s</code> Produces a timestamp in seconds since the epoch. e.g. <code>1597472305</code>.
| |
| *#* This is done using the DB root account as it always has full access to the entire database.
| |
|
| |
|
| == Download and Build Hayden == | | == Download and Build == |
| # Prepare location: | | # Prepare location: |
| #: <syntaxhighlight lang="bash"> | | #: <syntaxhighlight lang="bash"> |
Line 136: |
Line 90: |
| * Some options are implimented but not shown in the example JSON file. | | * Some options are implimented but not shown in the example JSON file. |
| ** The option to scan archive.json to save threads that have recently fallen off the board: <code"ReadArchive": true</code> | | ** The option to scan archive.json to save threads that have recently fallen off the board: <code"ReadArchive": true</code> |
| **: This value is a child of "source", as shown here:
| |
| **: <syntaxhighlight lang="json">
| |
| {
| |
| "source" : {
| |
| "type" : "4chan",
| |
| "boards" : [
| |
| "vmg"
| |
| ],
| |
| "apiDelay" : 0.8,
| |
| "boardDelay" : 30,
| |
| "ReadArchive": true
| |
| },
| |
| </syntaxhighlight>
| |
| * Some databses will not let you connect as the DB root account from a system user other than root. | | * Some databses will not let you connect as the DB root account from a system user other than root. |
| ** As a result of this, you should create a DB user specifically for Hayden to use. | | ** As a result of this, you should create a DB user specifically for Hayden to use. |
| * The database connection is defined by this string value: | | * The database connection is defined by this string value: |
| *: <syntaxhighlight lang="json">"connectionString" : "Server=DB_NETWORK_ADDRESS;Port=DB_PORT;Database=DATABASE;Uid=DB_USERNAME;Pwd=DB_PASSWORD;CharSet=utf8mb4;IgnorePrepare=false",</syntaxhighlight>
| | <syntaxhighlight lang="json">"connectionString" : "Server=DB_NETWORK_ADDRESS;Port=DB_PORT;Database=DATABASE;Uid=DB_USERNAME;Pwd=DB_PASSWORD;CharSet=utf8mb4;IgnorePrepare=false",</syntaxhighlight> |
| ** e.g. <syntaxhighlight lang="json">"connectionString" : "Server=localhost;Port=3306;Database=asagi;Uid=hayden;Pwd=some_secret_value;CharSet=utf8mb4;IgnorePrepare=false",</syntaxhighlight>
| | <br>e.g. <syntaxhighlight lang="json">"connectionString" : "Server=localhost;Port=3306;Database=asagi;Uid=hayden;Pwd=some_secret_value;CharSet=utf8mb4;IgnorePrepare=false",</syntaxhighlight> |
| * Image downloading is controlled by two options: | | * Image downloading is controlled by two options: |
| *: <code>"fullImagesEnabled" : true,</code>
| | <br><code>"fullImagesEnabled" : true,</code> |
| *: <br><code>"thumbnailsEnabled" : false,</code>
| | <br><code>"thumbnailsEnabled" : false,</code> |
| ** These values must either be <code>false</code> or<code>true</code>. | | ** These values must either be <code>false</code> or<code>true</code>. |
|
| |
|
| |
| == Running ==
| |
| # Ensure DB is running
| |
| #* Hayden requires the database to be available to function.
| |
| #* See the database sections of this document for information on setting up and running a database in a container.
| |
| # Start Hayden
| |
| #* It is STRONGLY RECCOMENDED to run Hayden inside a Ccreen / Tmux / Byobu session so Hayden persists after your terminal disconnects.
| |
| #*: e.g. One of the following three commands:
| |
| #*: <syntaxhighlight lang="bash">$ screen</syntaxhighlight>
| |
| #*: <syntaxhighlight lang="bash">$ tmux</syntaxhighlight>
| |
| #*: <syntaxhighlight lang="bash">$ byobu</syntaxhighlight>
| |
| #* Start Hayden itself:
| |
| #** <syntaxhighlight lang="bash">$ "/home/USER/hayden_2020/Hayden/Hayden/bin/Debug/netcoreapp3.1/Hayden" "/home/USER/hayden_2020/hayden-config.json"</syntaxhighlight>
| |
|
| |
| == Proxies ==
| |
| * UNTESTED BY AUTHOR
| |
| * Hayden supports proxies and these can be configured through the config file.
| |
| * The most common SSH server programs used by linux support acting as a proxy out of the box. You just have to tell it to start the proxy connection.
| |
| ** To test if your machine is rigged for automated SSH access to some remote host, run this command:
| |
| **: <syntaxhighlight lang="bash">$ ssh username@host</syntaxhighlight>
| |
| ** If it connects you to the remote host without needing to type things in, then your SSH key configuration is probably fine.
| |
| * To start a SOCKS proxy over SSH, giving you a proxy accessible at the address localhost:10000:
| |
| *: <syntaxhighlight lang="bash">$ ssh -D 10000 user@<remote_host_address></syntaxhighlight>
| |
| * To tell hayden to use a proxy:
| |
| *: <syntaxhighlight lang="json">
| |
| "proxies" : [
| |
| {
| |
| "type" : "socks",
| |
| "url" : "http://au393.nordvpn.com:1080",
| |
| | |
| "password" : "bigguy4u"
| |
| }
| |
| ],
| |
| </syntaxhighlight>
| |
| * To tell hayden to use a proxy:
| |
| *: <syntaxhighlight lang="json">
| |
| "proxies" : [
| |
| {
| |
| "type" : "socks",
| |
| "url" : "http://localhost:10000",
| |
| "username" : "TODO",
| |
| "password" : "TODO"
| |
| }
| |
| ],
| |
| </syntaxhighlight>
| |
|
| |
|
| |
| == Tips ==
| |
| * Use absolute filepaths instead of relative filepaths whenever you can, as it makes it harder to screw up as bad.
| |
| *: e.g. <code>/home/dir/subdir/file</code> instead of <code>./subdir/file</code> instead of <code>subdir/file</code>
| |
| * Back up your database frequently and regularly (i.e. mysqldump or similar)
| |
| ** Consider scheduling this via cron to occur automatically.
| |
| * Run your programs inside screen/tmux/byobu so they dont get closed from you disconnecting your terminal.
| |
| * Use a good password. <code>test</code> is a bad password.
| |
| ** Spaces need extra consideration regarding escaping, so try to avoid them.
| |
| ** Alphanumerical gibberish is hard to remember and to type in. <code>a2Ggkp5xx1</code>
| |
| ** Passphrases are easy to remember and to type in. <code>White_horse_lion_wave</code>
| |
| ** Write down your usernames and passwords so you dont lose them.
| |
| * Thumbnails can probably be generated from full images and thus can be turned off if you dont plan on running a public archive.
| |
| * Save any error messages to a plain text file via copy/paste otherwise you won't be able to get real help from any experts.
| |
|
| |
| == Example files ==
| |
| * These are what the configuration files should broardly look like.
| |
|
| |
| === Example config.json ===
| |
| * The following configuration file code instructs Hayden to download the 4chan /vmg/ board, including the old threads temporarily retained by 4chan's archive.json API endpoint.
| |
| *: <syntaxhighlight lang="json">
| |
| {
| |
| "source" : {
| |
| "type" : "4chan",
| |
| "boards" : [
| |
| "vmg"
| |
| ],
| |
| "apiDelay" : 0.8,
| |
| "boardDelay" : 30,
| |
| "ReadArchive": true
| |
| },
| |
|
| |
| "proxies" : [
| |
| ],
| |
|
| |
| "backend" : {
| |
| "type" : "Asagi",
| |
|
| |
| "connectionString" : "Server=localhost;Port=10306;Database=hayden_db;Uid=hayden_un;Pwd=hayden_pw;CharSet=utf8mb4;IgnorePrepare=false",
| |
|
| |
| "downloadLocation" : "/home/USER/hayden_2020/dl/",
| |
|
| |
| "fullImagesEnabled" : true,
| |
| "thumbnailsEnabled" : true,
| |
|
| |
| "sqlConnectionPoolSize" : 12
| |
| }
| |
| }
| |
| </syntaxhighlight>
| |
|
| |
| === Example docker-compose.yml ===
| |
| * The following configuration file defines two containers for docker-compose .
| |
| ** A MariaDB database container <code>db</code>
| |
| ** A adminer DB webUI container <code>adminer</code>
| |
| *: <syntaxhighlight lang="bash">
| |
| # https://hub.docker.com/_/mariadb/
| |
| version: '3.1'
| |
| services:
| |
| db: # SQL DB server
| |
| image: mariadb
| |
| restart: always
| |
| environment:
| |
| MYSQL_ROOT_PASSWORD: "root_super_secret_code"
| |
| MYSQL_DATABASE: hayden_db
| |
| MYSQL_USER: hayden_un
| |
| MYSQL_PASSWORD: "hayden_super_secret_cod"
| |
| ports:
| |
| - "10306:3306" # HOST:CONTAINER
| |
| volumes:
| |
| - "/home/USER/hayden-2020/mariadb_docker:/var/lib/mysql" # HOST:CONTAINER
| |
| - "/home/USER/hayden-2020/mariadb_docker_dump:/dump" # HOST:CONTAINER
| |
| adminer: # WebGUI for SQL DB server
| |
| image: adminer
| |
| restart: always
| |
| ports:
| |
| - 10088:8080
| |
| </syntaxhighlight>
| |