1,547 bytes removed, 3 months ago
Ayase Imageboard Archival Standard (Ayase)
Hayden source code :
= Ayase Imageboard Archival Standard (Ayase) =
The Ayase Imageboard Archival Standard was produced by the Bibliotheca Anonoma to handle the ever growing operations of Desuarchive and RebeccaBlackTech.
== Reference Implementation ==
* Operating System: CentOS/RHEL 8
* Database: PostgreSQL
* Scraper: Ena or Hydrus (.NET C#)
* Middleware: Ayase (Python PyPy)
* Frontends: 4chan X, Clover, iphone app
== Specifications ==
=== Files ===
* All files are to be named by shA256sum and file extension. This was chosen for the broad availability of hardware xtensions for the purpose nd its use by 8chan/vichan.
* They are to be stored in double nested folders.
=== Time ===
* Ayase requires time to be stored in PostgreSQL datetimes, which also store timezones.
* Only UTC should be used as the timezone for newly scraped data. The timezone support is not an excuse to store in other timezones.
* The timezone support is only meant for compatibility purposes with prior Asagi data, given that they store time as US time (maybe Eastern) due to their past HTML scraping. Future scrapes are strongly advised not to replicate this behavior, local time should be up to the frontend to determine.
=== PostgreSQL Schema ===
if we GET json from the 4chan API, and always serve the same json to the user, why deconstruct and reconstruct into post focused sql records every time?
=== Elasticsearch Engine ===
A seperate elastic search engine kept in sync with, but independent from the sql server, will replace Sphinxsearch which queries the mysql db