| Latest revision |
Your text |
| Line 1: |
Line 1: |
| = Fuuka Imageboard Archival Standard (Fuuka) =
| | How asagi does stuff |
|
| |
|
| <blockquote>Note: Not to be confused with the FoolFuuka frontend, which uses the Asagi scraper.
| | When asagi does a thread update: |
| </blockquote>
| |
| https://github.com/eksopl/fuuka/wiki/Sphinx-Search-Backend#gory-details
| |
|
| |
|
| = Asagi Imageboard Archival Standard (Asagi) =
| | in : YotsubaJSON.java, ln. 88: |
|
| |
|
| The Asagi Imageboard Archival Standard was developed by eksopl of Easymodo and the Foolz team under the direction of woxxy. It was developed to run the Foolz archiver, and has been the engine for the majority of archivers since the collapse of Archive.moe.
| | `public Topic getThread(int threadNum, String lastMod) throws ContentGetException, ContentParseException, CfBicClearParseException {` |
|
| |
|
| == SQL Schema ==
| | Loads thread JSON |
|
| |
|
| Two versions of the SQL Schema in use can be identified:
| | Decodes JSON |
|
| |
|
| * 1.0.0 (2013) - The final version by Eksopl for Foolz.us, and the SQL schema was unchanged in Archive.moe.
| | For each post in the decoded thread JSON: |
| ** Should be used by Fireden, but this is unknown.
| |
| ** Might be still used by Nyafuu, was in use by Loveisover.
| |
| * 1.3.0 (2019) - (only Mysql/triggers.sql was changed, no other structural SQL schema changes) The final reference standard consolidated from 4plebs repos by the Bibliotheca Anonoma for use in Desuarchive, in preparation for the development of new drop-in replacements.
| |
| ** Used by Desuarchive, Rbt, arch.b4k.co, maybe 4plebs?
| |
|
| |
|
| A refined version 2.0.0 is proposed that would eliminate SQL triggers for improved performance, instead leaving it up to the scraper engines to conduct similar operations as the triggers.
| | Check if resto value is zero, and if so create a new thread from that post, updating lastmodified time to the time from fetching the JSON. |
| | `t = this.makeThreadFromJson(pj);` |
|
| |
|
| == Reference Implementation ==
| | If resto is zero: Add the post to the current thread. |
|
| |
|
| * Operating System: Ubuntu 14.04/16.04 LTS
| | `t.addPost(this.makePostFromJson(pj));` |
| * Database: MySQL Compatible utf8mb4
| | (What if two posts were resto==0? We’d break!) |
| ** Despite the fact that PostgreSQL is supported by Asagi Scraper, it is not supported by FoolFuuka.
| |
| * Scraper: Asagi (Java) - https://github.com/bibanon/asagi
| |
| * Frontend: FoolFuuka (PHP) - https://github.com/bibanon/FoolFuuka
| |
| * PHP Engine: Historically HHVM, PHP5.x compatible. Desuarchive and 4plebs now uses PHP 7.
| |
| * Search: Sphinxsearch
| |
| | |
| == New Implementation ==
| |
| | |
| Proposed, still needs to be constructed.
| |
| | |
| * Database: Percona MariaDB TokuDB - Special variant of MariaDB with many unique optimizations.
| |
| * Scraper: baystdev/Hayden (.NET C#)
| |
| * Frontend: https://github.com/pleebe/FoolFuuka/tree/experimental
| |
| * New Frontend: some python based 4chan API compatible middleware
| |
| | |
| == Compilation and Usage ==
| |
| | |
| === Asagi ===
| |
| | |
| https://github.com/eksopl/asagi/wiki/Running-Asagi
| |
| | |
| Also check [[FoolFuuka/Install/Ubuntu16#Install_and_compile_Asagi_from_source.|FoolFuuka/Install/Ubuntu16#Install_and_compile_Asagi_from_source.]]
| |
| | |
| === FoolFuuka ===
| |
| | |
| https://blog.foolz.us/
| |
| | |
| Also check [[FoolFuuka/Install/Ubuntu16|FoolFuuka/Install/Ubuntu16]]
| |
| | |
| = How Asagi does stuff =
| |
| | |
| == Configuration ==
| |
| | |
| For example here is the config for Desuarchive:
| |
| | |
| <pre>{"settings": {
| |
| "dumperEngine": "DumperJSON",
| |
| "sourceEngine": "YotsubaJSON",
| |
| | |
| "boardSettings": {
| |
| "default": {
| |
| "engine": "Mysql",
| |
| "database": "asagi",
| |
| "host": "localhost",
| |
| "username": "asagi",
| |
| "password": "YOUR_PASSWORD_HERE,
| |
| "charset": "utf8mb4",
| |
| "path": "/srv/foolfuuka/boards",
| |
| "updateFileLastModified": false,
| |
| "useOldDirectoryStructure": false,
| |
| "webserverGroup": "www-data",
| |
| "thumbThreads": 2,
| |
| "mediaThreads": 2,
| |
| "newThreadsThreads": 6,
| |
| "deletedThreadsThresholdPage": 8,
| |
| "refreshDelay": 60,
| |
| "throttleAPI": false,
| |
| "throttleURL": "i.4cdn.org",
| |
| "throttleMillisec": 1050,
| |
| "threadRefreshRate": 50
| |
| },
| |
| | |
| "mlp": {},
| |
| "qa": {},
| |
| "aco": {},
| |
| "tg": {},
| |
| "d": {},
| |
| "co": {},
| |
| "a": {},
| |
| "an": {},
| |
| "k": {},
| |
| "fit": {},
| |
| "wsg": {"mediaThreads": 0},
| |
| "gif": {"mediaThreads": 0},
| |
| "r9k": {},
| |
| "int": {},
| |
| "c": {},
| |
| "m": {},
| |
| "vr": {},
| |
| "his": {},
| |
| "trash": {},
| |
| "cgl": {},
| |
| "g": {},
| |
| "mu": {}
| |
| }
| |
| }}</pre>
| |
| == Misc ==
| |
| | |
| Seperate thread for (thumbs, media, topic-fetch, topic-insert, post-delete) per board
| |
| | |
| AbstractDumper.java ln. 95: <code>public void initDumper(BoardSettings boardSettings) {</code>
| |
| | |
| == How Asagi decides to update a thread: ==
| |
| | |
| In <code>initDumper()</code>, <code>DumperJSON</code> spawns an instance of its inner class <code>BoardPoller</code> on a thread. In its <code>run()</code> method, <code>BoardPoller</code> loops indefinitely:
| |
| | |
| * Wake up from sleeping (duration set by <code>refreshDelay</code> in the configuration)
| |
| * <code>threadList = sourceBoard.getAllThreads(lastMod);</code>
| |
| * If the request 304s or errors, go to sleep
| |
| * Go over the previous threads:
| |
| ** If this thread is in the current threads and it’s been modified, mark its modification timestamp and page number. Then, push it to <code>newTopics</code>.
| |
| ** If this thread is not in the current threads, it’s been deleted. Push it to <code>newTopics</code>.
| |
| * Put the remaining threads in newTopics
| |
| * Sleep until the delay expires
| |
| | |
| The queue <code>newTopics</code> is processed by <code>AbstractDumper</code>’s inner class <code>TopicFetcher</code>.
| |
| | |
| == When Asagi does a thread update: ==
| |
| | |
| in : YotsubaJSON.java, ln. 88: <code>public Topic getThread(int threadNum, String lastMod) throws ContentGetException, ContentParseException, CfBicClearParseException {</code>
| |
| | |
| Loads thread JSON Decodes JSON For each post in the decoded thread JSON: Check if resto value is zero, and if so create a new thread from that post, updating lastmodified time to the time from fetching the JSON. <code>t = this.makeThreadFromJson(pj);</code> If resto is zero: Add the post to the current thread. <code>t.addPost(this.makePostFromJson(pj));</code> (What if two posts were resto==0? We’d break!)
| |
| | |
| == What does Asagi do with a post in a thread? ==
| |
| | |
| Relevant files: >YousubaJSON.java >YotsubaAbstract.java
| |
| | |
| The entire thread is processed at once.
| |
| | |
| === For OP: ===
| |
| | |
| In YousubaJSON.java, ln. 217: <code>private Topic makeThreadFromJson(PostJson pj) throws ContentParseException {</code>
| |
| | |
| Ensure post number is zero. Create a new Topic() object (ln222): <code>Topic t = new Topic(pj.getNo(), pj.getOmittedPosts(), pj.getOmittedImages());</code>
| |
| | |
| Add the supplied OP to the thread as any other would be (ln. 224): <code>t.addPost(this.makePostFromJson(pj));</code>
| |
| | |
| Return the Thread() object. DONE
| |
| | |
| === For reply: ===
| |
| | |
| In YotsubaJSON.java, (ln. 157:) <code>private Post makePostFromJson(PostJson pj) throws ContentParseException {</code>
| |
| | |
| Ensure post number is valid. Ensure time is valid. Create new Post() object.
| |
| | |
| If the JSON gave a filename not equal to null: Generate filename from JSON values. <code>p.setMediaFilename(pj.getFilename() + pj.getExt());</code>
| |
| | |
| Generate original filename from JSON values. <code>p.setMediaOrig(pj.getTim() + pj.getExt());</code>
| |
| | |
| Generate preview original filename from JSON values. <code>p.setPreviewOrig(pj.getTim() + "s.jpg");</code>
| |
| | |
| Find the post’s capcode, and if not null: Manager -> “G” Anything else: first character as uppercase.
| |
| | |
| Find the post hash (tripcode?) If the post hash is “Developer”, set the post hash value to “Dev”
| |
| | |
| If the post country is not null: Convert values “XX” or “A1” if they are used to null.
| |
| | |
| Pass through values from JSON YotsubaJSON.java (ln. 188 -> 212), doing the following conversions:
| |
| | |
| Thread num to the current thread number: <code>.setThreadNum(pj.getResto() == 0 ? pj.getNo() : pj.getResto());</code>
| |
| | |
| OP status YotsubaJSON.java (ln. 197): <code>p.setOp(pj.getResto() == 0);</code>
| |
| | |
| Sanitized title? YotsubaJSON.java (ln. 198): <code>p.setTitle(this.cleanSimple(pj.getSub()));</code>
| |
| | |
| See YotsubaAbstract.java (ln.83): <code>public String doClean(String text)</code>
| |
| | |
| Sanitized name YotsubaJSON.java (ln. 200): <code>p.setName(this.cleanSimple(pj.getName()));</code>
| |
| | |
| Date converted from NYC_TIMEZONE (ln. 202): <code>p.setDate(DateUtils.adjustTimestampEpoch(pj.getTime(), DateUtils.NYC_TIMEZONE));</code>
| |
| | |
| Sanitized? EXIF data YotsubaJSON.java (ln. 212): <code>p.setExif(this.cleanSimple(this.parseMeta(pj.getCom(), pj.getUniqueIps(), pj.getSince4pass(), pj.getTrollCountry())));</code> Return the Post() object.
| |
| | |
| == How Asagi handles an image in a thread? ==
| |
| | |
| Files of note: >Local.java - Saving image files
| |
| | |
| Local.java ln.201: <code>public void insertMedia(MediaPost h, Board source, boolean isPreview) throws ContentGetException, ContentStoreException, CfBicClearParseException {</code>
| |
| | |
| == How Asagi deals with post deletions? ==
| |
| | |
| Relevant files: >src.java >src.java >src.java
| |
| | |
| SQL That handles post deletion logic: SQL.java (ln. 94): <code>“this.updateDeletedQuery = String.format("UPDATE \"%s\" SET deleted = ?, timestamp_expired = ? WHERE num = ? AND subnum = ?", this.table);</code>
| |
| | |
| This is prepared into statement, SQL.java (ln.62): <code>updateDeletedStmt = conn.prepareStatement(updateDeletedQuery);</code>
| |
| | |
| Function that actually writes deleted flag to a post: SQL.java (ln. 267): <code>public synchronized void markDeleted(DeletedPost post) throws ContentStoreException, DBConnectionException {</code>
| |
| | |
| Class TopicFetcher has function run() which calls findDeleted() See AbstractDumper.java (ln. 268): <code>protected class TopicFetcher implements Runnable {</code>
| |
| | |
| And AbstractDumper.java (ln. 380): <code>findDeleted(oldTopic, topic, true);</code>
| |
| | |
| Function findDeleted() checks Posts in a Topic to check if they have been removed in AbstractDumper.java (ln. 103): <code>protected boolean findDeleted(Topic oldTopic, Topic newTopic, boolean markDeleted) {</code>
| |
| | |
| Function markDeleted() in Local.java calls function markDeleted() from SQL.java: Local.java (ln. 185): <code>public void markDeleted(DeletedPost post) throws ContentStoreException {</code>
| |
| | |
| SQL.java (ln. 267): <code>public synchronized void markDeleted(DeletedPost post) throws ContentStoreException, DBConnectionException {</code>
| |
| | |
| == How Asagi interacts with the database? ==
| |
| | |
| Relevant files: >SQL.java >Mysql.java
| |
| | |
| Inserting a thread: SQL.java, ln.196: <code>public synchronized void insert(Topic topic) throws ContentStoreException, DBConnectionException {</code> Each post in the current Topic object is fed through the insert statement sequentially.
| |
| | |
| == How each value is processed between 4ch and the DB ==
| |
| | |
| 4chan API JSON broken down to post level (YoutsubaJSON.java ln.97), then: ?PostJson are created/populated from the thread JSON data, with values keeping the names from the API? (PostJson.java, ln.4) (DISREGARD CAPITALIZATION, GOOG DOCS FUCKS WITH IT. EVERYTHING IS LOWERCASE) DB column names referenced here: (SQL.java ln.79):
| |
| | |
| <pre>(poster_ip,
| |
| num,
| |
| subnum,
| |
| thread_num,
| |
| op,
| |
| timestamp,
| |
| preview_orig,
| |
| preview_w,
| |
| preview_h,
| |
| media_filename,
| |
| media_w,
| |
| media_h,
| |
| media_size,
| |
| media_hash,
| |
| media_orig,
| |
| spoiler,
| |
| deleted,
| |
| capcode,
| |
| email,
| |
| name,
| |
| trip,
| |
| title,
| |
| comment,
| |
| delpass,
| |
| sticky,
| |
| locked,
| |
| poster_hash,
| |
| poster_country,
| |
| exif)</pre>
| |
| === 4ch → (Asagi) → DB ===
| |
| | |
| ==== “poster_ip” ====
| |
| | |
| Presumably always NULL? PROVEME Does not appear to ever be initialized to a value in the java source code as far as i can tell?
| |
| | |
| ==== “no” - > “num” ====
| |
| | |
| Post ID number from 4chan. Passed through as-is.
| |
| | |
| ==== “subnum” ====
| |
| | |
| Ghostpost ID number for foolfuuka. Always zero.?
| |
| | |
| ==== “Thread_num” ====
| |
| | |
| Thread IP number, always Post ID number of OP. “num” of thread OP (YoutsubaJSON.java, ln. 196): <code>“p.setThreadNum(pj.getResto() == 0 ? pj.getNo() : pj.getResto());”</code>
| |
| | |
| ==== “Time” ->“timestamp” ====
| |
| | |
| Timestamp of post. 4ch API provides as milliseconds since 1 JAN 1970 in USA NYC timezone. DB stores as ?(Probably milliseconds since 1 JAN 1970 UTC+0) (YoutsubaJSON.java, ln. 202): <code>p.setDate(DateUtils.adjustTimestampEpoch(pj.getTime(), DateUtils.NYC_TIMEZONE));</code>
| |
| | |
| ==== “resto” -> (If resto is 0, use value from “num”) -> op ====
| |
| | |
| Is this the OP of the thread? (YoutsubaJSON.java, ln. 197): <code>p.setOp(pj.getResto() == 0);</code>
| |
| | |
| ==== N/A -> Preview_orig ====
| |
| | |
| (YoutsubaJSON.java, ln. 170): <code>p.setPreviewOrig(pj.getTim() + "s.jpg");</code>
| |
| | |
| ==== “tn_w” -> “preview_w” ====
| |
| | |
| Width of media thumbnail (YoutsubaJSON.java, ln. 193): <code>p.setPreviewW(pj.getTnW());</code>
| |
| | |
| ==== “tn_h” -> “preview_h” ====
| |
| | |
| Height of media thumbnail. (YoutsubaJSON.java, ln. 170): <code>p.setPreviewH(pj.getTnH());</code>
| |
| | |
| ==== N/A -> media_filename ====
| |
| | |
| (YoutsubaJSON.java, ln. 168): <code>p.setMediaFilename(pj.getFilename() + pj.getExt());</code>
| |
| | |
| ==== “W” -> “media_w” ====
| |
| | |
| (YoutsubaJSON.java, ln. 191): <code>p.setMediaW(pj.getW());</code>
| |
| | |
| ==== “h” -> “media_h” ====
| |
| | |
| (YoutsubaJSON.java, ln. 192): <code>p.setMediaH(pj.getH());</code>
| |
| | |
| ==== “Media_size” ====
| |
| | |
| (YoutsubaJSON.java, ln. 190): <code>p.setMediaSize(pj.getFsize());</code>
| |
| | |
| ==== “Media_hash” ====
| |
| | |
| (YoutsubaJSON.java, ln. 189): <code>p.setMediaHash(pj.getMd5());</code>
| |
| | |
| ==== N/A -> “Media_orig” ====
| |
| | |
| (YoutsubaJSON.java, ln. 169): <code>p.setMediaOrig(pj.getTim() + pj.getExt());</code>
| |
| | |
| ==== “Spoiler” ====
| |
| | |
| (YoutsubaJSON.java, ln. 204): <code>p.setSpoiler(pj.isSpoiler());</code>
| |
| | |
| ==== N/A -> “Deleted” ====
| |
| | |
| Set initially as false, then later updated if post is later absent from thread during subsequent updates. (YoutsubaJSON.java, ln. 205): <code>p.setdeleted(false);</code>
| |
| | |
| ==== “Capcode” -> “Capcode” ====
| |
| | |
| (YoutsubaJSON.java, ln. 173):
| |
| | |
| <pre>String capcode = pj.getCapcode();
| |
| if (capcode != null) {
| |
| if (capcode.equals("manager") || capcode.equals("Manager")) {
| |
| capcode = "G";
| |
| } else {
| |
| capcode = capcode.substring(0, 1).toUpperCase();
| |
| }
| |
| }</pre>
| |
| ==== “email” -> “Email” ====
| |
| | |
| (YoutsubaJSON.java, ln. 199): <code>p.setEmail(pj.getEmail());</code>
| |
| | |
| ==== “Name” -> (cleanSimple()) -> “Name” ====
| |
| | |
| (YoutsubaJSON.java, ln. 200): <code>p.setName(this.cleanSimple(pj.getName()));</code>
| |
| | |
| ==== “trip” -> “Trip” ====
| |
| | |
| (YoutsubaJSON.java, ln. XX): <code>p.setTrip(pj.getTrip());</code>
| |
| | |
| ==== “sub” -> (cleanSimple()) -> “Title” ====
| |
| | |
| (YoutsubaJSON.java, ln. 198): <code>p.setTitle(this.cleanSimple(pj.getSub()));</code>
| |
| | |
| ==== “com” -> (doClean) -> “Comment” ====
| |
| | |
| (YoutsubaJSON.java, ln. 203): <code>p.setComment(this.doClean(pj.getCom()));</code>
| |
| | |
| ==== N/A -> “Delpass” ====
| |
| | |
| TODO (YoutsubaJSON.java, ln. XX): <code>TODO</code>
| |
| | |
| ==== “sticky” -> “Sticky” ====
| |
| | |
| (YoutsubaJSON.java, ln. 206): <code>p.setSticky(pj.isSticky());</code>
| |
| | |
| ==== “closed”, “archived” -> “closed” AND (NOT “archived”)) -> “Locked” ====
| |
| | |
| (YoutsubaJSON.java, ln. 207): <code>p.setClosed(pj.isClosed() && !pj.isArchived());</code> (SQL.java, ln. 78 - 86): <code>this.insertQuery = String.format(</code> (SQL.java, ln. 236): <code>insertStmt.setBoolean(c++, post.isClosed());</code>
| |
| | |
| ==== “id” -> “Poster_hash” ====
| |
| | |
| (YoutsubaJSON.java, ln. 182):
| |
| | |
| <pre>String posterHash = pj.getId();
| |
| if(posterHash != null && posterHash.equals("Developer")) posterHash = "Dev";</pre>
| |
| ==== “country” -> “Poster_country” ====
| |
| | |
| (YoutsubaJSON.java, ln. 185-186):
| |
| | |
| <pre>String posterCountry = pj.getCountry();
| |
| if(posterCountry != null && (posterCountry.equals("XX") || posterCountry.equals("A1"))) posterCountry = null;</pre>
| |
| ==== lots -> “exif” ====
| |
| | |
| (YoutsubaJSON.java, ln. 212): <code>p.setExif(this.cleanSimple(this.parseMeta(pj.getCom(), pj.getUniqueIps(), pj.getSince4pass(), pj.getTrollCountry())));</code> (YoutsubaAbstract.java, ln. 137): <code>public String parseMeta(String text, Integer uniqueIps, Integer since4pass, String trollCountry) {</code>
| |
| | |
| == Images table values: ==
| |
| | |
| These seem to be handled by triggers that run on post insert.
| |
| | |
| === 4ch -> (Asagi) -> DB ===
| |
| | |
| === N/A -> (Incremental integer) -> “media_id” ===
| |
| | |
| This is simply an autoincrementing integer value. Set by DB engine if new image. ''Set by trigger if already-seen image.'' (Triggers.sql ln.119-139): - ''If the md5 is already in the DB'' <code>media_id = LAST_INSERT_ID(media_id),</code>
| |
| | |
| === md5 -> (N/A) -> “media_hash” ===
| |
| | |
| The base64-encoded md5 hash of the media file as given by 4ch. (SQL.java ln.77-87) Omitted for brevity
| |
| | |
| === N/A -> (Local filepath to full image) -> “media” ===
| |
| | |
| The relative path to the image on disk. (Triggers.sql ln.119-139): <code>TODO</code>
| |
| | |
| === N/A -> (Local filepath to OP thumbnail) -> “preview_op” ===
| |
| | |
| The relative path to the image on disk. (Triggers.sql ln.119-139): <code>TODO</code>
| |
| | |
| === N/A -> (Local filepath to reply thumbnail) -> “preview_reply” ===
| |
| | |
| The relative path to the image on disk. (Triggers.sql ln.119-139): <code>TODO</code>
| |
| | |
| === N/A -> (incrementer) -> “total” ===
| |
| | |
| The number of posts that refer to this row.
| |
| | |
| (Triggers.sql ln.123): <code>INSERT INTO \"%%BOARD%%\_images\" (media\_hash, media, preview\_op, total)</code>
| |
| | |
| (Triggers.sql ln.127): <code>total = (total + 1)</code>
| |
| | |
| === N/A -> (N/A) -> “banned” ===
| |
| | |
| Not set by Asagi, but observed to prevent downloading banned files (Triggers.sql ln.119-139): <code>TODO</code>
| |
| | |
| ===== Table definition =====
| |
| | |
| (Boards.sql ln.38-49): - ''Table definition''
| |
| | |
| CREATE TABLE %%BOARD%%_images ( media_id SERIAL NOT NULL, media_hash character varying(25) NOT NULL, media character varying(20), preview_op character varying(20), preview_reply character varying(20), total integer NOT NULL DEFAULT ‘0’, banned smallint NOT NULL DEFAULT ‘0’, PRIMARY KEY (media_id), UNIQUE (media_hash) );
| |
| | |
| ===== Image insert procedure =====
| |
| | |
| (Triggers.sql ln.119-139): - Image insert procedure
| |
| | |
| DROP PROCEDURE IF EXISTS “insert_image_%%BOARD%%”; CREATE PROCEDURE “insert_image_%%BOARD%%” (n_media_hash VARCHAR(25), n_media VARCHAR(20), n_preview VARCHAR(20), n_op INT) BEGIN IF n_op = 1 THEN INSERT INTO "%%BOARD%%_images" (media_hash, media, preview_op, total) VALUES (n_media_hash, n_media, n_preview, 1) ON DUPLICATE KEY UPDATE media_id = LAST_INSERT_ID(media_id), total = (total + 1), preview_op = COALESCE(preview_op, VALUES(preview_op)), media = COALESCE(media, VALUES(media)); ELSE INSERT INTO "%%BOARD%%_images" (media_hash, media, preview_reply, total) VALUES (n_media_hash, n_media, n_preview, 1) ON DUPLICATE KEY UPDATE media_id = LAST_INSERT_ID(media_id), total = (total + 1), preview_reply = COALESCE(preview_reply, VALUES(preview_reply)), media = COALESCE(media, VALUES(media)); END IF; END; ```
| |
| | |
| == Data path: ==
| |
| | |
| === 4ch -> decode JSON -> make Post objects ===
| |
| | |
| API Date is retrieved from 4ch and put into TopicJson objects, ready for handling posts in each of those topics. (YoutsubaJSON.java ln .89): - ''Get 4ch API JSON'' <code>String[] wgetReply = this.wgetText(this.linkThread(threadNum), lastMod);</code> (YoutsubaJSON.java ln .97) - ''Decode JSON into TopicJson objects'' <code>topicJson = GSON.fromJson(threadText, TopicJson.class);</code> (YoutsubaJSON.java ln .93) V For a single topic, each post if decoded into a Post object Post object -> added to Topic object (YoutsubaJSON.java ln .93): - ''Create Topic object'' <code>Topic t = null;</code>
| |
| | |
| (YoutsubaJSON.java ln .109): - OP <code>t = this.makeThreadFromJson(pj);</code>
| |
| | |
| (YoutsubaJSON.java ln .116): - Reply <code>t.addPost(this.makePostFromJson(pj));</code>
| |
| | |
| V
| |
| | |
| (AbstractDumper.java ln 303): <code>topic = sourceBoard.getThread(newTopic, lastMod);</code>
| |
| | |
| V
| |
| | |
| Image processed somehow?
| |
| | |
| Full image fetching begins
| |
| | |
| (AbstractDumper.java ln 161): - Fullsize media downloader thread <code>protected class MediaFetcher implements Runnable {</code> (AbstractDumper.java lln. 169): - Grab from queue thing <code>mediaPost = mediaUpdates.take();</code> (AbstractDumper.java lln. 173): - Try to handle the media for one post <code>mediaLocalBoard.insertMedia(mediaPost, sourceBoard);</code> (Local.java ln. 201): - Handler for a post with media <code>public void insertMedia(MediaPost h, Board source, boolean isPreview) throws ContentGetException, ContentStoreException, CfBicClearParseException {</code> (Local.java ln. 201): - Interact with DB for this media <code>mediaRow = db.getMedia(h);</code>
| |
| | |
| If there is information for this media in the DB, retrieve it. If any new information exists about this media that is not already in the DB, add that to the DB entry. (SQL.java ln. 289): - Interact with DB for this media <code>public synchronized Media getMedia(MediaPost post) throws ContentGetException, ContentStoreException, DBConnectionException {</code>
| |
| | |
| (SQL.java ln. 342-347): - Decide if media row needs an update
| |
| | |
| <pre>boolean mediaUpdate = media.getMedia() == null;
| |
| boolean previewOpUpdate = media.getPreviewOp() == null && post.isOp();
| |
| boolean previewReplyUpdate = media.getPreviewReply() == null && !post.isOp();
| |
| // Update media row in _images table when any of its entries are null and we actually have it
| |
| if(mediaUpdate || previewOpUpdate || previewReplyUpdate) {</pre>
| |
| (SQL.java ln. 349-373): - Add values to DB for this media
| |
| | |
| <pre>if(mediaUpdate) {
| |
| updateMediaStmt.setString(1, post.getMedia());
| |
| updateMediaStmt.setString(2, post.getMediaHash());
| |
| updateMediaStmt.executeUpdate();
| |
| }
| |
| if(previewOpUpdate) {
| |
| updatePreviewOpStmt.setString(1, post.getPreview());
| |
| updatePreviewOpStmt.setString(2, post.getMediaHash());
| |
| updatePreviewOpStmt.executeUpdate();
| |
| }
| |
| if(previewReplyUpdate) {
| |
| updatePreviewReplyStmt.setString(1, post.getPreview());
| |
| updatePreviewReplyStmt.setString(2, post.getMediaHash());
| |
| updatePreviewReplyStmt.executeUpdate();
| |
| }
| |
| conn.commit();</pre>
| |
| (SQL.java ln. 62-66): - Preparing for selecting / updating media
| |
| | |
| <pre>selectMediaStmt = conn.prepareStatement(selectMediaQuery);
| |
| updateMediaStmt = conn.prepareStatement(updateMediaQuery);
| |
| updatePreviewOpStmt = conn.prepareStatement(updatePreviewOpQuery);
| |
| updatePreviewReplyStmt = conn.prepareStatement(updatePreviewReplyQuery);</pre>
| |
| (SQL.java ln. 96-103): - SQL for selecting / updating media
| |
| | |
| <pre>this.selectMediaQuery = String.format("SELECT * FROM \"%s_images\" WHERE media_hash = ?",
| |
| this.table);
| |
| this.updateMediaQuery = String.format("UPDATE \"%s_images\" SET media = ? WHERE media_hash = ?",
| |
| this.table);
| |
| this.updatePreviewOpQuery = String.format("UPDATE \"%s_images\" SET preview_op = ? WHERE media_hash = ?",
| |
| this.table);
| |
| this.updatePreviewReplyQuery = String.format("UPDATE \"%s_images\" SET preview_reply = ? WHERE media_hash = ?",
| |
| this.table);</pre>
| |
| == Threads table values: ==
| |
| | |
| ==== Files of note: ====
| |
| | |
| <blockquote>SQL.java YoutsubaJSON.java Topic.java - Class definition boards.sql (ln.52) - Table definition triggers.sql
| |
| </blockquote>
| |
| ==== Functions of note: ====
| |
| | |
| ===== SQL.java: =====
| |
| | |
| ===== YoutsubaJSON.java: =====
| |
| | |
| (ln.217) “private Topic makeThreadFromJson(PostJson pj) throws ContentParseException {”
| |
| | |
| ===== Topic.java =====
| |
| | |
| (Ln. 22) <code>public Topic(int num, int omPosts, int omImages) {</code>
| |
| | |
| Data flow: 4ch -> Asagi -> DB
| |
| | |
| ?Seems to be handled by DB triggers.?
| |
| | |
| == References: ==
| |
| | |
| Asagi source code used: Bibabon repo 2019-7 retrieved from https://github.com/bibanon/asagi with commit https://github.com/bibanon/asagi/commit/dace6f01664d887f9f60dfdec341e626685b542f
| |
| | |
| Hayden source code : https://github.com/bbepis/Hayden
| |
