Yotsuba Society/Github Site

The Bibliotheca Anonoma has put together a brand new, Jekyll (and Minimal Mistakes)-based website for the Yotsuba Society, which beats the rotting Drupal 7.x CMS it used to use.

And don't worry, all the old pagelinks and downloads still work (we worked hard to make that possible), so no need to change your bookmarks.

Situation
The Yotsuba Society, recently discovered that their website was getting spam sites injected into their aging Drupal 7.x installation.

Yes, with these messy, ancient CMS codebases, if you fail to update, security is like swiss cheese. It's a matter of time before your site gets exploited. And even if you think your site has nothing of value, it has valuable server space that spam sites can plaster their junk into.

The admin's original plan was simply to shut down the site indefinitely, but that would have meant the total death of the Yotsuba Society. A great example that sometimes, even the archivists have to archived. So I instead recommended that they abandon the CMS system entirely, and replace it with a Static HTML engine (such as Jekyll).

The CMS is often a necessary evil in large, account-based websites which implement forums, rich content, multiple posters, and such. However, by now the Yotsuba Society had significantly reduced it's operations, and was only using the CMS as a blog. Because Drupal 7 was now legacy code, it was starting to get spam ads injected into it to trick some poor saps.

Not everything has to be a giant Wordpress/Drupal/Joomla site. If you're only pushing information in a blog and some pages, perhaps with a few Disqus comments on the side, all you need is a static HTML engine, such as Jekyll.


 * No more complex interfaces to wrestle with. Just write a page and push (via FTP or Git).
 * No more bloated WYSIYWG editors. Just write in Markdown syntax, it's easy to learn and has massive potential. Dump YouTube embed codes in, dump internal image links, and be done with it.
 * No more security issues. Bloated CMS systems, with their accounts and SQL injection vulnerabilities, offer a massive attack surface, especially if it's not well updated. With Static HTML, there's nothing for attackers to attack, other than the web server admin console.
 * Static HTML uses significantly fewer resources. You don't need a PHP engine or SQL Database to generate pages on the fly anymore. The HTML pages are pregenerated on the author's computer, and just thrown at the viewer. In fact, you can serve them for free via Github Pages (which allows you to use your same ol' custom domains).

And yes, Static HTML engines support comments too, via Disqus.

Migration Rules
First, we must lay down some ground rules for website migration.


 * NEVER cause link rot. - Isn't it annoying when your bookmarked links no longer work? Even if the original content has been imported, you will break links if you change them to a different URL structure. Link rot causes a whole host of issues, from reduced accessibility to reduced Google listing presence.
 * We have to make sure that links such as  still work, or at least redirect in the new site. Luckily, Jekyll has a   option to set custom URLs, and a Jekyll Redirect-from plugin to create multiple redirects. Say goodbye to link rot.
 * Make sure all downloadable files remain in the same location. - Related to link rot. Ensure that any downloadable files can be accessed from the exact same URLs as you used to use.
 * Retain as much functionality as possible. - Don't drop features that the old site used to have: no matter how trivial, someone out there really does use them. Try to find some kind of alternative that the new site can provide.
 * RSS Feeds - Yes, there are people out there that still use these, despite that Google Reader is long gone. How else would you subscribe to of blogs? Thankfully, most Jekyll themes come with a templatable  that generates an RSS XML feed without any effort.
 * Comments - While the Yotsuba Society doesn't have comments, I would have used Disqus if it did, and imported the old comments with all history.
 * Multiple Authorship - This is a tricky one, given that it would be unsafe to give account access to all authors of the Yotsuba Society website. Since Jekyll is static HTML and best used with Git anyway, I would migrate the site to Github Pages, where you can give multiple users commit access to a repository.
 * Teach the Website Owners well. - Make sure that the owners of the site actually know how to use the engine, and provide plenty of documentation.
 * Jekyll is pretty simple to work with, but the actual command-line tools that help create posts can be a pain to install on Windows. Unfortunately, since most of the world uses Windows, you'll have to figure out a suitable alternative.
 * The Github Online Repo Editor is actually quite nice, and you can even create files using it.
 * We then created a README.md file that provides full documentation on how to create blog posts, edit articles, and modify the site structure.

Importing a Drupal Website into Jekyll
Jekyll provides a fully-functional blog import system that will grab all pages, tags, categories from Drupal, by communicating with the SQL server.

Note: This script can't grab image nodes or comments! Remember to grab those manually or obtain via some other method!

Also, it only grabs blog posts. Home Pages and other special nodes also need to be obtained manually.
 * 1) Install Ruby: Mac OS X - Linux
 * 2) Install Node.js: Mac OS X - Linux
 * 3) Open a command line.
 * 4) Install Jekyll-Import. Some manual installation may be necessary.


 * NOTE: This converter requires Sequel and the MySQL gems.
 * MySQL has been superseded by MariaDB in Linux. Make sure to install  and   (for Debian)
 * The MySQL gem can be difficult to install on OS X. Make sure to install MySQL, and use this argument instead:

$ sudo gem install sequel $ sudo gem install execjs $ sudo gem install mysql --with-mysql-config=/usr/bin/mysql_config  Use the import commands shown on Jekyll-Import documentation: Drupal 6 - Drupal 7

You should edit the fields in the command before using it.


 * - The name of your database, just as Drupal asked you during installation. (check PHPMyAdmin)
 * - The username of the database user. Check your hosting provider's admin console.
 * - The password of the database user.
 * - The SQL Server hosting all the website data.
 * - The table prefixes used in the database. Usually, it's, but you might have set something different.

There are two ways to work with the Drupal SQL Database:


 * 1) You can communicate directly with the curren website's Drupal SQL Database. However, it might be necessary to add your current IP as a trusted host (see your hosting provider for more details).
 * 2) Alternatively, you can make a SQL database dump using PHPMyAdmin, import it, and host your own database on your computer. The   entry should then be  . This can be a bit more challenging to set up, but it reduces security vulnerabilities, and is necessary when a backup is all you've got.

For example, here is the command we used to communicate with the Yotsuba Society MySQL Database:

Set up a SQL Backup and Export from it
Sometimes you might only have access to a SQL database backup. To export this database, you will have to create a local MySQL/MariaDB server, and import the database backup.

 Install MySQL/MariaDB. Some Linux distros may have already migrated to MariaDB, which is a drop-in replacement for MySQL.  Start the MySQLd Service. On Debian/Ubuntu: sudo service start mysqld   Run the MySQL/MariaDB Secure Installation Script as root, and follow the instructions. Make sure to drop the test database and reload the privilege tables. sudo mysql_secure_installation   Once you're finished, log in as the root MySQL user. mysql -u root -p   Create a user to manage the databases. For example, I used user:  with password:. 

MariaDB [(none)]&gt; CREATE USER 'localuser'@'localhost' IDENTIFIED BY 'password'; Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]&gt; GRANT ALL PRIVILEGES ON *.* TO 'localuser'@'localhost' -&gt;           WITH GRANT OPTION; Query OK, 0 rows affected (0.00 sec)  Now create a new database to import the backup into. In this example, we call it. Then quit.</li></ol>

MariaDB [(none)]&gt; create database yotsubasociety; Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]&gt; quit Now you can import the database. As the  user, we access the   SQL server and insert the backup.sql dump into the   database.

mysql -u localuser -p -h localhost yotsubasociety &lt; yotsubasociety352015.sql Finally, here's the command we used to export to Jekyll from the  SQL Server. Usually, the prefix is.

Cleaning up the Posts
Unfortunately, due to bugs in the Drupal import script:


 * The author names aren't imported.
 * The correct slugs aren't imported. In order to maintain compatibility with the old site's links, we need to grab the  table as shown in the Drupal 7 database schema.
 * The title is corrupted, as shown below:

--- layout: post title: !binary |- SW50cm9kdWN0aW9uIHRvIFlvdHN1YmEgU29jaWV0eQ== created: 1300717398 --- On the other hand, the correct title (well, lost underscores and Capitals) is always posted in the filename:

There must be some way we can make the title work. (though maybe it might actually display when compiled?)

Patching Jekyll-Import to work with Drupal 7
Jekyll-Import's Drupal 7 import scripts just aren't perfect or complete, as this bug report shows. We had to patch it to give us the correct title.

I have fixed this bug entirely by forcing the strings into Unicode format (and stripping any preceding and ending whitespace with ):

That way, it works even for those pesky titles with stray 2 junk that screws everything up.

Hacking the Jekyll-Import Gem
Before the change went live at the Jekyll-Import gem, I had to push the change to my existing gem to make the title work. First, we will have to be able to edit the gem, which is non-trivial. Thankfully, this user on StackOverflow has a handy Fork and Source method.

 Fork the Jekyll-Import script to your own Github account.</li>  Make your edits to the Drupal import script on your own repository. For example, I edited the following to add a  entry with spaces as underscores, and a better Title section: # Get the relevant fields as a hash, delete empty fields and convert # to YAML for the header data = { 'layout' =&gt; 'post', 'title' =&gt; '' + title, 'permalink' =&gt; '/' + title.strip.gsub!(' ', '_'), 'created' =&gt; created, 'excerpt' =&gt; summary }.delete_if { |k,v| v.nil? || v == ''}.to_yaml </li>  Now, clone your forked Jekyll-Import repository and add this to the Gemfile (also remove the  line): gem 'gem_name', :git =&gt; 'git://github.com/username/foo' </li> Run the command  to install the modified Gem.</li>  Run the Jekyll-Import command you planned to use as usual. If it works, submit a pull request to the authors. </li></ol>

(though for me, the changes didn't stick, so I just hacked the file directly at: )

Obtain url_alias
If your website used a, as it probably should have, you will definitely want to inject them into the blog pages as a Jekyll.

Sadly, as you see from the Drupal 7 Schema, the  table is in no way relationally related to a node, which is just plain stupid and makes life hard.

I guess the only way to insert this right is to do it by hand. That's what we were forced to do.

First, use the following MySQL command to open the table and grab everything from the  table.

MariaDB [(none)]&gt; USE yotsubasociety Database changed MariaDB [yotsubasociety]&gt; SELECT * FROM drupal_url_alias; You will obtain all the  from the database:

I created this nice markdown table by taking the MySQL Table SELECT query, and changing the middle bar to Markdown format (changed all the  signs to  ).

To ease the process of matching a node to a shortlink, I used a regex in the KDE Kate editor to match all  instances, and convert them into markdown links to the site in question. This way, I can just click a link as shown above to find the original title.


 * Find:
 * Replace:

Inject url_alias
Then, I open every single post dumped, and add  in the YAML section. This can take a while.

I also use the handy Jekyll Redirect-from gem to make the  links work as well:

redirect_from: - /node/1/ - /Page_Name_In_Underscores/ Note: Make sure that the links all end in a. This way, a folder with  will be created, instead of , which is just spit at the browser as a downloadable binary.

Choose a Jekyll Theme
Now that you have the data exported, choose a Jekyll theme that fits for you. Here's a nice listing of some free, amazing themes.

Since I needed something simple, and since the Bibliotheca Anonoma Blog and my own blog uses the same theme, I used the very nice, simple, and responsive Minimal Mistakes theme.

All I had to do was  the theme, paste all the pages into the theme, and do a bit of configuration, and the Jekyll site is ready.

Hosting the Jekyll Site
For the most part, you will want to host your Jekyll Site using Github Pages. Github has the best support for Jekyll (they created it), and offers free limitless Static HTML hosting.

Just push your website to the  branch on your   repository, or to the   branch on any random Github repo. Github will automatically generate the website at  or.

For the Yotsuba Society, we had them create a new Github Organization (which is free), whereby we pushed the Jekyll website to the  branch on the repo:. It was then instantly hosted at  without a hitch.

Notice that you can also use custom domains with Github Pages! We had to direct  to the Github Pages server, following the instructions provided by Github.

And now, the new Yotsuba Society Website is complete. What an amazing, difficult, but eye-opening journey.