System Administration

Linux System Administration is the fundamental skill to running the servers that quite literally form the Internet.

Note: Many of these articles are designed for Red Hat Enterprise Linux/CentOS, since that's the main operating system we use.

Infrastructure

The equipment and governors that make our applications possible.

• Providers - The right providers at the right price make all the difference.
• Amazon AWS - The Rolls Royce of cloud service providers.
• DigitalOcean - Low cost, relatively high bandwidth VPS systems.
• Hetzner - $30 dedicated servers on auction in Germany.
• Management - Managing a large hulking fleet of Linux servers can be a real challenge.
• OpenStack - Our app will probably be addicted to EC2 and S3 by the end of our subscription, so if we ever have to move to district run servers, set this up on there.
• Spacewalk - As our company grows, we are going to have to manage a ton of servers simultaneously. This makes the immense task possible.
• DevOps - A hated buzzword, but some actual interesting concepts.
• Linux - Linux, a free open source operating system, is our OS of choice.
• Red Hat Enterprise Linux/CentOS - Red Hat Enterprise Linux/CentOS is our preferred Linux distro.
• Ubuntu - Ubuntu can also be within our support lines for future clients.

Services

• Nginx - Nginx is a versatile web server that works best as a caching front proxy or static site host.
• PostgreSQL - PostgreSQL is our database of choice, combining the stability of SQL and the versatility of JSON. A non-admin user and a table should be created for each app.
• SQLite - A powerful database that consists of a single file.
• Gogs (Github Clone) - A Github clone written in Go. Great for confidential Git repositories.
• Mediawiki - The Mediawiki engine
• Semantic Mediawiki - The Semantic Mediawiki/Cargo engine is a powerful system for data organization in a familiar, easy to pick up wiki format.
• Realms - We use the Realms Wiki engine for Lynk Education company documentation.
• Docker - Containerization makes deploying, updating, and reinstalling apps with complex configurations easy.
• Kareha-psgi - An elegant anonymous textboard.
• Wordpress - Wordpress is a great CMS system that's relatively powerful and intuitive for graphical users. It uses MySQL as a database and PHP as its language.
• Info Pages - Info pages are very helpful for knowing what services are active. Anyone can figure this out through WHOIS and Nmap anyway: Security through obscurity is none at all.

Notifications

Have the system tell you whenever something is wrong, perhaps over email or something.

• Email/Notification - Get notified of a server event through email.
• SMS Notification - Get notified of a server event via cell phone SMS.
• RAID/Notification - Get notified whenever your RAID array experiences a detectable failure.

Security

When it comes to the work of a government contractor, data breaches simply cannot happen.

• SSL - Now that Let's Encrypt registers SSL certificates, there is no reason why any website should live without HTTPS.
• Amazon RDS Database SSL - If the database must be connected over TCP (such as on Amazon RDS), use SSL transport encryption.
• Firewall - The first line of defense, preventing connections through unused ports from reaching the server.
• On Amazon AWS, the Security Group is an additional network level defense perimeter.
• Discretionary Access Control - Defines ownership of files and directories by users/groups.
• Intrusion Detection System - Detects suspicious network behaviors, especially those that follow certain patterns.
• Honeypot - Let script kiddies have their fun, right until they discover that it's all just a trap.
• Mandatory Access Control - The final line of defense against application exploits. A program's expected behavior and directories of influence are registered, and deviations are blocked.
• SELinux - The US government standard for MAC, and is heavily integrated into RHEL.
• AppArmor - While we don't use AppArmor on our own systems, Ubuntu users might find some policy restrictions helpful.
• Contingency Plan - When failure is an option, we have to figure out what we as a company must do and what we are liable for.

Miscellaneous

• Play Framework - Set up the Java Play Framework on RHEL.
• Server Protips - Miscellaneous protips.