System Administration

Linux System Administration is the fundamental skill to running the servers that quite literally form the Internet. Check Home Servers as well for Debian related hosting information, as the process is quite similar.

Note: While we do try to provide vendor neutral commands, many of these articles are designed for Red Hat Enterprise Linux/CentOS, since that's the main operating system we use.

Infrastructure

The equipment and governors that make our applications possible.

Providers

See also: Hosting

The right providers at the right price make all the difference.

• Amazon AWS - The Rolls Royce of cloud service providers.
• Hosting/Azure - Gives free $150 a month for 2 years to startups in the BizSpark program.
• DigitalOcean - Low cost, relatively high bandwidth VPS systems.
• Hetzner - $30 dedicated servers on auction in Germany.

Management/DevOps

Managing a large hulking fleet of Linux servers can be a real challenge, and demands coordination like any navy.

• OpenStack - Our app will probably be addicted to EC2 and S3 by the end of our subscription, so if we ever have to move to our own servers, set this up on there.
• Spacewalk - As our organization grows, we are going to have to manage a ton of servers simultaneously. This makes the immense task possible.
• DevOps - A hated buzzword, but some actual interesting concepts.
• PaaS - Platform as a service. Sometimes we don't want to manage a whole server, and just need to set up an app. Rationale
  • Heroku
  • Red Hat Openshift - A good platform by Red Hat.

Linux

Linux, a free open source operating system, is our OS of choice.

• Red Hat Enterprise Linux/CentOS - Red Hat Enterprise Linux/CentOS is our preferred Linux distro.
• Ubuntu - Ubuntu can also be within our support lines for future clients.

Other OSes

Other operating systems have unique properties that give them merit.

• OpenBSD - The most paranoid variant of the BSD operating systems. It's also the native system for OpenSSL and OpenSSH.
• FreeBSD - One of the killer features of FreeBSD is it's ZFS filesystem support, which is immensely useful for RAID setups.

Services

• Nginx - Nginx is a versatile web server that works best as a caching front proxy or static site host.
• Apache - Apache is the most popular web server for applications by far, as a component of the LAMP stack.
• PostgreSQL - PostgreSQL is our database of choice, combining the stability of SQL and the versatility of JSON. A non-admin user and a table should be created for each app.
• SQLite - A powerful database that consists of a single file.
• Gogs (Github Clone) - A Github clone written in Go. Great for confidential Git repositories.
• Mediawiki - The Mediawiki engine
• Semantic Mediawiki - The Semantic Mediawiki/Cargo engine is a powerful system for data organization in a familiar, easy to pick up wiki format.
• Docker - Containerization makes deploying, updating, and reinstalling apps with complex configurations easy.
• Kareha-psgi - An elegant anonymous textboard.
• Wordpress - Wordpress is a great CMS system that's relatively powerful and intuitive for graphical users. It uses MySQL as a database and PHP as its language.
• NFS - Network attached storage across the network.

Organizational

• Indico - Event and Conference organizer system for large organizations and large events.
• SSO - Single sign on systems are the dream of any organization. But it can be fiendishly difficult to even find info on a full stack solution, let alone set one up.
• Realms - We used the Realms Wiki engine for documentation.
• Info Pages - Info pages are very helpful for knowing what services are active. Anyone can figure this out through WHOIS and Nmap anyway: Security through obscurity is none at all.

Notifications

Have the system tell you whenever something is wrong, perhaps over email or something.

• Email/Notification - Get notified of a server event through email.
• SMS Notification - Get notified of a server event via cell phone SMS.
• RAID/Notification - Get notified whenever your RAID array experiences a detectable failure.

Security

When it comes to the work of a government contractor, data breaches simply cannot happen.

• SSL - Now that Let's Encrypt registers SSL certificates, there is no reason why any website should live without HTTPS.
  • Amazon RDS Database SSL - If the database must be connected over TCP (such as on Amazon RDS), use SSL transport encryption.
• Firewall - The first line of defense, preventing connections through unused ports from reaching the server.
  • On Amazon AWS, the Security Group is an additional network level defense perimeter.
• Discretionary Access Control - Defines ownership of files and directories by users/groups.
• Intrusion Detection System - Detects suspicious network behaviors, especially those that follow certain patterns.
• Honeypot - Let script kiddies have their fun, right until they discover that it's all just a trap.
• Mandatory Access Control - The final line of defense against application exploits. A program's expected behavior and directories of influence are registered, and deviations are blocked.
  • SELinux - The US government standard for MAC, and is heavily integrated into RHEL.
  • AppArmor - While we don't use AppArmor on our own systems, Ubuntu users might find some policy restrictions helpful.
• Splunk & (Splunk/Lite) - Consume and process logs from all your servers. It is proprietary software though, so consider the ELK stack as an alternative. However, this does provide particularly strong job security.
• Contingency Plan - When failure is an option, we have to figure out what we as a company must do and what we are liable for.

Miscellaneous

• Contingency Servers - Consider setting up contingency servers that are replicated in real time and can spin up whenever needed.
• Server Protips - Miscellaneous protips.
• Licensing - Software Licensing tips.