Editing LTO Tape

LTO Tapes are hands down the best and cheapest way to archive large amounts of data.

Using LTO Tapes can be a bit of a challenge, but if you're willing to use some simple Linux commands you can do it. We've detailed the whole process of buying some tape drives and tapes, setting them up with a normal desktop with PCI-E (or better yet an ECC RAM Server), and how to actually write to the tapes.

It was a tremendous challenge to build this guide, but thankfully it won't be so hard for you once you read through the details below.

How tapes work: https://books.google.com/books?id=rGjkBQAAQBAJ&amp;lpg=PA76&amp;ots=DHpi1-f7K1&amp;dq=decompress%20before%20lto%20tape&amp;pg=PA75#v=onepage&amp;q=decompress%20before%20lto%20tape&amp;f=false

== Why use Tapes for Archival/Backup? ==

[http://searchdatabackup.techtarget.com/video/Why-a-tape-backup-system-is-still-popular Why a Tape Backup System is still popular]

https://youtu.be/RIDa2EobXqo

LTO Tapes are beyond your ol' VHS tapes: they can last up to 30 years in a good air conditioned room, and within 5 years you'll probably upgrade two generations up in LTO once the drives get cheaper.

You could take a hammer to a tape drive and rewind it on to a new tape. You'd lose what ever part got damaged but anything else would be salvageable.

They survive drops and falls much better than 4TB hard drives. They're easy to send off site. LTO tapes are backwards compatible for 3 generations.

Most of all, LTO-3 and LTO-4 tapes and drives are very inexpensive, at $10-15 for a 400-800GB tape, and around $100 for a good used drive.

There's a reason some very important stuff is backed up to Tape. I'll use 4 TB drives or Blu-rays to back up archived movies. I'll use tapes to back up family photos.

And large cloud data providers such as Google or Amazon would be teetering on hard drive destruction without cheap tape backups. [https://gmail.googleblog.com/2011/02/gmail-back-soon-for-everyone.html Google once lost a significant portion of Gmail data] from runaway bad code, but recovered everything just by picking out a tape from the shelf (with a robot arm).

=== Read/Write Speeds ===

One of the big reasons that LTO drives are so expensive when sold new (but not used) is that these drives use probably some of the fastest motors ever developed, and read data at very high speeds.

Notice that with each increasing LTO level, the minimum drive read/write speed increases as well. LTO-3 requires 40MB/s, while LTO-4 requires 100MB/s, and that's just uncompressed.

This can be mitigated by providing enough GBs of RAM for buffering, since RAM can write data very quickly. We recommend about 2-4GBs of RAM for LTO-3, and 4-8GBs for LTO-4.

Any modern desktop computer with a PCI-E slot would easily work with LTO 3, 4, and 5. We do recommend ECC RAM for mission critical data (standard on Xeon server blades), which conducts checksum verification on the data stored inside the RAM itself: But for movies, text, and images, you can probably live without error checking.

=== Data Migration ===

One aspect to watch out for is obseletion.

It's not hard to fight obseletion. Here's how you do it:

# '''Don't use proprietary software/standards.''' Stick to <code>tar</code>, which have been around forever and won't stop being supported on Linux. Or on LTO-5 and up, use <code>LVDS</code>, which looks like a standard hard drive filesystem. This way, you know what format your tapes used and you can recover a piece even if they are sliced up (not ideal, but try that with a hard drive).
# '''Migrate your old tapes if you're upgrading to a new Standard''' - You shouldn't be too worried about the coming of the next LTO standard. They have read backwards compatibility up to two generations back. In addition, an LTO tape one generation up can store 2 times the data, and 4 times for two generations up. Your old backups will just get smaller upon transfer.

== Alternatives ==

https://www.digitalrebellion.com/blog/posts/backup_options_for_filmmakers

* Hard drives are out of the question, since there's been big issues

== LTO Types ==

Note that all standards LTO-4 and lower must use tarballs for spitting data. No need to compress, it does hardware compression on it's own. LTO-5 and up supports &quot;partitioning&quot; and can utilize LTFS, an open backup format which is incredibly easy to work with.

=== Blu-ray ===

The first step is to compare against the price of typical Blu-ray. It's not an exact comparison, but it's a good marker. While you do have to switch out disks and use DVDisaster using Blu-ray, that compares easily against the non-intuitiveness of tapes. Thus, to be practical, the tape price must beat the cost per gigabyte

* $23 - Plexdisc 50 pack, 1,250GB, 1.84 cents per gigabyte, 54GB for $1, 46 cents per 25GB disc.

=== LTO-3 ===

The absolute minimum you should go for. It's 3.75 cents per gigabyte at $15, or [http://www.amazon.com/HP-C7973A-Compressed-Transfer-Cartridge/dp/B0006N3ETK $1 for 22-45GB]. That's actually not that cost effective against Blu-rays (LTO-4 would be much better value), but if the data is compressible you can fit more, at least the drives are significantly cheaper, and what you get is volume and longevity over burned Blu-Rays.

I could in fact salvage one for Justin, get all the parts except for the drive (which he buys himself). Most importantly, a second PCIe to SCSI card from a Dell PowerEdge.

* Examples
* https://www.youtube.com/watch?v=zmYq71gKDd8
* https://www.reddit.com/r/freenas/comments/30cbpd/any_one_else_backup_their_server_to_lto_tape/
* [http://h10032.www1.hp.com/ctg/Manual/c04571431.pdf Configuration Guide for Linux]
* Parts - Rings up to about $50-70, depending on which items you already have.
* [http://www.ebay.com/gds/Top-5-LTO-3-Tape-Drives-/10000000177715643/g.html Drive Comparison]
* $33 [http://www.ebay.com/itm/410645-001-HP-StorageWorks-960-LC-UF6QC-HP-LTO-3-Ultrium-Tape-Drive/151663976187 HP StorageWorks 960] - This drive is dirt cheap and uses SCSI (68-pin Internal).
* $30 - [http://www.ebay.com/itm/HP-439946-001-LSI-Logic-LSI20320IE-PCIe-x4-U320-SCSI-HBA-Card-L3-01130-05D-/321861120942?hash=item4af06af3ae LSI Logic PCI-E to SCSI Adapter] - This is used to connect to the drive in question. If you want to be extra cheap, find a 2000s era computer and buy a PCI-X SCSI card for just $10.
** I'm going to need to find more of these at the scrapyard, they're so valuable. Find at least two more.
* $10 [http://www.ebay.com/itm/NEW-Dell-Precision-Workstation-360-370-68-Pin-SCSI-Cable-Assembly-J0957-0J0957-/381211649336 Internal SCSI Cable with built-in Terminator] - If you're using an internal drive, this is what you need.
* $10 [http://www.ebay.com/itm/Genuine-Dell-SCSI-68-PIN-TO-VHDCI-4m-Large-Cable-JJ003-0JJ003-/331567913512?hash=item4d32fcda28 68pin to VHDCI SCSI cable] - If you're lucky enough to have a drive equipped to work externally (with it's own terminator), this cable allows you to wire up the drive with the SCSI board to the small VHDCI port.
* $0 - Molex Power Supply - Use a good ol' PC Power supply to fire it up.

=== Other LTO Types ===

* '''LTO-4''' - The best value when it comes to media, at 1.85 cents per gigabyte, and [https://www.tape4backup.com/lto-4-tapes.php $1 for 40-80GB.] This handily beats Blu-ray. The drives are more expensive than LTO-3 at $200, but if you bid you might get lucky with $50-100. At least you're not scraping the bottom of the barrel.
* Think about it: you have double the storage at the same price as LTO-3, so it you would save $15 per tape. Weigh that against the cost of an LTO-4 drive.
* '''LTO-5''' - [https://www.tape4backup.com/lto-5-tapes.php $1 for 75-150GB] - At this point, the price of drives starts to climb up significantly, to $500. Whether that will pay for itself in a year is your challenge. Hopefully we can find it on a Dell PowerEdge blade, but don't push your luck.
* '''LTO-6''' [https://library.creativecow.net/kobler_helmut/LTO6-Review/1 Photographers and Moviemakers] have been using external LTO drives for years.

=== Silicon Mechanics Server ===

The best way is to get a mini SATA server to run the tape library. This little blade server is the ideal system

Upgrades will be duly necessary:

* '''2TB Hard Drive''' - The Hard Drive functions as the data cache, while the data is being written over.
* '''4Gigabit Fiber Channel HBA''' - I already have one of these, and
* LTO-5 Effect - It looks like LTO-5 will only saturate to 150mbps, so [http://www.veritas.com/community/forums/impact-lto-5-or-6-connected-4-gb-fc-port 4GB is more than enough.]

= Usage =

== Prerequisites ==

Make sure you have chosen your LTO type. Then, get a Host Bus Adapter that is compatible with your chosen LTO drive, and wire it up following one of the directions below:

* [[LTO Tape SCSI]]
* [[LTO Tape FC]]
* [[LTO Tape SAS]]

== Dependencies ==

<code>tar</code> and <code>mt</code> come by default with all Linux systems. However, you will want the <code>mt-st</code> and <code>mtx</code> packages to provide more detailed tapeinfo, and control robotic autochangers (if equipped).

We will also want <code>pv</code> to view progress, and <code>mbuffer</code> to buffer the data before pushing.

<pre>sudo apt-get install mtx mt-st pv mbuffer</pre>
== Kernel Module ==

You will need to set the kernel module <code>st</code> on boot before you will be able to detect tape drives.

This will enable it temporarily:

<pre>sudo modprobe st</pre>
However, we want to enable it on every boot. Check how to do so. On Ubuntu 14.04 just add <code>st</code> on a line to <code>/etc/modules</code>.

== mtx Tapeinfo ==

We want to figure out specific information about the tape drive we're using, so let's run the following command to get a status report:

<pre># tapeinfo -f /dev/nst0

Product Type: Tape Drive
Vendor ID: 'HP      '
Product ID: 'Ultrium 3-SCSI  '
Revision: 'G63W'
Attached Changer API: No
SerialNumber: 'HU1061001U'
MinBlock: 1
MaxBlock: 16777215
SCSI ID: 3
SCSI LUN: 0
Ready: yes
BufferedMode: yes
Medium Type: Not Loaded
Density Code: 0x44
BlockSize: 0
DataCompEnabled: yes
DataCompCapable: yes
DataDeCompEnabled: yes
CompType: 0x1
DeCompType: 0x1
BOP: yes
Block Position: 0
Partition 0 Remaining Kbytes: 400308
Partition 0 Size in Kbytes: 400308
ActivePartition: 0
EarlyWarningSize: 0
NumPartitions: 0
MaxPartitions: 0</pre>
== <code>tar</code> and <code>mt</code> ==

LTO-4 drives and lower only support classic UNIX tarballs. Yeah, you got that right: <code>tar</code> stands for Tape ARchive, and is a sequential archive format as a result.

You will also need the <code>mt</code> command to operate the drive (our version provided by the package <code>mt-st</code>), such as checking status or rewinding.

=== /dev/st0 and /dev/nst0 ===

<blockquote>The major difference is after perform a task, /dev/st0 will rewind to begining of tape, but the /dev/nst0 won't. normally we use nst0 for daily backup(unchange tape), st0 for weekly/monthly backup (using single tape for each backup).
</blockquote>
* Use <code>/dev/st0</code> if you only want to push a single directory to fill the entire tape (it rewinds upon completion).
* Use <code>/dev/nst0</code> if you want to add multiple directories to the tape incrementally.

<blockquote>'''Note:''' The moment you use <code>/dev/st0</code>, when a command completes the system will automatically rewind it.
</blockquote>
=== Operation ===

<blockquote>'''Note:''' Run all commands as root.
</blockquote>
Before loading a new tape (at least for an HP LTO3 Drive), you need to run the following to set up <code>mt-st</code>:

<pre># mt -f /dev/nst0 stsetoptions scsi2logical</pre>
If you've already written data to the tape, make sure to wind to the last written block, otherwise existing data could be lost. If this is a new tape, the drive will stay at 0.

(Alternatively, if you are fine with overwriting the whole tape, you can start from the beginning.)

<pre># mt -f /dev/nst0 eod</pre>
Figure out what block you are at:

<pre># mt -f /dev/nst0 tell</pre>
Now send a folder to the tape drive using tar:

<blockquote>'''Tip:''' We don't need to set software compression with tar, because the LTO drive will do hardware compression on its own.
</blockquote>
<pre># tar -cvf /dev/nst0 /path/to/dir</pre>
However, even an LTO-3 tape drive requires 40-80MB/s of sustained RW speeds or it will undergo a &quot;shoe shining&quot; effect, where not enough data is being pushed to fill the LTO drive's buffer, and the head has to stop or rewind the tape to wait. This can cause serious wear to both tape and drive.

Although today's hard drives are just barely able to push 50-80MB/s, there's a better method: buffer to RAM.

=== Buffered Writes ===

A better method is to buffer the data to 2-4GB RAM before pushing (of course, this means you need that much free RAM).

We also want to make checksums of the tar to ensure that everything went well.

Here's a good one-liner (check the source link for more info about what it does). First, go to the directory where you want to store the write logs.

In the following command, enter a name in <code>bkname=&quot;name-of-backup-YYYY-MM-DD&quot;</code>, then enter folders/files in <code>tobk=&quot;/home /var/www&quot;</code> Make sure that if your folder names contains spaces, use single quotes <code>'</code> within: <code>tobk=&quot;'/mnt/extdisk/Disc 7/ISOs' /var/log&quot;</code>. Finally, run it as root.

<pre>bkname=&quot;001_external-backup-items_2016-02-25&quot;; tobk=&quot;/mnt/extdisk/Backup&quot; ; totalsize=$(du -csb $tobk | tail -1 | cut -f1) ; tar cvf - $tobk | tee &gt;(sha512sum &gt; $bkname.sha512) &gt;(tar -tv &gt; $bkname.lst) | mbuffer -m 3G -P 100% | pv -s $totalsize -w 100 | dd of=/dev/nst0 bs=256k</pre>
The buffered writes command we used above will write a <code>tar</code> archive with a block size of (bs=256k), so don't forget it. Make sure to write this block size on your tape case, so you can use it later.

<blockquote>'''Tip:''' Make sure to put the resulting <code>.lst</code> and <code>.sha512</code> files into a small flash drive and/or CD-R alongside the LTO tape. (consider online storage as the third backup) It will be extremely helpful, if not crucial, to refer to these next time you have to extract data, so they need to be accessible.
</blockquote>
<blockquote>'''Tip:''' Always number the <code>bkname</code> above, so this way you will know the order of the files on the tape. Also write down a sequential, numbered list of each record you've pushed to the tape (the plastic case usually has a notepad). That way you'll be able to figure out which record to access by that same index number.
</blockquote>
<blockquote>'''Tip:''' If at all possible, have a primary hard drive where the OS can store the generated logs and programs (e.g. a small SSD), and keep your backup data on a separate hard drive. This way you can run this command from a directory on the primary hard drive, to reduce lag.
</blockquote>
<blockquote>'''Warning:''' Mbuffer will take up 3GBs of RAM in the above example. If you have less than 3GBs of RAM, performance will actually suffer, not improve. Reduce the buffer size to fit your RAM amount: or buy more RAM.
</blockquote>
<blockquote>'''Tip:''' If the data you want to push is already in <code>.tar</code> format (no compression), replace <code>tar cvf - $tobk</code> with <code>cat $tobk</code>. If the data is in <code>.tar.gz</code> format (gzip compression), replace with <code>zcat $tobk</code>. If the data is in <code>.tar.xz</code>, use <code>xzcat $tobk</code>, and for <code>.tar.bz2</code> use <code>bz2cat $tobk</code>.
</blockquote>
* [http://www.commandlinefu.com/commands/view/13582/backup-to-lto-tape-with-progress-checksums-and-buffering Command Line Fu - Backup to LTO Tape with Progress, Checksums, and buffering]
* http://dampfnudel.blogspot.com/2007/10/tape-backups.html

=== Tape Record Navigation ===

Once you have multiple records on the tape, you can navigate between them using the following commands:

Forward 1 record:

<pre># mt -f /dev/nst0 fsf 1</pre>
Go to previous record:

<pre># mt -f /dev/nst0 bsfm 1</pre>
Go to the last written byte of the tape:

<pre># mt -f /dev/nst0 eod</pre>
<blockquote>'''Caution:''' If you're appending a record to a previously written tape, make sure to wind to the last written byte of the tape (<code>eod</code>), or you will overwrite existing data.
</blockquote>
=== Reading Tape Archives ===

Since we piped in data using <code>dd</code>, we might as well pipe out data using it and the same block size (bs=256k) as well. Here's how you list all files in one section of the tape (navigate to your desired section first):

<pre>dd if=/dev/nst0 bs=256k | tar tvBpf -</pre>
Once you know that this section is what you want, rewind (<code>mt -f /dev/nst0 bsfm 1</code>), and extract with the following command (drops in the current directory):

<pre>dd if=/dev/nst0 bs=256k | tar xvBpf -</pre>
=== Ejecting the Tape ===

<blockquote>'''Note:''' We strongly recommend that the tape is ejected when it is not in use. Otherwise, dust may enter the cartridge. The drive will also have to wind the motor every hour or so, to keep the tape taut. The LTO Drive can run pretty darn fast, so it won't take much time to wind forward.
</blockquote>
Finally, when you are finished with the tape, rewind and eject it with the following command:

<pre># mt -f /dev/st0 offline</pre>
=== Hardware Compression ===

The LTO Drives support basic, but very fast hardware compression algorithms, burned into the chips of the tape drive. This way, if you have tons of text logs to push, you don't have to compress your data before putting it in the drive, the drive will do it for you. You also don't have to decompress afterwards.

For the most part, the LTO drive has compression on by default, but if it is off for some reason, run the following commands to enable them.

Enable compression until drive is turned off:

<pre># mt -f /dev/nst0 compression 1</pre>
Enable compression for all writes by default:

<pre># mt -f /dev/nst0 defcompression 1</pre>
Note that only some kinds of data will compress well enough for 2:1 compression ratio (as they advertise to transform 400GB to 800GB). This includes text, logs, documents, etc.

Since images (jpg, png) and video (mp4, mpeg) are already compressed, further compression is much less effective. In fact, due to the primitive compression algorithm used, [https://forums.veeam.com/tape-f29/lto6-capacity-t26331.html#p137077 it might even make compressed data bigger, at 15% loss].

As a result, you might want to consider disabling hardware compression whenever you're storing such data, or using your own software compression: <code>gz</code> or <code>xz</code> algorithms (with pigz or parallel xz for speed, modern CPUs should handle it fine). Before you use these, make sure to turn compression off:

Disable compression until drive is turned off:

<pre># mt -f /dev/nst0 compression 0</pre>
Disable compression for all writes by default:

<pre># mt -f /dev/nst0 defcompression -1</pre>
Notice that the tape drive can automatically detect HW compressed data on an existing tape and automatically enable it. Thus, if you aren't using a fresh tape, you'll need to relabel it accordingly.

https://wiki.zmanda.com/index.php/Hardware_compression

=== Sources ===

* [http://ubuntuforums.org/showthread.php?t=1437815 Ubuntu Forums - mt-st HP LTO Input/Output Error]
* http://www.cyberciti.biz/faq/linux-tape-backup-with-mt-and-tar-command-howto/
* http://www.linuxquestions.org/questions/linux-hardware-18/difference-between-dev-st0-and-dev-nst0-493760/

== LTFS ==

LTO-5 tapes and after support tape partitioning, with the open source LTFS filesystem. This makes the tape look like an ordinary filesystem, rather than tape archives back to back, making it easier to deal with.

The flip side of this is that tape is still a sequential medium, so beware if you try to use it like a disk. Avoid random reads and just push and pull data sequentially.

=== Sources ===

* https://www.cs.swarthmore.edu/~knerr/helps/lto5.html

== LTO Autoloaders using MTX ==

http://surf.ml.seikei.ac.jp/~nakano/dump-restore/dump-restore-mini-HOWTO.en.html#ss2.2

[[Category:Home Servers]]