Difference between revisions of "LTO Tape/6"

(Backup Software)
(Add information about successful test commands)
 
(17 intermediate revisions by 2 users not shown)
Line 3: Line 3:
 
LTO-6
 
LTO-6
 
SAS Drive is assumed to be used (for now).
 
SAS Drive is assumed to be used (for now).
 +
 +
RHEL/CentOS 8, or Fedora 28 and up is assumed to be used in this guide, but tape drivers are part of the mainline Linux kernel and supported by all distros.
  
  
Line 12: Line 14:
  
 
== HBA (Host Bus Adaptor) ==
 
== HBA (Host Bus Adaptor) ==
 +
 +
Currently using:
 +
LSI 9267-8i 6Gb/s PCI-Express 2.0 512MB 8Port SATA/SAS + 8087 SATA Cable Battery
 +
 +
RAID-on-Chip Controller: LSISAS2208 Dual-Core RAID on Chip (ROC)
 +
Host Bus Type: x8 lane PCI Express® 2.0
 +
Cache Memory:512MB 1333MHz DDRIII SDRAM
 +
Internal Ports: 8
 +
Data Transfer Rates: Up to 6Gb/s per port
 +
Devices Supported: Up to 128 SAS and/or SATA devices
 +
Internal Connectors: 2 Mini-SAS SFF8087 (side mount)
  
  
Line 19: Line 32:
 
LTO5 and up support LTFS. But all support Tar and BareOS.
 
LTO5 and up support LTFS. But all support Tar and BareOS.
  
First check that /dev/tape/by-id exists and contains an nst0 device. If so, the Linux kernel already detected your tape drive and no further drivers are necessary. As for software though...
+
LTO Tape Drive support is part of the mainline Linux kernel through the `st` module thanks to contributions from IBM, so no drivers are actually needed for a tape drive to work.
 +
 
 +
To check to see that a tape drive is accessible, first check that /dev/tape/by-id exists and contains an nst0 device. If so, the Linux kernel already detected your tape drive and no further drivers are necessary. As for software though...
 +
 
 +
Also add your normal user to group `tape`.
  
 
== Backup Software ==
 
== Backup Software ==
Line 27: Line 44:
 
=== Tar (Simplest) ===
 
=== Tar (Simplest) ===
  
See [[LTO Tape]]
+
It is good to run tar to test write speed, though using it as a backup method leaves much to be desired, so rely on bareos after.
 +
 
 +
See [[LTO Tape]]. Make sure to always disable hardware compression using mt.
 +
 
 +
<pre>
 +
# mt -f /dev/nst0 compression 0
 +
# mt -f /dev/nst0 defcompression -1
 +
</pre>
  
 
=== LTFS (Medium) ===
 
=== LTFS (Medium) ===
  
Build from the upstream source code rather than using vendor source code, which sucks. A handy source rpm is provided.
+
Build from the upstream IBM source code rather than using vendor source code, which sucks. A handy source rpm is provided.
  
 
https://github.com/LinearTapeFileSystem/ltfs/wiki
 
https://github.com/LinearTapeFileSystem/ltfs/wiki
 +
 +
https://github.com/piste2750/rpm-ltfs
 +
 +
https://fedoramagazine.org/how-rpm-packages-are-made-the-source-rpm/
 +
 +
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/packaging_and_distributing_software/packaging-software_packaging-and-distributing-software#building-source-rpms
 +
 +
All commands should be done as normal unprivileged user: the sudo commands are elevated to root.
 +
 +
In RHEL/CentOS 8, install the dependencies:
 +
 +
{{bc|sudo dnf install rpm-build rpmdevtools}}
 +
 +
First, create the rpmbuild setuptree in your home directory.
 +
 +
{{bc|rpmdev-setuptree}}
 +
 +
Download [https://github.com/piste2750/rpm-ltfs/blob/master/ltfs.spec this .spec file, and place it into ~/rpmbuild/SPECS/ltfs.spec]. Then run the following src.rpm packaging command.
 +
 +
{{bc|rpmbuild -bb ~/rpmbuild/SPECS/ltfs.spec}}
 +
 +
Install the dependencies needed to build the source rpm.
 +
 +
{{bc|sudo dnf builddep ~/rpmbuild/SRPMS/ltfs-2.4.1.2-10254.src.rpm}}
 +
 +
Build an rpm from a .src.rpm file:
 +
 +
{{bc|rpmbuild --rebuild ~/rpmbuild/SRPMS/ltfs-2.4.1.2-10254.src.rpm}}
 +
 +
Now the rpms can be installed
 +
 +
{{bc|
 +
cd ~/rpmbuild/RPMS/x86_64
 +
sudo dnf localinstall ltfs-2.4.1.2-10254.x86_64.rpm  ltfs-debuginfo-2.4.1.2-10254.x86_64.rpm  ltfs-debugsource-2.4.1.2-10254.x86_64.rpm
 +
}}
 +
 +
Now the LTO Drive can be used. Skip all the build and install instructions in this post and use that:
 +
 +
https://www.reddit.com/r/sysadmin/comments/3qfyhm/how_to_start_using_an_lto6_tape_backup_drive_with/cyxpzo5/
  
 
=== BareOS (Enterprise) ===
 
=== BareOS (Enterprise) ===
Line 40: Line 103:
  
 
https://www.bareos.org/en/
 
https://www.bareos.org/en/
 +
 +
 +
 +
<nowiki>  bareos-sd:
 +
    container_name: bareos-sd
 +
    image: barcus/bareos-storage
 +
    devices:
 +
      - /dev/tape/by-id/scsi-3500e09efff0f9bb6:/dev/tape/by-id/scsi-3500e09efff0f9bb6
 +
</nowiki>
 +
 +
 +
 +
=== plain tar, dd, ect ===
 +
Install mt-st package for tape commands mt:
 +
$ sudo yum install mt-st
 +
$ sudo mt -f /dev/nst0 status
 +
$ sudo tapeinfo -f /dev/nst0 # Get information about drive and tape.
 +
 +
Creating a junk data file for testing:
 +
$ head -c 10G < /dev/urandom > junkfile_10GB.jnk
 +
 +
Disabling compression as it will only result in bottlenecks and lower odds of successful file retreival if the tape is damaged.
 +
$ sudo mt -f /dev/nst0 compression 0
 +
$ sudo mt -f /dev/nst0 defcompression -1
 +
 +
Getting status to confirm the drive has compression disabled:
 +
$ sudo mt -f /dev/nst0 status
 +
$ sudo tapeinfo -f /dev/nst0
 +
There should be a line indicating is is not enabled
 +
'DataCompEnabled: no'
 +
 +
LTO6: Density code 0x5a
 +
 +
Setting the block size to something the drive likes results in very significant speedups. (from 7MB/s to 170MB/s):
 +
I determined the value used by going down a power of two from 1024 to 512
 +
$ sudo /usr/bin/time --append --output=tape_time10gb.txt tar -c  -v -b 512 -f /dev/st0 junkfile_10GB.jnk
 +
 +
 +
 +
 +
 +
 +
=== Labelling Tapes ===
 +
https://tapelabel.de/
 +
 +
https://tapelabel.de/lto-sample/lto-sample.html
 +
 +
[http://www-01.ibm.com/support/docview.wss?uid=ssg1S7000429 IBM LTO Ultrium Cartridge Label Specification]
 +
 +
 +
I have been using Avery brand model "J8167" A4 size "Premium inkjet" "White Trueblock" "Mailing Labels" label sheets to reasonable short-term success.
 +
Long-term properties are not yet tested.
 +
 +
Printing was done in a decent quality multifunction household printer.
 +
 +
A metal ruler and craft knife was used to cut out the labels by gently cutting along the lines between the labels.
 +
 +
Settings:
 +
 +
"Stock label type": "blank"

Latest revision as of 09:09, 11 November 2019

Don't trust this information.

LTO-6 SAS Drive is assumed to be used (for now).

RHEL/CentOS 8, or Fedora 28 and up is assumed to be used in this guide, but tape drivers are part of the mainline Linux kernel and supported by all distros.


Links:

https://www.reddit.com/r/sysadmin/comments/3qfyhm/how_to_start_using_an_lto6_tape_backup_drive_with/cyxpzo5/ https://www.reddit.com/r/sysadmin/comments/3qfyhm/how_to_start_using_an_lto6_tape_backup_drive_with/ https://www.quantum.com/serviceandsupport/softwareanddocumentationdownloads/ltfs/index.aspx?whattab=Third https://www.ibm.com/support/knowledgecenter/en/STQNYL_2.2.2/ltfs_reference_tools_linux.html

HBA (Host Bus Adaptor)

Currently using: LSI 9267-8i 6Gb/s PCI-Express 2.0 512MB 8Port SATA/SAS + 8087 SATA Cable Battery

RAID-on-Chip Controller: LSISAS2208 Dual-Core RAID on Chip (ROC) Host Bus Type: x8 lane PCI Express® 2.0 Cache Memory:512MB 1333MHz DDRIII SDRAM Internal Ports: 8 Data Transfer Rates: Up to 6Gb/s per port Devices Supported: Up to 128 SAS and/or SATA devices Internal Connectors: 2 Mini-SAS SFF8087 (side mount)


Drive

LTO5 and up support LTFS. But all support Tar and BareOS.

LTO Tape Drive support is part of the mainline Linux kernel through the `st` module thanks to contributions from IBM, so no drivers are actually needed for a tape drive to work.

To check to see that a tape drive is accessible, first check that /dev/tape/by-id exists and contains an nst0 device. If so, the Linux kernel already detected your tape drive and no further drivers are necessary. As for software though...

Also add your normal user to group `tape`.

Backup Software

Backup Software determine the format of the data being pushed to the tape. We will only describe open source software available at no cost, because if you are an actual enterprise willing to pay big bucks stop reading this guide and start calling your IBM sales rep to purchase IBM Spectrum Protect(tm).

Tar (Simplest)

It is good to run tar to test write speed, though using it as a backup method leaves much to be desired, so rely on bareos after.

See LTO Tape. Make sure to always disable hardware compression using mt.

# mt -f /dev/nst0 compression 0
# mt -f /dev/nst0 defcompression -1

LTFS (Medium)

Build from the upstream IBM source code rather than using vendor source code, which sucks. A handy source rpm is provided.

https://github.com/LinearTapeFileSystem/ltfs/wiki

https://github.com/piste2750/rpm-ltfs

https://fedoramagazine.org/how-rpm-packages-are-made-the-source-rpm/

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/packaging_and_distributing_software/packaging-software_packaging-and-distributing-software#building-source-rpms

All commands should be done as normal unprivileged user: the sudo commands are elevated to root.

In RHEL/CentOS 8, install the dependencies:

sudo dnf install rpm-build rpmdevtools

First, create the rpmbuild setuptree in your home directory.

rpmdev-setuptree

Download this .spec file, and place it into ~/rpmbuild/SPECS/ltfs.spec. Then run the following src.rpm packaging command.

rpmbuild -bb ~/rpmbuild/SPECS/ltfs.spec

Install the dependencies needed to build the source rpm.

sudo dnf builddep ~/rpmbuild/SRPMS/ltfs-2.4.1.2-10254.src.rpm

Build an rpm from a .src.rpm file:

rpmbuild --rebuild ~/rpmbuild/SRPMS/ltfs-2.4.1.2-10254.src.rpm

Now the rpms can be installed

cd ~/rpmbuild/RPMS/x86_64
sudo dnf localinstall ltfs-2.4.1.2-10254.x86_64.rpm  ltfs-debuginfo-2.4.1.2-10254.x86_64.rpm  ltfs-debugsource-2.4.1.2-10254.x86_64.rpm

Now the LTO Drive can be used. Skip all the build and install instructions in this post and use that:

https://www.reddit.com/r/sysadmin/comments/3qfyhm/how_to_start_using_an_lto6_tape_backup_drive_with/cyxpzo5/

BareOS (Enterprise)

Not an operating system, but "Backup Archiving REcovery Open Sourced" with tape as one key storage backend. It was forked from Bacula. Does not require LTFS.

https://www.bareos.org/en/


bareos-sd: container_name: bareos-sd image: barcus/bareos-storage devices: - /dev/tape/by-id/scsi-3500e09efff0f9bb6:/dev/tape/by-id/scsi-3500e09efff0f9bb6


plain tar, dd, ect

Install mt-st package for tape commands mt: $ sudo yum install mt-st $ sudo mt -f /dev/nst0 status $ sudo tapeinfo -f /dev/nst0 # Get information about drive and tape.

Creating a junk data file for testing: $ head -c 10G < /dev/urandom > junkfile_10GB.jnk

Disabling compression as it will only result in bottlenecks and lower odds of successful file retreival if the tape is damaged. $ sudo mt -f /dev/nst0 compression 0 $ sudo mt -f /dev/nst0 defcompression -1

Getting status to confirm the drive has compression disabled: $ sudo mt -f /dev/nst0 status $ sudo tapeinfo -f /dev/nst0 There should be a line indicating is is not enabled 'DataCompEnabled: no'

LTO6: Density code 0x5a

Setting the block size to something the drive likes results in very significant speedups. (from 7MB/s to 170MB/s): I determined the value used by going down a power of two from 1024 to 512 $ sudo /usr/bin/time --append --output=tape_time10gb.txt tar -c -v -b 512 -f /dev/st0 junkfile_10GB.jnk




Labelling Tapes

https://tapelabel.de/

https://tapelabel.de/lto-sample/lto-sample.html

IBM LTO Ultrium Cartridge Label Specification


I have been using Avery brand model "J8167" A4 size "Premium inkjet" "White Trueblock" "Mailing Labels" label sheets to reasonable short-term success. Long-term properties are not yet tested.

Printing was done in a decent quality multifunction household printer.

A metal ruler and craft knife was used to cut out the labels by gently cutting along the lines between the labels.

Settings:

"Stock label type": "blank"