Tape drives

SGI hardware problems, solutions, tips, hacks, etc.
Forum rules
Any posts concerning pirated software or offering to buy/sell/trade commercial software are subject to removal.

Best tape archiver tool?

tar
9
56%
cpio
1
6%
xfsdump
6
38%
 
Total votes: 16

User avatar
mia
Posts: 976
Joined: Wed Feb 19, 2003 1:54 pm

Tape drives

Unread postby mia » Sat Nov 03, 2012 10:31 pm

It is said that you should spend as much in storage than in backups.

Some companies went completely out of business due to an inappropriate/in-existent backup process (notice: I said "process", not software or script). Today, I will review how I'm currently archiving "nearline" backups, but I will purposely not cover storage, software and tape inventory in this thread, simply hardware, the reason for sharing this is not simply that it could potentially benefit others, it is also that, in my own selfishness perhaps, I hope some might help me improve the current implementation.

As far as the software is concerned, it is supposed to fit one's needs; therefore the solution you adopt on any system should fit your own needs and retention policy; my needs are likely to be very different from yours; which is why I'm not going to give details on which software is being used; although I will give the following short overview:

Backups are handled in multiple ways here, nearline storage gets replicated remotely, and backups are performed on an "large SGI", we use Amanda for some servers, as it worked well for us in the past; but also some specific storage archival software (commercial) which works in adequately with our NFS and FCP servers (no, not ftp; fcp: fiber channel protocol).

The tape drive used is a HP LTO-4 (model 1760, mounted externally, scsi id #3, no terminator). For the sake of testing, I plugged this same drive on a few machines using few different cards and performed a few tests (using a LTO-4 tape), I will review the results I got, archiving a 45GB directory.

Test #1: External Tezro* #1 "IO9" Qlogic 12160, gnu tar 1.26, no compression.
Integral SCSI controller 1: Version QL12160, low voltage differential
Tape drive: unit 3 on SCSI controller 1: unknown
Time to tape: 31m14s (24MB/s).

Test #2: LSI 22320-R on Tezro* #1, gnu tar 1.26, no compression.
Integral SCSI controller 4: Version LS1030, low voltage differential
Tape drive: unit 3 on SCSI controller 4: unknown
Time to tape: 29m43s (25MB/s).

Test #3 (Irix control group): Tezro* #1, tar to local disk (on Qlogic 12160), gnu tar 1.26, no compression.
Integral SCSI controller 0: Version QL12160, low voltage differential
Disk drive: unit 1 on SCSI controller 0 (XVM Local Disk) (primary path)
Disk drive: unit 2 on SCSI controller 0 (XVM Local Disk) (primary path)
Time to tar: 33m36s (22MB/s)

Test #4: dd of the tarball created during test #3 to the tape drive, block size of 64k, tape drive on LSI 22320-R on Tezro* #1 (using Irix's dd).
Integral SCSI controller 4: Version LS1030, low voltage differential
Tape drive: unit 3 on SCSI controller 4: unknown
Time to dd: 19m07s (39MB/s)

Test #5 (for reference): Qlogic 12160 on Altix 350, gnu tar 1.23, no compression.
scsi 0:1:3:0: Sequential-Access HP Ultrium 4-SCSI W51D PQ: 0 ANSI: 5
scsi(0:1:3:0): Sync: period 9, offset 14, Wide, DT, Tagged queuing: depth 31
Time to tape: 24m43s (30MB/s)

Test #6 (Linux control group): tar to local disk, Qlogic 12160 on Altix 350, gnu tar 1.23, no compression.
ata1.00: ATA-8: WDC WD5002ABYS-02B1B0, 02.03B03, max UDMA/133
scsi 1:0:0:0: Direct-Access ATA WDC WD5002ABYS-0 02.0 PQ: 0 ANSI: 5
Time to tar: 26m59s (28MB/s)

Test #7: LSI 22320-R on Tezro* #1, xfsdump to tape.
Integral SCSI controller 4: Version LS1030, low voltage differential
Tape drive: unit 3 on SCSI controller 4: unknown
Time to tape: 20m01s (37MB/s)

Test #8: External Tezro** #2 "IO9" Qlogic 12160, gnu tar 1.26, no compression.
Integral SCSI controller 1: Version QL12160, low voltage differential
Tape drive: unit 3 on SCSI controller 1: unknown
Time to tape: 31m30s (24MB/s)

Test #9: External Tezro** #2 "IO9" Qlogic 12160, xfsdump to tape.
Integral SCSI controller 1: Version QL12160, low voltage differential
Tape drive: unit 3 on SCSI controller 1: unknown
Time to tape: 30m31s (24.5MB/s)

Test #10: External Tezro** #2 "IO9" Qlogic 12160, cpio (using Irix's cpio).
Integral SCSI controller 1: Version QL12160, low voltage differential
Tape drive: unit 3 on SCSI controller 1: unknown
Time to tape: 29m11s (25.5MB/s)

The cpu load on the tezro was 100% on only one cpu during tar (~10% on the altix, with 2% IOwait), so I think this is the current bottleneck. Tar (at least the GNU implementation) can not be threaded, however, thanks to hundreds of thousands of users using it (on different architectures and operating systems), it is reliable, a few efforts to replace GNU tar have been done already and it seems that all failed so far.

During the xfsdump test, the load on the tezro was shared amongst processors, which explains the discrepancy; nevertheless, less people run xfs (and xfsdump) than tar, so I'm not completely sure this would be more reliable than tar.

Regarding APD (asynchronous personality daemon), the tests were done on a tezro with apd installed, but disabled. For consistency, test #8 was done on a tezro where apd has never been installed.

I welcome comments, thoughts, spare tapes and concerns.

Notes:
(*) tezro #1 where apd has been installed but disabled.
(**) tezro #2 where apd has never been installed, identical hardware as tezro #1, except drives: tezro #1 has a pair of 300GB seagate and tezro #2 has a pair of 300GB HP/Compaq drives; which are a little slower than their seagate counterpart; the IOwait is noticeably higher on tezro #2 than #1 during archival.
gnu tar options: tar cvf /dev/rmt/tapedevice /path/to/directory
xfsdump options: xfsdump -o -F -l0 -J -f /dev/rmt/tapedevice -L 20121104 -s path/to/directory /
cpio options: find /path/to/directory -print | cpio -oKv > /dev/rmt/tapedevice
_______________________________________________________________________________________________________
Edit (Sat Nov 3 22:58:37 PDT 2012): added tests #6 and #7, comments on xfsdump.
Edit (Sun Nov 4 10:20:41 PST 2012): added tests #8, #9, #10, comments on apd, backup arguments used.
_______________________________________________________________________________________________________
Last edited by mia on Sun Nov 04, 2012 12:46 pm, edited 1 time in total.
:Onyx2:

hamei
Posts: 10002
Joined: Tue Feb 24, 2004 4:10 pm
Location: over the rainbow

Re: Tape drives

Unread postby hamei » Sun Nov 04, 2012 3:16 am

Thank you for the valuable information (xfsdump for me) .... and the kick in the ass reminder to do backups !!!

kubatyszko
Posts: 352
Joined: Sat Nov 12, 2011 3:18 am
Location: Tokyo
Contact:

Re: Tape drives

Unread postby kubatyszko » Sun Nov 04, 2012 5:09 am

Is that with SGI APD or without ?

Thanks
[click for links to hinv] JP: :Fuel: |:O2: | :Indy: || PL: [ :Fuel: :O2: :O2+: :Indy: ]

User avatar
recondas
Moderator
Moderator
Posts: 5287
Joined: Sun Jun 06, 2004 5:55 pm
Location: NC - USA

Re: Tape drives

Unread postby recondas » Sun Nov 04, 2012 6:08 am

Thanks for sharing - that's very useful info.
mia wrote:My conclusions on the results above are as follows: 37MB/s is bearable, if not satisfactory.
Be interesting to see how a fiber channel connected tape drive would fit into those results. An FC connection might also allow the use of a fiber channel switch to share a single tape drive between multiple systems.

To test both of those ideas I recently acquired (but haven't yet tested) a 1GB FC-to-SCSI bridge from the Keeper. That one is a bare PCB I plan to install in a 2U tape drive enclosure, but FC-to-SCSI bridges can be had in free-standing rackmount enclosures. I haven't done any research into compatibility or function, so solely as an illustration of the type of device I'm describing, here's an example.

kubatyszko wrote:Is that with SGI APD or without ?
Good question. I've speculated that APD might have been at least partially a marketing tool, so it would be interesting to see the results of testing with and without APD.
***********************************************************************
Welcome to ARMLand - 0/0x0d00
running...(sherwood-root 0607201829)
* InfiniteReality/Reality Software, IRIX 6.5 Release *
***********************************************************************

User avatar
mia
Posts: 976
Joined: Wed Feb 19, 2003 1:54 pm

Re: Tape drives

Unread postby mia » Sun Nov 04, 2012 8:30 am

I would like to move to LTO-5 or LTO-6, although, external or rackmount (no library) FC drives are difficult to find, even LTO-4 ones; it's all SAS now. Note that I have APD installed but disabled. In any case, I don't think APD supports LTO-4 drives.
:Onyx2:

User avatar
mia
Posts: 976
Joined: Wed Feb 19, 2003 1:54 pm

Re: Tape drives

Unread postby mia » Sun Nov 04, 2012 12:47 pm

added more tests to my original post.
:Onyx2:

User avatar
hhoffman
Posts: 71
Joined: Fri Apr 01, 2011 7:45 am

Re: Tape drives

Unread postby hhoffman » Wed Nov 07, 2012 2:06 am

I'm also using a HP LTO-4 (model 1760, mounted externally, scsi id #5, no terminator) with my tezro. The entry for it in my /var/sysgen/master.d/scsi looks like that:

Code: Select all

 /* HP LTO4 / Ultrium 4 */
   { DATTAPE, TPDAT, 2, 9, "HP", "Ultrium 4", 0, 0, {0},
          MTCAN_BSF|MTCAN_BSR|MTCAN_APPEND|MTCAN_SETMK|MTCAN_PART|MTCAN_PREV|
          MTCAN_SYNC|MTCAN_SPEOD|MTCAN_CHKRDY|MTCAN_VAR|MTCAN_SETSZ|
          MTCAN_SILI|MTCAN_AUDIO|MTCAN_SEEK|MTCAN_CHTYPEANY|MTCAN_COMPRESS,
          40, 5*60, 20*60, 3*3600, 512, 256*512, 512*512,
          tpsc_default_dens_count, tpsc_default_hwg_dens_names,
          tpsc_default_alias_dens_names,
          {0}, 0, 0, 0,
          0, (u_char *)0 },


mt status says:

Code: Select all

tezro 1#mt status
        Controller: SCSI
        Device: HP: Ultrium 4-SCSI  W54D
        Status: 0x20262
        Drive type: DAT
        Media : READY, writable, at BOT

User avatar
recondas
Moderator
Moderator
Posts: 5287
Joined: Sun Jun 06, 2004 5:55 pm
Location: NC - USA

Re: Tape drives

Unread postby recondas » Wed Nov 07, 2012 9:27 am

hhoffman wrote:I'm also using a HP LTO-4 (model 1760, mounted externally, scsi id #5, no terminator) with my tezro. The entry for it in my /var/sysgen/master.d/scsi looks like that:

Thanks! I added your SCSI definition file to those listed in the LTO Anyone thread. Can I ask if your LTO-4 definition file originated with HP?
***********************************************************************
Welcome to ARMLand - 0/0x0d00
running...(sherwood-root 0607201829)
* InfiniteReality/Reality Software, IRIX 6.5 Release *
***********************************************************************

User avatar
mia
Posts: 976
Joined: Wed Feb 19, 2003 1:54 pm

Re: Tape drives

Unread postby mia » Wed Nov 07, 2012 10:16 am

Amazingly, this provided file doesn't work with my 1760:

Code: Select all

root@plum:~# tar cvf /dev/rmt/tps4d3 /tmp/tape_benchmark
tar: /dev/rmt/tps4d3: Cannot write: Invalid argument


This doesn't seem to be a tar issue:

Code: Select all

root@plum:~# dd if=/dev/zero of=/dev/rmt/tps4d3
Write error: Invalid argument


However, changing this:
DATTAPE, TPDAT, 2, 9, "HP", "Ultrium 4", 0, 0, {0},
[...no changes here...]
40, 5*60, 20*60, 3*3600, 512, 256*512, 512*512,

to this:

TPUNKNOWN, TPUNKNOWN, 2, 9, "HP", "Ultrium 4", 0, 0, {0},
[...no changes here...]
40, 5*60, 20*60, 20*60, 3*3600, 512, 512*512,


Makes it work, why?

This is what "mt blksize" reports with hhoffman's definition file:

Code: Select all

 Recommended tape I/O size: 262144 bytes (512 512-byte blocks)                 
 Minimum block size: 1 byte(s)                                                 
 Maximum block size: 16777215 bytes                                           
 Current block size: Variable

This is with the one I use (and works):

Code: Select all

 Recommended tape I/O size: 262144 bytes (512 512-byte blocks)
 Minimum block size: 1 byte(s)
 Maximum block size: 16777215 bytes
 Current block size: 512 byte(s)

My complete definition file:

Code: Select all

         { TPUNKNOWN, TPUNKNOWN, 2, 9, "HP", "Ultrium 4", 0, 0, {0},
           MTCAN_BSF|MTCAN_BSR|MTCAN_APPEND|MTCAN_SETMK|MTCAN_PART|MTCAN_PREV|
             MTCAN_SYNC|MTCAN_SPEOD|MTCAN_CHKRDY|MTCAN_VAR|MTCAN_SETSZ|
             MTCAN_SILI|MTCAN_AUDIO|MTCAN_SEEK|MTCAN_CHTYPEANY|MTCAN_COMPRESS,
           40, 5*60, 20*60, 20*60, 3*3600, 512, 512*512,
           tpsc_default_dens_count, tpsc_default_hwg_dens_names,
             tpsc_default_alias_dens_names,
           {0}, 0, 0, 0,
           0, (u_char *)0 },
:Onyx2:

User avatar
recondas
Moderator
Moderator
Posts: 5287
Joined: Sun Jun 06, 2004 5:55 pm
Location: NC - USA

Re: Tape drives

Unread postby recondas » Wed Nov 07, 2012 11:18 am

mia wrote:

Code: Select all

TPUNKNOWN, TPUNKNOWN, 2, 9, "HP", "Ultrium 4", 0, 0, {0},
My guess is the edits to cause LTO to be identified as TPUNKNOWN are going to cause more than your share of headaches.

mia wrote:Amazingly, this provided file doesn't work with my 1760

Is this with the Tezro that has APD installed, or the one you mentioned that didn't have APD?

If you haven't, I'd suggest trying it on the non-APD Tezro, and if possible, with a *copy* of the original unedited /var/sysgen/master.d/scsi.

If you have a copy of the original unedited /var/sysgen/master.d/scsi
  • restore a *copy* of the original unedited master.d/scsi;
  • as root run autoconfig;
  • reboot with the LTO-4 attached and running;
  • as root, run ./MAKEDEV tape. followed by mt status;
  • double check that the "device" line in mt status matches the one in hhoffman's mt status (the firmware revision "W54D" might be different);
  • paste a copy of hhofman's LTO-4 device file into the copy of master.d/scsi and save the change;
  • re-run autoconfig and reboot again;
  • with the LTO-4 still attached and running, re-run ./MAKEDEV tape;
  • then run mt status and post the results here.
If your mt status matches the one hhoffman posted, I'd suggest first testing the drive by running a small back up via the IRIX back up GUI. If that works then try tar.
***********************************************************************
Welcome to ARMLand - 0/0x0d00
running...(sherwood-root 0607201829)
* InfiniteReality/Reality Software, IRIX 6.5 Release *
***********************************************************************

User avatar
mia
Posts: 976
Joined: Wed Feb 19, 2003 1:54 pm

Re: Tape drives

Unread postby mia » Wed Nov 07, 2012 6:19 pm

Recondas, Hhoffman,

I'm as surprised and confused by this discrepancy; I have removed APD on this machine; yet I am unsure why I do not get identical results, similarily I get identical results on another Tezro, but this could be related to the fact that both (and the others) were installed from the same Irix netinstall.

Recondas, I have followed your procedure, to the same outcome.

Hhoffman, which throughput do you get, when writing to a tape, either with tar or xfsdump (or dd, really), and which scsi card are you using?

My mt status reports this following firmware: Ultrium 4-SCSI W51D, while Hhoffman is running W54D. This is the only possible explanation at this time.

Please note that the latest firmware for this drive is: W62D.
:Onyx2:

User avatar
mia
Posts: 976
Joined: Wed Feb 19, 2003 1:54 pm

Re: Tape drives

Unread postby mia » Wed Nov 07, 2012 8:01 pm

I've upgraded the drive's firmware to W62D, which is the latest firmware available for the 1760; and the repeated the test on a very fast Linux/amd64 machine; the results are astounding: 19m04s (39MB/s) for a tar archive of this same directory (which was placed on very fast storage).

I have redone the "tar" test using this new firmware, on the Tezro, and was able to do it in 24m00s (31 MB/s) which is really good, so I'm fairly certain the firmware changed a few things, so I'm going to try different scsi parameters now.

Stay tuned.
:Onyx2:

User avatar
mia
Posts: 976
Joined: Wed Feb 19, 2003 1:54 pm

Re: Tape drives

Unread postby mia » Wed Nov 07, 2012 8:52 pm

I've repeated the tests, this time with a different tar blocking factor, as such:

timex tar cvb 4096 -f /dev/rmt/tps4d3 /path/to/files


Tezro: 15m53s (47MB/s)
"Very fast" Intel*: 19m04s (39MB/s)

Now I'm happy. I'll run multiple tests overnight to collect some statistics with different blocking factors; this is really much better, now I can write a 800GB tape in less than 5 hours. The payload written to tape is hardly compressible, so I'm not going to bother trying compression on this dataset, this will be the subject of another test.

(*) 8 cpus, pci-e scsi card, tons of ram, raid-6, a disgusting marvel of technology; yet beaten by a Tezro.
:Onyx2:

User avatar
hhoffman
Posts: 71
Joined: Fri Apr 01, 2011 7:45 am

Re: Tape drives

Unread postby hhoffman » Thu Nov 08, 2012 8:56 am

Can I ask if your LTO-4 definition file originated with HP?

Yes of course. I tried to get support from hp for the unit with IRIX, but they where not willing to help. My version for the SCSI file simply came out of the LTO 3 version in the 'LTO Anyone thread'.
Anyway, with my scsi file entry, I noticed some issues with tar (original IRIX version) and mt. But cpio (with the IRIX backup interface) and gnu tar worked fine. With mia's version, these issues disappeared.

User avatar
mia
Posts: 976
Joined: Wed Feb 19, 2003 1:54 pm

Re: Tape drives

Unread postby mia » Thu Nov 08, 2012 8:58 am

Hhoffman,

Glad it helped.

Could you tell us which throughput you currently get with tar and/or cpio/xfsdump? Finally, could you describe those tar & mt issues?

Thanks!
:Onyx2:


Return to “SGI: Hardware”

Who is online

Users browsing this forum: No registered users and 2 guests