Nekochan Net

I've mentioned in earlier posts in the hardware forum work I am doing on two Altix clusters (running SLES 9). To be brief the older cluster has 48gb ram and a TP900 with 7x 146gb storage. The newer one has 64gb ram and 16x 146gb storage (in two TP900s). I started copying home directories from the old array to one of the new arrays recently after building them into RAID-5 using mdadm.

I noticed that the home directories were taking a very long time, and using a fair bit more space than they used before. At first I thought the space difference might just be a file cluster size issue, so I didn't worry about it too much. Then eventually the home directories (which did not finish copying) became enormous - some 10 times their original size.

A little investigation lead me to what was causing this.

It appears that every user on the system who has logged in to an SGI workstation in person has a directory "Desktop" in their home. In that Desktop directory is a symlink that is named the same as their user name - hence if I had a user "Bob Smith" called "smithb", his home directory would be /home/smithb. Furthermore the problem link would be /home/smithb/Desktop/smithb, pointing at /home/smithb.

I was originally using scp to copy the files from one cluster to another, and I specifically chose the "-r" option for recursive copying. Unfortunately, scp would traverse the directories to the problem link, which lead it back to the home directory. Hence pretty soon poor Mr. Smith's home directory had directories like /home/smithb/Desktop/smithb/Desktop/smithb/Desktop/smithb/Desktop ... and so on. I never bothered to figure out just how many levels down it went before I killed the scp jobs.

I have since killed off those symlinks; at least for the users whose home directories I plan to copy any time soon. It does appear that IRIX will re-create those links whenever needed so I don't think we're losing anything by unlinking them.

Good reminder about scp. Scp ain't Unix. There's a general scp issue when using the -r option and encountering symbolic links.

Assuming you have enough disk space on the original system, I would have kicked everyone off the machine (single user mode if possible) and made a tarball of the whole /home directory and then copied that over. Copying a single large file is much more efficient with respect to network overhead than copying hundreds or thousands of files.

Alternately, assuming that everything is on a local, trusted net, you could have kicked everyone off the systems, mounted via NFS, and done a direct tar create/extract as follows (executed from the old machine, with the new machine's /home NFS mounted on the old machine as /newhome):

(cd /home ; tar cvf . - ) | (cd /newhome ; tar xvf - )

You could also do the same type of thing with xfsdump, cpio, etc., modifying the command lines accordingly.

That's the same method I used when migrating Nekochan from IRIX to FreeBSD - rolled tarballs of directories I needed to move and sftp'd them over afterwards. If space is tight, compressing with gzip/bzip may help.

I usually use rsync when copying lots of files over. If you specify the "-a" flag it not only preserves user, group, owner and time, but also copies symlinks as symlinks instead of following them. There are lots of other useful flags, which the man page of course explains

If available, I like fs dump/restore utilities such as xfsdump, ufsdump, zfs send, etc. You can send the data across a network connection or create a file and restore from the file.

Ahem. scp is the rcp protocol over an ssh connection, to the letter. So while it's a shame rcp is quite limited, I wouldn't pretend scp isn't unixy...

Ahem. I didn't say that it wasn't unixy, just that it wasn't Unix.

For system to system, rsync is the best tool (if available), as you can always Control-C it if necessary and restart from where you left off (the -a option covers most bases). Otherwise, you can use the basic ssh and tar commands to transfer data across the network, preserving permissions, groups, devices and links... something like:

tar cf - /source | ssh user@host "cd /dest ; tar xf -"

I found tarballing stuff, while inconvenient, is easiest. Everything else is prone to hick-ups. Even rsync (or especially?). Then again, I don't do heterogenous anymore. I mostly just use OpenSolaris and a bit of Win7, which gets stuff pushed to it via a FAT partition.

We are on a very local, very trusted, very small, network here. While I often criticize the network admins above me for their heavy-handed firewall rules, it does keep out the riff-raff that could interfere with my work. And in this case not only are the two clusters on the same network, they are in the same room (closet, actually, but that is another story) and hooked up to the same switch.

So I decided to go ahead with the NFS option. I mounted the NFS shares from the old cluster on to the new cluster and started copying away. In this case I was only looking to copy two home directories - my own and my boss's - and we were both logged out at the time. It took a while since the directories were so large, but it seems pretty content now. I've even rebooted the new cluster a few times just to make sure it would be content and things look good.

Eh? Wuh? Wha? Gah?

I guess you haven't been paying attention! Check out viewtopic.php?f=2&t=16722649

Yeah, haven't been around much lately. Somehow I was convinced that nekochan itself was still running on a production IRIX system...