Nekochan Net

Mac'er,

here are some notes on the Mac Pro RAID card (from Early 2008 and prior Mac Pro, this is version with 256MB Cache, current Mac Pro 2009 has 512MB Cache and uses PCI base based lines for SATA/SAS connectivity).

Aplogies in advance for the length of the post, it just the the rather strange behavior of this is not that well decribed...

1. Installation requires significant effort, as you have to pull out the fan module to get to system board and detach the SATA cabling from system board to replug it into Mac Pro RAID card. This has an advantage of leaving you with 4 free SATA additional SATA lanes which can be used for eSATA, SATA DVD/Blu-Ray etc (see my post about MCE Blu-Ray for details on this)

2. To configure RAID use "RAID Utility". This allows you to establish a number of RAID configurations based on no. of disks and whether performance or data protection is the priority.

3. When you first get a systems from Apple it will have the Boot hard disk set set up as a single drive which is configured as an "Enhanced JBOD" raid volume (or J+). So what is an "J+". It is essentially a volume that can be migrated to an alternative RAID configuration.

4. Using "RAID Utility" you can migrate the "J+" to an multi-disk configuration, however I do not think this will provide what you might expect....
When I converted a single disk "J+" configuration to a 4 disk RAID 5 I ended up with 2 volumes... not a single volume.
The first Volume was the same size as the original "J+" volume but now "smirred" over 4 disks... with a second volume titled "Untitled" taking up the rest of the space... This is not a very useful behavior as what I wanted was to end up with a single "big disk" holding all that was on prior "J+" volume. To achieve this you need to take an image of the original "J+" disk using "Disk Utility" or "Carbon Copy Cloner" and then boot from Install DVD and run "RAID Utility" to delete the existing "J+" volume then create a completely new RAID Set using the desired number of disks, then use "Disk Utility" to restore your original image backup onto the new RAID volume. If you used "Carbon Copy Cloner" this will require that you reboot back to clone disk and then reclone back onto the new RAID Volume.
Note that this means if you wish to use all 4 Disks in a RAID volume, then you must have some other disk system attached to your Mac that you can copy the image/clone to and from... I would recommend that this not be a USB based disk.. as it will be way to slow.

5. Disk failure scenario 1. This is the really interesting one. Proceed with extreme caution. Be very pessimistic on what will happen. My original RAID setup had 4 Seagate 1TB drives. These drives are notoriously unreliable and have very high failure rates due to a bug in the firmware. So if you have 4 of them the probability of failure on your RAID is very high. Do not just stripe your disks if you have Seagate 7200.11 series SATA disks , you are likely to experience a failure and hence data lose. On my first disk failure I got a notification from the "RAID Utility" and I immediately did a Time Machine backup (to my Fibre Channel based disk set, but any other thing including Time Capsule, external FireWire drive etc would have been ok). I then shutdown the machine and removed the failed disk and added a new 1.5TB Seagate 7200.11 disk. This was recognised by "RAID Utility" and by then selecting the disk as a spare via "RAID" -> "Spare". This will result in the system automatically doing a RAID Set rebuild. This occurs as a background task and you are still able to use the computer as the "RAID Utility" churns away in the background. And churn it does... it churned away for over six hours until... the entire computer crashed! At this point the entire machine was unbootable and I had to boot it off the install DVD. I tried to recover the RAID Set and the "Verify RAID Set" command on the "RAID Utility" comes back ok. I also did a check of the desk via "Disk Utility" and it appeared ok, but computer still hung on boot. So I rebooted via Install DVD again and completely deleted and recreated the RAID set across the 4 disks and then did a "Utilities" -> "Restore System from Backup" keeping my fingers cross that the OX restore was more reliable than the "RAID Utility". It was after much playing around and a very long time (large RAID Sets can take half a day to a full day to complete...) my machine was back in working condition. Until ...

6. Disk failure scenario 2. So still having 3 of the dodgy 1TB 7200.11 Seagate SATA drives in my machine, not surprisingly another one failed. So learning from previous experience I did an immediate backup via Time Machine from remaining viable 3 disk RAID set. I went and purchased another 1.5TB Seagate SATA drive as prices of these is now so low that I just got the largest available. Being a bit more cautious this time, I booted and the "RAID Utility" came up automatically and I once more did a "RAID" -> "Make Spare" from the menu and left the "RAID Utility" to churn away leaving computer untouched to avoid any possibility of conflict with other running programs. Again it churned away for most of the afternoon and well into the evening.... and crashed. Once more rendering the computer unbootable. Once more the rebuilt option on disk failure had made the entire system unusable. In this case it appears that the brand new Seagate 1.5TB disk had failed on rebuild and so rendered the entire RAID Set unusable. Pending receiving a replacement of the new disk and getting replacements for further two old 1TB drives, I decided that I would restore my system to single "Extended JBOD" disk, on assumption that this would then provide a simple "Migrate RAID Set" option once I got rest of the disks back into the machine. This led me to discover the rather useless capability of the "Migrate RAID Set" as described in point (4) above.

7. Next option and to try to avoid the long time required for Time Machine based restore and now very much going on the cautious side due to complete unreliability of both the Seagate disk and the Mac Pro RAID controller, I have decided to go with 3 disk RAID 5 with supporting "J+" manually managed emergency spare and backup. To do this I deleted the useless "Untitled" volume created by the "Migrate RAID Set" option and then pulled the last disk out of my 4 disk RAID 5 set to force a failure (while machine was off). On reboot the "RAID Utility" detected a failure. I then re-shutdown machine and reinserted the disk, which because there is no automatic rebuilt then come up as "Roaming" in its status. I then did a "Carbon Copy Cloner" replication of now "smirred" across 3 disks remaining RAID volume to new single J+ volume I created from 4th disk and then completely deleted 3 disk yet single disk sized volume crated by "Migrate RAID Set" option. I created a new 3 disk RAID 5 Volume and copied back from single "J+" volume using "Carbon Copy Cloner". The result is I have now have 3 disk RAID 5 with a spare backup disk I can use in the event of a single disk failure on the RAID 5 Volume.

8. Suggestions
(a) Never ever attempt to rebuilt RAID set without first ensuring you have a done a back up of the damaged volume first. It is likely to result in total loss of data
(b) I suggest that 3 disk RAID 5 with manually managed backup via single J+ is a safe option, as it allow recovery without risk of rebuilt failure, while still providing a significant increase in the level of data protection
(c) Avoid "Migrate RAID Set" option, what this does is not really that useful.
(d) Best manage your RAID Sets and Volumes manually than rely on current functionality

9. Now I have moved from 4 Disk RAID 5 to 3 Disk RAID 5 I will retest the disk performance to see the difference.

10. I have also used Apple's software RAID that is available via "Disk Utility" on my Fibre Channel connected array and this does automatic rebuilt and recovery without a hitch. So software RAID is much more reliable than hardware RAID. Unfortunately this only does Stripes and Mirror not RAID 5, which is a shame as the disk on Fibre Channel are much smaller than those available for SATA and so space is much more at premium.

I hope this will help some people who have the misfortune to have some low quality drives....

_________________
jwhat - ask questions, provide answers

I'm curious whether anybody has a similar report of experiences with disk failure using mirrored drives with the onboard SATA ports (which is what I'm doing).

_________________

Hi Kjaer,

if you are talking about inbuilt SATA drives set up to RAID 1 mirror via "Disk Utility" then the results should be same as per my Fibre Channel case as the drives are just a bunch of disk (JBOD) set up as RAID via software driver. As I posted the software RAID has proved to be very reliable, I have had two cases where it went to automatic mirror set rebuilt (caused by RAID Set not starting correctly because of delay in time starting up Brocade Fibre Channel switch vs Mac Pro boot time I believe). In both cases it detected problem and went into automatic rebuilt. So far the Fibre Channel disks (mostly Hitachi's) have been much more reliable than the SATA disks...).

I only really use the Fibre Channel when backing up and archiving or needing serious i/o rates from non redundant striped set (RAID 0).

I do all my Time Machine backups to RAID set on Fibre Channel and also have archived all my original IRIX based code and data stored there on RAID 1+0 (ie Striped set of 4 mirrored disks).

If the Fibre Channel setup was not so noisey (FC Switch & 16 Disk Array Blow out a lot of air...), then I would not use the Apple RAID card at all.

Cheers,

_________________
jwhat - ask questions, provide answers

Nekochan Macers,

Following over 30 hours rebuilt of 3 x 1.5TB RAID 5 Volume - 3 hours backup, approx 24 hours to initialise new Volume and 3 hours restore...

Here are XBench Disk performance figures with new Mac Pro RAID 5 across 3 x 1.5TB Seagate 7200 RPM disk.
I am using a single volume to store everything (ie OS, Apps and User Files).

For comparison purpose I have also rerun the disk bench marks on Fibre Channel attached storage as well.



So results show about the same performance as prior 4 x 1TB RAID 5 configuration, which was posted here: viewtopic.php?f=9&t=16719046
Though Random results look better in going from 86 to 99.
Again Apple Mac Pro RAID with "inexpensive" SATA II 7200 performs very well compared to significantly more expensive FC based option.

Cheers,

_________________
jwhat - ask questions, provide answers