More RAID rant

This post is an update to my previous post about my RAID-5 experience. Also see the continuation of this saga.

I found some disturbing problems with the nForce semi-software RAID-5 fault tolerance - without the fourth drive, the array seemed to work normally, but any modifications to the drive contents seemed to corrupt data on the drive. For example, in order to back up the data, after I had copied a few folders of data, I started to verify the copied data and delete it as I moved on. I always do it this way - first copy the data over, then run byte-per-byte verify (or MD5 hash verify) to see if the copy process worked and only after successful verify I will delete the data at the source.

Now, however, after I had verified and deleted the first folder of data from the RAID-5 array, the next folder showed an error in couple of files. Since they were picture files I was able to visually check the difference and the result was puzzling - the copied data seemed to be OK, but the source data on RAID-5 was corrupted. I tried a reboot, but the source file contents were still corrupted. I dismissed this as random happening and deleted the source (after all the errors were in the source data). Now the next folder showed even more corruptions and again the corrupted data was at the source.

A RAID-5 array with N drives saves actual data on N-1 of the drives and calculates a parity information to the remaining one drive. The data is saved in rotating stripes, so that the parity information of each next stripe is always on a different disk.
In case of reading data, if any one of the drives have failed, then N-1 times out of N, the the failed part of the data for each stripe is constructed by using the parity.

For example, when a stripe has parity on the drive 4, but drive 2 has failed, then the contents of drive 1 and 3 as well as the drive 4 with parity info is read and the contents of the failed drive 2 is constructed from the three other drives. Only in case the drive 2 held the parity data, is the reconstruction not needed.

The writing, however, is much more difficult. Assuming the above scenario, if the data is updated in the drive 1 region, then in addition to updating data on drive 1, the contents of drives 3 and 4 are read, the contents of drive 2 is temporarily reconstructed and new parity data is generated, which is updated on drive 4. If the data is updated in the failed drive 2 region, then similarly the data from drives 1, 3 and 4 are read, the original data of drive 2 is reconstructed, updated in memory and new parity is created and updated on drive 4.

Since the corruption was only few dozen bytes at a time and with no detectable pattern of changed bits I dismissed any further drive failure and assumed that one of the SATA cables could be faulty. I bought full set of new cables and tried again, recreated new RAID-5 array, copied a few dozen gigabytes of data on it. Made a copy of the data and then another copy of the same data. While the second copy was in progress, I disconnected one of the drives. At first, everything seemed to work normally. The array was in degraded mode and the copy was finished normally. Then I started to verify the copies of the data. At first things looked good, but then one of the bigger files had a few kilobytes of data totally differing in the middle of the file.

At this point I decided to power off the system, re-attach the drive I removed earlier and try the rebuilding. However, the nForce RAID BIOS reported error for the fourth drive and did not integrate it to the array. Booting Windows was also broken. After several attempts, somehow, I was able to boot and log into Windows at which point most things I tried to run either crashed, reported access violations or did not run at all.

For me, all this means that I will never trust the chipset semi-software RAID anymore. It could well be, that it is my motherboard that is at fault here, or the BIOS version... But still, having the RAID array is all about the ability to save important data on it without the need to worry if it will still be available after something happens to one of the drives. When the RAID array starts to corrupt data while in degraded mode then there really is no point in having a redundant RAID array.

If anyone has had a success using a chipset provided BIOS RAID array where a drive has failed in the middle of using it, please let me know the type and model of your motherboard.

Next I will try the software RAID offered by Linux. I'll try to do the same kind of trick - power down one of the disks while in use and see if and how much data I'll lose.