raid5 issue (2 disks died, don't want to lose all data)

S

spotter

Weaksauce
Joined
Jul 29, 2007
Messages
95
so I had 2 disks that kicked the bucked close to each other in time.

of the 5 disks (0 through 4, each 400GB, for a total of ~1.5TB of disk space), disk 1 kicked first, as it was rebuilding disk 0 died.

with ddrescue, I was able to recover all but 1 512byte block of disk 1, and (so far) all but about 20kb (spread over 35 blocks) of disk 0.

This is a raid5 w/ a perc 5/i (aka LSI megaraid). The raid array was built with 128kb stripes.

Is there any way for me to reimage a replacement drive and force it in place, with the knowledge that I'll have about 4MB of corrupted information on disk (willing to have 4MB of corruption over losing 1.5TB). I calculate 4MB as 35 blocks of bad data * 128KB (if each block is on an individual stripe).

thanks!
 
as an update, I've managed with ddrescue to get disk 0 down to 1 512byte block.
 
Was this a system drive?

If not, can't you have your recovery software simply pull all intact files from the remaining good discs and place them on a temporary drive. Then you simply recreate the array and move the files from the temp drive back to the array, right?

If it's the system drive, you should have been imaging it (Acronis or Symantec) or be prepared to reinstall the OS in such a situation, IMHO.
 
not the system drive. I don't have an intact array, that's the point. the Q is can I force megaraid (LSI) to use the "bad" array with the knowledge that one stripe of data will be corrupted (due to one bad block that I haven't been able to recover yet).
 
Seems unlikely, unless some hardcore data recovery situations are employed; that's like trying to recover a RAID 0 volume...the striping across the array is on such a low-level that many files will have corruption to some point. :(
 
I've been able to recover all the data beside 1 512byte block, I would assume corruption should be minimal then.
 
I think this falls under trying to take the easy way out and probably paying for it later. Even if you can do what you're proposing, if you're smart, you'll back up the recoverable data before you try it, just in case it fails today, tomorrow, etc.

So if you have to make a backup anyway, why not just remake the array (instananeous but deletes all contents) and then restore from your backup?
 
I did. read my original post. I have a ddrescue made image of both drives where it so far has only bee unable to read 1 512byte block on each.

I wont attempt to rebuild the disk before I dd that image to a new replacement disk.
 
But what you need to understand and you are missing this, is even if you got all but 1 block on the two drives, you still NEED data off the third drive to complete the files. The files are corrupted without it.
 
I have 5 drives. 3 are completely fine.

2 have problems.

RAID-5 needs 4 disks. With 1 disk with one bad block that is reimaged to a good disk, I'm looking at worst at N*stripe width of corrupted data (assuming each disk eventually gets rebuilt with corrupted data), though more likely just stripe width worth of data (corresponding blocks on disk 5 that gets rebuilt) . In my case that's 128K. Considering I have 5 400GB drives in this array, I'm thinking that's a worthwhile trade off.

what am I thinking wrong?
 
I'd argue I can be even smarter.

I have 2 disks that are bad in different areas.

I could do this

1) get replacement disk 0, image it w/ image made from original disk 0 with its 512byte bad block
2) place replacement disk 0 (imaged) and replacement disk 1 (empty) in raid array and force a rebuild
3) disk 1 is now mostly correct except for one stripe
4) if disk 1 image is accurate (probably can take a few MB chunk image to verify), I can overwrite that bad area with the part from the image
5) remvoe disk 0 from array, add back and rebuild, it should rebuild its bad block correctly as well.

actually doesn't seem that hard. wish raid manfuacturers supplied utilities that did this.
 
Ok well you need data off at least 4 of the 5 drives is what I meant.

But the thing is, unless you can get your raid controller to accept the new drive after you transfer most of the data I still don't know/think it will work right. Besides trying to mix good data from the two bad drives to make one whole set is going to be near impossible. Do you know what blocks are which parity data? I doubt it.

You are going to have to try to reintegrate it all so nothing is corrupt.
 
Redo the array from scratch and transfer the backup-ed files back to the array. What you didn't make a backup? Oh you tought Raid-5 was sure way to keep your files safe?

Oh well...

Good luck with your data but I am 95% sure even if you try your best your attempt to rebuild your data is lost.
 
Cyant said:
Redo the array from scratch and transfer the backup-ed files back to the array. What you didn't make a backup? Oh you tought Raid-5 was sure way to keep your files safe?

Oh well...

Good luck with your data but I am 95% sure even if you try your best your attempt to rebuild your data is lost.
Click to expand...

I wasn't going to be as blunt but yea. This...
 
anything I lost that wasn't backed up, isn't "needed" (for instance, probably about 500GB of media center recorded tv shows). Its stuff that while not simply replacable isn't a great loss. Other things include stuff that I can get again, but will take time (it served as my local software archive backup, iso images of all the software I have, while I can recreate the iso images from originals, that will take A LOT of manual time).

Though this makes me much more keen on mdadm in linux now, I know I could have recovered from this with it.
 
You could try RAID Reconstructor; you'd need a 2tb disk or second array to hold the data that it pulls off. I believe their products have a free trial that will show you what it would be able to grab from the drives.

I backed up my array onto 2tb drives because while 99% of the material is just DVD rips, ripping 400 discs back onto the system and cutting the credits and such out would take ~200 hours (damned TV series...) I value my time at much more than the <$1/hour that would work out to after getting a 2tb drive for $170.
 
From what I remember of LSI megaraid, there is a function to force an array online. You might have to do it from the commandline utility. The LSI megaraid tends to play it conservative and marks disks as foreign if they've been moved or messed around with, so it probably won't recognize your cloned drive as part of the original raid set. I don't know if there's a way to force it in. I wouldn't be too optimistic.

The other way is to delete the raidset and recreate the raidset *without* initializing. Then forcibly remove or mark bad the dead disk. This is basically trying to fix the metadata saying the raid is dead back to the state where only 1 disk is dead so you're still OK but in critical condition. If you do this, you have to use all the exact same parameters (block size, disk order, etc.) I don't remember if LSI megaraid supports no initialization on creating raid - this function is supported on Adaptec, Areca, Highpoint, so I assume it's available on LSI.

The other way is to just connect the drives to standard SATA ports and use disk recovery software (I use R-Studio personally) to try and reassemble the data.
 
You guys that don't back up yer shit kill me. n00b mistakes 101.

Unbelievable this still goes on ...
 
Sigh.

I'm the one who beats people up over crap. Gods. Now step aside.

Image your new 400GB disk with the raw blocks off the failed disk. Repeat for disk 2. Reboot and immediately go into MegaRAID BIOS. You should show the array as OFFLINE-DEGRADED. If you do NOT see the array as OFFLINE-DEGRADED, try to rescan disks first. Add another 400GB and immediately configure it as a hotspare - the lack of hotspare is what got you here.

Remember that MegaRAID stores configuration data on disk blocks; we don't know if we got it though. Presuming it got the configuration data, you'll be OFFLINE-DEGRADED.
Do NOT ONLINE the array. You should show two faulted disks. First, online the individual disks. Do NOT say they have been replaced, and only online the disks you replaced.
If you're still not OFFLINE versus OFFLINE-DEGRADED, VERIFY the array. This should pull in your replacement disks, and start verifying the configuration blocks first. After the configuration blocks, it should start doing full parity checks. It may ask you if the disks have been replaced - DO NOT SAY YOU REPLACED THE DISKS. That will move it to rebuild. We don't want it to rebuild, we want it to do parity checks with repairs first, if possible.
Worst, worst case scenario is that config data blocks are missing on one of the disks. With a bit of luck, you should be able to recover from that by admitting the disk is replaced and rebuilding the array.
EDIT: I should explain, rebuild as in "total array rebuild" which can potentially take days. Individual disk rebuild and array rebuild are very different; we want to only do individual disks.

That should pretty much cover all the steps you need to do.
 
Last edited:
RAID is not backup. period. You might try raid reconstructor as suggested and / or start redownloading all that.
 
after a whole lot of effort, I was finally able to get one disk fully read with ddrescue! so now I have hopefully 4 good disks, will ddrescue the other 3 disks to ensure.

now, hopefully I can force the array to be built. otherwise I'll have to look into the third party utility
 
If you've got four complete disks out of a 5 disk array, all you need to do is add a hotspare disk at this point, and boot (but immediately go to MegaRAID BIOS.) You REALLY need a hotspare. It should work it's magic automatically and rebuild the failed disk.
 
Last edited:
I have forced online disks on raid controllers before (and ended up with a small amount of corruption)

But the RAID controller had this option in it's BIOS.. (and warned all kinds of doom when I did it)

So there might be a way to tell it to go ahead and make one of the dead drives (preferably the one that died 2nd) live again.. But it is dependent on the card.
 
Lazn_Work said:
I have forced online disks on raid controllers before (and ended up with a small amount of corruption)

But the RAID controller had this option in it's BIOS.. (and warned all kinds of doom when I did it)

So there might be a way to tell it to go ahead and make one of the dead drives (preferably the one that died 2nd) live again.. But it is dependent on the card.
Click to expand...

There's no need to here. No need at all. In fact, that's the absolute last damn thing he should be doing. If you don't know MegaRAID, please don't advise people to wreck their arrays.

All he needs to do is get the restored disk in a slot, the failed disk, and add a hotspare. The MegaRAID will take care of the repairs itself. Forcing an OFFLINE-DEGRADED array online will trigger an unnecessary multi-day rebuild.
 
I'm waiting on seagate to send me the replacement disks. I'm actually going to image the other 3 disks to make sure they are ok (And get some SMART data on them). Though after I recover the array, I might just get rid of them as a raid array and use the card for jbod usage, unsure I really need this 5*400 raid array anymore, have a mdadm 5*1500 (deadly seagates, but they seem to be treating me ok) in my linux box.
 
Hardware RAID is ALWAYS better. You should move the 1.5's to the MegaRAID, and add a 6th as a hotspare.

I cannot emphasize this enough. RAID5 and RAID6 REQUIRE a hot-spare drive!! That is why you are playing this game and not running with a "DEGRADED-ONLINE" array. If you had had a hot-spare, the array would have handled the first failure gracefully, and the second would have gone to limp-in mode.
Remember that the controller pre-fails drives, which is why your recovery rate is 99.99% and not 0%. It will fail the drive before it is unreadable, to ensure that you do not lose data, by migrating data to the hot-spare. Limp-in gives you enough time to replace one of the failed drives, which is done by normal replacement procedure. Then you let the second drive fail to the hot-spare, replace it online, and you're back to normal.
 
in this case, I don't see how a hot spare wouldn't have helped.

I have Disks 1-5

Timeline

T1: Disk 2 dropped.
T2: Rebuild started.
T3: Disk 1 died w/ major problems

even with a hot spare, if it the rebuild would have been to the hot spare, would have still had the same problem.
 
AreEss said:
There's no need to here. No need at all. In fact, that's the absolute last damn thing he should be doing. If you don't know MegaRAID, please don't advise people to wreck their arrays.

All he needs to do is get the restored disk in a slot, the failed disk, and add a hotspare. The MegaRAID will take care of the repairs itself. Forcing an OFFLINE-DEGRADED array online will trigger an unnecessary multi-day rebuild.
Click to expand...

So RAID 5 array with two dead drives can be rebuilt to the hotspare with a MegaRAID?

In my experience once two drives of a RAID 5 set have been marked failed by a controller you either have to rebuild from scratch, or force one of the failed drives live again. (with all the risks involved with this bad idea)
 
Lazn_Work said:
So RAID 5 array with two dead drives can be rebuilt to the hotspare with a MegaRAID?

In my experience once two drives of a RAID 5 set have been marked failed by a controller you either have to rebuild from scratch, or force one of the failed drives live again. (with all the risks involved with this bad idea)
Click to expand...

Yeah. No.
Controllers pre-fail drives. Without anything short of catastrophic failure (the only kind you can have in SSD,) the drive is failed before data is inaccessible. Again, look at the data recovery rate spotter got - 100% on disk 1, 99% on disk 2. That means the disks did NOT have catastrophic failure.

What makes the MegaRAID different is how it handles a two disk failure in an array with one hot spare. Upon the first disk pre-fail, it does a normal rebuild, just like everyone else. On the second disk prefail, it does not take the prefailed disk offline. Instead, it throws up lots of alarms, but continues to operate, because the disk is not truly failed yet.
Obviously, this doesn't happen in true catastrophics e.g. PCB failures, refusal to spin up, etcetera, only in mechanical pre-failures. (Bad block triggered.) There's also a ceiling to how long it will hold the disk in that state. So you need to replace the first failed disk as quickly as possible. IIRC, you get 2-3 bad blocks past the first on a pre-fail before it drops.

This is also why SSD RAID without hotspare is suicide; SSDs can only have catastrophic failure. When an SSD goes, it's inaccessible, period. You have lost access to the data completely. They also can't be pre-failed. The worst of all worlds, basically.
 
so, to resurrect my thread. as have 4 disks, the raid is rebuildable. However, seagate replaced the 400GB drives w/ 500GB drives.

megaraid didn't seem to like it.

as I had 3.5 TBs of room unallocated on my linux raid server, I create an lvm partition, wrote the images to it (w/ the help of an esata dock), and r-studio is rebuilding it as I speak. Going to take a long time!

r-studio is very powerful, and the fact that it lets you "preview" the file system and recover individual files, is very useful for recovering data, but more importantly for figuring out what order the disks go in and verifying that its right.
 
You must log in or register to reply here.
Back
Top