The [H]ard Forum Storage Showoff Thread - Post your 10TB+ systems

Just finished up my build.

30.3TB Advertised
24.56TB Usable

Case: Norco 4020
PSU: CORSAIR CMPSU-750TX
Motherboard: ASRock M3A780GXH/128M
CPU: AMD Athlon II X2 245 Regor
RAM: G.SKILL 4GB (2 x 2GB) 240-Pin DDR3 SDRAM DDR3 1600 (PC3 12800)
Controller Card: Highpoint 2340 (software raid6 via mdadm)
Hard Drives: 300gb Seagate (OS), 20x SAMSUNG Spinpoint F2EG HD154UI 1.5TB
Operating System: Ubuntu 9.04 (Jaunty Jackalope)

After much thought I settled on software raid via mdadm in linux. I picked up the highpoint used for $115 shipped. I really wanted to go hardware raid, but the price was too good to pass up. I had a lot of issues with the card and the samsung drives, but after a few emails to highpoint they sent me a brand new card unopened and it works wonderfully.

All in all I'm pretty satisfied with the build.





I will try to add some more pictures later.

@Sulfuric

Nice build. Is that a 20-drive RAID6? I'm thinking of something similar using Seagate Barracuda LPs, but I was gonna split the drives into two 10-drive RAID 6 arrays, just to be extra safe.

Hold on a sec...24TB usable? So that is two 10-drive RAID 6 arrays? Also, how come your first four devices report different drive sizes to the rest of the drives? Controller discrepancy? What are those first four attached to, the on-board SATA controller? Does that mean your boot drive is IDE?

hehe, all those guesses from a /dev listing...

Is that a 20-drive RAID6?

Yes. 16 on the 2340 and 4 on the motherboard.

Hold on a sec...24TB usable? So that is two 10-drive RAID 6 arrays?

Its 1 20 drive Raid6 1.5TB drives are roughly 1.36TB after formatting. 1.36 x 18 usable drives after subtracting the 2 parity leave roughly 24TB usable. Also I'm not sure why the size on the controller card is different. A-D are on the motherboard F-U are on the controller. E is the system drive which is also sata.

Hope that answers your questions.

Creation Time : Fri Oct 2 03:09:35 200
Raid Level : raid6
Array Size : 23440915456 (22355.00 GiB 24003.50 GB)
Used Dev Size : 1465057216 (1397.19 GiB 1500.22 GB)
Raid Devices : 20
Total Devices : 20
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Tue Oct 6 22:12:52 2009
State : clean, recovering
Active Devices : 20
Working Devices : 20
Failed Devices : 0
Spare Devices : 0

Status is still recovering from the final expansion. I had to do it parts due to the original card failure.

@Sulfuric
What drive format are you using? You've done my future setup, and I've been contemplating whether to use ext3 or xfs for the build.

What drive format are you using? You've done my future setup, and I've been contemplating whether to use ext3 or xfs for the build.

Originally I wanted to go with ZFS and raid-z, but decided against it once I couldn't build the whole array at once. I ended up using xfs and I think it was a better choice.

Also I'm pretty sure you mean ext4 since ext3 has a 16TB limit.

enjoy losing all your data as soon as one drive dies and another 2 die during rebuilds.
24 drives is a lot for one array.

Id post my server but its only up to 4TB. You people are freaks.

Originally Posted by GLSauron

enjoy losing all your data as soon as one drive dies and another 2 die during rebuilds.
24 drives is a lot for one array.

Id post my server but its only up to 4TB. You people are freaks.

lol, no. The likelihood of 3 drives falling over is kinda small, even with a 20-drive array.

@Colonel_Panic

I've used JFS to great effect with my workstation's 3.2TB array, it's a good solid filesystem format. I'm also currently using ext4 under Ubuntu on this work machine's external drive, and it's also been solid.

@Sulfuric

If you were gonna use ZFS, would you have deployed OpenSolaris or used it with FUSE under Linux?

Originally Posted by parityOCP

lol, no. The likelihood of 3 drives falling over is kinda small, even with a 20-drive array.

What if your PSU goes or even having a drive fall out of the array while it is rebuilding?

what if there is a crazy lightning storm and your whole house catches on fire....

damn you'll need RAID 666 to be safe from that!


stop being lame little jealous people and either make constructive comments about peoples builds or stay out...

RAID 6 is sweat, I am looking at my 4TB raid 5 array and wanting more..

congrates dude

Yeah seriously. I think raid6 is fine for 20 drives although any more and I would want something like raidz3 (tripple parity raid) which ZFS supports. you have a good PSU with (PFC?) support or whatever the PSU dieing should not take out your drives. I had had a raid6 rebuild complete just fine even when another drive was getting read-errors during the rebuild process which I replaced a day or two later after the rebuild.

Originally Posted by protias

What if your PSU goes or even having a drive fall out of the array while it is rebuilding?

lol, ffs If your PSU goes (and doesn't fry your drives) you replace it. Linux software RAID is rock bloody solid - it'll simply carry on from where it left off. If it does fry your drives, it's your own fault for buying an el cheapo PSU.

If one drive falls out of the array during a rebuild, it'll take parity data from the second parity drive.

If you're worried about a lightning strike, get an UPS or a surge protector. Or both.

Originally Posted by houkouonchi

Yeah seriously. I think raid6 is fine for 20 drives although any more and I would want something like raidz3 (tripple parity raid) which ZFS supports. you have a good PSU with (PFC?) support or whatever the PSU dieing should not take out your drives. I had had a raid6 rebuild complete just fine even when another drive was getting read-errors during the rebuild process which I replaced a day or two later after the rebuild.

No need for triple-parity RAID, my friend. Just split the array into more manageable sizes, like 30 drives would be split into 2 15-drive RAID 6 arrays. Lower your liability.


@sulfuric

I'd also suggest a weekly scrubbing of your array, just for surety.

Originally Posted by parityOCP

lol, ffs If your PSU goes (and doesn't fry your drives) you replace it. Linux software RAID is rock bloody solid - it'll simply carry on from where it left off. If it does fry your drives, it's your own fault for buying an el cheapo PSU.

Even quality PSUs can go up in smoke.

Originally Posted by protias

Even quality PSUs can go up in smoke.

Yep, but quality PSUs don't take your computer with them.

EDIT:

@sulfuric

Did you know you could have used an HX520? Or did you have that 750TX lying about anyway?

Originally Posted by parityOCP

Yep, but quality PSUs don't take your computer with them.

Actually, any PSU can take your computer with it. I just hope it never happens. Anyway, enough thread derailment. I can't wait to build a 40TB server sometime next year, which I will also be using RAID6.

If you were gonna use ZFS, would you have deployed OpenSolaris or used it with FUSE under Linux?

I had it working on a 4 drive array with no issues using fuse in linux. I haven't messed with solaris much so I wasn't too comfortable going that route.

Did you know you could have used an HX520? Or did you have that 750TX lying about anyway?

I've seen a lot of people using lower watt psus on 20 drive arrays, but I went with the corsair 750w to be on the safe side. Plus it was on sale.

To all the other comments if something happens it happens. I can't worry about all the what ifs.

My current rig:

AMD Phenom II X4 955 BE 3.20GHz
MSI 790FX-GD70 Motherboard
4 x 2GB Corsair Dominator PC3-12800 DDR3-1600 9-9-9-24
Zotac GeForce 8800 GT 512MB AMP!
Coolermaster Real Power Pro 1000W
Pioneer BDR-203BK BD-R Drive

Storage:

1 x 160GB WD Caviar Blue 7200 RPM 16MB
1 x 500GB Maxtor 7200 RPM 32MB
4 x 1.5TB WD Caviar GP 5400 RPM 32MB
4 x 1.5TB Samsung EcoGreen F2 5400 RPM 32MB
2 x 2TB WD Caviar Green 5400 RPM 32MB
1 x 2TB WD RE4-GP 5400 RPM 64MB

Originally Posted by Fahim1982

My current rig:

Pics, please?

Miguel

Pics or shens!

I'm currently debating wether to go for 16x or 12x 1.5TB Samsung F2 for the 10x500GB box upgrade...

this time i have to agree with Miguel,
PICS or it didnt happen!

blank stats of systems we could post in some diff thread :P

p.s.
when this thread was forming there was a though that every pic should be with paper sheet with your nickname on it ...

Wilba, start small ... just 10x

its a mess to move from those small drives
next time ill have to chose between moving to higher density or leaving old drives be, im chosing second
after cleaning 24x750, selling most of them and getting sick of having 30+ 1.5s tossed around my room, im not gona do this again! i think

Originally Posted by asgards

this time i have to agree with Miguel,
PICS or it didnt happen!

Why "this time"? Are my posts that unrelateable?

Also, I didn't say "it didn't happen". (Just messing with you, we're cool, OK?)

Originally Posted by asgards

p.s.
when this thread was forming there was a though that every pic should be with paper sheet with your nickname on it ...

lol

I seriously thought that rule only applied to the "for sale" section... It does seem a good idea, though, as one might be tempted to visit the nearest ISP and snap a couple of photos...

Now, who was it that was coming with me to the exclusive Google datacenter tour?

Originally Posted by asgards

its a mess to move from those small drives

From that description, I'm very glad I'm starting a NAS when the smallest drive I'll fit in it will be 1TB... NOT funny.

However, if you have enough ports available, given the ratio of newer-to-older drives (1:3 on a 500GB-to-1.5TB migration, and even better if you go with 2TB drives), it shouldn't have to be THAT much of a pain...

Rebuilding and expanding arrays and partitions, though, is bound to cause severe headache problems if you have only a few ports available... But even then, most of the time you could go the "change one, rebuild" route, and only expand the partition once at the end, right?

Oh, and btw, if you don't want to be tossing drives around, I'll be happy to pay P&P for them, and toss them around...

Cheers.

Miguel