best zfs setup for 10 SAS drives

D

deaddawg

Weaksauce
Joined
Apr 17, 2006
Messages
111
I am setting up a small NFS-based SAN for VMware. I am using a PE 2900 III with 10 x 300GB 15k SAS drives. It will be running nexentastor, based on opensolaris.

My first thoughts were to run 5 x mirrors striped. (300x5 = 1.5tb usable)

Then I didn't like the thought of no hotspare.

So now I am thinking 3 x 3drive raidz, + 1 hotspare. (600x3 = 1.8tb usable)


Any thoughts on this?
 
6/i, I have each drive in its own Raid 0, so it is all being handling in software.
 
if you want to maximize space and still have redundancy you could go 9 in raidz2 and one hotspare.

Keep in mind that 9 drives in one raid configuration is the maximum recommended by the zfs admin guide. If you go lower you may see somewhat better performance. Depending on your usage this may or may not be negligible.

I'm using sets of 10 in raidz2 which is over the recommended number but it works just fine for my needs, which is streaming media over the network.

You could also do 2 sets of 4 in raidz and have 2 hot spares, or 2x5 in raidz2 which is almost the same thing. This would give you the same amount of storage as your 3x3 idea.

If you can buy one more drive you could have 2x5 + 1, which would be a good setup.

10 is a hard number to work with. It doesn't divide out nicely. One of the small complaints I have about the norco case.
 
2x RAID1 boot
1x Hotspare
7x RAID6

Gives you the same space and mirrored pairs and will be faster.
 
I am booted off embedded USB flash.

7 drives in raidz2 is better performance? My understanding of raidz is the IO preformance of a single pool is not that great unless you mirror, hence my idea of 3 x 3 raidz
 
So I tested some file copies from a 3-drive raid0 to a 2 x 3raidz (6 drives) ... i think the speed is decently fast.

sasisfast.jpg


So I am thinking the 3 x 3raidz is the way to go, should be 700+ MB/s write, and even higher read. I will need to get some 10gigE nic's for this thing. Although, with a lot of activity (read, 20 - 50 VM's), that number could go down pretty significantly.
 
It depends on the drives and RAM in your system. If yor have an anemic 2GB main board, go with 5 mirror groups. If you have quad-core and 8GB of RAM then expect to get about 50-60% (worst case) performance (writes) from three 3-disk raidz groups vs. five 2-disk mirrors. With reads, you'll not see a real difference at that scale... Under VMware, NFS performance could be dominated by synch writes against raidz.

Beware deduplication and raidz groups - go mirror if that is a factor unless you have flash for DDT space... This is especially true in low ram situations. With good RAM and SSD offload, single RAIDZ2 could even work for some workloads. I'd go with the mirror groups for such a small array...
 
Last edited:
deaddawg said:
So I tested some file copies from a 3-drive raid0 to a 2 x 3raidz (6 drives) ... i think the speed is decently fast.

sasisfast.jpg


So I am thinking the 3 x 3raidz is the way to go, should be 700+ MB/s write, and even higher read. I will need to get some 10gigE nic's for this thing. Although, with a lot of activity (read, 20 - 50 VM's), that number could go down pretty significantly.
Click to expand...
Don't confuse write bandwidth to disk as realizable performance. ZFS queues write and bulks them out to disk every 20-30 seconds. This makes random IO behave like sequential IO. In the case of mirror, to-disk bandwidth on write is twice the effective rate. For raidz, it's harder to calculate, bit those peaks will look like full stripe writes to N disks, so the periodic writes to disk from ZFS will look pretty huge.

Here's a test: run IOmeter against your array and watch the meters. The disk activity should look nothing like the iometer traffic on the wire. That's how different ZFS disk access works from direct disk access. If you have enough ram, you'll see very little disk activity at all - everything will come from ARC...
 
You must log in or register to reply here.
Back
Top