ZFSguru NAS fileserver project

S

sub.mesa

2[H]4U
Joined
Feb 16, 2010
Messages
2,508
ZFSguru NAS fileserver project

Website Links

Quick Download links:
Warning: the ZFS v28 system image, including the one included on the LiveCD, is considered very experimental, and you should not use this on your existing valuable data unless you got a very good backup! This release is for testing only! For real data, use the stable image instead.


 
Last edited:
Frequently Asked Questions

What is ZFSguru?
ZFSguru is the name of the project that i wrote. It is based on FreeBSD, a highly advanced operating system that includes the ZFS filesystem. The intend for ZFSguru is to be a NAS-capable operating system with web-interface, suitable for people new to the world of UNIX and open source in general.

The primary goal of my distribution is to make the step towards ZFS more easy and less scary. Thus, i want ZFS to be accessible to a broader range of users, allowing them to benefit from this great technology. The aim is to provide a very user friendly web-interface that should be usable by anyone interested in a NAS.

ZFSguru would feel very much like a clone of FreeNAS; but they are completely separate projects with different intents.


What is ZFS?
Sun Microsystems developed the Zettabyte FileSystem, which first showed up in Sun´s Solaris operating system in mid-2005. ZFS is different from other filesystems like FAT, NTFS, Ext3/4, UFS, JFS and XFS because ZFS is both a filesystem and RAID-engine in one package. This yields emerging properties; things that aren't possible if the two systems were separate.



Why is it so cool?
ZFS is the most advanced single-system filesystem available, with unique properties:
  • ZFS is both Filesystem and RAID-engine.
  • ZFS protects your data from corruption using checksums.
  • ZFS is maintenance-free and requires no filesystem check; it automatically fixes any problems or corruption.
  • ZFS can also act as (incremental) backup using snapshots, much like Windows Restore Points.
  • ZFS is versatile; it allows you to grow your filesystem by adding more disks.

Some more technical features:
  • It allows you to make any combination of RAID0 (striping), RAID1 (mirroring), RAID5 (single parity) and RAID6 (double parity)
  • Because on its Copy-on-Write design, its very resilient against crashes and other problems, and won't need a filesystem check ever!
  • Because ZFS is both RAID and Filesystem, it can use dynamic stripesizes to adapt to the I/O workload.
  • Aggressive caching and buffering make ZFS consume lots of RAM, but benefits I/O performance.
  • Transparent compression can significantly reduce size in some cases.
  • ZFS can use SSDs as cache device, increasing the performance of the entire array and adapts to your usage pattern.

In essence, ZFS offers excellent protection against data-loss of any kind; other than disasters which affect the whole system physically like flooding/fire. Its also very flexible, allowing you to change almost anything.



What do i need to use ZFS?
Sadly, ZFS doesn't run on Windows, and Apple withdrawn their plans to include it into MacOS. Windows falls short in offering advanced storage technology - all the hot stuff is either on Linux or UNIXes like FreeBSD. So for the purpose of building our fileserver, we want something more modern. To use ZFS, you need to have:

  • 64-bit dual-core/multi-core processor (AMD64)
  • Lots of RAM (2GB minimum; 4GB+ recommended)
  • Modern motherboard with onboard SATA + gigabit ethernet and PCI-express
  • Operating system that supports ZFS. Currently: OpenSolaris, FreeBSD and FreeNAS


Tell me more about FreeNAS
FreeNAS is an open source OS based on FreeBSD, but adapted to function as an easy NAS operating system that is configurable over the network. FreeNAS installs very easy and should be easy to configure, but it has limited features. It also offers ZFS, but a slightly older version; v6 instead of v13/14 which FreeBSD/OpenSolaris use. Generally your best option if you want something setup quick; but for large filesystems or more complicated setups, a full-fledged Operating System is required.



So what about FreeBSD?
FreeBSD version 7.0 - 7.2 supports ZFS version 6. This is what FreeNAS uses.
FreeBSD version 8.0 supports ZFS version 13.
FreeBSD version 8.1 supports ZFS version 14.
FreeBSD version 8.2 supports ZFS version 15.
FreeBSD version 9.0 supports ZFS version 28 (with experimental ZFS patchset).

FreeBSD is a UNIX operating system, and may be difficult to master. Especially the installation part is a bit tricky, as it uses an outdated text-based installer.
Since my LiveCD can do all installation for you, you don't really need to bother with the details here.



Wait, hold on! So you're saying this FreeBSD is command line stuff?
Well yes, especially the installation and setup will be hard and you should follow a howto document or be guided by someone familiar with FreeBSD. After that, you can use a windows terminal like PuTTY to connect to your FreeBSD server; so you can configure the server from your Windows pc; but still have a command-line.



So what do i need to do in this command-line; i prefer something graphical!
Yes, well, it can be useful to work with commands so you know exactly how you got there. Meaning, if you write down the commands you used to create your filesystem, you know how to do that a second time. So while commands may be scary at first, it will be more logical on the long run.



So how do these commands look like, can you give me an example?
Certainly!

Code: 
# creates a RAID-Z (RAID5) array called "tank" from disks 1, 2 and 3
zpool create tank raidz label/disk1 label/disk2 label/disk3
# create filesystems
zfs create tank/pictures
zfs create tank/documents
# enable compression for our documents directory only
zfs set compression=gzip tank/documents
# also store each file in the documents directory on all 3 three disks; for maximum safety
zfs set copies=3 tank/documents
# snapshot the documents directory, creating a "restore point"
zfs snapshot tank/documents@2010-03-04
# made a mistake? simply roll back to the last snapshot
zfs rollback tank/documents@2010-03-04
# get status from your array
zpool status tank



But ZFS is still Software RAID right? Should i get a real Hardware RAID card instead?
No, certainly not! Doing so would mean you lose the portion of ZFS where it can heal itself. You should let ZFS do the RAID, and just use the onboard SATA ports. If those are not enough, expand with PCI-express controllers which can act as normal SATA controller to ZFS. Never use PCI for anything! Only PCI-express.



But still isn't software RAID slower than hardware RAID?
On paper its not, on paper software RAID is superior to hardware RAID. Hardware RAID has increased latency that is unavoidable - software RAID has access to very fast hardware already and is easier to implement an advanced RAID controller in software. Note that in the case of Hardware RAID, its still the firmware software that actually implements the RAID; and its implementation may be simple/unsophisticated when compared to ZFS.

As for speed, ZFS is speedy enough; but never without making sure the data is safe. Some unprotected filesystems may be a fraction faster, but ZFS adds a lot of reliability without sacrificing on speed all that much.



So exactly how do i setup this ZFS?
I will be explaining this in detail later, but generally:
  • First, install your operating system, i'm assuming FreeBSD here. The OS should be on a different system drive, which can be a USB pendrive or compactflash card, a parallel ATA disk or just a SATA drive. It's best if the system drive is completely separate from the disks that will be used by ZFS.
  • Then, connect your HDDs, let FreeBSD find them. Label the drives so each drive has a name like label/disk1 or label/disk2 etc. This avoids confusion, and makes sure that it will be found and identified correctly, regardless of how it was connected.
  • ZFS is already pre-installed in FreeBSD; so no need to install FreeBSD.
  • Create ZFS RAID pool using the "zpool create" command
  • Create ZFS filesystems
  • Set various ZFS options
  • Set permissions
  • Setup Samba/NFS so you can use the filesystem from your networked computers



Can i expand my ZFS RAID array?
Yes, but some restrictions apply.

What you cannot do, is expand an existing RAID-Z (RAID5) or RAID-Z2 (RAID6) array with one or more disks.
But, you can add new disks or RAIDs to an existing pool. So if you have a 4-disk RAID-Z, you can add another 4-disk RAID-Z so you have 8 disks. The second array would share free space with the first; in essence it would be a RAID0 of two RAID5 arrays. ZFS can expand this way.

What you can do, is expanding mirrors and RAID0's. In the example above that's what actually happened: a new array is RAID0-ed with the existing array. New created files will be written to both devices, for additional speed. Setting copies=2 would make files in that directory be stored on both RAID arrays; for extra redundancy.



What if a disk fails?
Then you identify which disk is causing problems with the zpool status command. Then replace the disk for a new one.
As long as the failures do not exceed the redundancy offered by ZFS, everything will continue to work, including write access.



Do i need to use TLER or RAID edition harddrives?
No and if you use TLER you should disable it when using ZFS. TLER is only useful for mission-critical servers who cannot afford to be frozen for 10-60 seconds, and to cope with bad quality RAID controller that panic when a drive is not responding for multiple seconds because its performing recovery on some sector. Do not use TLER with ZFS!

Instead, allow the drive to recover its errors. ZFS will wait, the wait time can be configured. You won't have broken RAID arrays, which is common with Windows-based FakeRAID arrays.



How future-proof is ZFS?
As Sun got acquired by Oracle, the future of ZFS may be uncertain. However, it is open source code and still in development. Several non-Sun operating systems now have ZFS integrated, and projects like kFreeBSD may port ZFS to Linux-distributions like Ubuntu.

But, ZFS is not very portable; only few systems can read it.
However, you can connect the disks to windows, and use VirtualBox/VMWare to let FreeBSD inside a VM access the RAID, and export over the network. That works, but Windows should not touch your disks in any way. Simply selecting to 'initialize' the disks would lead in data-loss and perhaps total corruption as key sectors get overwritten. ZFS is resilient, but such tampering may exceed the metadata redundancy of 3 copies per metadata block.



How do i maintain ZFS? Defragment etc?
You don't. You don't need to. :)
The only thing you need to do, is make sure you get an email/update when one of your drive fails or has corruption, so you are aware and can intervene at earliest opportunity.



Can ZFS replace a backup?
RAID alone can never replace a backup; RAID doesn't protect against accidental file deletion, filesystem corruption or a virus that wipes the drive. But ZFS can protect against that, using snapshots you can make incremental backups so you can go back in time and get each day's version of the filesystem.

A nightly snapshot is very useful and snapshots do not use additional storage space, unless you modify your files since the latest snapshot.

So yes, ZFS can replace a backup. But note that ZFS is advanced software with many lines of code, and any bug may still threaten your data. So for data you can't afford to lose, a real backup at another physical location is still highly recommended.



What about the ZFS support in Linux?
Linux has the GPL-license, and it is not compatible with the CDDL-license that ZFS uses. That means ZFS can't be directly integrated into the Linux Kernel; the best implementation possible. Instead, the FUSE project aims at implementing ZFS in userspace; something that has huge drawbacks and is generally not suitable for most users.

Another effort to implement a kernel-level implementation of ZFS as a CDDL-module, linked to the Linux kernel has a working prototype but appears unmaintained. If you want ZFS, you need either FreeBSD, FreeNAS or OpenSolaris.



How fast does ZFS go?
Real performance is too complicated to be reduced to simple numbers. The buffering and caching of ZFS also make benchmarking it quite hard. But its very fast in real-world scenario's and its speed should never be any issue. As long as you do not use PCI in your system!



What are ZFS cache devices?
ZFS is able to use SSDs in a special configuration, where it acts as cache for the HDDs. This is like having more RAM as filecache, but then SSDs are much bigger than your RAM can hold. Whenever you read something from the RAID array that is cached, the SSD will perform the read request instead; with very low access times. ZFS tracks which data is most accessed and puts those on the SSD. This means it automatically adapts to your usage pattern. You

can have an array of many TeraBytes and have a small SSD that serves the files you access every day and make a real improvement to the performance of the array.



Can i use hot-spares with ZFS?
Yes, you can add one or more hot-spare disks as 'spare' device. These will be available to any array that is degraded; so you can share one hot-spare disk across multiple RAID-Z arrays, for example.



How much RAM can ZFS use?
A lot, largest i've seen was 6.3GB. The RAM space depends on settings, number of disks, stripesize and most of all - the workload. The faster you can make ZFS work, the more memory it will consume. It will be well spent though. For low-memory systems, you can limit the memory ZFS uses; but this also limits performance. Generally, you should not use ZFS with less than 2GB without conservative tuning disabling alot of fancy ZFS features.



Why can't i use ZFS on 32-bit?
You can, but memory constraints mean ZFS is limited to 512MiB memory; where only minimum settings would work. In those conditions, heavy workload would cause ZFS to panic and crash. It wouldn't really be bad; just reboot and it works again without you needing to do anything. But that's not the way you should use ZFS. ZFS is an 128-bit filesystem and feels at home with an 64-bit CPU and 64-bit operating system.



How do i access ZFS from my Windows PC?
For that you need a network protocol. Windows filesharing is common, which uses CIFS/SMB protocol. Samba can be used to export your ZFS filesystem to your windows PC's; you would have a drive letter like X:\ which contains your ZFS filesystem. Other protocols are recommended though, especially NFS and iSCSI work very well. Unfortunately they are not natively supported by Windows. While Samba works, it may limit throughput speed. Its a shame if your ZFS array does 400MB/s internally but over the network you're stuck at 40MB/s. That's an a common issue with Samba.



How do you access your ZFS filesystem?
I use ZFS as mass-storage disk and access it using NFS or Network File System - the preferred way to share files on Linux and alike.
I also use ZFS to store my system volumes of my five Ubuntu workstations. So my desktop PCs don't have internal drives - everything is on the network, on ZFS. This makes using backups much easier as i can perform snapshots on my system disks. The system drives are accessed using iSCSI, which also works on ZFS using FreeBSD. Booting also happens over the network, using "PXE" and specifically, "pxelinux".
The upside is i have a lot of control over my data - especially because i can make incremental snapshots really easy. The downside is that performance is capped by the network bandwidth, as i'm still using 1Gbps ethernet. 10Gbps NICs are available but at supreme cost - more than $500 per NIC; and switches are even more exotic. I suspect prices will drop significantly in 2011; getting 10Gigabit to enthusiasts as well as the server market.



Please use this thread to discuss setting up ZFS and talk about its features. Feel free to ask questions.

Version history:
1.0 - initial version
1.1 - added Hot-Spare section, added section about how i access my ZFS
 
Last edited:
I have a few questions before I move into building a computer to run ZFSGuru for my future plans.


1. More cores or Higher clocks? (2.0 Ghz 8-Core, or 3Ghz 4-Core)
2. Do you use a single SSD for cache, or can you RAID0 a few for increased cache performance?
3. Because of the cache, would this be a viable solution for VMs? (booting from ZFS,over the network, with like ESX? or Hyper-V?)


So basically can I use this like a SAN?

I want to run either Hyper-V or ESX and need a storage solution so I don't have to store the VMs on the servers themselves. This way if a system goes down, a secondary system can just boot the VM back up with little to no downtime.


Also because of this, is it possible to link NICs together in FreeBSD to maximize my network traffic in this situation?
 
Great writeup sub.mesa

Have a few questions

1. Does ZFSGuru allow the user to utilize the block level Deduplication or is that a feature of Solaris ZFS distros only
2. Does ZFSGuru allow the user to configure ARC, L2ARC, and ZIL or does this still have to be done on the command line?

BTW NFS and iSCSI are natively supported on Windows Vista and 7....not sure about XP.
 
Greatone123 said:
1. More cores or Higher clocks? (2.0 Ghz 8-Core, or 3Ghz 4-Core)
Click to expand...
More cores of lower frequency = more SMP performance, better performance-per-watt and better idle power. In short, the pro solution would be 8x2GHz or something like that. ZFS is fully threaded; so 64 cores would mean 64 in use when doing CPU intensive stuff. ZFS without the advanced functions isn't THAT CPU-intensive, though.

Enough memory (12/16GB+) would be more important than preferring a quad/octo core over dualcore. The RAM should be the most expensive of your build, aside from the disks.

2. Do you use a single SSD for cache, or can you RAID0 a few for increased cache performance?
Click to expand...
You can add as many as you want, which will be striped for extra performance. If they fail or become corrupted, ZFS detects that and will read from the primary storage instead; so you don't need to worry about L2ARC corrupting or otherwise be dangerous; it's not!

SLOG/ZIL device is a different story; you should wait with that until ZFS pool version 19 and third generation SSDs with supercapacitor (SF2000, Intel G3, Micron C400).
3. Because of the cache, would this be a viable solution for VMs? (booting from ZFS,over the network, with like ESX? or Hyper-V?)
Click to expand...
L2ARC and SLOG would be crucial to high IOps performance. But how exactly are you planning to do this?

Would ZFSguru itself be virtualized? I recommend against that, if you want a high-performance solution. There may also be issues with virtualization and ZFS (write flushing).

However, if you just want ZFSguru to provide the VMs via like NFS (NAS) or iSCSI (SAN) so you have the 'system disks' of your other machines done by the ZFS server instead, then yes this is exactly what i do.
I have my system disks on SSD primary storage, without L2ARC. You can use your SSDs for both primary storage, L2ARC and/or SLOG.

Speed increases nearly linearly when doubling the number of SSDs, which will become striped. But perhaps you should explain a bit in detail what you want. Keep in mind i'm not familiar with ESX or Hyper-V; though i would not recommend virtualizing your ZFS server; instead use another box for virtualization and let it pick up the storage over gigabit network, managed by ZFS via either NFS/CIFS/iSCSI.

Also because of this, is it possible to link NICs together in FreeBSD to maximize my network traffic in this situation?
Click to expand...
It supports lagg link aggregation/failover driver, supporting either manual or LACP or FEC link aggregation, depending on what type of switch you employ. Configuring lagg-devices is not yet implemented in the Web-GUI, though.
 
Last edited:
nitrobass24 said:
1. Does ZFSGuru allow the user to utilize the block level Deduplication or is that a feature of Solaris ZFS distros only
Click to expand...
De-duplication is ZFS pool version 21, so this is supported by the latest experimental version, which has ZFS pool version 28. Keep in mind, that even Solaris thinks this version is not stable yet; their commercial clients don't run De-duplication yet, only when it is deemed production ready.

2. Does ZFSGuru allow the user to configure ARC, L2ARC, and ZIL or does this still have to be done on the command line?
Click to expand...
Configure ARC? You mean memory tuning? Is done on System->Tuning page. Configuring L2ARC or SLOG/ZIL devices is not yet supported via the GUI; should appear in 0.1.8!

BTW NFS and iSCSI are natively supported on Windows Vista and 7....not sure about XP.
Click to expand...
I thought only certain versions of Windows 7 supported NFS? Not sure about iSCSI either. Would Vista Home Basic support that?
 
Sub.mesa definately not virtualizing zfsguru, planned on the 2nd option just booting from zfs like you do. Physical zfsguru, then 3 hyper-v servers in a cluster.

8-core 16Gb ram for zfs, with ssds for l2arc.

1 nic for management/webgui and 4 nics for data if possible atleast.
 
I finally got around to testing this out in Virtual Box last night. I had a strange issue where it wouldn't boot using 1GB or RAM, but bumping it up to 1.25GB solved it for some reason. I didn't really look into why it was doing this yet, as I was happy enough with the solution for now :p I'll look into what was happening more though as time permits.

Once I got it booted, all went well however. I found it to be quite intuitive, and after just looking through all the pages that the interface included, I felt I had a strong understanding of how most things were done. I created three virtual disks, and set them up in a RAID5 configuration, and added them all to a pool. All I got around to last night unfortunately.

I think this project is fantastic. As I said, I feel your interface is very intuitive, and very easy to use. Keep up the great work.
 
Greatone123 said:
Sub.mesa definately not virtualizing zfsguru, planned on the 2nd option just booting from zfs like you do. Physical zfsguru, then 3 hyper-v servers in a cluster.
Click to expand...
Alright so another system than ZFSguru will run the ESXi/Hyper-V solution, right? That would be good.

ZFS should be able to provide very high IOps to your VMs, using a shared set of SSDs to accelerate your normal (5400rpm?) HDDs. You can use 5400rpm + high-performance SSD and get gigantic boosts to performance. This is a killer feature of ZFS; though in fairness i have to add that Btrfs project on Linux also has an option, but i'm not sure how it compared to ZFS. ZFS also includes SLOG feature, which would accelerate random writes and also sequential writes, but not reads.

8-core 16Gb ram for zfs, with ssds for l2arc.
Click to expand...
That's an awesome configuration. Though as stated before, i would wait with the SSDs if possible, so you can buy new third generation SSD with supercapacitor so you can use it for SLOG feature as well. Any existing SSD you have should probably only be used for L2ARC, but would still be extremely useful. L2ARC is mainly multiqueue random read; so JMicron/Toshiba/Indilinx wouldn't do so well here, while Sandforce, Intel and Marvell/Crucial/Micron would be able to provide the highest performance here. Intel SSDs go as far as 200MB/s multi-queue random reads (4KiB aligned). If you RAID0 them you can go very high; i've peaked 1234MB/s of random reads with 5 Intel SSDs in RAID0 (not ZFS config tho; raw I/O).

1 nic for management/webgui and 4 nics for data if possible atleast.
Click to expand...
Have a managed or 'smart' switch supporting LACP? L4 hashing?
 
Jesse B said:
I finally got around to testing this out in Virtual Box last night. I had a strange issue where it wouldn't boot using 1GB or RAM, but bumping it up to 1.25GB solved it for some reason.
Click to expand...
Yes, the memory filesystem has grown with the new packages; it now consumes more than 620MiB which is quite alot. Once you perform Root-on-ZFS installation it won't be an issue though, and a little more than 1GiB would be enough; but that means officially i should say 1.5GiB RAM minimum i guess.

Perhaps i should add a warning message saying something like "Warning: detected less than 1.5GiB of internal memory; this could lead to boot and installation issues, please add more RAM!". That would give the user a quick hint at what the curlpit is if something goes wrong after seeing that message.

I think this project is fantastic. As I said, I feel your interface is very intuitive, and very easy to use. Keep up the great work.
Click to expand...
Sure will! Thanks! :)
 
Just wondering if there will be an .img for USB installation like there was with version 0.15?

Also, might be worth putting something about support for 4k drives in the FAQ?

Thanks for the great work!

ML
 
I am currently trying to find a switch that supports LACP that wont kill my wallet to start with. With only a few systems at first I wont need anything major.

I haven't bought any of the SSDs yet so no worry there.

How would you suggest backing up the files? Im used to windows so freebsd is all new territory.
 
Greatone123 said:
I am currently trying to find a switch that supports LACP that wont kill my wallet to start with. With only a few systems at first I wont need anything major.

I haven't bought any of the SSDs yet so no worry there.

How would you suggest backing up the files? Im used to windows so freebsd is all new territory.
Click to expand...

Dell 27xx will do lacp and has other nice features like vlans for around 100 or less depending on the deal you find
 
Something you may want to add to your FAQ: While the L2ARC is awesome, something most don't take into account is the amount of RAM required to use an L2ARC. Usage differs depending on how you access your data, but a rule of thumb is 200 bytes per record. Came as a nasty shock to me when I added a 40gb SSD as cache device in my server with 4GB of RAM and ZFS wouldn't use it! :)

Edit: http://www.mail-archive.com/[email protected]/msg34674.html
 
Mossy Lawn said:
Just wondering if there will be an .img for USB installation like there was with version 0.15?
Click to expand...
Not anymore; the new installation methods includes a new 'Embedded' installation method which would do the same thing as the old binary image, but now you don't need to download a separate package anymore; the LiveCD is fine. You can create this inside Virtualbox then move the USB pendrive over to the target system.

However, the embedded installation is not finished yet, though most things are already setup i have yet to finish and properly test this new installation method; you should see this arriving in 0.1.8!

Until then, you can do a normal Root-on-ZFS installation on your USB media; ZFS is a copy-on-write filesystem so it wouldn't be so bad for wear leveling since it doesn't overwrite stuff rather write to new space; just what SSDs prefer. It just writes more often than the Embedded distribution would; which only sees writes on shutdown for saving your settings.

Greatone123 said:
How would you suggest backing up the files? Im used to windows so freebsd is all new territory.
Click to expand...
Several methods are available:
  • Use your own Windows/Mac OSX desktop to proxy all files, using some application that reads from the source mount and writes to the target (backup server) mountpoint. This may be slower since your Windows computer has to both send and receive data, and the data travels through your own PC meaning that corruption due to RAM is possible.
  • Much better would be a direct transfer from the main ZFSguru server to the backup server. This can be done with rsync; remote file synchronisation. This connects over the network and allows to sync a file tree, where it can use checksums to verify the integrity of the files that are sent.
  • The pro solution would be to use zfs send/receive, which transfers raw ZFS data over the wire including all checksums and ZFS properties intact.

The first one you can already do, the second one is possible with manual commands (rsync is already installed) but both the second and third options would need to be implemented in the web-gui for easy backup and data migration options.

Can't promise, but i would love rsync to be manageable via the web-interface, in 0.1.8.
 
Baddreams said:
Came as a nasty shock to me when I added a 40gb SSD as cache device in my server with 4GB of RAM and ZFS wouldn't use it! :)
Click to expand...
Were you using a very low recordsize?

ZFS should simply not use a part of your SSD if you don't have enough RAM to hold the L2ARC tables.

Example maximum L2ARC sizes with 1GiB RAM dedicated to L2ARC:
128KiB recordsize -> 1GiB RAM = 640GiB L2ARC (SSD)
64KiB recordsize -> 1GiB RAM = 320GiB L2ARC (SSD)
32KiB recordsize -> 1GiB RAM = 160GiB L2ARC (SSD)
16KiB recordsize -> 1GiB RAM = 80GiB L2ARC (SSD)
8KiB recordsize -> 1GiB RAM = 40GiB L2ARC (SSD)
4KiB recordsize -> 1GiB RAM = 20GiB L2ARC (SSD)

Example RAM requirements to fully utilize L2ARC of 100GiB:
128KiB recordsize -> 100GiB L2ARC = 160MiB RAM
64KiB recordsize -> 100GiB L2ARC = 320MiB RAM
32KiB recordsize -> 100GiB L2ARC = 640MiB RAM
16KiB recordsize -> 100GiB L2ARC = 1,25GiB RAM
8KiB recordsize -> 100GiB L2ARC = 2,5GiB RAM
4KiB recordsize -> 100GiB L2ARC = 5,0GiB RAM
 
From http://www.zfsbuild.com/2010/06/03/howto-add-cache-drives-to-a-zpool/comment-page-1/#comment-121

...One thing to keep in mind when adding cache drives is that ZFS needs to use about 1-2GB of the ARC for each 100GB of cache drives. If you are planning to run a L2ARC of 600GB, then ZFS could use as much as 12GB of the ARC just to manage the cache drives. You will want to make sure your ZFS server has quite a bit more than 12GB of total RAM. I’d personally recommend using 24GB-48GB of system RAM if you are using 600GB of L2ARC.
Click to expand...
 
I'm currently running a raidz2 pool of 6 Samsung HD204UI with 4K sector override.

I also have a WD 2GB EARS (4K sector) drive that I freed up from my old (WHS) server, that I wanted to add as a hot spare. Since ZFSGuru doesn't support hot spares yet, I wanted to do it manually, and also take into account the 4K alignment, so here's what I did. I'm kind of a noob at this so I hope I did this correctly. Comments/critiques/suggestions would be welcome!

1) Format the new drive:
Used the ZFSGuru Physical Disk page to format as a GPT disk labeled "hotspare" with 1MiB reserved space (i.e. the default).

2) Create 4K sector provider at command line:
# gnop create -S 4096 /dev/gpt/hotspare

3) Attach spare to my main pool 'tank':
# zpool add tank spare gpt/hotspare.nop

That was it. zpool status returns:
Code: 
  pool: tank
 state: ONLINE
 scrub: none requested
config:

	NAME                STATE     READ WRITE CKSUM
	tank                ONLINE       0     0     0
	  raidz2            ONLINE       0     0     0
	    gpt/disk1       ONLINE       0     0     0
	    gpt/disk2       ONLINE       0     0     0
	    gpt/disk4       ONLINE       0     0     0
	    gpt/disk5       ONLINE       0     0     0
	    gpt/disk3       ONLINE       0     0     0
	    gpt/disk6       ONLINE       0     0     0
	spares
	  gpt/hotspare.nop  AVAIL   

errors: No known data errors

Hope that looks okay.
 
Those are indeed proper procedures for doing it manually, but two things to take into consideration:

1. On reboot, the .nop entry will be gone and it will be detected by gpt/hotspare instead. At this point, the sectorsize override will be gone. Read on to 2:

2. The whole point of the sectorsize feature is not to keep the disks with a 4KiB sector size; but rather to have a 4KiB sector size on the moment that ZFS creates the pool, where the vdev would get an ashift=12 value instead of ashift=9.

In other words:
- only one disk actually needs the sectorsize override feature, in order to trigger ZFS into using a different ashift value on the vdev
- or in english: the procedure is not necessary for your hot-spare, only when creating the initial pool or when expanding a pool with a new vdev, in both cases the sectorsize override feature is supported by the Web-GUI.
 
Is the ZFSguru web-interface only compatible with Lighttpd, or does Apache2 work as well? Any deamon-specific tunnels perhaps?
I think Apache2 is bundled with Squeezebox Server (but could not find it in ports requirements; strange) so it would be nice to use the same deamon for ZFSguru.
 
I used Apache 2.2 before i switched to Lighttpd, so yes it should work. The services page would replace Lighttpd with Apache if you use Apache. Not tested with recent releases though, but it should work. Keep in mind you need sudo working properly as well as some other stuff which the GUI expects.

In the future i'll make a package/port of it so you can 'compile' it and it would download, compile and install dependencies just like other ports do.
 
Just like to share this: my wishlist for 0.1.8 release. Not all items on this list are guaranteed to make it into 0.1.8, though, and not all items are on the list. Nevertheless, feel free to comment!

Wishlist for 0.1.8:
  • Welcome wizard on first access, selecting your expertise level as well as a password for authentication
  • Password-based authentication (possibly also IP-based authentication)
  • Feedback bar, telling exactly what happened as you click stuff
  • Disk spindown & AAM/APM control
  • More disk information: TRIM capability etc.
  • Replacing failed disks in your ZFS pool; downgrading mirrors; upgrading single disks to mirrors
  • Torrent client integration, which the GUI would use to download new stuff (system images, web GUI, Extensions)
  • Proper sanity checking (for example user can create a pool with a space; this breaks things and need to be sanitized)
  • Improvements to installation method
  • Finishing the Embedded installation method
  • Exporting system configuration like Samba configuration (so you can keep a backup and not have to worry about losing your system disk; which shouldn't be a real problem anyway)
  • Migrating existing configuration when upgrading Root-on-ZFS/Embedded systems
  • Structural under-the-hood changes
  • Structural framework for ZFSguru extensions, allow for cool things like Media Streaming packages (which are very big)
 
sub.mesa said:
Just like to share this: my wishlist for 0.1.8 release. Not all items on this list are guaranteed to make it into 0.1.8, though, and not all items are on the list. Nevertheless, feel free to comment!

Wishlist for 0.1.8:
  • List of awesome stuff
Click to expand...

Any chance for FTP that supports scheduled downloading? A graphical approach to this is the only thing keeping me Windows atm.
 
No1451 said:
Any chance for FTP that supports scheduled downloading? A graphical approach to this is the only thing keeping me Windows atm.
Click to expand...

What does this mean?

I have setup rsync in cron that seems to keep my data wherever i want it.
 
I ran a scrub and now this is what I see

capturehol.png
 
That would mean some corruption was detected and fixed by ZFS. For the moment everything is alright; but if the underlying issue is not gone then you may see additional checksum errors. So run scrub regularly and see if the number changes; if not then the issue that once was is now gone.

Now imagine what this would have meant on another filesystem; some small corruption somewhere. Ever noticed a hickup in your videos, showing a bit of corruption but gone after half a second? Well corruption is a very likely cause for just that.

Read errors = I/O error when trying to read something
Write error = no or negative confirmation about whether the write I/O was successful
Checksum error = the data that the drive returned does not match the expected data, as judged by the checksum engine and the data on the other disks.

ZFS can fix all these problems, as long as you got some redundancy left. But if you become degraded and you rebuild with a disk, and then some of your remaining disks have corruption/checksum errors, then some files will 'fail' and couldn't be accessed; but the filesystem always stays consistent since metadata is always replicated. So all that could happen is losing some files, but still something to remember! For valuable data like text documents, consider using copies=2 to protect against BER issues when rebuilding degraded RAID-Z or double-degraded RAID-Z2.

Have you run a 24-hour MemTest on that machine to rule out any bad RAM? That would be highly advisable if you haven't yet done so. I've seen some very silent bad RAM that has like one error per 4 hours of memtest testing, so quite sneaky! That's why i recommend 24 hours. Also ECC memory is a clear advantage here of course, but good thing ZFS can correct most casual errors here!
 
Looking to experiment with ZFS. I was thinking of using the following hardware:

Core i5 2 cores/4 threads
Intel P55 motherboard
6GB Ram
4 1TB 5400RPM Samsung Drives
60GB Vertex 2 (sandforce 1200 controller) as a cache drive
1GBit Intel Ethernet (PCI-Express)

Would you comment on this configuration. Right now at work we have implemented a small FreeNas box using a dedicated Raid card 3ware 9650SE and 4 2TB 7200RPM Enterprise Hitachi drives. Performance is so so despite everything being 1GBit Ethernet and NFS protocol. Our main software uses millions of tiny files (Insurance company) So can ZFS provide better performance in random reads and writes? This will mainly be for for our fail over site. I look forward to testing your software.
 
sub.mesa said:
Were you using a very low recordsize?

ZFS should simply not use a part of your SSD if you don't have enough RAM to hold the L2ARC tables.
Click to expand...

I believe I was using IOMeter over samba so yeah it was probably 4kb or 8kb records. I had to manually limit the ARC to get ZFS to use the L2ARC with my setup, it wasn't attempting to cache anything in the L2ARC until I strangled the ARC down to 2GB.

Regardless, folks should be aware that there could be issues when either working with limited memory or large cache devices.
 
THE JEW (RaVeN) said:
Does the Oracle link work with one's old Sun login?
Click to expand...
I see they've replaced the original pages with an Oracle login page, how nice! :rolleyes:

So i guess i should redirect to a ZFSguru URL, describing that issue.

Vengance_01 said:
Looking to experiment with ZFS. I was thinking of using the following hardware:
Click to expand...
Looks good. Disable the hyperthreading though, it causes lower performance, and potential security issues (though not very significant).

6GB RAM is decent; especially since you have only 4 disks. If you want to do 10+ disks you would want more memory perhaps.

Don't use your SF-1200 SSD as ZIL/SLOG; only as L2ARC or primary storage. You can create primary storage by partitioning it (GPT partitions are preferred); so 50% primary storage and 50% L2ARC is an option. Note that SSD primary storage is best kept RAID0 and backed up properly, since current generation SSDs can corrupt data - even data that was not recently written or modified. Using as L2ARC is fine since corrupt data in L2ARC will be detected by checksums and primary storage will be used instead; no problem here!

Right now at work we have implemented a small FreeNas box using a dedicated Raid card 3ware 9650SE and 4 2TB 7200RPM Enterprise Hitachi drives. Performance is so so despite everything being 1GBit Ethernet and NFS protocol.
Click to expand...
ZFS works best on plain harddrives, so ZFS can do the RAID. Using on Hardware RAID means you outsource the RAID to the controller, and if it does a bad job then ZFS can't help you either! For ZFS that is just a single disk without any redundancy.

Hardware RAID5 needs BBU (Battery Backup Unit) in order to enable write-back safely, and also needs to obey buffer flush commands that ZFS sends, or data integrity may still be in question!

The best setup is one where ZFS controls the disks directly, without anything in between. That means a normal AHCI or SAS controller, like your onboard ports on your motherboard (set BIOS to AHCI) or an add-on controller without RAID, called a HBA. Popular HBA is LSI 1068E like Intel SASUC8i; quite cheap, too. Might need a firmware flash to IT-firmware.

Our main software uses millions of tiny files (Insurance company) So can ZFS provide better performance in random reads and writes?
Click to expand...
Enable compression! Compression not only reduces the size of your files, it also enhances performance on tiny text files since they can be compressed into chunks, meaning that one sector could hold multiple files; without compression that is not possible as far as i know.

If your dataset is only text files, consider using gzip (heavy) compression; if it contains both binary and textual files, then consider LZJB (light) compression, to ease your CPU workload.

For compression tasks, which are 100% multithreaded in ZFS (at least up to 64 CPUs), you can multiply your cores and frequency:
2* 3.0GHz dualcore = 6.0GHz total
4* 2.0GHz quadcore = 8.0GHz total
6* 3.0GHz sixcore = 18.0GHz total

This is a rough estimate of course, but the compressing performance increases linearly with the SMP performance available to the system. Perhaps on larger SMP systems memory bottlenecks would also come into play, but i've found this rule of thumb to scale perfectly on my singlecore, dualcore and quadcore systems.

Baddreams said:
Regardless, folks should be aware that there could be issues when either working with limited memory or large cache devices.
Click to expand...
Indeed! I will add a graphical memory estimation on the L2ARC page which automatically adapts to the size of the L2ARC you've chosen. That would be nice.

Also, memory tuning like ARC should be made easier with a 'slider' widget of some kind, with red zones (too high) and green zones (stable) with yellow (performance tuning) in between. Good idea?
 
Excellent idea! I have a feature request :p Online raid expansion! I'll be giving it a shot as soon as I get this stupid serveraid 1015 working, if ever.
 
Expansion should be working on the expansion page, but expansion on ZFS works differently:

What you cannot do:
  • you cannot expand an existing RAID-Z (and RAID-Z2/3) with one or more disks
  • you cannot shrink a RAID-Z
  • Convert a RAID-Z to Z2 or Z3 or vice versa

What you can do:
  • add a new vdev like add 4 drives in RAID-Z to an existing pool, kind of the same as normal expansion but you add one additional drive as parity drive.
  • add a new vdev of different configuration (a mirror, a RAID-Z, RAID-Z2/3, you don't have to stick to the same thing for every vdev)
  • upgrade a single disk to a mirror or downgrade a mirror to a single disk
  • replace each disk in a RAID-Z by larger capacity drives, increasing the storage capacity without really expanding.
 
You forgot the baddreams nutjob option: create a new pool with atleast as much storage as the actual data on the existing pool, plus one sparse file as a device for each disk in the existing pool. After copying over all data, replace each sparse file with an actual disk from the old pool. Voila! Expansion :D

Edit: Just to be clear. I'll be turning a 3TB (3x1.5TB) RaidZ into a 7.5TB (7x1.5TB) RaidZ2. Strickly speaking I'm not expanding a pool. I'm just creating a larger pool without losing any data or needing to offload it somewhere.
 
You must log in or register to reply here.
Back
Top