File Server: Setup w/best results ??

I am currently looking into building a file server for my network. I have been using multiple computers to store numerous files across my network for it's entire existance. When ever I try to condense them to one computer I come accross the same issues, space and of course the files always end up being moved around to accomendate this space issue and I lose track of files.

I have a "file server" now but it is merely a computer with a bunch of Hard drives and a legacy processor. I was wondering if anyone else has this problem and if you have looked into building a computer especially for this and what parts and setup did you go with.

I have looked into getting a computer with raid 5 support so I may stretch 4-5 hard drives across this computer and have a "fat" 750 - 1000" gigs of space. This would allow for redundency and have a speed boost, this is my understanding? I was wondering if a computer with 4 400gig drives in 2 separate raids with redundancy would boast better backup and speed.

Currently I have an Intel motherboard and an 820 chip just sitting on my counter collecting dust. I got it from an Intel website "contest." It was a great deal 4 months ago but I am wondering if I will ever use it. I was going to sell it for a profit but it seems as though it is losing it's market value quickly and selling it would not be worth the price to buy the AMD equivilant to replace it for this setup. The motherboard has raid 5 and sata II.

Would dual cores even give my file server a speed boost?

I am looking for some design ideas and most importantly hard drive configs - since that is going to be the bulk of the price tag.

BenLoomis said:

Would dual cores even give my file server a speed boost?

Click to expand...

In a machine that's a file server for you at home? No, it won't give it a speed boost. The machine is bound by the speed of network I/O and disk I/O. If you've got lots of users, or you're doing software encryption or compression for the drives, or you're doing network encryption (IPSEC, for instance), then maybe... but even then, a very modest machine should do just fine.

You need to sort out what your backup strategy will be. Is the data recoverable or rebuildable? If your house burns down, do you mind losing your data? How will you recover from a bad virus infection? What will you do if one drive in the array fails? Two?

Answering these questions will lead you to the answer you want about your drive configuration.

Thanks for you input. I didn't think that the CPU would make a difference.

For your input in the Hard disks, I am confused. The reason I want to put the disks in redundancy is for back up purposes, correct? That was my idea. I was only asking if raid 5 would be beneficial. If not then I could very well use a current computer for the job and spend money on just hard disks instead.

The files stored are going to be mostly multimedia. Dvd images, CD images, music and huge wav files and video files.

I will use this PC to house the images while my other PCs mount the images or play the files. I also want to feel safe enough to back up important files (school work, or word documents ) onto this machine incase of a failure on another computer.

Also anyone with experience using such a server like this: Did you use Linux or Windows?

I am sorry if this thread's material stray from the topic "hard drives" but that is really the focus point of this build. If I don't need raid 5 or sata II then i'll use a current PC.

BenLoomis said:

Thanks for you input. I didn't think that the CPU would make a difference.

For your input in the Hard disks, I am confused. The reason I want to put the disks in redundancy is for back up purposes, correct? That was my idea. I was only asking if raid 5 would be beneficial. If not then I could very well use a current computer for the job and spend money on just hard disks instead.

The files stored are going to be mostly multimedia. Dvd images, CD images, music and huge wav files and video files.

I will use this PC to house the images while my other PCs mount the images or play the files. I also want to feel safe enough to back up important files (school work, or word documents ) onto this machine incase of a failure on another computer.

Also anyone with experience using such a server like this: Did you use Linux or Windows?

I am sorry if this thread's material stray from the topic "hard drives" but that is really the focus point of this build. If I don't need raid 5 or sata II then i'll use a current PC.

Click to expand...

well, if you get a virus, all your data may be screwed, even with RAID 5 or NOT.. backup is always on another location (tape drives, another computers), so if something happens to your data, your not totally screwed..

you also need gigabit LAN, if u want fast access

Right. Like MJZ says, RAID doesn't help with operator error or destructive changes.

Think of the failure categories:

Operator error: you accidentally delete a file or directory by mistake, or overwrite a file. RAID 5 doesn't help that; the file is delete from the volume. An image backup does.
Virus: same thing. If a virus modifies all your EXE files on your RAID 5 volume, they're modified and that's that.
A single drive dies. RAID 5 does help this, assuming you can find a compatible drive to put in and the rebuid firmware works. What if another drive fails while rebuilding the first?

"Image backup" can mean a lot of different things. For me, I run Acronis TrueImage on each of my machines at least once a week. It runs at night, and makes an image of a drive on the machine. That image is copied over my network to a server which has a lot of disk storage. That server has an image of any drive of any computer I own, and that image is never more than a week old.

If I accidentally delete a file, I can probably go to that machine, mount the image as a virutal disk, and then copy a version of the file off. If I notice a virus infection before the next backup, I can disable the backup job and rebuild the machine from a previous image that doesn't have the infection. For a drive failure on an individual machine, I can burn a CD that boots, reads the image over the network to restore a new drive, and then I'm back online.

A virus doesn't affect this strategy because the disk images aren't mounted as a file unless I'm actualy restoring or loading. The virus can't "See" the infectable files inside the large image file.

Different backup strategies will protect against different things. Here's another set of failure categories:

Your house burns to the ground. All your machines are ruined at the same time.
There's a large-scale disaster, like a big earthquake, which affects the whole geographic area where you live.

I defend against this by putting my most important data on an FTP server somewhere else. You can do a similar strategy with tapes. If you bakup the most important stuff you have, it probably fits on a managable number of DVDs or tapes. You can then take those into work with you and leave them in a drawer under your desk, or put them in a safe-deposit box, or leave them in a storage locker. Maybe that won't defen against a large-scale disaster.

Other factors to consider while deciding on a backup strategy include the confidentiality of the data. You'll want to think of how often data changes; for some machines, a three year-old backup is useful. For other machines, a backup from yesterday might be too old. You have to think of the risks and decide which are acceptable to you and which aren't.

Only once you've made those decisions can you decide on what you want to do for the system. I run RAID 0 on many of my machines, for example, since the data is either recreatable (because it comes from a program or a process I use to create sample data to study databases, data warehousing, and algorithms) or because I have a good backup strategy and can recover promptly. For other applications, such a strategy might not be acceptable, and I'd want RAID 5 or a redundant distributed file system, and so on.

Wow are you guys overthinking this. He just wants a fileserver that will be robust against drive failure. He's asking about hardware setups, not backup methodology and philosophy.

So, with that in mind, here's an answer - go with RAID 5. 4 drives in RAID 5 only loses the storage on one of them, whereas 4 drives in RAID 1 loses the storage of 2 of them. Plus, you can run RAID 5 off just 3 of the 4 drives if one of them fails (usually, not sure if lower end RAID implementations can handle that). Plus, two RAID 1 disks would still make for two different places you need to look for files.

I recommend linux as it is infinitely less likely to get hit by a virus than windows (though, really, anyone reading HardOCP is probably smart enough not to run executables from their email). Plus, linux is free. Ubuntu linux is supposed to be pretty user friendly... google it, they have pretty extensive forums which are a great resource. I haven't used it, but heard good things.

Use the P4 chip and motherboard you have... the CPU in these servers is almost inconsequential. A friend of mine is running one on a Pentium 450MHz.

I also recommend putting the OS on a separate drive from the storage, so if you end up having to rebuild the OS drive, you won't lose the storage as well. If you can swing it, use RAID 1 on the OS drive (a couple 80 Gig drives is only $100, and you don't need more than that for just the OS), then you won't have to rebuild the whole OS if one drive fails.

250 gigabyte hard drives are the sweet spot for price for performance right now. If you have plenty of SATA connections on the board, you're better off getting 6 250s for $600 than 4 400s for $800. However, most motherboards only have like 4 SATA connections, so that might not work.

I don't know what your price range is, but if you have money to spare, get an 8 port RAID card ($200-ish) - it'll allow more expansion in the future by just adding drives (one of the beauties of RAID 5 - adding new drives adds 100% of their storage space - assuming they're the same size as the rest of the drives in the array). It's also good because it means the RAID array isn't tied to that motherboard. If the motherboard dies, you can just move the array and the card to another machine with almost zero effort. If the RAID is run off the motherboard, you're probably hosed (unless you go out and buy another identical motherboard). In general, the "PCI-X" RAID cards can be run just fine in a regular PCI slot. I can't vouch for all of them, but I know several that work that way.

So... ideal setup
RAID Card
2x 80gig OS drive (RAID 1)
4x400gig or 6x250gig storage drive (RAID 5)

Hope this helps.

-Nate

Thanks guys.

I really didn't need to know all the backup solutions, since backing up 600 - 800 gigs of data on a weekly or even yearly basis would take way too long and I don't need that much security or backup for the information. I can always pull the info off my cds and dvds again.

To get into more detail but not a full blown explanation, I have backed up all my dvds (seriously, not stolen) and playstation/consoles images on to my computers and I mount the images and watch the movies or play the games directly over my network. I put the cases and media away in my colletion wall. If I were to have an earthquake or tsunami sometime soon I would be more worried about the physical product as apposed to the images. (Pennsylvania doesn't see many of those). I also do a lot of video and sound editing. I have almost 200 gigs of just Halo videos I have captured, this is unrecoverable, but it's backup on 3 hard drives and put away in my closet.

Sorry to get you guys into a disscussion about backup solutions, I didn't make myself clear in the beginning.

Now, what is the likelyhood of getting 6 hard drives all the same and all of them functioning? hehe. I was thinking about the 250 hitachi's sata II, any qualms? I could always get a sata II card, which was mentioned by Nate, great idea, and use that in my other PC. My only question to that would be, what would be the speed difference from PCI instead of directly from the mainboard?

Nate Finch said:

Wow are you guys overthinking this. He just wants a fileserver that will be robust against drive failure. He's asking about hardware setups, not backup methodology and philosophy.

Click to expand...

Perhaps you're under-thinking it: only a fool would dismiss the idea that backup strategy doesn't influence the design of a file server.

BenLoomis said:

I really didn't need to know all the backup solutions, since backing up 600 - 800 gigs of data on a weekly or even yearly basis would take way too long and I don't need that much security or backup for the information. I can always pull the info off my cds and dvds again.

Click to expand...

If your back up strategy is a resore, that's fine. Be mindful that restoring 800 gigs of data from DVDs requires 200 disk swaps. How fast can you swap disks? At what rate would your machine read from those disks, as fast as it could?

BenLoomis said:

Now, what is the likelyhood of getting 6 hard drives all the same and all of them functioning?

Click to expand...

Nearly 100%, if you're buying new drives.

mikeblas said:

Perhaps you're under-thinking it: only a fool would dismiss the idea that backup strategy doesn't influence the design of a file server.

Click to expand...

The problem is, joe average HardOCP reader can't afford weekly offsite tape backup for all the personal crap he keeps on his home file server. Tapes and tape drives are expensive (heck, spare hard drives are more economical) and a hassle to use.

So, most people do what they can on a budget - use RAID 1 or 5, burn the occasional backup DVD of important documents, and just hope the house doesn't burn down.

I think that's a perfectly reasonable strategy for most people. It's by no means perfect, but the next step up is large enough to be a real barrier to a lot of people.

-Nate

ok, well I guess this post has gone south. I do understand everything you are saying. Seriously, if I had a natural disaster the only way I could keep my data would be to have it stored on a remote server somewhere. But to upload at 40-80k a sec would take a long time. (roughly 60-120 days) Plus the idea of tapes makes me think about how much and how long? I setup a tape back up for a guy and I see him on a weekly basis for computer checkups. The tape is setup to only back up the data that is different, for a once a week fix. For his senario, this is the best practice. I have so much data and it will have gigantic files added and that will require more tapes. The largest tape back up I have seen is 36gb per tape with an maximum tape speed of 20gb per hour. I would be better off buying extra hd drives and putting important files on them, and storing the disks some where. This is my current practice and so far it is working. Plus with a disaster the tapes would be ruined.

It would take too much time to back up and to restore, think about it, it would take longer to restore off a tape than the original disk. The tape will move at roughly 6000kb per sec where my dvd drives more anywhere from 3000kb to 15000kb from start up to finish. Yeah I would have to switch disks, but I would have to switch tapes too. Plus why am I bothering putting the media onto tapes when they are already on dvds/cds?

So the only logical move would be to focus on speed for the inital setup with back up as a procautiounary step. Raid 5 seemed to handle this for me, and I was asking if anyone else in this situation would use that for speed and redundancy? If not, what disk strategy would someone use for the build? Barring in mind that all the data is already backed up and to make another back up is redundant and a waste of money. I am not saying I don't care about a precautionary step, this is where raid 5 would come into play and I would be able to rebuild the drive data, incase of a failure.

The reason why I want some drive safety, is because I have had numerous drives go bad on me. I am sure all of you have had you fair share of drives go bad, it seems like I have one go once every 2 years and I have always been able to rescue the data before I lose the drive. Plus I have had my fair share of files deleted, which I have always been able to rescue with a file restoration program.

As you can see file restoring is not the problem, it is speed and of course some back up incase things go belly up.

Please, I am not attacking anyone! I think what every has said is correct in every way; however, for my situation it is different. I am sorry I didn't explain myself properly before.

Quote:
Originally Posted by BenLoomis
Now, what is the likelyhood of getting 6 hard drives all the same and all of them functioning?

Nearly 100%, if you're buying new drives.


I said that because almost everytime I buy drives in pairs, atleast one is DOA. One time I bought 2 80gigs and both were bad. I was getting weary about 4-6 drives and no issues?

I guess I have bad luck?

Where are you buying from? Zipzoomfly and Hypermicro are practically the only retailers to package their drives right. Newegg and many others use bubble wrap and foam peanuts around either a manufacturer-made enclosure like Seagate's Seashell or a static-proof bag to package their drives, which is specifically non-recommended by all 5 of the big drive makers. ZZF uses foam boxes as the big 5 all recommend. That could cause all the fatalities right there.

Newegg has 400gb tapes, albeit at $90 a pop. 200GB ones are about 40, or 100s for $25, but of course you need N times as many. And tape drives are *not* cheap - see what I mean? The lowest price on that page is $2k, and the average seems to be around $5k. I personally think tape is dead except for huge-volume applications and offsite storage. Household machines don't need a thousand dollars' worth of backup, let alone five - buy some external drives, and give one to a (reliable) relative to hold on to at Christmas, then next year bring them the updated version and swap back. Or put it in a security deposit box, you get the idea.

BenLoomis said:

I have looked into getting a computer with raid 5 support so I may stretch 4-5 hard drives across this computer and have a "fat" 750 - 1000" gigs of space. This would allow for redundency and have a speed boost, this is my understanding? I was wondering if a computer with 4 400gig drives in 2 separate raids with redundancy would boast better backup and speed.

Currently I have an Intel motherboard and an 820 chip just sitting on my counter collecting dust. I got it from an Intel website "contest." It was a great deal 4 months ago but I am wondering if I will ever use it. I was going to sell it for a profit but it seems as though it is losing it's market value quickly and selling it would not be worth the price to buy the AMD equivilant to replace it for this setup. The motherboard has raid 5 and sata II.

Would dual cores even give my file server a speed boost?

I am looking for some design ideas and most importantly hard drive configs - since that is going to be the bulk of the price tag.

Click to expand...

If I were you I'd look at a single core processor, board with gigabit, and a sata controller that does hardware raid 5.

Personally I'd look at the Highpoint 1640 4 port sata raid5 controller and I'd grab 4 320gb hard drives (320gb is much less than 400's prices and still packs a good punch of data capacity). This setup would yeild you 960 gigs of storage. I'd recommend any drive for a system drive


That should put you well on your way. And no, you wont feel the dual core being faster on a home file server.

Nate Finch said:

The problem is, joe average HardOCP reader can't afford weekly offsite tape backup for all the personal crap he keeps on his home file server. Tapes and tape drives are expensive (heck, spare hard drives are more economical) and a hassle to use.

Click to expand...

There's many other affordable solutions that do the same thing but cost less. FTPing to a remote site, which I think I mentioned in my post. Or using recordable DVDs, for example. Tapes aren't that expensive; you can get used DAT systems pretty cheap, and they'll hold four or eight times more data than a DVD... though they're slower.

Nate Finch said:

So, most people do what they can on a budget - use RAID 1 or 5, burn the occasional backup DVD of important documents, and just hope the house doesn't burn down.

I think that's a perfectly reasonable strategy for most people. It's by no means perfect, but the next step up is large enough to be a real barrier to a lot of people.

-Nate

Click to expand...

Of course it is. That's thinking it through, not ignoring the issue.

BenLoomis said:

It would take too much time to back up and to restore, think about it, it would take longer to restore off a tape than the original disk. The tape will move at roughly 6000kb per sec where my dvd drives more anywhere from 3000kb to 15000kb from start up to finish. Yeah I would have to switch disks, but I would have to switch tapes too. Plus why am I bothering putting the media onto tapes when they are already on dvds/cds?

Click to expand...

Ya'll seem to have latched onto the tape idea a little too tightly; I brought the issue up as one alternative, and suggested different alternatives to catastrophe-proofing the strategy, not as an end-all recommendation. But I'm glad to see you thinking it through: if measuring the data rates leads you to the conclusion that a particular backup strategy isn't for you, that's progress towards a comprehensive solution.

And price is a cosnideration, particularly relative to the value of the data. There's nothing unique about CDs or DVDs if they're commercial products—you can just buy new ones, and have the data back. Up to a point, that'll actually be cheaper than getting a backup system in order. If the data is not replaceable, then its value and replacement has to be more carefully considered.

BenLoomis said:

Raid 5 seemed to handle this for me, and I was asking if anyone else in this situation would use that for speed and redundancy? If not, what disk strategy would someone use for the build? Barring in mind that all the data is already backed up and to make another back up is redundant and a waste of money.

Click to expand...

Yep, RAID 5 would work. You could even use RAID 0, since you have a full backup of everything on the original mdeia. RAID 5 trades an extra drive for faster recovery, and at your data volumes is probably worth it.

BenLoomis said:

I said that because almost everytime I buy drives in pairs, atleast one is DOA. One time I bought 2 80gigs and both were bad. I was getting weary about 4-6 drives and no issues?

I guess I have bad luck?

Click to expand...

Wow, I'd say so! Or, you're buying from an iffy vendor—maybe there's a problem with their pacakaging. We bought 20 drives at work for some RAID towers back in 2002. Only one of the drives arrived DOA; I think that's the biggest bunch I've ever personally taken delivery of at once. Over time, I've probably only seen one or two percent of drives I've ever bought DOA.

Thanks guys.

Yeah I purchase most of my products through Newegg and I have noticed how the packaging has gotten cheaper and cheaper over the years. I have gotten products from ZipZoomFly, but I don't think they were HDs.

So far it seems like I will keep my current build and purchase a raid 5 controller card and 4-5 drives. I will have to investigate drive costs and speed. I will also purchase 2 80s for the main partition, maybe less?

BenLoomis said:

Thanks guys.

Yeah I purchase most of my products through Newegg and I have noticed how the packaging has gotten cheaper and cheaper over the years. I have gotten products from ZipZoomFly, but I don't think they were HDs.

So far it seems like I will keep my current build and purchase a raid 5 controller card and 4-5 drives. I will have to investigate drive costs and speed. I will also purchase 2 80s for the main partition, maybe less?

Click to expand...

Run a single drive for your system itself. Just dont keep data on it other than the OS. That way in the event of a failure, you can just install a new disk and reformat

Ockie said:

Run a single drive for your system itself. Just dont keep data on it other than the OS. That way in the event of a failure, you can just install a new disk and reformat

Click to expand...

While I agree with this in theory, in practice (at least in my case) setting up a new OS tweaked to perform just the way you like it (especially for a specialized server like this) could take days, especially given that this is a home machine, so all work is done during evenings and weekends. For $55 you can get another 80gig drive for RAID 1 and save yourself the hassle if you have an OS drive fail.

BTW, I've been looking at RAID cards, and the one that interests me the most is the Highpoint RocketRAID 2220. It's PCI-X with downwards compatibility to PCI, it has 8 internal SATA II ports, and allows for online expansion and upgrading of RAID arrays. 8 is a good number of ports if you want 2 for the OS drive, then you have 6 left for storage, which is good for 4 starter drives and room to add 2 more when you need the space and/or have the money. It's $250, which isn't cheap, but isn't all that expensive compared to a lot of RAID cards out there, and it's the least expensive option that allows online expansion and upgrading (so you don't have to rebuild the array and lose all your data every time you want to add a drive).

-Nate

Nate Finch said:

While I agree with this in theory, in practice (at least in my case) setting up a new OS tweaked to perform just the way you like it (especially for a specialized server like this) could take days, especially given that this is a home machine, so all work is done during evenings and weekends. For $55 you can get another 80gig drive for RAID 1 and save yourself the hassle if you have an OS drive fail.

BTW, I've been looking at RAID cards, and the one that interests me the most is the Highpoint RocketRAID 2220. It's PCI-X with downwards compatibility to PCI, it has 8 internal SATA II ports, and allows for online expansion and upgrading of RAID arrays. 8 is a good number of ports if you want 2 for the OS drive, then you have 6 left for storage, which is good for 4 starter drives and room to add 2 more when you need the space and/or have the money. It's $250, which isn't cheap, but isn't all that expensive compared to a lot of RAID cards out there, and it's the least expensive option that allows online expansion and upgrading (so you don't have to rebuild the array and lose all your data every time you want to add a drive).

-Nate

Click to expand...

Correct.


Now factor in heat and power usage of the drive and the potential of corruption. Also consider the fact that a lot of drives exceeds their warranty term so this drive might never end up failing in the life of the server.

In any event, having just your os on reduces your installation to a matter of minutes, not days. I have a slipstreamed disk and all I do is just pop it in and I'm good to go, I've mastered formatting and installation You also have other alternatives as a simple ghosting which costs you in essence nothing.

Ockie, you make a good point. For a drive that should change infrequently (which ought to be the case for a file server), ghosting the drive is probably a fine backup plan in case of drive failure, plus it's a lot easier to set up, like you said.

-Nate

Ghosting is a great idea. I am currently using 2 80gb hard drives in the machine as it is. They are not sata II, but I dont think that will make that much of a difference.

Yeah, SATA II is just a marketing gimmick right now. Maybe eventually it'll be useful, but not on any drives made today.

-Nate