Re-introducing FlexRAID (Flexible RAID)

S

spectrumbx

[H]ard|Gawd
Joined
Apr 2, 2003
Messages
1,647
Old thread: http://hardforum.com/showthread.php?t=1293651
Note: because FlexRAID has evolved so much, much of what's in that old thread no longer applies.
Click to expand...

I ended up writing a RAID software since nothing out there fit my needs.

Things I was looking for were:
- Parity data protection (of course)
- No need to run a specialized OS
- No need for additional hardware
- Ability to turn off my hard disks when not in use
- Use hard drives of any size in the RAID
- Losing any number of disks would never cause me to lose all of my data (unless I lost all disks)
- Lot of flexibility

Well, FlexRAID has all that and many more features. :)
Click to expand...

FlexRAID Basic 1.4 beta is out! http://www.openegg.org/forums/posts/list/268.page
More info at: www.flexraid.com

Web client screenshots (old): FlexRAID Web Client UI preview teaser
Video tutorials (old): Using the WebUI


FlexRAID as it currently stands:
  • FlexRAID currently only works as a snapshot RAID.
  • Nevertheless, FlexRAID Live! is getting ready for its first release.
  • FlexRAID now includes FlexRAID-View (consolidates all your data into a single massive drive).

Features:
- Supports multiple RAID engines (RAID 4, RAID 6, and RAID∞)
- Fast! (faster than many hardware RAIDs)
- Works on Windows and Linux (no need for a specialized OS or a separate system - use what you've already got).
- Imposes no performance penalty on your drives (you get to use your drives at full speed).
- No disk limit (RAID as many disks as you want).
- Supports Unit of Risk (UoR - A UoR can be anything you want it to be).
- You can RAID more than hard drives (anything with data on it - CD/DVDs, USB drives, etc).
- You can RAID network data.
- You can use drives of any size in the RAID and can use multiple small drives for parity space (some FlexRAID users even put their parity data on a network share).
- Only the drive where content is being read from spins, while the other drives can remain in sleep mode.
- Free. :)


Installation instructions:

FlexRAID Host: http://www.openegg.org/forums/posts/list/289.page

FlexRAID-View: http://www.openegg.org/forums/posts/list/0/290.page

WebUI: (standalone) http://www.openegg.org/forums/posts/list/173.page OR (as a service) http://www.openegg.org/forums/posts/list/191.page
 
Last edited:
I'll try it out when i get my new server up and running. I like the fact that you don't loose all data if you loose more disks then the parity, as I would now with my Areca raid 5.
 
Poorly named because it is not RAID.. It's a fancy backup / uptime solution (which RAID is not, RAID is an uptime / performance solution and not backup)
 
Looks like spam.

Post a link to a youtube promo video displaying your product in action. A screen capture or w/e. It would actually grab my attention a little better.
 
Lazn_Work said:
Poorly named because it is not RAID.. It's a fancy backup / uptime solution (which RAID is not, RAID is an uptime / performance solution and not backup)
Click to expand...

LOL!
Clearly your mind isn't "flexible" enough.
FlexRAID Basic is a snapshot RAID (new concept - I invented it - or at least, I am coining it :) ).

I have said somewhere else; and I quote:
FlexRAID is a program I created because I was looking for a RAID solution that nothing out there satisfied.

Many of us are storing more and more data (DVD backups, movies, MP3s, pictures, OS image backups, etc.).
As we start to accumulate terabytes of data, data protection becomes even costlier.

Imagine yourself with a file server that has 5 hard drives storing your movies and other data.
If you use something like RAID 5, accessing a single file will cause all 5 disks to spin.
Losing more than 1 disk will cause you to lose all of your data too.

Well, I wanted a cheap solution:
- where only the disk where the content was being accessed would spin while the remaining disks could be put into sleep mode (saving energy)
- where losing any number of disks would never cause me to lose all of my data (unless I lost all disks)
- where I could take some of the disks offline without affecting parity protection
- where I could make use of network drives in my RAID

FlexRAID solves these problems and then some.

The program is free (no commercial aspirations).
I am sharing it with others because it has proven to be very useful to me, and I expect others to have the same problems I had.
Click to expand...

BurntToast said:
Looks like spam.

Post a link to a youtube promo video displaying your product in action. A screen capture or w/e. It would actually grab my attention a little better.
Click to expand...

Web client screenshots: http://www.openegg.org/forums/posts/list/10.page
Video tutorials: http://www.openegg.org/tutorialFlexRAID.curi
:)
 
spectrumbx said:
LOL!
Clearly your mind isn't "flexible" enough.
FlexRAID Basic is a snapshot RAID (new concept - I invented it - or at least, I am coining it :) )
Click to expand...

Just saying FlexBackup or BackupNow would be more accurate than FlexRAID to describe the job it performs. But no worries, it's a neat concept.
 
Lazn_Work said:
Just saying FlexBackup or BackupNow would be more accurate than FlexRAID to describe the job it performs. But no worries, it's a neat concept.
Click to expand...

I disagree. :)

"Backup" would be misleading as you should be able to get 100% of your data (from a given point) if you were to lose all of your original data.
Just like tolerance level RAIDs (RAID 5/6/etc.), FlexRAID has tolerance levels (from 1 to infinity ).
The key difference with FlexRAID is that you don't lose all of your data even if you lose more disks beyond the tolerance level (unless you lost all of your disks).

Just because FlexRAID Basic does not do real-time parity synchronization does not mean it is not RAID.
FlexRAID Basic is RAID... only a snapshot RAID.
Just like any RAID solution out there, there are advantages and disadvantages to this design.

That said, only one version of FlexRAID is out; and, that's FlexRAID Basic.
The next version will have real-time parity synchronization as you are used to seeing (like in RAID 5 but with many of the features of FlexRAID Basic).
 
spectrumbx said:
I disagree. :)

"Backup" would be misleading as you should be able to get 100% of your data (from a given point) if you were to lose all of your original data.
Just like tolerance level RAIDs (RAID 5/6/etc.), FlexRAID has tolerance levels (from 1 to infinity ).
The key difference with FlexRAID is that you don't lose all of your data even if you lose more disks beyond the tolerance level (unless you lost all of your disks).

Just because FlexRAID Basic does not do real-time parity synchronization does not mean it is not RAID.
FlexRAID Basic is RAID... only a snapshot RAID.
Just like any RAID solution out there, there are advantages and disadvantages to this design.

That said, only one version of FlexRAID is out; and, that's FlexRAID Basic.
The next version will have real-time parity synchronization as you are used to seeing (like in RAID 5 but with many of the features of FlexRAID Basic).
Click to expand...

Hmm, I see what you are saying, it still doesn't "feel" like RAID.. lol :p

I commend you on an interesting project!
 
I haven't seen a clear description of how this works, so I'm going to take a guess here. Please correct me if I'm wrong, but this sounds similar to RAID 3 or 4, but running on top of the file system rather than underneath it and only calculating parity on demand rather than on every write. Probably an oversimplification, but hopefully close enough.

If I'm correct, and this is running on top of the file system, I have a few questions:

Does it work with any available file system? Has it been tested on Reiserfs, XFS, EXT3/4, etc.?

I've seen you repeatedly say "your data is safe", without an explanation that's made sense to me. If I were writing this, it would never write to the data drives except in the event of a restoration, but only write to the parity drives when told to. Since I haven't seen your code, I have to ask, is this what FlexRAID does?

How does it handle things such as symbolic links and hard links? Has this been thoroughly tested? Hard links are an integral part of my current backup system.

Is there any provision for presenting a single file system to the operating system? I assume not, but then I'm making a lot of guesses about how this works.
 
Steve01S4 said:
I haven't seen a clear description of how this works, so I'm going to take a guess here. Please correct me if I'm wrong, but this sounds similar to RAID 3 or 4, but running on top of the file system rather than underneath it and only calculating parity on demand rather than on every write. Probably an oversimplification, but hopefully close enough.

If I'm correct, and this is running on top of the file system, I have a few questions:

Does it work with any available file system? Has it been tested on Reiserfs, XFS, EXT3/4, etc.?
Click to expand...

A lot of those questions have been answered in the forums (both here [old thread] and on the FlexRAID forum).

Regardless, yes this has been tested on different filesystems.
Currently only the RAID 4 engine has been enabled.
Nevertheless, FlexRAID also supports RAID 6 and RAID∞ .

I've seen you repeatedly say "your data is safe", without an explanation that's made sense to me. If I were writing this, it would never write to the data drives except in the event of a restoration, but only write to the parity drives when told to. Since I haven't seen your code, I have to ask, is this what FlexRAID does?
Click to expand...

Your data is indeed as safe as it gets.
I never write to the data drives unless doing a restore.
Even then, you can choose to restore to a different path or set of paths than the original.

How does it handle things such as symbolic links and hard links? Has this been thoroughly tested? Hard links are an integral part of my current backup system.
Click to expand...

Handles them fine.
Yes, tested extensively.

Is there any provision for presenting a single file system to the operating system? I assume not, but then I'm making a lot of guesses about how this works.
Click to expand...
FlexRAID Basic does not do JBOD.

Read on how people are using FlexRAID Basic with WHS: http://www.openegg.org/forums/posts/list/28.page

The best thing about FlexRAID Basic is that it is very easy to test out.
It is non-intrusive, installs and configures in minutes, and you can use only a small portion of your data to test it out.

Check out the Web UI client video tutorial for a quick view of how it works: http://download.openegg.org/static/vid/FlexRAID Web UI/FlexRAID Web UI.html
 
Thanks for the response. I'm not sure yet whether it will work out for me... it'd take some changes to my current backup method, but it's definitely a cool program.
 
I hope your app gets popular, as i have followed it for a long time but not had the time to try it yet. :)

Btw. is the instruction video up to date with the current enhancements?
 
I havn't read up on it, but isn't this similar to matrix storage stuff?
 
spectrumbx - just a suggestion - I'm completely losing what your product is because I can't understand what it is quickly after reading your post and clicking on your url. Maybe some explanatory graphics would help.

I've lost interest before I even know what it is. I remember looking at your original post back when, I think I was interested then. I'm not trying to be an ass here, I just read a bunch of keywords and came away thinking, "What did I just read?"
 
[LYL]Homer;1033490108 said:
spectrumbx - just a suggestion - I'm completely losing what your product is because I can't understand what it is quickly after reading your post and clicking on your url. Maybe some explanatory graphics would help.

I've lost interest before I even know what it is. I remember looking at your original post back when, I think I was interested then. I'm not trying to be an ass here, I just read a bunch of keywords and came away thinking, "What did I just read?"
Click to expand...

You're very right.
Bad habits from my day job. :)

I edited the first post.
Hopefully, it is a little more clearer.
 
Justintoxicated said:
Can this be run WITH OpenFiler? So long as open filer is installed on a seperate disk or partition?
Click to expand...

I am tempted to just say yes, since OpenFiler runs on Linux and it does not restrict you on what RAID you should use.
However, I have never played with OpenFiler to understand the implications of all its features.
So, I will say yes with reservations.

There are many users using FlexRAID with WHS (Windows Home Server) leveraging both of their powers.
WHS provides the JBOD like view (unified view of the storage) and data management while FlexRAID provides the data parity protection.

I would say try it out and report back.
Once you have OpenFiler setup, setting up FlexRAID and creating your first RAID should take you less than 5 minutes.
 
spectrumbx said:
I am tempted to just say yes, since OpenFiler runs on Linux and it does not restrict you on what RAID you should use.
However, I have never played with OpenFiler to understand the implications of all its features.
So, I will say yes with reservations.

There are many users using FlexRAID with WHS (Windows Home Server) leveraging both of their powers.
WHS provides the JBOD like view (unified view of the storage) and data management while FlexRAID provides the data parity protection.

I would say try it out and report back.
Once you have OpenFiler setup, setting up FlexRAID and creating your first RAID should take you less than 5 minutes.
Click to expand...

I plan to setup open filer tonight, however its on an old box with only an 80GB drive until I can buy some storage, it will be a while before I can create an array unless someone wants to donate drives for testing :p This would work really awesome together if possible.
 
If I use flexraid with WHS, lets say that I lose my OS drive, will my data still be in tacked once I reinstall WHS and flexraid? Whats the procedure?

Performance/Speed wise, how is flexraid (software) any different from hardware raid?

When I setup flexraid, I choose what drives I want for parity and what drives I want for data storage. Say I have 20 data drives and 1 parity drive. How much of my data is protected if any? Is the only reason to add additional parity drives to protect me against a catastrophic failure (2 or more disks).
 
Justintoxicated said:
I plan to setup open filer tonight, however its on an old box with only an 80GB drive until I can buy some storage, it will be a while before I can create an array unless someone wants to donate drives for testing :p This would work really awesome together if possible.
Click to expand...

You don't need multiple drives to test out FlexRAID.
Remember, FlexRAID is flexible.

FlexRAID doesn't just protect drives, it protects anything that is a unit of risk (hence, the term UoR I use often).
Your RAID can include files, folders, partitions, disks, networked shares, USB drives, Floppies, CD/DVD, etc.

For instance, for your test, you can imagine that you have 5 folders on your disk that are at risk (someone could accidentally delete files in those folders from the network).
Let's say, you want to protect those folders from the accidental file deletions.
Well, each of those folders will be a unit of risk that FlexRAID can protect.

So, you can protect files (individual files), folders (and all the files/folders in them), partitions, disks, network shares, etc. with FlexRAID.
It just happens that, for most of us, our unit of risk is a disk, but FlexRAID does not makes that restriction.

For testing, setup something like this:
-C:\Test
|- Data1
|- Data2
|- Data3
|- Parity

Then fill the data folders with dummy/test data.

BurntToast said:
If I use flexraid with WHS, lets say that I lose my OS drive, will my data still be in tacked once I reinstall WHS and flexraid? Whats the procedure?

Performance/Speed wise, how is flexraid (software) any different from hardware raid?

When I setup flexraid, I choose what drives I want for parity and what drives I want for data storage. Say I have 20 data drives and 1 parity drive. How much of my data is protected if any? Is the only reason to add additional parity drives to protect me against a catastrophic failure (2 or more disks).
Click to expand...

If you lose your OS and have to reinstall it, your RAID will be unaffected.
In fact, you could re-install a totally different OS, and your RAID will be unaffected.
You will be able to pick right where you left off.

FlexRAID is highly multi-threaded. So, it scales very well.
It is faster than most hardware RAIDs that offload to the CPU that are out there.
On most system, you will be I/O bound.

The 20 drives will all be protected.
If you choose the T1 engine (RAID 4), you can only afford to lose 1 drive.
Even then, the worst case scenario, is that if you lose more than 1 drive, you will not lose all of your data as if it would be if you were using RAID 5.

FlexRAID also supports other RAID engines.
However, don't have the other engines enabled yet.
I wanted to leave something out to look forward to in version 2. :p

The T2 engine (RAID 6) supports losing up to 2 disks.
The T∞ engine supports losing any number of disks.

I am trying to file for a patent for my T2 and T∞ engines. :)

[LYL]Homer;1033490845 said:
Awesome! It is much clearer now, thanks!
Click to expand...

Well, thank you for pointing it out. :)
 
so if I have 10 disks with fault tollerance of 2 disks.

I lose 2 disks at once, I can rebuild the array and restore the data?
And if I lose 3 drives at once I lose the data on these 3 drives, but the additional 7 drives will still have their data.

Any documentation onhow exactly this works? Not sure if I trust it if I odnt understnad how it works :(
 
Justintoxicated said:
so if I have 10 disks with fault tollerance of 2 disks.

I lose 2 disks at once, I can rebuild the array and restore the data?
And if I lose 3 drives at once I lose the data on these 3 drives, but the additional 7 drives will still have their data.
Click to expand...

Yep, correct.

Any documentation onhow exactly this works? Not sure if I trust it if I odnt understnad how it works :(
Click to expand...

How it works depends on the RAID engine you pick.
The RAID 4 engine does simple XOR operations.
However, how the other engines work is top secret (patent pending). :)

Again, it is really easy and safe to test out how it works.
Just install it and give it a test drive.

There is a lot of documentation on the site and its forum.
Make sure to check those out.
 
Say I install FlexRAID Basic. Will I be able to upgrade to FlexRAID Live without rebuilding?

If I install FlexRAID today, will I ever have to pay a subscription fee or retail fee for the application down the road?

Down the line what is your intentions for this program? To be open source, to license out to hardware raid manufactures or to be a pay to use program?
 
BurntToast said:
Say I install FlexRAID Basic. Will I be able to upgrade to FlexRAID Live without rebuilding?
Click to expand...

No. FlexRAID Live! will work quite differently.
So, you would need to recreate the RAID, which is very safe and simple.

If I install FlexRAID today, will I ever have to pay a subscription fee or retail fee for the application down the road?
Click to expand...
Not likely.

Down the line what is your intentions for this program? To be open source, to license out to hardware raid manufactures or to be a pay to use program?
Click to expand...
Open source.
 
I'm really interested in your product, as I'm starting to feel some of the limitations of standard raid 5. I've considered WHS, but really don't feel like loosing my current linux server environment and the flexibility that it brings.

There is one bit that bothers me though:

spectrumbx said:
However, how the other engines work is top secret (patent pending). :)
Click to expand...

Later in the thread you state that it is your intention for FlexRAID to go open source, how can this software be both open source and patented? IMO this offers much more than WHS, mdadm, hardware raid, etc. and would greatly benefit from going open source,. This would also result in more developer contributions towards a finished product, and better acceptance by end users.
 
Any time frame on Real-Time Parity?

I really like what this product has to offer, but not having real-time parity is a major drawback for me.
 
Rakinos said:
I'm really interested in your product, as I'm starting to feel some of the limitations of standard raid 5. I've considered WHS, but really don't feel like loosing my current linux server environment and the flexibility that it brings.

There is one bit that bothers me though:



Later in the thread you state that it is your intention for FlexRAID to go open source, how can this software be both open source and patented? IMO this offers much more than WHS, mdadm, hardware raid, etc. and would greatly benefit from going open source,. This would also result in more developer contributions towards a finished product, and better acceptance by end users.
Click to expand...
Err... where did you read that open sources do not get patents? :confused:

nitrobass24 said:
Any time frame on Real-Time Parity?

I really like what this product has to offer, but not having real-time parity is a major drawback for me.
Click to expand...

Still working on it. :)


Everyone should check out MHDDFS.
Linux + MHDDFS is looking out to be a good alternative to WHS.

http://www.openegg.org/forums/posts/list/88.page
 
FYI, FlexRAID Basic 1.1 RC1 is out.

Code: 
Changes:
-----------
- Various bug fixes
- Memory optimizations (mostly for large RAIDs)
- RAID engine performance optimization (at least 15% and up to 50% performance improvement)
- I/O API re-work for better scaling
- Advanced RAID reconfiguration: Remove locked DRU / Rename locked DRU
- Option to enable read errors retry
- Support for transaction (ability to undo/rollback a failed operation automatically so that the parity data is not left in an unknown state)
- New data pickup delay feature (a delay since the last modified date specifying when a new file is ready to be added to the RAID)
 
Interesting to read about implementing storage pooling.

While Basic (I thought) is sufficient for me, I might have to re-think that...

Definitely going beyond where I thought it would mature to (including Live...)
 
Seems like people are knocking this without remembering what RAID stands for. Redundant Array of Inexpensive (or Independent) Disks.

Raid 0 isn't technically raid at all. I think too many people hear RAID and don't know or bother to find out the origin of the term.

FlexRAID sounds like a great idea. My question is... How much storage does the parity drive require? Can you use a smaller HD than your "set"? For example, if you have 4HDDs to protect, 2x 1TB, 1x 750GB, 1x 500GB, how big does the parity drive need to be to cover the data if all drives are at 75-90% capacity?

[edit] nevermind, I answered my own question by reading a bit... Parity drive needs to be larger than the largest data set on a single drive.
 
Last edited:
Logan321 said:
[edit] nevermind, I answered my own question by reading a bit... Parity drive needs to be larger than the largest data set on a single drive.
Click to expand...

Then you might want to read again...

The area available for your PPU just has to be bigger than your largest DRU.

A DRU doesn't have to be the entire size of a drive, or even just a single drive.

Same for the PPU...

From http://www.openegg.org/referenceFlexRAID.curi

data=DRU1{/Path1;Path2;...;/etc.}|DRU2{/Path1;/Path2;...;/etc.}|...|DRUxxx{/Path1;/Path2;...;/etc.}

parity=PPU1{/Path1;Path2;...;/etc.}|PPU2{/Path1;/Path2;...;/etc.}|...|PPUxxx{/Path1;/Path2;...;/etc.}

Just clarifying in case you didn't catch that Flexraid isn't at the disk level at all...
 
That's well and good but if you "raid" two DRUs on the same drive and only have a single parity drive you won't be able to recover the data on the drive with two DRUs, since it would count as two failed DRUs with a single drive lost.

I did say "largest data set" not largest drive. True, single drive wasn't accurate, but why would you combine them? It would just require a larger parity set to compensate.

I also was under the impression that at this time, only a single PPU is available for use with Basic, and thus, limited to a single drive.

"If you choose the T1 engine (RAID 4), you can only afford to lose 1 drive.
FlexRAID also supports other RAID engines.
However, don't have the other engines enabled yet."

Additionally, if you're looking for protection from hardware failure, you cannot have more than 1 DRU on any single HD until you are able to have more than 1 PPU, which is currently not available.

As well, you are not able to have your PPU on a drive you also have a DRU on without putting that PPU at risk in the case of drive failure.

So while it's theoretically moving away from a drive-based designation, it's current implementation is very limited in it's ability to do so.
 
I was merely pointing out where the "flex" part of flex-raid comes from...

Logan321 said:
How much storage does the parity drive require? Can you use a smaller HD than your "set"?
Click to expand...

Logan321 said:
I also was under the impression that at this time, only a single PPU is available for use with Basic, and thus, limited to a single drive.
Click to expand...


You said "parity drive" twice now... Your parity can span across multiple drives...

if you have 1TB drives, you could have four 250GB drive hold the parity for the 1TB drives...

I thought with your example of having drives of varying sizes in a raid, you might find that capability of interest since you had used the term "drive"...

I should have emphasized the "path" parms in my examples...

parity=PPU1{/Path1;Path2;...;/etc.}|

Each PPU can have multiple paths...
 
That makes sense, of course. I guess I hadn't considered setting up multiple locations for the parity since it adds complexity and power draw and storage space that's not really necessary for the kind of storage usage I have. Just because I wouldn't do it that way though, doesn't mean having that option is a poor one.

I am considering setting up flex-raid myself, but I don't have a dedicated HD for it's use atm, so I'd need to purchase one.
 
Logan321 said:
I guess I hadn't considered setting up multiple locations for the parity since it adds complexity and power draw and storage space that's not really necessary for the kind of storage usage I have.
Click to expand...

I'm sure you already know this, but (especially if the data on your DRUs is going to be static...) you don't need to have your parity drives on-line all of the time. If the DRUs having static data, the PPUs are only needed for when the parity's created or when it's needed for a rebuild...

(I'm just trying to save you the expense of buying a dedicated drive in the event you're like me and have other, albeit smaller, drives sitting around...)



Logan321 said:
Just because I wouldn't do it that way though, doesn't mean having that option is a poor one.
Click to expand...

Flex-raid's all about options. :)


But, in the end, just in case you're missing my intentions, yes, Flex-raid does work simplest when each DRU and each PPU is its own drive... I'm just pointing out that the "flex" part is that it doesn't have to be...

(FWIW - your DRUs and PPUs can even be across a network... So you could have your PPU drive(s) completely off-line and off of the main system, but be able to access the PPU without having to take that main system down as well...)

I'm just trying to point out capabilities that flex-raid has that others don't tend to offer (if at all)... Not every flexible option seems useful, but I'd rather a new (or curious) user acknowledge it all before they start setting things up, or spending money or extra hardware that they may not need...

I like flex-raid, and I just want anyone new to understand it. :)
 
You must log in or register to reply here.
Back
Top