Completed 62 TB FlexRAID Media Server

[AshleyAshes]

28 minutes ago, mrbilky said:

Dig the reuse of the ram didn't know they had such an option to install them and I too bought the elcheapo sata card why not it only uses the x1 slot so still have options with the rest of the x4 x8 x16 slots I'm am using unRAID though but happy so far, nice build!

Thanks.  Using only the parts I needed and recycling what I could was definitely a major consideration.  In all seriousness the only part that disappoints me is that since it's a P67 board it requires a discrete graphics card and that uses up a PCIE slot and adds to the overall power consumption.  Though the costs and effort of swapping in any other motherboard makes it a minor concern.

 

So the 2TB drive was pulled and the 8TB installed, I kept the 500GB and 1TB drive.  Interestingly, FlexRAID had never gotten around to PUTTING anything on that drive OTHER than parity data, so it would have been inconsequential had the drive failed in even the most critical way.

 

Funny thing, only TODAY did I realize how to set FlexRAID to do 'Fast Expansion' and rapidly add a new drive rather than have it spend 3 days regenerating ALL of the parity.  Oops. >_>

 

But here we have it.  41.5TB or 37.7 TiB.

 

[AshleyAshes]

The old 500GB Seagate 7200 RPM drive started having some early SMART issues and was rated 'Caution' in CrystalDiskInfo so I pulled it.  Since 8TB externals were $179 CAD for Black Friday I picked up an 8TB to replace it.  Nothing like a replacement that's 16x larger than the original, right?

 

As a result we are now at 49TB of storage.  1TB short of a nice round 50... I have a nearly new 1TB 2.5" drive pulled from my new laptop that I'm eyeballing hard at installing just to make it an even 50.

 

 

 

[timdine]

How do you have your shucked 8TB drives connected?  Are they connected to the SI-PEX40064 you described?  The website suggests that it supports up to 6TB drives, I'm on the hunt for a controller card myself.

[dalekphalm]

Just now, timdine said:

How do you have your shucked 8TB drives connected?  Are they connected to the SI-PEX40064 you described?  The website suggests that it supports up to 6TB drives, I'm on the hunt for a controller card myself.

Generally speaking, when a website lists (6TB max support) or something similar, that's simply because that was the biggest drive size available and/or tested with at the time of manufacture.

 

If a controller can handle 3TB+ drives, it can usually handle anything bigger as well with no problems.

[AshleyAshes]

Just now, timdine said:

How do you have your shucked 8TB drives connected?  Are they connected to the SI-PEX40064 you described?  The website suggests that it supports up to 6TB drives, I'm on the hunt for a controller card myself.

Yeah, I have zero issue with multiple 8TB Archive or Baracuda Pros hung off the SI-PEX40064.  You're right though, I never looked at SYBA's specs page before but they do state a 6TB limit.  However this is not a new card so maybe that page is just out of date and consumer 8TB's were not on the market and untested.

I can however say that mine work fine and they just use the default drivers in Windows 10.

[prouser]

Here's my personal recommendation - if you value this data in any way whatsoever, migrate it somewhere else and run away from FlexRAID as fast as possible. The product is dead, and is full of critical bugs that corrupt all your data, along with bugs that prevent you from restoring under the most basic of drive failure options.

 

There hasn't been a bug release in 2 years. Let that sink in, 2 years! It's riddled with critical bugs. If you read their support forums, you will see page after page of people who had drive failures or other issues and simply lost all their data. I personally had a single drive failure, and it refused to restore the data. Even after having a nightly job to update the parity, and email me the results saying things were OK. That's the killer part of all the bugs in that program - you THINK everything is fine. Your test failures may work. But when you really need to restore... it's a total crapshoot. And there's no support, and nobody to help you.

 

I literally just won a PayPal Fraud case against them. I purchased the "24 hour or less premium support plan" after getting nowhere in their support forums for 2 weeks. Nobody responded to me for 5 days. I will repeat that, I had a 24 hour or less paid premium support plan - and nobody responded for 5 days. And when they did, it was a total scam. PayPal refunded me all my money, and is investigating shutting down their PayPal account.

 

The tech was interesting a few years ago, but it's just not relevant today as there are far better solutions from Windows Storage Pools, to really good RAID cards at dirt cheap prices. All of which have actual support, large communities, etc. It's simply not worth the risk.

 

Just my .02, but I cannot stress highly enough you are just asking for trouble if you keep using that product.

[dalekphalm]

18 minutes ago, prouser said:

Here's my personal recommendation - if you value this data in any way whatsoever, migrate it somewhere else and run away from FlexRAID as fast as possible. The product is dead, and is full of critical bugs that corrupt all your data, along with bugs that prevent you from restoring under the most basic of drive failure options.

 

There hasn't been a bug release in 2 years. Let that sink in, 2 years! It's riddled with critical bugs. If you read their support forums, you will see page after page of people who had drive failures or other issues and simply lost all their data. I personally had a single drive failure, and it refused to restore the data. Even after having a nightly job to update the parity, and email me the results saying things were OK. That's the killer part of all the bugs in that program - you THINK everything is fine. Your test failures may work. But when you really need to restore... it's a total crapshoot. And there's no support, and nobody to help you.

 

I literally just won a PayPal Fraud case against them. I purchased the "24 hour or less premium support plan" after getting nowhere in their support forums for 2 weeks. Nobody responded to me for 5 days. I will repeat that, I had a 24 hour or less paid premium support plan - and nobody responded for 5 days. And when they did, it was a total scam. PayPal refunded me all my money, and is investigating shutting down their PayPal account.

 

The tech was interesting a few years ago, but it's just not relevant today as there are far better solutions from Windows Storage Pools, to really good RAID cards at dirt cheap prices. All of which have actual support, large communities, etc. It's simply not worth the risk.

 

Just my .02, but I cannot stress highly enough you are just asking for trouble if you keep using that product.

http://dl.flexraid.com/Transparent-RAID-Change-Log.txt

Quote

2017-11-22: Refresh release with many improvements and bug fixes including:

- Fixed defect whereas storage pool would in some merge cases move data to a hidden folder when URB is enabled

Perhaps there have not been a lot of bug fixes in the past, but to say there hasn't been one in 2 years is patently false, as one came out a week ago.

 

In my past experience with FlexRAID, it ran pretty well. Though it's been a while since I used it.

[AshleyAshes]

2 hours ago, prouser said:

if you value this data in any way whatsoever

To be honest, no, I don't value this data much at all.  It would be rather inconvenient to replace most of it if somehow all of it was lost somehow or even if a single drive was lost but even Medusa, SickRage, CouchPotato and Transmission would keep on ticking with new stuff.  So even if there are risks associated with FlexRAID in some circumstances the level of protection that FlexRAID offers is 'good enough' for this task.

For sure if I was storing actually IMPORTANT data then I'd be using a separate ZFS setup plus at least one additional physical backup but a machine that is filled with, let's admit it, a bunch of pirate videos or rips of my own disc media, it's not worth the effort of ZFS or something, especially when it comes to dealing with growth of the data set.

 

50% of what this server represents is a 'Hobby'.  It's the 'fun project' of maintaining the MySQL, Medusa, Transmission and stuff and getting it all humming.  Outside of that stuff it's just a really dumb NAS with laptop RAM in it. 

[prouser]

21 hours ago, dalekphalm said:

http://dl.flexraid.com/Transparent-RAID-Change-Log.txt

Perhaps there have not been a lot of bug fixes in the past, but to say there hasn't been one in 2 years is patently false, as one came out a week ago.

 

In my past experience with FlexRAID, it ran pretty well. Though it's been a while since I used it.

 

I'm aware one just came out, but IMHO the only reason that came out is to avoid their PayPal getting shut down from fraud cases against them. They have been selling a product that knowingly corrupts your data for years, along with paid "premium support" in order to help you when it happens. But then when you pay for support, they say it's a bug and there is nothing you can do. That's a scam, and again I literally just won a fraud case against them.

 

People who installed FR 1-2 years ago are starting to see more drive failures (MTBF of the drives) as this is entering the sweat spot for drive failures. And more and more people are realizing (like I did) that the protection they thought they had, simply doesn't work and they are losing data.

 

Any software project that goes 2 years without a single bug fix, or maintenance release despite countless critical bugs is a dead product and something I wouldn't put any trust in whatsoever.

 

It's easy to say the data isn't valuable and it's just a hobby, but when you wake up one morning and your 49TB is gone I assure you that will not be a good day. I personally moved my home blu ray collection to Windows Storage Pools. It's gotten really good, there's a ton of support around it, constant updates, it's free (with Windows), ReiFS integrity validation, etc. ZFS would also be good, although I would probably just go to a Synology or something before that just for simplicity as well.

 

Again though, just warning you and others... if you are using FlexRAID, you are asking for trouble - this is coming from someone who just got burned by them. Tread carefully!

 

[AshleyAshes]

Updating this thread with some more changes.  A 1TB drive was removed, the 2TB system/temp download drive was replaced and that drive put to task in the storage pool.  So let's begin.

 

First, I ordered replacement SATA cables, all in blue to match the mobo, and like a grownup I got the zip ties and did neat and tidy cable management.  I also got a custom 9 drive SATA power cable  so that all nine drive stacked up in the case are powered off a single connector on the power supply.  It all makes it a lot cleaner and easier to work in.

 

 

Though using mostly 100cm SATA cables they aren't perfect to length but it's a lot better than the spiders nest of SATA cables I used to have.

 

Next, the system drive.  The 2TB Seagate Green drive did a lot of things, it hosted the OS (ANd all database functions), an image library for my screensavers, all the download applications, it's where all data was downloaded to, where it was read from for parity and unraring, what it was written to once repaired or unrared, and where it was read from to move data to the final storage pool.  Suffice it to say, large download queues could SLAM that drive.  The MySQL DB could even become sluggish.  Since this case also supports 4x2.5" drives in snap on holders over the back of it, I decided to solve that.

 

1 x 256GB Samsung EVO 850 for OS, database, and all that.

1 x 500GB Samsung 2.5" Drive as the 'Temp Download' drive, all downloads go there first.

1 x 2TB Seagate 2.5" drive as the 'Holding Tank' drive, when unraring, the rared files are read off the 500GB and written two the 2TB, reducing the load and allowing the unraring stage to be less of a bottleneck.  This is where anything waits before being copied to it's final destination or any other unsorted media.

 

 

Witht hat, I could still add one more 2.5" drive, maybe a 4TB to add additional pool storage.  There's a single 3.5" bay free.  And the 3x5.25 bays could hold 4x3.5" drives with the right adapter.  That said, there's some 13TB free currently so it'll be over a year till the next drive is added.

[dalekphalm]

6 minutes ago, AshleyAshes said:

It hurts! It hurts my eyes! ahhhhggggg!

 

I love how you went out and got blue Sata cables to match the MB colour, yet there's still green, black and red PCB cards all over the place

 

But hey, it's a start

 

Gotta say, I do love the build - keep up the good work! This frankenmoster still makes most of our server builds look like small fries.

[AshleyAshes]

Just now, dalekphalm said:

It hurts! It hurts my eyes! ahhhhggggg!

 

I love how you went out and got blue Sata cables to match the MB colour, yet there's still green, black and red PCB cards all over the place

 

But hey, it's a start

 

Gotta say, I do love the build - keep up the good work! This frankenmoster still makes most of our server builds look like small fries.

Yeah well I'm not about the spend the money to get blue SATA controllers and I don't think that there are even any passive GPUs I could use 'just for video' that even come in blue.

 

...

...

...

 

Well shit...

http://www.dx.com/p/ati-radeon-hd5450-2gb-ddr3-pci-e-x16-graphic-card-blue-125114#.Wn9TY6inFIA

[dalekphalm]

1 minute ago, AshleyAshes said:

Yeah well I'm not about the spend the money to get blue SATA controllers and I don't think that there are even any passive GPUs I could use 'just for video' that even come in blue.

 

...

...

...

 

Well shit...

http://www.dx.com/p/ati-radeon-hd5450-2gb-ddr3-pci-e-x16-graphic-card-blue-125114#.Wn9TY6inFIA

I mean... just do like Linus... Plastidip EVERYTHING!

 

lmao sweet card. Although ~$70 is pretty steep for an HD5450... The last one I bought was like $40 brand new at Canada Computers

 

... no blue though

[AshleyAshes]

3 minutes ago, dalekphalm said:

I mean... just do like Linus... Plastidip EVERYTHING!

 

lmao sweet card. Although ~$70 is pretty steep for an HD5450... The last one I bought was like $40 brand new at Canada Computers

 

... no blue though

Yeah seems Sapphire has some passive ones that are also blue.  Not that it HAS to be a 5450, it's just that that was a good candidate since there were passive models, native HDMI, and still decently supported with drivers.  I could probably get away with a really, really old PCI card (And that'd free up the PCIE slot) if I wanted to.

 

But to be clear, I have no plans to replace the GPU just for the color blue.  I got Blue Sata cables since I needed to order more anyway so I was like 'I'll get them all blue to match the mobo'

[dalekphalm]

4 minutes ago, AshleyAshes said:

Yeah seems Sapphire has some passive ones that are also blue.  Not that it HAS to be a 5450, it's just that that was a good candidate since there were passive models, native HDMI, and still decently supported with drivers.  I could probably get away with a really, really old PCI card (And that'd free up the PCIE slot) if I wanted to.

 

But to be clear, I have no plans to replace the GPU just for the color blue.  I got Blue Sata cables since I needed to order more anyway so I was like 'I'll get them all blue to match the mobo'

Not to mention that SATA cables tend to be pretty damn cheap, and are available in just about any colour, so sure - why not?

[AshleyAshes]

Yet another update.  Basically a 4 year old 4TB drive gave out some uncorrectable errors.  The data was all copied off fine but it was replaced with an 8TB drive so we've moved from 58TB to 62TB.  With like 16TB -free- though this is excessive.  It'll take 18-24 months MINIMUM to even really need to add more storage.  What am I doing with my life? D:

 

[Ginger_]

4 hours ago, AshleyAshes said:

Yet another update.  Basically a 4 year old 4TB drive gave out some uncorrectable errors.  The data was all copied off fine but it was replaced with an 8TB drive so we've moved from 58TB to 62TB.  With like 16TB -free- though this is excessive.  It'll take 18-24 months MINIMUM to even really need to add more storage.  What am I doing with my life? D:

 

 

Because something just looks spectacular about seeing tons of available space and stacks of hdds? 

[AshleyAshes]

1 minute ago, Ginger_ said:

Because something just looks spectacular about seeing tons of available space and stacks of hdds? 

To a degree yes, but at the same time I could have pulled the 4TB drive and NOT replaced it, and it still woulda taken 12+ months to use up the remaining 8TB

[dalekphalm]

1 minute ago, AshleyAshes said:

To a degree yes, but at the same time I could have pulled the 4TB drive and NOT replaced it, and it still woulda taken 12+ months to use up the remaining 8TB

obsesssssssssssssssssssssssed

 

JK (or am I?)

[Ginger_]

4 minutes ago, AshleyAshes said:

To a degree yes, but at the same time I could have pulled the 4TB drive and NOT replaced it, and it still woulda taken 12+ months to use up the remaining 8TB

Let me know when you figure it out, I have 3tb used and fill about 300gb a month, and yet still built my server with 12tb of storage.... 

[AshleyAshes]

Just now, dalekphalm said:

obsesssssssssssssssssssssssed

 

JK (or am I?)

I can't lie, it's fun to maintain the machine and I'm keen to see it with a third additional PCIE SATA controller and all the slots full.  But of course that's also BAD cause once this box is physically full I have a BIG problem to solve.  How do I grow past that?  but since my consumption is, REALISTICALLY, maybe 6TB/year (I wish there was an easy automated tool the CHART that.  Like something that could poll a drive's storage every week and build a projection chart?) I hope that I can more or less 'age out' older, smaller HDDs for larger ones.   But on that note, 8TB drives have been the 'sweet spot' for $/GB and may remain there for a while.  Even Seagate's recently introduced 10TB external drives a a LOT more per GB.

[dalekphalm]

10 minutes ago, AshleyAshes said:

I can't lie, it's fun to maintain the machine and I'm keen to see it with a third additional PCIE SATA controller and all the slots full.  But of course that's also BAD cause once this box is physically full I have a BIG problem to solve.  How do I grow past that?  but since my consumption is, REALISTICALLY, maybe 6TB/year (I wish there was an easy automated tool the CHART that.  Like something that could poll a drive's storage every week and build a projection chart?) I hope that I can more or less 'age out' older, smaller HDDs for larger ones.   But on that note, 8TB drives have been the 'sweet spot' for $/GB and may remain there for a while.  Even Seagate's recently introduced 10TB external drives a a LOT more per GB.

In your particular case, expanding could be done as follows (Gotta love FlexRAID for using NTFS as the base file system):

1. Pull the smallest drive, replace with a larger one. Put the pulled drive into a USB dock or enclosure, and copy the data back.

2. Repeat.

 

I guess you could also just replace the drive and rebuild from parity, but I imagine that's not as fast as simply copying the data straight over.

 

You're still limited by total drive number capacity of your setup, but as disks get larger, you can just replace the smaller ones with larger ones.

 

Another possible solution: SAS Expander Bays

 

Buy a SAS HBA (not RAID card, though you could flash RAID card to IT mode) with external SAS connectors. Buy a SAS Expander Bay. Connect them. The host system treats all the drives in the Expander Bay as local drives.

 

Personally I do this with an LSI 9207-8e paired with a Dell MD1200. Granted, I got the MD1200 (12x 3.5" bays) for free (they seem to go for around $450 CAD on eBay), but you can buy cheaper brands or even off brands. I used to use a RackableSystems SE3016, which was some knockoff SAS Expander Bay that has 16x 3.5" bays, and these are usually only $150-$200. Though shipping/import fees are a bitch for anything this big and heavy.

 

FreeNAS and FlexRAID both work in the same way in that they want low level access to the drives. I don't think it's as important with FlexRAID, but it's still better to be able to properly see SMART data, etc.

[AshleyAshes]

4 minutes ago, dalekphalm said:

In your particular case, expanding could be done as follows (Gotta love FlexRAID for using NTFS as the base file system):

1. Pull the smallest drive, replace with a larger one. Put the pulled drive into a USB dock or enclosure, and copy the data back.

2. Repeat.

 

I guess you could also just replace the drive and rebuild from parity, but I imagine that's not as fast as simply copying the data straight over.

 

You're still limited by total drive number capacity of your setup, but as disks get larger, you can just replace the smaller ones with larger ones.

Yeah, since I have 3x4TB drives I could replace those with 8TB drives and have a net gain of 12TB.  Of course 8TB is my limit till I get a larger parity drive.  On the topic of updating a parity drive I'm not sure what the best course of action is.  FlexRAID can actually happily move to a cloned Parity drive.  You could literally just drag and drop an 8TB's drive worth of parity onto a larger drive and rebuild the setup and it should 'just work'.  The downside is that the data must be static while copying, you'd have to stop FlexRAID -entirely- first cause any changes during would break it.  That's a long time to copy 8TB of data. =X  A rebuild would almost be easier since at least the files are accessible while it's rebuilding.

 

But for now, I do have 5 drives I could still add, not even counting potential 4TB drive failures due to age in the meanwhile, so I'm not too worried.

[dalekphalm]

25 minutes ago, AshleyAshes said:

Yeah, since I have 3x4TB drives I could replace those with 8TB drives and have a net gain of 12TB.  Of course 8TB is my limit till I get a larger parity drive.  On the topic of updating a parity drive I'm not sure what the best course of action is.  FlexRAID can actually happily move to a cloned Parity drive.  You could literally just drag and drop an 8TB's drive worth of parity onto a larger drive and rebuild the setup and it should 'just work'.  The downside is that the data must be static while copying, you'd have to stop FlexRAID -entirely- first cause any changes during would break it.  That's a long time to copy 8TB of data. =X  A rebuild would almost be easier since at least the files are accessible while it's rebuilding.

 

But for now, I do have 5 drives I could still add, not even counting potential 4TB drive failures due to age in the meanwhile, so I'm not too worried.

I would only do the copy of parity data if you shut down the FlexRAID service completely (or, as a precautionary measure, even disconnected all of the data drives).

 

Just rebuilding the parity drive from scratch sounds way easier - even if the rebuild time is longer.

[AshleyAshes]

4 hours ago, dalekphalm said:

I would only do the copy of parity data if you shut down the FlexRAID service completely (or, as a precautionary measure, even disconnected all of the data drives).

 

Just rebuilding the parity drive from scratch sounds way easier - even if the rebuild time is longer.

Either way, I wouldn't look at including any 10TB drives till they are cheaper per GB.

But for example, Seagate's STEL10000400 10TB external is $380 CAD vs $240 for the 8TB at Best Buy.  Granted I've gotten the 8TB as cheap as $179 on Black Friday and other sales.  But even for 25% more space, that'd only make it worth $300 rather than $380.  It's a huge step in $/GB for another 2TB even if it's more efficient physically.