Large (60TB?) NAS questions

[Syntaxvgm]

I want to do 10 6TB drives, 2 or 3 drive parity. 

Everyone and their mother uses freeNAS, which requires 1GiB of memory per TB of storage, which is doable with 4 16GB sticks, but I have no experience with freenas an ZFS, only hardware based RAID6 for something like this. 

-What do you suggest?

-Is ZFS Parity raid susceptible to fucking up on unexpected power loss like other software raid solutions? I do intend to have a good UPS, but I'm concerned about this, as I know nothing about ZFS. I will research more before making a final decision, but for now I'm just considering options 

-Is RAIDZ3 (triple parity) worth trying? I hear it's a newer thing, and I don't want to use anything half baked. But It does interest me. 

-How messy are RAIDZ2 rebuilds in the case of 2 failed drives? Is this a reason to consider RAIDZ3 because of the high number of high capacity drives? 

-Does FreeNas rely on direct smart data access? 

-Does FreeNas Support having a hot spare?

-Can I script the system to safely power down upon a single drive failure, failure to read/write, or SMART errors? 

 

Thanks

[vanished]

Linus has said he's really been liking unRAID lately... have you given that any though?

[Syntaxvgm]

8 minutes ago, Ryan_Vickers said:

Linus has said he's really been liking unRAID lately... have you given that any though?

It's a consideration for a mixed-drive and expandable solution, but I'm not sold on it. Don't really know way too much about it, and people seem to love it or scoff at it. I'm not even sure how their raid solutions work. 

[failblox]

I don't have any experience with RAID or ZFS, unfortunately, but from what I've heard, RAID has certain disadvantages that ZFS is supposed to improve on (one specific example I can name off the top of my head is bitrot).

[scottyseng]

1 hour ago, Syntaxvgm said:

-snip-

I probably would vote for FreeNAS...Unraid is pretty good as well. If you have Windows, Storage Spaces is a option

 

Yes, all RAID (software / hardware without the on RAID card battery) does not play well with sudden power loss. ZFS especially, because it relies heavily on the RAM. I would say to get a UPS for sure.

 

FreeNAS does rely on direct SMART access, so you need either onboard motherboard SATA ports or need to use a HBA (Not RAID) card to hook up your drives.

 

The rest of the questions I don't know about. I'm on the hardware RAID side of things and my knowledge of FreeNAS is pretty limited.

[Syntaxvgm]

6 hours ago, scottyseng said:

I probably would vote for FreeNAS...Unraid is pretty good as well. If you have Windows, Storage Spaces is a option

 

Yes, all RAID (software / hardware without the on RAID card battery) does not play well with sudden power loss. ZFS especially, because it relies heavily on the RAM. I would say to get a UPS for sure.

 

FreeNAS does rely on direct SMART access, so you need either onboard motherboard SATA ports or need to use a HBA (Not RAID) card to hook up your drives.

 

The rest of the questions I don't know about. I'm on the hardware RAID side of things and my knowledge of FreeNAS is pretty limited.

this is my worry. I do not want problems on power loss if I can avoid it, 2 reasons- I lose power frequently (will have a nice UPS), and I don't want 40+TB of data to at risk for a single component failure. That sounds crazy. I'm paranoid enough to be planning multiple PSUs and potentially 2 small UPSes on different line from my breaker. . Not only have I seen power supplies take drive controller boards with them, my house has really nasty power, it's already taken victims. I'm not sure if a single module of a redundant psu fails it reduces the risk to components, but I think it does. Will do research. 

I need to research this. I DO NOT care if data being written while the power is lost is gone, as long as the integrity of the array and the existing data is maintained.  

While the most important data, probably 10TB worth give or take, will be in multiple locations (probably a bank deposit box), I don't want to loose any of it if I can avoid it. 

[KuJoe]

I use RAID5 for a production storage server and it's fine even with a cheap RAID card, RAID6 has better redundancy and RAID50 has better redundancy and performance. I personally have no experience with ZFS or any of that other stuff and trust plain old RAID for all of my stuff. If you want to find a good build guide for a high capacity storage server, do a search for "Backblaze Pod" on Google and there's a few different versions out there that they document each hardware piece and configuration. Of course their pods hold a lot more than 10 drives, but you can scale it down to fit your needs.

 

 

[Syntaxvgm]

Just now, KuJoe said:

I use RAID5 for a production storage server and it's fine even with a cheap RAID card, RAID6 has better redundancy and RAID50 has better redundancy and performance. I personally have no experience with ZFS or any of that other stuff and trust plain old RAID for all of my stuff. If you want to find a good build guide for a high capacity storage server, do a search for "Backblaze Pod" on Google and there's a few different versions out there that they document each hardware piece and configuration. Of course their pods hold a lot more than 10 drives, but you can scale it down to fit your needs.

 

 

45 Drives was a consideration. But given I want 10 drives and 40+TB of usable should last me until it's time to upgrade. 

[Syntaxvgm]

6 hours ago, scottyseng said:

I probably would vote for FreeNAS...Unraid is pretty good as well. If you have Windows, Storage Spaces is a option

 

Yes, all RAID (software / hardware without the on RAID card battery) does not play well with sudden power loss. ZFS especially, because it relies heavily on the RAM. I would say to get a UPS for sure.

 

FreeNAS does rely on direct SMART access, so you need either onboard motherboard SATA ports or need to use a HBA (Not RAID) card to hook up your drives.

 

The rest of the questions I don't know about. I'm on the hardware RAID side of things and my knowledge of FreeNAS is pretty limited.

Ok, little reading before I leave here and it turns out that ZFS uses copy on write, which should prevent the array taking a shit on power loss, and people claim to have no problem with unplugging it in tests. A bit of conflicting information out there really, as people mention it as a risk (Maybe it's just a blanket statement for software raid?)

[Aekim]

Unraid only has only one parity drive, so I would go with freenas or Windows.

Freenas would have better performance (storage pools have poor write speeds) but probably is not as easy to setup as Windows.

 

 

 

[leadeater]

Just now, Aekim said:

Unraid only has only one parity drive, so I would go with freenas or Windows.

Freenas would have better performance (storage pools have poor write speeds) but probably is not as easy to setup as Windows.

Only parity configurations with Storage Spaces have bad write performance. My 4x 850 Pro simple pool gets 2GB/s read/write and my 2x 840 Pro + 5x 3TB two-way mirror Auto Tier pool gets 1GB/s read and 500MB/s write.

[Syntaxvgm]

I would like to mention read/write performance is not a huge factor here, as long as it's reasonable. 

[KuJoe]

12 minutes ago, Syntaxvgm said:

I would like to mention read/write performance is not a huge factor here, as long as it's reasonable. 

Not to mention if you're network is only 1Gbps then you're limited to about 110MB/s anyways. If you're running RAID5 even with a cheap RAID card you'll be fine (I have a heavily populated server with 6x 2TB drives in RAID5 with a really old Dell PERC5 and it's doing 135MB/s write speeds and close to 6000 IOPS without a problem).

[Syntaxvgm]

5 minutes ago, KuJoe said:

Not to mention if you're network is only 1Gbps then you're limited to about 110MB/s anyways. If you're running RAID5 even with a cheap RAID card you'll be fine (I have a heavily populated server with 6x 2TB drives in RAID5 with a really old Dell PERC5 and it's doing 135MB/s write speeds and close to 6000 IOPS without a problem).

That's about the minimum target speed. I will be the only one to use it internally, and I while I would like more than ~100MB/s, that's only for local drive imaging via hot swap bays. I will probably add multiple 1Gb/s nics with link aggregation if the performance IS faster, so I can do fast transfers without hot using the hot swap bays.

[Syntaxvgm]

I didn't realize there was a new section for 'servers and NAS', so I'm gonna have a mod move it. 

[scottyseng]

7 hours ago, Syntaxvgm said:

Ok, little reading before I leave here and it turns out that ZFS uses copy on write, which should prevent the array taking a shit on power loss, and people claim to have no problem with unplugging it in tests. A bit of conflicting information out there really, as people mention it as a risk (Maybe it's just a blanket statement for software raid?)

Well, as far I remember, ZFS does rely heavily on the RAM (It trusts the RAM and not the drives). As you imagine, when the power goes out the RAM also goes off. I don't know how ZFS handles it, but I would imagine it's similar to running a hardware RAID card with a RAM cache without a on RAID card battery backup. You'll lose the data that was on the RAM cache that was supposed to be written. It might not take out the array, but you lose what data was supposed to be written back. The worst case scenario would be if you lost power while ZFS was writing something back to the array. Either way, I wouldn't want to take a chance, and I would heavily say to invest in a really good UPS unit first and plug that UPS unit into a really good surge protector. Eaton and APC make really nice UPS units. My knowledge of ZFS is pretty limited, but I wouldn't risk playing with power loss.

 

Yeah, the on board battery on my hardware RAID card saved me multiple times during power loss at my house. It also saved me when I had my RAID configuration inside of my PC, because the AX860i I bought decided to go nuts and randomly turn off / go into a restart loop (I'm sure you can imagine that's like 100% on a hell RAID) because of a faulty temp sensor on the PSU. It also took me two RMAs to get a working PSU, so there were quite a lot of sudden shut downs while waiting to get a working PSU. Every time the power turned off, the RAID controller's onboard battery kicked in. I have the fancier LSI Cachevault system that uses a tiny SSD cache / capacitor bank to store the data from the RAM, and it activated every time (You can tell it's recovering the data because it'll light up blue when the system has power restored.

 

And yes, with 1Gb/s Ethernet, you're crippled to 80-100MB/s (At least for me because my main PC only has one Ethernet port). If I stop downloading stuff, the speed does increase to 110ish MB/s. The RAID array is quite fast at 550MB/s read and write on the server side (Six 4TB WD Reds). I'll invest into 10Gb/s networking eventually, but my priority is to get a UPS and a external drive as a proper backup (Yeah, I've never had a proper backup....I'm surprised I haven't lost any data over the years...).

[leadeater]

7 hours ago, scottyseng said:

Either way, I wouldn't want to take a chance, and I would heavily say to invest in a really good UPS unit first and plug that UPS unit into a really good surge protector. Eaton and APC make really nice UPS units. My knowledge of ZFS is pretty limited, but I wouldn't risk playing with power loss.

A true inline UPS does the surge protection itself and will do a much better job than a standard surge protector, not that it would hurt to have two.

 

For Eaton the difference between the 5 series an 9 series is the number for power protection types it protects you from, funnily enough 5 vs 9. Link below explains them etc.

http://www.upspower.co.nz/component/content/article/38

[scottyseng]

1 hour ago, leadeater said:

A true inline UPS does the surge protection itself and will do a much better job than a standard surge protector, not that it would hurt to have two.

 

For Eaton the difference between the 5 series an 9 series is the number for power protection types it protects you from, funnily enough 5 vs 9. Link below explains them etc.

http://www.upspower.co.nz/component/content/article/38

Oh, really? Yeah, I remember reading online that the inbuilt surge protection on a UPS isn't that great so you'd be better off plugging it into a separate surge protector and using that as a first layer surge protection. I've never actually tried researching it any further though. Very interesting link though.

[leadeater]

3 hours ago, scottyseng said:

Oh, really? Yeah, I remember reading online that the inbuilt surge protection on a UPS isn't that great so you'd be better off plugging it into a separate surge protector and using that as a first layer surge protection. I've never actually tried researching it any further though. Very interesting link though.

Standby/Switching UPS's are generally terrible at everything so that information would be correct for those. Line-Interactive is a set up from this and have better electrical power components in them. Line-Interactive UPS surge protection would be equivalent to most dedicated surge protectors except for the most expensive top tier products that well, actually do protect you from surges unlike everything else which is just for show and gives you the warm and fuzzes.

 

Online Double-Conversion UPS's are the premium choice and come with the cost of such a product. These are better since power to devices is always delivered from the batteries, never directly from the mains unless in bypass to service the battery. The power components are generally much better in Online UPS's too and any surges get dealt with as a normal surge protector does but any extra input power or residual that makes it through only charges the battery. These can pretty much run forever since all parts in them are serviceable/replaceable, but other than the battery almost never need to be.

 

The Eaton 5 series UPS we talked about are Line-Interactive type, the 9 series are Online Double-Conversion. The 9 series are also the only ones I buy for servers and switches, I personally think they are worth the extra for a business and will pay themselves off over the total life of the product compared to lesser products.

[Mikensan]

What do you suggest?

unRaid is great as a backup vault, but you will only ever get the performance of a single drive. Data is not striped across disks so even if 2 drives fail, you only lose data that was on those 2 drives vs the whole pool.

ZFS/BTRFS is great if data integrity is important to you. Various solutions use either or (FreeNAS=ZFS, unRaid=BTRFS etc...)

FreeNAS which strictly uses ZFS now, offers a lot of features at the click of a mouse. zPools will outright die if any of the vdevs die. Careful planning is required.

Nas4Free is a fork of FreeNAS - still offers support for UFS (as well as ZFS) in case you are willing to gamble on data integrity.

Windows - very user friendly, can either use hardware raid or storage spaces depending on your end goal.

 

-Is ZFS Parity raid susceptible to fucking up on unexpected power loss like other software raid solutions? I do intend to have a good UPS, but I'm concerned about this, as I know nothing about ZFS. I will research more before making a final decision, but for now I'm just considering options 

    The entire raid is not, but any active file transfers might have an issue (said file might just simply never make it to disk)
-Is RAIDZ3 (triple parity) worth trying? I hear it's a newer thing, and I don't want to use anything half baked. But It does interest me. 

     You'll need enough horsepower so transfer speeds aren't impacted. You could also look into just doing a Raid10 since you cannot grow vdevs (in case you want to add space in the future).
-How messy are RAIDZ2 rebuilds in the case of 2 failed drives? Is this a reason to consider RAIDZ3 because of the high number of high capacity drives? 

      Time consuming for larger drives especially with two resilvers. I've had a sinlge drive fail, 4TB disk, took less than a day but not sure exactly how long - I didn't babysit it. 
-Does FreeNas rely on direct smart data access? 

      In order to report SMART data, yes.

-Does FreeNas Support having a hot spare?

     Hmmm, not sure honestly lol.
-Can I script the system to safely power down upon a single drive failure, failure to read/write, or SMART errors? 

     You can setup email alerts, and you do have shell access, but I do not see a way within the GUI to do this.

 

 

**I personally user FreeNAS 9.3, very happy with it. I have two pools, one with a single RaidZ1 vdev, the other pool has a single Raidz2 vdev (used for replication of the main pool). Both vDevs are using different brands/models of disks. My little Celeron G1610 doesn't do well with the RaidZ2, I get about 90mbytes per second whereas my RaidZ1 gets 400-500mbytes per second.

 

 

[bobhays]

If you do decide to with freeNAS either use 10 drives and Z2 (2 drive parity), or if you really want 3, then get 11 drives and use 3 drive parity. The reason is that having a power of 2 increases performance (8 is 2^3).