Raid performance with HDDs

[Commander_Dork]

I just want to hear it straight out from someone who has done this before. What is the best raid configuration for HDDs prioritizing speed over redundancy? I hear constantly that HDDs are terrible ideas for NAS but if configured properly they can have good speed? Is it possible to saturate a 2.5gigabit or even 10gigabit connection with the right raid configuration with HDDs?

[AbydosOne]

2 minutes ago, Commander_Dork said:

Is it possible to saturate a 2.5gigabit or even 10gigabit connection with the right raid configuration with HDDs?

You can saturate 2.5G with two HDDs in a mirror (assuming the mirror is smart enough to read as if it's a striped, i.e. read data blocks from each drive in an alternating fashion).

10G would take a little work, but could probably be done consistently with 10-12 (modern, fast, CMR) disks in a RAIDZ2 array (ZFS, which gets you at least a little redundancy).

 

3 minutes ago, Commander_Dork said:

What is the best raid configuration for HDDs prioritizing speed over redundancy?

RAID-0.

 

6 minutes ago, Commander_Dork said:

I hear constantly that HDDs are terrible ideas for NAS but if configured properly they can have good speed?

I don't understand the people who want their NASes to run at 1GBps for any task. NASes (outside of multi-user video editing workflows and a few other) are usually used as glorified external HDDs and content vaults. Most uses of NASes other than moving data around for the sake of moving data around don't need anywhere near 10G ethernet speeds.

I have a pair of mirrored NVMe SSDs in my server for funsies (and 10G networking), but all the mass storage is done on HDDs, and I can still get >200MBps for most tasks.

[tkitch]

2 minutes ago, Commander_Dork said:

I just want to hear it straight out from someone who has done this before. What is the best raid configuration for HDDs prioritizing speed over redundancy? I hear constantly that HDDs are terrible ideas for NAS but if configured properly they can have good speed? Is it possible to saturate a 2.5gigabit or even 10gigabit connection with the right raid configuration with HDDs?

 

Yes, HDDs can absolutely saturate 2.5 or 10gbit with enough drives.

 

Now, here's where it gets sketchy:

 

For pure speed, you'd want to RAID 0 your drives.  That means there's 0 data security.  If one drive dies, they ALL die instantly.  (Well, all data dies, the other drives would still be working fine.  You can probably saturate 10gbit with this on 8-10 drives, give or take.  

 

This is a TERRIBLE idea.

 

Now, assuming you don't want one failure to utterly hose all your data instantly:

 

You need enough drives to get the bandwidth.  2.5gbit?  You can probably do that on a pool of 4-5 drives.

10gbit?  you'd need 10+, if not 15+ to saturate that much bandwidth.

SECOND CAVEAT:  You also need enough CPU Power to process the parity that's getting made for your data.  And the higher the speed, the more CPU Power you need for it to run properly.

 

For 10gbit, you'd need some decent CPU Power if you're doing a ton of data writes constantly.

 

.

 

As for other concerns of yours?

Uhhh, anyone saying HDDs are bad for NAS use?  Uhhhh, they're high.  LTT has Petabytes of HDDs in multiple NAS Servers.  HDDs are *still* the go-to for bulk storage, since SSDs are *expensive* for large data volumes.

 

8TB HDD?  You can do that for 100 bucks on sale.

 

8TB SSD?  Uhhh....  3-400$?  More?  

 

20TB HDD?  250$ or so?

20TB SSD?  Good luck, you're gonna go broke. 

 

For bulk storage, NAS w HDDs is still the way to go.  Anyone saying otherwise is either richer than they should be, or they're just delusional. 

[Eigenvektor]

Just now, AbydosOne said:

10G would take a little work, but could probably be done consistently with 10-12 (modern, fast, CMR) disks in a RAIDZ2 array (ZFS, which gets you at least a little redundancy).

Wouldn't you be limited by the SATA bus, which tops out at a theoretical 6 Gbps?

[johnt]

You can saturate anything with a good enough cache. HDD spinners can come close to saturating a 2.5 line.
 

I think it all depends on your use case. I have RAID1 configured on my server/nas and it works for me. I don’t plan to do any editing from my nas. However you could also do different drives. For example use flash storage for active/temporary projects and spinners for long term repository. However, many people will say RAID is not meant to be a back up. 

but if you want the best of both worlds, speed and minimal redundancy, then you need to consider four drives in RAID10. Otherwise RAID0 is for speed but not usually a great idea. 

[AbydosOne]

1 minute ago, Eigenvektor said:

Wouldn't you be limited by the SATA bus, which tops out at a theoretical 6 Gbps?

Each individual drive gets 6Gbps; the total bandwidth would be limited by the controller's link to the CPU. ZFS can spread this bandwidth over multiple controllers.

[manikyath]

7 minutes ago, Commander_Dork said:

prioritizing speed over redundancy?

RAID0.

 

8 minutes ago, Commander_Dork said:

I hear constantly that HDDs are terrible ideas for NAS

does anyone ever say this?

 

8 minutes ago, Commander_Dork said:

Is it possible to saturate a 2.5gigabit or even 10gigabit connection with the right raid configuration with HDDs?

striping a LOT of disks.. or realisticly, striping a LOT of RAID6 arrays, because at this number of disks, disk failure is a routine thing.

 

realisticly, with sequential reads, you'll do 2.5G fast enough, and 10G is just 4x that.. but the moment seek times come into play, spinning rust struggles hard.

 

1 minute ago, Eigenvektor said:

Wouldn't you be limited by the SATA bus, which tops out at a theoretical 6 Gbps?

SATA isnt a bus, it's 6Gbps to each port, you might reach PCIe speed limitations depending on the controller though.

[Commander_Dork]

1 hour ago, tkitch said:

 

Yes, HDDs can absolutely saturate 2.5 or 10gbit with enough drives.

 

Now, here's where it gets sketchy:

 

For pure speed, you'd want to RAID 0 your drives.  That means there's 0 data security.  If one drive dies, they ALL die instantly.  (Well, all data dies, the other drives would still be working fine.  You can probably saturate 10gbit with this on 8-10 drives, give or take.  

 

This is a TERRIBLE idea.

 

Now, assuming you don't want one failure to utterly hose all your data instantly:

 

You need enough drives to get the bandwidth.  2.5gbit?  You can probably do that on a pool of 4-5 drives.

10gbit?  you'd need 10+, if not 15+ to saturate that much bandwidth.

SECOND CAVEAT:  You also need enough CPU Power to process the parity that's getting made for your data.  And the higher the speed, the more CPU Power you need for it to run properly.

 

For 10gbit, you'd need some decent CPU Power if you're doing a ton of data writes constantly.

 

.

 

As for other concerns of yours?

Uhhh, anyone saying HDDs are bad for NAS use?  Uhhhh, they're high.  LTT has Petabytes of HDDs in multiple NAS Servers.  HDDs are *still* the go-to for bulk storage, since SSDs are *expensive* for large data volumes.

 

8TB HDD?  You can do that for 100 bucks on sale.

 

8TB SSD?  Uhhh....  3-400$?  More?  

 

20TB HDD?  250$ or so?

20TB SSD?  Good luck, you're gonna go broke. 

 

For bulk storage, NAS w HDDs is still the way to go.  Anyone saying otherwise is either richer than they should be, or they're just delusional. 

Okay cool! Supposing I wanted to have a bit of redundancy like, a drive or two go down and I can just power down the server replace the drives and keep it going? I've heard about Vdevs is it possible to have  Vdev with raid 0 on the drives and then have it backing up to another Vdev as well? so if it a drive in the primary VDEV fails, I can power down replace it and then restore the data from the backup Vdev? if that makes any sense lol

[Commander_Dork]

1 hour ago, manikyath said:

does anyone ever say this?

it was in a previous forum post, judging by your responses imma guess I might've misunderstood or they didn't realize I was thinking of running them in a raid config? idk lol

[tkitch]

2 minutes ago, Commander_Dork said:

Okay cool! Supposing I wanted to have a bit of redundancy like, a drive or two go down and I can just power down the server replace the drives and keep it going? I've heard about Vdevs is it possible to have  Vdev with raid 0 on the drives and then have it backing up to another Vdev as well? so if it a drive in the primary VDEV fails, I can power down replace it and then restore the data from the backup Vdev? if that makes any sense lol

well, running a RAID 0 and duplicating it to another is called:  RAID 0+1 

But it's very inefficient.  Just do Raid 5 or 6 at that point.  

[AbydosOne]

6 minutes ago, Commander_Dork said:

I've heard about Vdevs

VDEVs are a part of ZFS, for clarification.

 

6 minutes ago, Commander_Dork said:

is it possible to have  Vdev with raid 0 on the drives and then have it backing up to another Vdev as well?

You could configure your OS to run periodic backups, but there isn't an inbuilt ZFS layout for that.

In spirit, it's like RAIDZ1/2/3, which have 1/2/3 disks of redundancy and read performance similar to pure RAID-0 (roughly scaling with (N - n), where N is number of disks and n is RAIDZ level). For more info, read up on Parity disks.

[Commander_Dork]

1 minute ago, tkitch said:

well, running a RAID 0 and duplicating it to another is called:  RAID 0+1 

But it's very inefficient.  Just do Raid 5 or 6 at that point.  

oh ok thanks for the help! If I decide on raid 5 or 6 could I get away with like an i5 6500?

[tkitch]

2 minutes ago, Commander_Dork said:

oh ok thanks for the help! If I decide on raid 5 or 6 could I get away with like an i5 6500?

should be plenty

[da na]

3 minutes ago, Commander_Dork said:

oh ok thanks for the help! If I decide on raid 5 or 6 could I get away with like an i5 6500?

Yeah absolutely.

 

I hit 850mbps read/writes on just four 15,000RPM 2.5" SAS drives. Those are excellent options if you just need raw speed from HDDs and capacity per drive is not a concern; each of mine are only 143gb. At that rate, you could saturate gigabit Ethernet with 6 of those drives, and only two of them would max out 2.5gb.

[LIGISTX]

12 minutes ago, Commander_Dork said:

oh ok thanks for the help! If I decide on raid 5 or 6 could I get away with like an i5 6500?

RAID 5 or 6 is not the right terminology for "vdevs". vdevs are part of the ZFS architecture which is a software RAID solution (its a fantastic solution, I have been running ZFS via truenas for almost a decade).

 

I would advise you do A LOT more research before you buy or impliment anything. I would look in the truenas forums for beginner guides and explinations of whats what so you can get a better idea of pitfalls that are common, issues people run into, and gain a better understanding of what all is going on. These things are very important to understand up front because if you make config setups in the beginig, once you have the data populated, its really difficult to change things later since you will need to offload your data and start over...

 

This forum definitely can help a lot, but to get the mots out of said help, you need to do a good bit of reserach and homework on your own so we can all be speaking the same language per say :).

 

Things to understand first are:

• How much storage space do you think you need?
  • how long will it take to fill this up, and how do you plan to increase capacity?
• how much money do you want to spend
• what hardware do you already have
• what is ZFS
  • what are vdev's
• look into unraid so you know what your two main options are

[Commander_Dork]

1 minute ago, LIGISTX said:

RAID 5 or 6 is not the right terminology for "vdevs". vdevs are part of the ZFS architecture which is a software RAID solution (its a fantastic solution, I have been running ZFS via truenas for almost a decade).

 

I would advise you do A LOT more research before you buy or impliment anything. I would look in the truenas forums for beginner guides and explinations of whats what so you can get a better idea of pitfalls that are common, issues people run into, and gain a better understanding of what all is going on. These things are very important to understand up front because if you make config setups in the beginig, once you have the data populated, its really difficult to change things later since you will need to offload your data and start over...

 

This forum definitely can help a lot, but to get the mots out of said help, you need to do a good bit of reserach and homework on your own so we can all be speaking the same language per say :).

 

Things to understand first are:

• How much storage space do you think you need?
  • how long will it take to fill this up, and how do you plan to increase capacity?
• how much money do you want to spend
• what hardware do you already have
• what is ZFS
  • what are vdev's
• look into unraid so you know what your two main options are

There's @LIGISTX!! I was hoping you'd save me again LOL almost every post I make your advice is always easy to understand and super helpful! thanks dude!

[wseaton]

On average your vanilla 8TB 7200 spinner is going to read about 130MBps....assuming it's data written without the drive being too full. Some drives are faster but most are slower.

 

In theory 3 drives in RAID 0 would be able to saturate a 2.5Gb link given each drive can do about 1Gb.

 

However, we have other factors to deal with, like SMB over head, if it' one large files or bunches of smaller ones, etc. Your client device would also have to be able to receive data that fast, but any SATA SSD should be able to write ~500MBps.

 

On a SAN we just devote a lot of spinners on the LUN, but it's really inefficient. The lower latency of SSD is so much easier to crank higher Xfer rates and still have fault tolerance. 

 

Don't get me going on the trash reliability of 15k drives. So glad that tech is dead.