Raid 5 with Mirroring - Possible?

[sivasunken]

I have 10 3TB WD Red NAS Drives. I am planning to use them in Raid, preferably in RAID5 for 5 disks and then mirror the same across the other 5. Is this possible? If so, how is it achievable in Linux (preferably Ubuntu)?

 

Thanks in Advance

[Votivee]

Would raid 10 be a better option?

[Windows7ge]

You can pick a software RAID solution. I like ZFS so you can install ZFS and then use zpool create to setup the pool. I find the parity performance of ZFS isn't bad especially if you're only on a 1Gbit network. Might as well use parity to keep the additional usable disk space.

[sivasunken]

15 minutes ago, Windows7ge said:

You can pick a software RAID solution. I like ZFS so you can install ZFS and then use zpool create to setup the pool. I find the parity performance of ZFS isn't bad especially if you're only on a 1Gbit network. Might as well use parity to keep the additional usable disk space.

I am already using a FreeNAS system which uses ZFS and a larger storage pool in that. This is for my home lab/media server/cloud. This is going to keep my media and cloud information and will backup critical information across the NAS and BackBlaze. Just a curiosity to see how the community responds to my question.

[Windows7ge]

49 minutes ago, sivasunken said:

I am already using a FreeNAS system which uses ZFS and a larger storage pool in that. This is for my home lab/media server/cloud. This is going to keep my media and cloud information and will backup critical information across the NAS and BackBlaze. Just a curiosity to see how the community responds to my question.

It's a popular file system which isn't limited to FreeNAS. If this is just going to be for general data storage RAID51 (what you're asking for) should be fine. If you plan to use it for virtualization you'll be much better off with RAID10. Outside of what you plan to use it for ZFS & SAMBA can be installed on Ubuntu. If you want something else you can wait and see what other people come up with.

[brwainer]

31 minutes ago, Windows7ge said:

It's a popular file system which isn't limited to FreeNAS. If this is just going to be for general data storage RAID50 (what you're asking for) should be fine. If you plan to use it for virtualization you'll be much better off with RAID10. Outside of what you plan to use it for ZFS & SAMBA can be installed on Ubuntu. If you want something else you can wait and see what other people come up with.

He’s actually asking for RAID51 not RAID50.

[Windows7ge]

23 minutes ago, brwainer said:

He’s actually asking for RAID51 not RAID50.

Whoops, error on my part  still doable with ZFS.

 

Scratch that. Researching the command I actually don't think it's possible. raidz is already providing fault tolerance. It'd actually be quite the waste to mirror a raidz inside the same pool were it possible. You could in theory create two independent vdevs or raidz pools and attach one to the other but if anything were to go wrong within the pool or the system as a whole the failure could take out data on both which makes this kind of double resiliency pointless.

 

You'd be better off running two independent servers each with their own 5 drive raidz then configuring a replication task to mirror the data between the two over a network.

 

If the goal were to increase performance & usable storage (probably why my mind defaulted to RAID50) it'd make more sense which should be possible using ZFS.

[leadeater]

14 hours ago, Windows7ge said:

Scratch that. Researching the command I actually don't think it's possible. raidz is already providing fault tolerance. It'd actually be quite the waste to mirror a raidz inside the same pool were it possible. You could in theory create two independent vdevs or raidz pools and attach one to the other but if anything were to go wrong within the pool or the system as a whole the failure could take out data on both which makes this kind of double resiliency pointless.

Yea actually using RAID 51 and 61 is very rare, hardware RAID can do it easily but it's highly questionable as to why one would do it. Just use RAID 6 instead with hot spares.

 

Normally you'd have a shelf with 24 bays and have 22 in RAID 6 (or even triple parity) with 2 hot spares, then you add more trays configured the same in to the pool/aggregate creating a striped parity group. This way it's easy to expand and you know that each shelf is it's own vdev/RAID group which makes it easier to not make an operator error by removing too many disks from a disk group taking it offline causing data loss.

[NelizMastr]

16 hours ago, Windows7ge said:

You can pick a software RAID solution. I like ZFS so you can install ZFS and then use zpool create to setup the pool. I find the parity performance of ZFS isn't bad especially if you're only on a 1Gbit network. Might as well use parity to keep the additional usable disk space.

I use ZFS using a RAIDZ1 across 3 480GB SSDs in a test server machine, and it's quite good. I get 400MB/s+ sequential writes. Across disks it was better than a software RAID like in Windows Storage Spaces, but not by a ton.

[dalekphalm]

17 hours ago, sivasunken said:

I have 10 3TB WD Red NAS Drives. I am planning to use them in Raid, preferably in RAID5 for 5 disks and then mirror the same across the other 5. Is this possible? If so, how is it achievable in Linux (preferably Ubuntu)?

 

Thanks in Advance

Yes, you should be able to do this via MDADM in Linux. ZFS should also in theory be capable of doing this.

 

First you'd create the two RAID5 arrays as per normal in MD RAID, then, you'd create a RAID1 using the names of the two RAID5 arrays you created in the first step. I doubt you'll find any guides on how to do this though, since even when RAID5 was super popular, a RAID51 array was super niche.

 

My main question is why, though?

 

If this is to backup your data? Just stop right there. RAID is not a backup. Never was, never will be. It doesn't protect against accidental deletions, malware infections, etc. RAID is for hardware redundancy, meaning it's designed simply to reduce downtime in the event a drive dies.

 

You'd be better off just creating one larger RAID6 with the entire disk set - or a RAID10 array if you want higher performance and don't mind losing half the capacity.

[Windows7ge]

1 hour ago, NelizMastr said:

I use ZFS using a RAIDZ1 across 3 480GB SSDs in a test server machine, and it's quite good. I get 400MB/s+ sequential writes. Across disks it was better than a software RAID like in Windows Storage Spaces, but not by a ton.

Read/write performance using parity in Windows Storage Spaces is terrible. If the goal is to maximise performance and maintain redundancy RAID10 is your only option. And I'm aware the parity performance of ZFS is more than good enough for a 10Gig network. I've done that myself. I'm just emphasizing that if he's only on a 1Gig it's all the more reason to not be bothered by what performance is lost by not going with RAID10.

 

 

[NelizMastr]

9 minutes ago, Windows7ge said:

Read/write performance using parity in Windows Storage Spaces is terrible. If the goal is to maximise performance and maintain redundancy RAID10 is your only option. And I'm aware the parity performance of ZFS is more than good enough for a 10Gig network. I've done that myself. I'm just emphasizing that if he's only on a 1Gig it's all the more reason to not be bothered by what performance is lost by not going with RAID10.

Never disputed anything you said, I fully agree. Just wanted to add my experience with the matter

[Windows7ge]

21 minutes ago, NelizMastr said:

Never disputed anything you said, I fully agree. Just wanted to add my experience with the matter

I couldn't tell if you were or not. Most people who reply to me reply because they disagree.

[Electronics Wizardy]

1 hour ago, Windows7ge said:

Read/write performance using parity in Windows Storage Spaces is terrible. If the goal is to maximise performance and maintain redundancy RAID10 is your only option. And I'm aware the parity performance of ZFS is more than good enough for a 10Gig network. I've done that myself. I'm just emphasizing that if he's only on a 1Gig it's all the more reason to not be bothered by what performance is lost by not going with RAID10.

 

 

parity speeds seem to have gotten better with 2019 for me. Im getting around 200mB/s write with parity on hdds in server 2019, and thats pretty usable to me.

[leadeater]

5 hours ago, Windows7ge said:

I couldn't tell if you were or not. Most people who reply to me reply because they disagree.

I disagree that most people who reply to you is because they disagree with you 

[Windows7ge]

1 hour ago, leadeater said:

I disagree that most people who reply to you is because they disagree with you 

Agggh, I see what you did there. Disagreeing with me. Well what about that one guy who said "You don't know what you're talking about." because I gave OP a vague answer? (Not this topic)

[brwainer]

1 hour ago, Windows7ge said:

Agggh, I see what you did there. Disagreeing with me. Well what about that one guy who said "You don't know what you're talking about." because I gave OP a vague answer? (Not this topic)

I am replying to you to agree with you, but also to say this is off-topic.

 

The consensus of everyone who has replied so far is that making a mirror comprised of two RAID 5 arrays is not very useful. I agree with this, but I'll add some details as to why:

• A backup is a separate, not-instantly-replicated, copy of your data. Best practices say to have 2 backups, one of which is offsite. 3-2-1: 3 copies of your data, 2 onsite and 1 offsite. Anything that automatically replicates all changes does not count as a backup, because it offers no protection against accidental deletion or cryptolocker malware. A periodic sync that maintains at least one changed version can be a backup, depending on how many versions it tracks and how long it keeps them.
• Recovering your data from a backup is a pain, therefore RAID exists. One meaning of RAID is "Redundant Array of Inexpensive Disks" - meaning instead of buying one large disk that will fail eventually and require you to recover from backups, you can create an array of smaller disks so that when one fails, you can replace it and repair the array while it still provides access to the data, meaning that to the users of the data there is no downtime (but it will be slower while rebuilding)
• RAID1 and RAID5 provide protection against one drive dying, however experience has shown that during the rebuild of a large array, a second drive is likely to die, meaning the array is lost and the data needs to be recovered from backups. This is because the rebuild process is hard on the remaining drives, and drives in an array tend to be of the same age and possibly from the same batch.
• RAID6 provides protection against two drives dying, to make the above issue with RAID1 and RAID5 less likely to occur.
• RAID10 or RAID1+0 (RAID0 or stripe made up of two or more RAID1 arrays) and RAID 50 (RAID0 made up of two or more RAID5 arrays) can potentially withstand one lost drive per inner array, but if two drives in the same inner array are lost then all the data in the entire set is lost.
• RAID60 can withstand two lost drives per inner array.
• Your proposed "RAID51" (RAID1 or mirror made up of two or more RAID5 arrays) would be able to lose all of the drives in one array plus up to one drive in the other array, however it cannot withstand two lost drives in both inner arrays at the same time. This makes it less useful than RAID60, and barely better than RAID50.

Usable capacity assuming 10x 2TB drives with 2 inner arrays [formula for x drives of y size, with z inner arrays]:
RAID50: 16TB (2TB * 4 *2) [y * ((x/z) -1) * z]

RAID60: 12TB (2TB * 3 *2) [y * ((x/z) -2) * z]

RAID51: 8TB (2TB * 4 *1) [y * ((x/z) -1) * 1]

 

Therefore, RAID51 provides less protection than RAID60, as well as less storage space. Additionally, comparing the stripe versus mirror at the top, read speeds should be equal, but RAID60 would be twice as fast at writing than RAID51.

[Windows7ge]

10 minutes ago, brwainer said:

-snip-

I agree with brwainer's summary. If you already planned to abandon 50% of your pool to redundancy you really may as well opt for RAID60. Better performance, more usable storage, and pool resiliency would still be optimal.

[sivasunken]

Thanks for the feedback. I know that RAID is not a backup. My backup plan was to use FreeNAS system and BackBlaze. This is going to be my Home Server with Plex/Media, Home Automation, Webserver and Security System Monitoring. I guess i shall go with RAID 6 or 60. 

Thanks for you responses so far

[Windows7ge]

2 hours ago, sivasunken said:

Thanks for the feedback. I know that RAID is not a backup. My backup plan was to use FreeNAS system and BackBlaze. This is going to be my Home Server with Plex/Media, Home Automation, Webserver and Security System Monitoring. I guess i shall go with RAID 6 or 60. 

Thanks for you responses so far

I believe you said you have a FreeNAS box correct? FreeNAS supports both replication via SSH & RSYNC both of which can be used with any Debian distro (maybe others as well like Arch/RHEL I don't know) of Linux. Regardless of how you form the RAID6/60 or other (hardware/software, combination of the two) you can use this to sync the data between the servers. It would save you the cost of an off-site backup service.