External disk parity in small software raid

[The clueless enthusiast]

Hi

 

Im building a small software riad for my increasing bluray backup (plus all other valuable data) the array will be built of 8tb HDDs, in a cooler master silencio 550 (i went to a fiar bit of effort to find a case with 5.25" bays as well as many hdd space). I have 2x 5.25 blueray drives and an 3rd 5.25" bay with an hotswap bay. Hdds go upto 18tb currently in the consumer space, i considered however that it may be useful to create an external parity drive for the array in the pc.

 

The array will be running windows as my backup software will need to be run on this ofc and i dont understand enough about unraid or similiar things to worry about running containers and feeding the hdd space to it etc (seems too complex to worry about tbh)

 

This is more a thought experiment atm but due to the fact that a single large drive is also very expensive- based on the fact the array is unlikely to be updated very often- could an drive be created for parity to keep in as an "external" spare to help rebuild the array should one drive fail?

 

I know most of you will say nas or synology but i really dont need anything like a nas just a vault to protect my data and hold my bluray drives etc

 

May also be worth mentioning this pc i will need to run headlessly which i will be looking into so any advice here would be great (no i dont want a subscription).

 

Any thoughts on this would be welcome.

 

Thanks

 

alex

[LIGISTX]

20 minutes ago, The clueless enthusiast said:

Hi

 

Im building a small software riad for my increasing bluray backup (plus all other valuable data) the array will be built of 8tb HDDs, in a cooler master silencio 550 (i went to a fiar bit of effort to find a case with 5.25" bays as well as many hdd space). I have 2x 5.25 blueray drives and an 3rd 5.25" bay with an hotswap bay. Hdds go upto 18tb currently in the consumer space, i considered however that it may be useful to create an external parity drive for the array in the pc.

 

The array will be running windows as my backup software will need to be run on this ofc and i dont understand enough about unraid or similiar things to worry about running containers and feeding the hdd space to it etc (seems too complex to worry about tbh)

 

This is more a thought experiment atm but due to the fact that a single large drive is also very expensive- based on the fact the array is unlikely to be updated very often- could an drive be created for parity to keep in as an "external" spare to help rebuild the array should one drive fail?

 

I know most of you will say nas or synology but i really dont need anything like a nas just a vault to protect my data and hold my bluray drives etc

 

May also be worth mentioning this pc i will need to run headlessly which i will be looking into so any advice here would be great (no i dont want a subscription).

 

Any thoughts on this would be welcome.

 

Thanks

 

alex

Why not run a parity drive in the array? 
 

Also, I’d definitely look into unraid or truenas. Plenty of tutorials on setting basic SMB shares. No need for containers, you can run the backup software on the client PC’s, the NAS is just a network target. 
 

Fir your setup, I’d likely recommend unraid. Just watch a few YouTube videos, you will see how simple it is. 

[tikker]

Obligatory RAID is not a backup. It's just to protect the data should a drive fail.

24 minutes ago, The clueless enthusiast said:

The array will be running windows as my backup software will need to be run on this ofc and i dont understand enough about unraid or similiar things to worry about running containers and feeding the hdd space to it etc (seems too complex to worry about tbh)

The backup software likely just needs a folder path. It won't care which device it's on or what OS that device is running, only that it can access it and write to it.

24 minutes ago, The clueless enthusiast said:

This is more a thought experiment atm but due to the fact that a single large drive is also very expensive- based on the fact the array is unlikely to be updated very often- could an drive be created for parity to keep in as an "external" spare to help rebuild the array should one drive fail?

Why would an external drive be more attractive than one in the array? An external drive won't change the fact that you'll still need a drive as big as each drive in the array for parity. If you want to use 8 TB drives to their full capacity, then the parity drive will have to be 8 TB or more. You cannot use that drive for anything else either, since parity must reflect what is on the array, so an external solution doesn't make much sense. The parity drive is part of the array. In some RAID(-like) setups the parity may also be striped across the array in which case there isn't even a dedicated parity drive.

[The clueless enthusiast]

32 minutes ago, LIGISTX said:

Why not run a parity drive in the array? 
 

Also, I’d definitely look into unraid or truenas. Plenty of tutorials on setting basic SMB shares. No need for containers, you can run the backup software on the client PC’s, the NAS is just a network target. 
 

Fir your setup, I’d likely recommend unraid. Just watch a few YouTube videos, you will see how simple it is. 

Thanks for the reply, i wasnt thinking of having the external instead of an internal parity more a seperate cold back up parity disk

 

In terms of why the answer i thought would be fairly simple in theory if the array is raid 5 but eventually grows to greater than 18tb (beyond the limit of a single hdd back up- which i will have ofc) i can effectively keep a drive spare with an image of the data that could act as a cold back up to be swapped out should 2 drives fail internally (am i making sense or is this a waste of brain power?) in my head if 2 failed i can swap out one of the failed drives with the cold parity drive and allow the array to rebuild the blank one i replace one of the failed disks  with.

 

31 minutes ago, tikker said:

Obligatory RAID is not a backup. It's just to protect the data should a drive fail.

The backup software likely just needs a folder path. It won't care which device it's on or what OS that device is running, only that it can access it and write to it.

Why would an external drive be more attractive than one in the array? An external drive won't change the fact that you'll still need a drive as big as each drive in the array for parity. If you want to use 8 TB drives to their full capacity, then the parity drive will have to be 8 TB or more. You cannot use that drive for anything else either, since parity must reflect what is on the array, so an external solution doesn't make much sense. The parity drive is part of the array. In some RAID(-like) setups the parity may also be striped across the array in which case there isn't even a dedicated parity drive.

Ill certianly look into it if you believe it best, the two software i use are makemkv (for bluray and 4k films) and aimersoft (a generic dvd ripper) the point of the case was a single machine that can handle it all, i dont really want to have 2 machines if 1 can be built to do both.

Unless, are you saying truenas can run a windows os to run the programs and access the raid array which is running zfs?

[LIGISTX]

7 minutes ago, The clueless enthusiast said:

in my head if 2 failed i can swap out one of the failed drives with the cold parity drive and allow the array to rebuild the blank one i replace one of the failed disks  with.

That’s unfortunately not at all how RAID works. All of the drives need fully up to date data to be used for rebuild. You can’t take an old cold spare that is not reflective of the current array and use it as a means to rebuild the array. 
 

Technically, if your a literal guru, you could. One example is how Linus media groups ZFS array pooped the bed due to bad management of the array, was not recoverable, and literal ZFS guru (Wendell) has more or less help get almost all of the data back - this is not normal, this is not to be expected, and unless you have an extremely deep knowledge of how ZFS works your not going to be able to recover anything. 
 

TLDR; if you use RAID 5 (or whatever software equivalent, in ZFS it would be Z1) if two drives die, your SOL. This is why I run Z2 (raid 6 in normal terms); I have actually had a second drive fail while rebuilding another drive. If I only had Z1, buh bye data. 
 

But, as stated above, the first 2 rules of RAID is “RAID IS NOT A BACKUP”, and the third rule is, you guess it, “RAID IS STILL NOT A BACKUP”. It is used for increased uptime. It can function as a graceful way to retain data when a drive dies, but anything mission critical (your personal data, pictures, documents etc) need to be backup up elsewhere, such as the cloud. Backblaze is extremely affordable and offers unlimited backup for like 6 bucks a month. Use your own encryption key for more security, and pump all of your important data up to the cloud! 

[tikker]

1 hour ago, The clueless enthusiast said:

Thanks for the reply, i wasnt thinking of having the external instead of an internal parity more a seperate cold back up parity disk

 

In terms of why the answer i thought would be fairly simple in theory if the array is raid 5 but eventually grows to greater than 18tb (beyond the limit of a single hdd back up- which i will have ofc) i can effectively keep a drive spare with an image of the data that could act as a cold back up to be swapped out should 2 drives fail internally (am i making sense or is this a waste of brain power?) in my head if 2 failed i can swap out one of the failed drives with the cold parity drive and allow the array to rebuild the blank one i replace one of the failed disks  with.

You may be confusing a backup of the data with parity. What parity tells you is how to reconstruct the data when parts are missing, but in order to do that it must be up to date at all times. It basically indicates if the sum of 1s and 0s over all drives, at a specific location, is even or odd. Say you have three data drives and one parity drive. If the data is 1, 0, 1 then the parity bit could be 0 to indicate even. If the first drive fails you are left with X, 0, 1, but since you know X + 0 + 1 must be even (as the parity bit was 0) you can reconstruct that X=1. Now imagine me asking you whether X + 0 + 1 was even or odd yesterday with the knowledge that 4 months ago it was odd (your cold spare parity drive). It's impossible to say for sure what it was yesterday, because the cold spare isn't up-to-date with the data on the array.

 

What parity or RAID offer you is protection against data loss in case of drive failure and if needed keep the system going in the mean time. Should a drive fail, you can replace the failed drive and reconstruct what was on it using the parity information. If more drives fail than the parity is set up to replace then data loss still occurs, however. If you have RAID5 and two drives fail your data is gone, because such a setup is for a single drive failure. That is why RAID isn't a backup. If you want a backup of your blu-ray rips, then you should make a separate copy of it all. You can use a RAID(-like) setup to protect either or both the original data and the backup.

1 hour ago, The clueless enthusiast said:

Ill certianly look into it if you believe it best, the two software i use are makemkv (for bluray and 4k films) and aimersoft (a generic dvd ripper) the point of the case was a single machine that can handle it all, i dont really want to have 2 machines if 1 can be built to do both.

Unless, are you saying truenas can run a windows os to run the programs and access the raid array which is running zfs?

It depends a bit on what you want to do with it. If it just needs to be storage and nothing else (like serving the blu-rays somewhere else) then you could have it on once PC. The nice thing about a separate NAS is that it's it separate entity. You would rip your discs on your PC, then copy them over to the NAS when done. The benefit of that is that if your PC goes tits up for whatever reason, your blu-rays remain accessible and unaffected.

[The clueless enthusiast]

1 hour ago, tikker said:

You may be confusing a backup of the data with parity. What parity tells you is how to reconstruct the data when parts are missing, but in order to do that it must be up to date at all times. It basically indicates if the sum of 1s and 0s over all drives, at a specific location, is even or odd. Say you have three data drives and one parity drive. If the data is 1, 0, 1 then the parity bit could be 0 to indicate even. If the first drive fails you are left with X, 0, 1, but since you know X + 0 + 1 must be even (as the parity bit was 0) you can reconstruct that X=1. Now imagine me asking you whether X + 0 + 1 was even or odd yesterday with the knowledge that 4 months ago it was odd (your cold spare parity drive). It's impossible to say for sure what it was yesterday, because the cold spare isn't up-to-date with the data on the array.

 

What parity or RAID offer you is protection against data loss in case of drive failure and if needed keep the system going in the mean time. Should a drive fail, you can replace the failed drive and reconstruct what was on it using the parity information. If more drives fail than the parity is set up to replace then data loss still occurs, however. If you have RAID5 and two drives fail your data is gone, because such a setup is for a single drive failure. That is why RAID isn't a backup. If you want a backup of your blu-ray rips, then you should make a separate copy of it all. You can use a RAID(-like) setup to protect either or both the original data and the backup.

It depends a bit on what you want to do with it. If it just needs to be storage and nothing else (like serving the blu-rays somewhere else) then you could have it on once PC. The nice thing about a separate NAS is that it's it separate entity. You would rip your discs on your PC, then copy them over to the NAS when done. The benefit of that is that if your PC goes tits up for whatever reason, your blu-rays remain accessible and unaffected.

I do know the difference, the point of raid was that i will need a solution that is already bigger than most HDDs and as they get alot more costly per terrabyte i wanted something reasonably cost effective to act as storage so a windows storage pool acting in raid 5/6 seemed the obvious choice. I used to have a usb enclosure for my 5.25 buray drive but i found this too combersome and as the enclosure began to fail i thought it best to move onto something more permanent.

 

As im sure your aware not many cases have 5.25 bays anymore and my phanteks enthoo evolv is no esception. But yes your correct that mostly what im looking for is a storage array and some protection would be good (the case i has 8 bays so i guess investing in raid 6 would be no big deal, good redundancy and still space to grow)

[tikker]

20 minutes ago, The clueless enthusiast said:

I do know the difference, the point of raid was that i will need a solution that is already bigger than most HDDs and as they get alot more costly per terrabyte i wanted something reasonably cost effective to act as storage so a windows storage pool acting in raid 5/6 seemed the obvious choice. I used to have a usb enclosure for my 5.25 buray drive but i found this too combersome and as the enclosure began to fail i thought it best to move onto something more permanent.

I understand the choice of a RAID array, just wanted to reiterate that an parity drive is intrinsic to your array and can't (shouldn't) be an external cold spare thing. I've never used Windows storage spaces myself, I run Unraid, but sounds like it should work.

29 minutes ago, The clueless enthusiast said:

As im sure your aware not many cases have 5.25 bays anymore and my phanteks enthoo evolv is no esception. But yes your correct that mostly what im looking for is a storage array and some protection would be good (the case i has 8 bays so i guess investing in raid 6 would be no big deal, good redundancy and still space to grow)

Yeah 5.25 bays are pretty much a thing of the past. Whatever storage solution you go for, do read up how expansion works. Not all solutions allow for easily adding drives to expand capacity.