NAS Backups and Hard Drive Duplication

[Benny Biscuit]

Ok so I'm new to the forum but have been watching Linus videos for several years now. My boyfriend is the techy guy in our home. He has built all of our gaming computers as well as our server. When it comes to backing up our server I get so confused by the things he tells me. And he seems to be having a difficult time picking the best back-up solution that meets his needs.

 

Our server is used for media ONLY. All of our movies, TV shows, music, etc. We run the server and then use it with Plex to access all of this media. But only use IS media. We have separate PC's/laptops for gaming and anything else. 

 

First of all we don't have an external backup, which I think is stupid because if the house Burns down, everything is gone. Everything is backed up internally. There are 9 hard drives totaling 47 terabytes. For every piece of media on the hard drive, there is an exact duplicate copy of it on a separate hard drive. Our media totals about 22 terabytes of space and there is 22 more terabytes of ALL of that media, backed up onto separate drives. So it's redundant storage within the same computer but on different drives. 

 

So every time we add something new, whether it be a new show, a new television episode, or new music we need an exact copy of whatever it was, backed up onto the appropriate drive. 

 

To me, the best solution is to build another server and back it up in a separate location. Because I don't think it should all be backed up internally. However, this is currently the way he has it setup. I've looked into so many solutions but can't quite find what works best. There are hard drive duplicators, which are external, there are programs like native windows programs, or RAID or the various others. We don't want something that will run constantly. But we do need a good solution to duplicate the media onto appropriate drives so they when he clicks on the drives, they match up byte for byte. 

 

So there are 9 drives total: 3 drives have all the media, 3 more have an EXACT copy of all that media (they are cloned.) There is then a 2TB drive which is for our cameras and doesn't need backed up. There is a 3TB drive that is a backup for various items around the house (gaming console hard drives.) The 9th drive, I believe is the boot drive. But the 6 drives with the media need to be matched up, every time we add something to the 3 that Plex regularly accesses

 

Can someone PLEASE explain this to me, because I feel like there is a MUCH better solution which is faster, doesn't require removing the drives to duplicate them, and doesn't require sacrificing a butt load of time once a week or once a month to copy and make the changes. He keeps saying it's not that simple but to me, it seems like there SHOULD be a simple solution?

[emosun]

well if you're worried about fire just put the server you have in a fire box , i imagine it would cost a lot to buy another server that can match the storage capacity. instead just fire and flood proof the server you have already.

[Benny Biscuit]

Ok that seems like an option to look into for that and thank you!! But what about backing up all of this media internally? Best way to do it? It just seems so complicated, the way he is doing it?

[emosun]

1 minute ago, Benny Biscuit said:

It just seems so complicated, the way he is doing it?

well making direct copies of all the drives internally is probably the most simple solution. so technically he's doing the easiest method without having to set up any form of software.

might be a bit labor intensive but it does eliminate links in the chain which is smart when backing up data. an automatic backup solution would add links to the chain

[Benny Biscuit]

The other day he formatted all the backup drives and then literally copy and pasted files from one drive to the other. Normally what he does when new media is added is to put one copy on the drive it goes on and one copy on the backup drive. However, what about just scanning for changes? I think he has used various native programs. See the thing is; when I add new files or new music, it sits in a folder called 'Ready to Go' until he is ready to make the needed copy and put those onto both drives. This means that until he has time, my stuff is just sitting there and isn't really accessible until it is moved to where it needs to be. So I take care of renaming things for plex and getting it ready to be moved and then it just sits there; waiting to be moved to each location.

 

And a few days ago, as I was adding music, I wasn't thinking about it and just moved the music into my music and realized "oh crap, he needs to back that up." and then of course I couldn't remember exactly what I had added or removed. 

 

So I was wondering what the simplest process was. That way all of these new files aren't just sitting there, waiting to be moved and I could move them into the necessary folders and on the necessary drives, and he would be able to find the changes and back those drives up; without me having to write down everything I moved. 

[emosun]

Well from what you describe , I get the gist of what he's most likely trying to avoid. By manually moving files , he's preventing unknown losses. It seems he's most likely concerned about file redundancy and safety more so than speed and convenience .

in a situation as such , you most likely want every backup to be intentional , it would eliminate another link in the chain being the possibility of an accidental backup. While automatically cloning drives is easier , it introduces the possibility of cloning mistakes. meaning if you make a mistake on the first drive and something is deleted or corrupted , the mistake will be copied to your only backup. 

I do understand what they are doing and why they choose to take the more manual method.

however , for YOU in your situation , I would simply suggest using your own drives/server system that is separate from theirs. that way you can use whatever methods you'd like to automatically back up your own files to your own devices and not have to worry about them being on this media server they have.

[Benny Biscuit]

Thank you so much for your input! I was just having a very hard time understanding all of this. I just felt that it should be simpler. All of our media is contained within one place (the server,) as it frees up space on the laptops and is unnecessary to have any of this media on either of the Gaming PC's. So all of my media is with his on our home server I just felt that there HAD to be an easier way to duplicate all of this media onto this other internal drives without going through all of this trouble. But it seems as though it is the safest way to do so without encountering errors and accidental deletion of files.

As for this fire box which you mentioned, what about air flow and how that will affect it? The server is cooled with fans; 4 of them. 

[Benny Biscuit]

3 hours ago, emosun said:

well if you're worried about fire just put the server you have in a fire box , i imagine it would cost a lot to buy another server that can match the storage capacity. instead just fire and flood proof the server you have already.

It also wouldn't be so much the cost of matching those drives. He would just build another server and remove those backup hard drives out of this server and put them into another. So there are 9 drives total: 3 drives have all the media, 3 more have an EXACT copy of all that media (they are cloned.) There is then a 2TB drive which is for our cameras and doesn't need backed up. There is a 3TB drive that is a backup for various items around the house (emulated programs for gaming consoles.) The 9th drive, I believe is the boot drive. But the 6 drives with the media need to be matched up, every time we add something to the 3 that Plex regularly accesses.

[emosun]

2 hours ago, Benny Biscuit said:

As for this fire box which you mentioned, what about air flow and how that will affect it? The server is cooled with fans; 4 of them. 

it can be a fire box that only contains the backup drives themselves and not the machine , seeing as the drives are copies , they only need to spin up when backups are made and can be turned off most of the time , so really no airflow would be needed. but i'll leave that up to them to design and build. but basically if only the drives are in the box and they arent on most of the time then they dont really need air.

[Benny Biscuit]

3 minutes ago, emosun said:

it can be a fire box that only contains the backup drives themselves and not the machine , seeing as the drives are copies , they only need to spin up when backups are made and can be turned off most of the time , so really no airflow would be needed. but i'll leave that up to them to design and build. but basically if only the drives are in the box and they arent on most of the time then they dont really need air.

Oh ok! I guess I didn't really understand what a fire box was there for a second!

 

This has all been VERY helpful and definitely gives me a better understanding of how all of these things in my home work! He does try to explain these processes to me, but sometimes I DO get a little lost.

[jde3]

This is all... no.

 

Get a real backup solution for the stuff you need most. (baby pictures, documents etc, the stuff you value most) This data should be pretty small. Their are lots of options out there but you need to get this stuff "off site" to protect it from loss. Combining a backup program like Duplicati and Amazon S3 would be a solid solution for this or you can buy a service plan from someone such as TarSnap, Noton, Dropbox, SpiderOak, Mega etc. Critical data goes here.. My world ends if it's gone, things. (You should encrypt this before it's sent to the cloud, Duplicati, Tarsnap and SpiderOak do this automatically, not sure about the others.)
 

As for media Plex/Emby library. Large stuff that is too heavy to move over the internet.

 

If your pirating (? anyhow I don't want to know) there is no need to archive any of it as it's transient and (at least for the moment) can be replaced. A simple solid NAS on reliable equipment (such as FreeNAS) with "non-dumb"(ie not raid 5) RAID layout would be fine. This is one option.. and it's plausible, risky and cheep but.. at least you do have a backup for the important stuff in the cloud.

 

Your second option for large data is to mirror this data on to a second FreeNAS box and have that as a backup. So a two NAS system. You have one system that is live that is accessed as your media server.. It's powerful and has transcoding abilities, then a second is a light weight low power NAS that can receive ZFS snapshots from the first. And that's all the second does.. it just receives the ZFS snapshot (or rsync job) from the first system every night. It should have no user level access to make a buffer from virus/ransomware accidental deletion issues.. (Also. as before.. You still need to have the offsite plan above as this does not protect you from fire, theft, flood/water, act of god etc)

 

Is that expensive to do?

Meh, not really.. Here is a ~$100 FreeNAS build (without drives) that consumes 6-9w of power.

 

If you were a small business (for completeness) then the next step would be to push the second NAS's drive array down to tape drive once a week and have a service plan with an offsite tape vault provider.

 

So your first system could snapshot every hour, your second every day, and the tape every week.

 

The reason I like Duplicati (or Duplicity/Deja Dup on Unix) is because being they are just a collection of opensource tools, you don't need the actual backup program to exist to be able to restore the backup. So in 10-15-20 years if you can't get the backup program anymore, or can't get it to run, you can still restore them.

[Benny Biscuit]

55 minutes ago, jde3 said:

This is all... no.

 

Get a real backup solution for the stuff you need most. (baby pictures, documents etc, the stuff you value most) This data should be pretty small. Their are lots of options out there but you need to get this stuff "off site" to protect it from loss. Combining a backup program like Duplicati and Amazon S3 would be a solid solution for this or you can buy a service plan from someone such as TarSnap, Noton, Dropbox, SpiderOak, Mega etc. Critical data goes here.. My world ends if it's gone, things. (You should encrypt this before it's sent to the cloud, Duplicati, Tarsnap and SpiderOak do this automatically, not sure about the others.)
 

As for media Plex/Emby library. Large stuff that is too heavy to move over the internet.

 

If your pirating (? anyhow I don't want to know) there is no need to archive any of it as it's transient and (at least for the moment) can be replaced. A simple solid NAS on reliable equipment (such as FreeNAS) with "non-dumb"(ie not raid 5) RAID layout would be fine. This is one option.. and it's plausible, risky and cheep but.. at least you do have a backup for the important stuff in the cloud.

 

Your second option for large data is to mirror this data on to a second FreeNAS box and have that as a backup. So a two NAS system. You have one system that is live that is accessed as your media server.. It's powerful and has transcoding abilities, then a second is a light weight low power NAS that can receive ZFS snapshots from the first. And that's all the second does.. it just receives the ZFS snapshot (or rsync job) from the first system every night. It should have no user level access to make a buffer from virus/ransomware accidental deletion issues.. (Also. as before.. You still need to have the offsite plan above as this does not protect you from fire, theft, flood/water, act of god etc)

 

Is that expensive to do?

Meh, not really.. Here is a ~$100 FreeNAS build (without drives) that consumes 6-9w of power.

 

If you were a small business (for completeness) then the next step would be to push the second NAS's drive array down to tape drive once a week and have a service plan with an offsite tape vault provider.

 

So your first system could snapshot every hour, your second every day, and the tape every week.

 

The reason I like Duplicati (or Duplicity/Deja Dup on Unix) is because being they are just a collection of opensource tools, you don't need the actual backup program to exist to be able to restore the backup. So in 10-15-20 years if you can't get the backup program anymore, or can't get it to run, you can still restore them.

Thanks so much for ALL of this information! I don't really sweat the small stuff (pictures and personal documents) as it is backed up in numerous places; both in our home and on several off-site (Google, Drop-Box etc.) Definitely not pirating but everything I own is digital, and it is all that heavy stuff that he keeps an identical copy of that I'm worried about being contained within the same server/place. 

It is really more of annoyance when I add something, that I can't just put it into the folder where it goes as he needs to back it up to whichever drive it's supposed to be on. That's why I was wondering about programs to check for those changes and I am aware they do exist but he's pretty picky about how he wants it done. 

[jde3]

8 minutes ago, Benny Biscuit said:

Thanks so much for ALL of this information! I don't really sweat the small stuff (pictures and personal documents) as it is backed up in numerous places; both in our home and on several off-site (Google, Drop-Box etc.) Definitely not pirating but everything I own is digital, and it is all that heavy stuff that he keeps an identical copy of that I'm worried about being contained within the same server/place. 

It is really more of annoyance when I add something, that I can't just put it into the folder where it goes as he needs to back it up to whichever drive it's supposed to be on. That's why I was wondering about programs to check for those changes and I am aware they do exist but he's pretty picky about how he wants it done. 

Yeah, you want to get out of that situation. Simply copying data is.. well better than nothing but kind of bad. You want it to be automatic and you want it to be transnational. (meaning verified, checksumed etc) You do need to verify it's working from time to time.. and you need to actually test if you can restore it. (One way to do that is to say.. restore / reinstall your OS occasionally. Having all your data in well organized files and folders in your user directory helps this.)

 

Backup sucks.. it's a hassle and nobody likes to do it. This is why it's such a pain. For me, it's part of my job and policy and training as a sysadmin but your a pretty smart person thinking about this stuff. Most don't.

[Mikensan]

Many ways to skin a cat, and likely if he's done all of that he's probably considered other means. Your options vary on the backup server's OS, some natively do it better than others.

 

I do something similar, I'm running FreeNAS which uses ZFS and I use replication to backup my data to a separate volume frequently. I feel it's fairly rare that short of a fire/flood that all disks would die together. Even in the event of a power surge I've never had every single disk get fried, but fingers crossed the giant UPS takes the brunt of anything that bad.

 

It's too expensive to replicate my movies off-site in every way imaginable. From cloud solutions, to co-location, and to having another physical box somewhere else - just too costly. I have plenty of time to re-download all my digital content that. If you purchased a digital copy, you should be entitled to download it again and again for free. Free > not-free.

 

[Benny Biscuit]

14 hours ago, Mikensan said:

Many ways to skin a cat, and likely if he's done all of that he's probably considered other means. Your options vary on the backup server's OS, some natively do it better than others.

 

I do something similar, I'm running FreeNAS which uses ZFS and I use replication to backup my data to a separate volume frequently. I feel it's fairly rare that short of a fire/flood that all disks would die together. Even in the event of a power surge I've never had every single disk get fried, but fingers crossed the giant UPS takes the brunt of anything that bad.

 

It's too expensive to replicate my movies off-site in every way imaginable. From cloud solutions, to co-location, and to having another physical box somewhere else - just too costly. I have plenty of time to re-download all my digital content that. If you purchased a digital copy, you should be entitled to download it again and again for free. Free > not-free.

 

Thanks so much for the help! Like I said it seems like it was more of a hassle to me, being unaware how all of this worked. I was just expecting some magical solution that would be efficient and not seem like such a headache! This stuff all comes naturally to him; as he's been in the biz for over a decade, but sometimes I can get a little lost while he explains these complicated processes to me. 

 

As for the movies, yes the recently purchased ones are easy to retrieve over and over again. When we moved in together I started the painful process of converting 500 some DVD's and those wouldn't be so easy to retrieve; as they no longer exist in my home. 

 

But thank you for all the help, it has been VERY insightful and gives me a better understanding of why he chooses to do things the way he does!

[Mikensan]

13 hours ago, Benny Biscuit said:

Thanks so much for the help! Like I said it seems like it was more of a hassle to me, being unaware how all of this worked. I was just expecting some magical solution that would be efficient and not seem like such a headache! This stuff all comes naturally to him; as he's been in the biz for over a decade, but sometimes I can get a little lost while he explains these complicated processes to me. 

 

As for the movies, yes the recently purchased ones are easy to retrieve over and over again. When we moved in together I started the painful process of converting 500 some DVD's and those wouldn't be so easy to retrieve; as they no longer exist in my home. 

 

But thank you for all the help, it has been VERY insightful and gives me a better understanding of why he chooses to do things the way he does!

Well one far-reaching option is to convert everything to H265 which will greatly reduce the file size while keeping the quality (used for 4k these days). Then buy a couple large external HDDs and keep say 1 at your house and 1 at a friend's house. Rotate them every month while putting the latest and greatest on them.

 

Just more options to consider :-) Glad we've been a help to some amount.