media server Plex Media Server Backup

[ArchGabriel]

I am wondering that backup techniques are people using or what techniques are recommended for backing up a media server.

Personally, I have a Plex media server with some 4TB of media files that I am looking to back up. Specifically, I'm only interested in strategies for backing up all the movies and tv series on it as I'm already backing up the Plex database and the OS separately. I am thinking any kind of hardware/software RAID/real-time file sync-ing would not be appropriate for large media files. I am considering a file sync-ing backup that runs every 24 hours or every couple of days (I am not adding files very often and changes to files are rare) and that syncs the media files from the main media hard drive to a secondary hard drive in the server would be the best option.

I am very curious about this as these types of backups of movies quickly eat through a lot of available storage space. My server has 20TB of storage, but with all the 4K movies and now even 4K TV series out there, that media library can grow fairly quickly and hard drives don't seem to be getting any cheaper (rather the contrary).

 

What backup techniques for media files are you guys using and what other options should I consider?

[Firewrath9]

Buying a bunch of external HDDs, or cheap enterprise HDDs., although if you lose data, why do you care that you have to re download some movies? 

[Lady Fitzgerald]

20 minutes ago, Firewrath9 said:

...although if you lose data, why do you care that you have to re download some movies? 

Um, the amount of bandwidth required to redownload movies may exceed an ISP's data cap? The amount of time required to redownload movies? Movies may not be available for download anymore? The movies were not downloaded but were ripped and reripping iss time consuming? And I'm sure there are plenty of other possible reasons.

[ArchGabriel]

31 minutes ago, Firewrath9 said:

Buying a bunch of external HDDs, or cheap enterprise HDDs., although if you lose data, why do you care that you have to re download some movies? 

I had external HDDs and cheap HDDs previously and they worked well for many years. Then they suddenly died last year (two of them within a day of each other) and I lost all my media library at the time, despite having it backed up. I'm not doing that again. Also, re-downloading all the movies and TV series was a huge headache, especially as some are more difficult to find (especially the older ones or the ones from non-English speaking countries) and some took ages to download. For some TV series, I had to go and buy the DVDs from Amazon to recover them (and then rip them), so that was both a huge headache and an additional cost.

 

11 minutes ago, Lady Fitzgerald said:

Um, the amount of bandwidth required to redownload movies may exceed an ISP's data cap? The amount of time required to redownload movies? Movies may not be available for download anymore? The movies were not downloaded but were ripped and reripping iss time consuming? And I'm sure there are plenty of other possible reasons.

Fortunately, I don't have a data cap, but you are right about the rest.

[dalekphalm]

On 3/20/2019 at 6:50 PM, ArchGabriel said:

I am wondering that backup techniques are people using or what techniques are recommended for backing up a media server.

Personally, I have a Plex media server with some 4TB of media files that I am looking to back up. Specifically, I'm only interested in strategies for backing up all the movies and tv series on it as I'm already backing up the Plex database and the OS separately. I am thinking any kind of hardware/software RAID/real-time file sync-ing would not be appropriate for large media files. I am considering a file sync-ing backup that runs every 24 hours or every couple of days (I am not adding files very often and changes to files are rare) and that syncs the media files from the main media hard drive to a secondary hard drive in the server would be the best option.

I am very curious about this as these types of backups of movies quickly eat through a lot of available storage space. My server has 20TB of storage, but with all the 4K movies and now even 4K TV series out there, that media library can grow fairly quickly and hard drives don't seem to be getting any cheaper (rather the contrary).

 

What backup techniques for media files are you guys using and what other options should I consider?

First thing's first, RAID is not a backup. If you want to backup your Media, I'd probably just get a single drive large enough for all your media plus room to grow (If you're using 4TB now, I'd suggest a 6TB or 8TB drive). Keep it simple. If you're running Windows, just use built in backup tools to backup those files. You could also create a script and schedule it to run daily, using something like RoboCopy or XCopy.

 

If running linux, there are many ways to do the same thing, including RSync.

 

It really depends on how good you want your backup to be, what features you want, do you need file history/versioning, etc.

[paddy-stone]

Personally, I have anything personal/rare/time-sucky, backed up in 3 places, plus the server that's being used to share the files/media. It's a real pain as you know to try and replace, even if you can replace at all, some things. So it's mega important for those things to be backed up in paranoid mode

For the other stuff, I either just have the one backup, just to save time in the event of a server failure, or not backed up at all if it's just some crap I got on a whim and don't care if I lose it.

At the moment I have around 6TB of data in 2 places (the serving, and backup server), and around 500GB - 1TB of paranoidal backups in several places... I try to keep these as low as possible because of the amount of copies, especially as one of them is off-site backup at my brothers house that I sync on a weekly basis usually.

[Lady Fitzgerald]

27 minutes ago, paddy-stone said:

Personally, I have anything personal/rare/time-sucky, backed up in 3 places, plus the server that's being used to share the files/media. It's a real pain as you know to try and replace, even if you can replace at all, some things. So it's mega important for those things to be backed up in paranoid mode

For the other stuff, I either just have the one backup, just to save time in the event of a server failure, or not backed up at all if it's just some crap I got on a whim and don't care if I lose it.

At the moment I have around 6TB of data in 2 places (the serving, and backup server), and around 500GB - 1TB of paranoidal backups in several places... I try to keep these as low as possible because of the amount of copies, especially as one of them is off-site backup at my brothers house that I sync on a weekly basis usually.

I love it when I see someone who understands the reason for backups and has a good concept of what backups are! You appear to have a solid backup scheme in place. 

[paddy-stone]

Just now, Lady Fitzgerald said:

I love it when I see someone who understands the reason for backups and has a good concept of what backups are! You appear to have a solid backup scheme in place. 

Yes, thank you, me too!

 

I feel paranoid sometimes, but us people that were born before online backups were a thing, usually know the pain of having files lost... and vow never to have it happen again if possible. It comes at a price of course, but if the data matter to you at all, then it should be worth the cost IMO. Just limit as much as you can for multiple backups space to try and save a little.

Oh, and I've been a sinner when it comes to this too. don't overestimate your backup needs if possible. Having 20TB free space on your server(s) when you only expand by 2-5TB per year is not ideal. I didn't go that far luckily, just was going to have a raid 5/zfs1 array, but in the end decided to just go raid 0/striped instead... because of the backups. So ended up with around 9TB free at the moment on my main "server".

 

I am waiting for Ryzen 2 to drop at the moment, and then rebuilding my main server... at the moment I am just using a windows storage space (striped), until I get that up and running again. I have been experimenting with different setups, and have almost decided for sure that I may stick with windows, as I can still share my media and files on there perfectly fine. Also, if the worst happens and my Nvidia Shield TV borks, then at least I have an HTPC all ready to go too for plex/KODI etc. I may also get around finally to having a retro gaming/emulator too with it

 

Sorry for the wall of text.

[Lady Fitzgerald]

I'm the kind of person who would much rather have something and not need it than to need something and not have it (based on personal experience, I assure you the latter totally sucks when it happens!).

 

One can never have too many backups (up to the point it becomes too cumbersome or too expensive). Redundancy (RAID, etc.) by itself is not a backup since it only protects from drive failure and drive failure is not the only way one can lose data. Redundancy can be inclued as part of a backup scheme as long as the data still exists in at least three separate places (which is what paddy-stone appears to have done).

 

One also has to determine what warrants being backed up. Paddy-stone has determined only certain data is worthy of being backed up, which is perfectly fine because, per himself, he doesn't care if he loses the data that is not backed up or is only marginally backed up.

 

My personal feeling is, if data is worthy of gracing my computers, it is worthy of being fully backed up so I back up the snot out of everything. Rather than deal with the added complications and expense of RAID or other, similar redundancy schemes, my redundancy scheme is akin to sneaker net (which is what I use to sync data between computers, btw). Instead of striping or using parity over multiple drives, I simply have duplicate backups. For each data drive in my desktop machine, I have a set of four backup drives (bare drives I access via hot swap bays in the computer): two of each set kept onsite in a drawer and the other two kept offsite in my safe deposit box at my credit union. That way, if a backup drive dies, I have another one. Cheap and simple (there are other advantages). I use folder/file syncing for up dating backups so updates take little time or effort.

 

To sync data between my desktop computer and a notebook computer, I just use one of the notebook's backup drives to move the data with (aka sneakernet). I already have the backup drive so, by using folder/file syncing, it's quick and easy to sync the data on the backup drive with the corresponding data on the desktop's drive, then sync the backup drive with the notebook.

 

Most people will not need schemes as elaborate as paddy-stone's and mine (ok, mine might be considered anal) but the goal should be to have important data exist in at least three, separate places, such as on the computer, on an onsite backup drive, and on an offsite backup drive. If budget is limited, however, even only one, partial backup is considerably better than no backups until additional backup capacity can be afforded.

[Mikensan]

If you have 20TB then you are likely working off a raid already for active storage. Buying an external for backups would be my first recommendation, despite your past experience. Save for a lightning strike taking everything out (one hopes you have a beefy surge protector), the likelyhood of the external and raid suffering a catastrophic failure is pretty rare.

 

To suppliment this I would find a friend who is like-minded that you could either agree to store each other's data, or allow you to put something like a readynas / qnap / synology at their house. If you want to store data on one another's servers, you could use rclone + encryption if you want privacy. Assuming you're downloading 1 4k movie a week I'd ball park 10-15gb (HEVC) which should sync overnight without much problems.

 

Storage providers (AWS/Google/Backblaze) are very costly once you're storing several terabytes. AWS glacier is definitely cheaper but 4TB is still going to run the bill up.

[ArchGabriel]

On 3/22/2019 at 4:24 PM, dalekphalm said:

First thing's first, RAID is not a backup. If you want to backup your Media, I'd probably just get a single drive large enough for all your media plus room to grow (If you're using 4TB now, I'd suggest a 6TB or 8TB drive). Keep it simple. If you're running Windows, just use built in backup tools to backup those files. You could also create a script and schedule it to run daily, using something like RoboCopy or XCopy.

 

If running linux, there are many ways to do the same thing, including RSync.

 

It really depends on how good you want your backup to be, what features you want, do you need file history/versioning, etc.

 

Thanks for the advice. I know RAID is not technically a backup, but for the purposes of a media library where files might get changed once or twice per week, there is not a lot of difference, in real terms, between RAID, backup or file-sync-ing. I have decided to go with a software called Bvckup 2 for file-syncing my media library and my backup media library every 24 hours.

 

On 3/24/2019 at 12:56 PM, paddy-stone said:

Personally, I have anything personal/rare/time-sucky, backed up in 3 places, plus the server that's being used to share the files/media. It's a real pain as you know to try and replace, even if you can replace at all, some things. So it's mega important for those things to be backed up in paranoid mode

For the other stuff, I either just have the one backup, just to save time in the event of a server failure, or not backed up at all if it's just some crap I got on a whim and don't care if I lose it.

At the moment I have around 6TB of data in 2 places (the serving, and backup server), and around 500GB - 1TB of paranoidal backups in several places... I try to keep these as low as possible because of the amount of copies, especially as one of them is off-site backup at my brothers house that I sync on a weekly basis usually.

Yes, I am paranoid about backing up personal data (including work, study, projects etc) as well. I am actually just as "worse" as you as I back it up on 3 different places as well: on two separate hard drive on my server and on OneDrive. I would also recommend you get OneDrive for you files. I'm paying something around $5-7 per month for 1TB of OneDrive which is ridiculously cheap. Amazon is much more expensive (I think double the price per TB for their cheapest option).

 

Might I also suggest on taking a look on this other post of mine from earlier today about overall backuk strategies? 

 

 

On 3/24/2019 at 1:27 PM, Lady Fitzgerald said:

I love it when I see someone who understands the reason for backups and has a good concept of what backups are! You appear to have a solid backup scheme in place. 

Same for you, I would love to hear your opinions on my backup strategy on my other post.

 

10 hours ago, Mikensan said:

If you have 20TB then you are likely working off a raid already for active storage. Buying an external for backups would be my first recommendation, despite your past experience. Save for a lightning strike taking everything out (one hopes you have a beefy surge protector), the likelyhood of the external and raid suffering a catastrophic failure is pretty rare.

 

To suppliment this I would find a friend who is like-minded that you could either agree to store each other's data, or allow you to put something like a readynas / qnap / synology at their house. If you want to store data on one another's servers, you could use rclone + encryption if you want privacy. Assuming you're downloading 1 4k movie a week I'd ball park 10-15gb (HEVC) which should sync overnight without much problems.

 

Storage providers (AWS/Google/Backblaze) are very costly once you're storing several terabytes. AWS glacier is definitely cheaper but 4TB is still going to run the bill up.

I don't have a RAID configuration for redundancy as it would create inherent inneficiencies. I use a combination of back-ups and file-syncing.

In regards to an outside server, yes, that is a great idea and I do have a location available. As long as I'm prepared to spend the money on building a small server with enough storage to deploy there, I will take this approach.

[Mikensan]

@ArchGabriel You've piquied my ineterest regarding your current 20TB. How many disks is this comprised of and in what configuration are they (being nosey and curious)?

 

If it's a backup device that's going to be remote, it can be underpowered and "dumb". Something like a WD cloud with 2 disks would work (10TB x 2), these types of things usually have plugins or run on linux and support rsync. Or you can spice it up, get a Dell R210 off ebay for ~$200 etc.. but then that'll draw a lot more power than a WD cloud / buffalo NAS would.

[ArchGabriel]

4 hours ago, Mikensan said:

@ArchGabriel You've piquied my ineterest regarding your current 20TB. How many disks is this comprised of and in what configuration are they (being nosey and curious)?

 

If it's a backup device that's going to be remote, it can be underpowered and "dumb". Something like a WD cloud with 2 disks would work (10TB x 2), these types of things usually have plugins or run on linux and support rsync. Or you can spice it up, get a Dell R210 off ebay for ~$200 etc.. but then that'll draw a lot more power than a WD cloud / buffalo NAS would.

 

 

 

 

No, it is not just a backup device. It's my main server, so it has various other services runing on it as well as a couple of VMs. I have a Seagate Surveillance 4TB, Seagate IronWolf 6TB and Seagate Exos 10TB. From now on, I will buy just Exos 10TB for simplicity.  They are not noisy as all. You cannot hear the server at all unless you're literally sitting with your ears next to it in which case you can only hear the fans. Obviously, besides the 20TB for storage, I have some SSDs for OS, virtual machines, Plex Transcoding etc.

[Mikensan]

27 minutes ago, ArchGabriel said:

No, it is not just a backup device. It's my main server, so it has various other services runing on it as well as a couple of VMs. I have a Seagate Surveillance 4TB, Seagate IronWolf 6TB and Seagate Exos 10TB. From now on, I will buy just Exos 10TB for simplicity.  They are not noisy as all. You cannot hear the server at all unless you're literally sitting with your ears next to it in which case you can only hear the fans. Obviously, besides the 20TB for storage, I have some SSDs for OS, virtual machines, Plex Transcoding etc.

I meant if the remote storage is going to do nothing other than be used for storage / backups, not what you currently have*

[ArchGabriel]

1 hour ago, Mikensan said:

I meant if the remote storage is going to do nothing other than be used for storage / backups, not what you currently have*

Oh, yes, I see. No, it is the hard drives that would cost me much more than the actual system. That's why I am not building it yet.

[Mikensan]

17 hours ago, ArchGabriel said:

Oh, yes, I see. No, it is the hard drives that would cost me much more than the actual system. That's why I am not building it yet.

Ah yes, that is one of my problems. I want to expand my storage and a disk shelf is cheap, but buying 12x of even 3TB is going to cost a good bit.

[ArchGabriel]

3 hours ago, Mikensan said:

Ah yes, that is one of my problems. I want to expand my storage and a disk shelf is cheap, but buying 12x of even 3TB is going to cost a good bit.

 

 

I think you're looking at around $30 per TB* (or more with NAS grade HDDs), so yeah, it won't be cheap.

* That's in UK. Hard Drives tend to be cheaper in the US.

[Mikensan]

1 hour ago, ArchGabriel said:

I think you're looking at around $30 per TB* (or more with NAS grade HDDs), so yeah, it won't be cheap.

* That's in UK. Hard Drives tend to be cheaper in the US.

That's about the same here, $30USD for "NAS" drives, consumer drives I've seen hover around $20-25 or so. I'll cross that bridge after a few vacations first lol.