Home NAS/Plex server RAID/Backup Advice

[xl3b4n0nx]

Hello,

 

I am looking for some advice on the best setup for my NAS/Plex server. I am mostly focused on security for my data. Speed is of no concern (not doing a lot of R/W ops and I am remote most of the time so network is the bottleneck). Currently I have Ubuntu 14.04 with a 500GB boot drive and 2 4TB drives (no RAID) internal drives. Then I have 1.5 TB, 1 TB and 750 GB external drives. 2 are used as back up for all non Plex files and 1 TB is for a single Samba drive for my family. All of this is in a desktop setup with an FX 6300 on an ASRock 970 a-g/3.1 mobo (6 SATA 3 ports).

 

I am in college (poor) so I can't spend a lot at once. I need something I can expand in the future and offers good price to capacity ratio. Ideally I would have nearly equal storage to back up ratios. I would like to move away from using the externals for backup ASAP.

 

I am considering a RAID 5 array for storage and possibly a second array for backup. I am fine with getting RAID cards if I need to (reccommendations are welcomes). Is this a good idea? Is there a better design? Any help is appreciated.

 

Thank you.

 

[AshleyAshes]

For media files, FlexRAID is probably the 'cheapest' solution for what it does and the level of expandability it offers

[Electronics Wizardy]

You might also want to look at used tape drives for backup.

 

A raid card, tape drive and 8tb(tapes are compressed, storage could be as low as 4tb if your data can't be compressed) is about 200 dollars. Tapes are cheap if you want to add more too.

 

http://www.ebay.com/itm/Dell-Powervault-LTO3-LT04-EH1-External-Tape-Drive-8024-/132253596506?epid=1809218797&hash=item1ecaedbb5a:g:ihAAAOSw2iNZX9Uh

tape drive, external 1 tape, manual loading

http://www.ebay.com/itm/Box-of-five-5-Quantum-LTO-4-tapes-UNUSED-800GB-1600GB-All-Sequential-numbers-/322553932039?hash=item4b19b66907:g:CbYAAOSwr~lYoR1l

tapes(lto-4, 1.6tb)

 

http://www.ebay.com/itm/Tyco-EMC-038-003-787-Cable-Mini-SAS-SFF-8088-to-SFF-8088-2M-/371488174209?epid=1120536505&hash=item567e6bb081:g:QFEAAOSwBLlVPo~p

cable to connect tape drive to pc

 

http://www.ebay.com/itm/LSI-Logic-SAS3801E-Dual-SFF-8088-SAS-SATA-PCIe-Storage-Controller-HBA-Card-/182197136207?epid=1577543291&hash=item2a6bcbab4f:g:AFIAAOSwdGFYuaeQ

Hba to use the card

 

[xl3b4n0nx]

1 hour ago, Electronics Wizardy said:

You might also want to look at used tape drives for backup.

 

A raid card, tape drive and 8tb(tapes are compressed, storage could be as low as 4tb if your data can't be compressed) is about 200 dollars. Tapes are cheap if you want to add more too.

 

http://www.ebay.com/itm/Dell-Powervault-LTO3-LT04-EH1-External-Tape-Drive-8024-/132253596506?epid=1809218797&hash=item1ecaedbb5a:g:ihAAAOSw2iNZX9Uh

tape drive, external 1 tape, manual loading

http://www.ebay.com/itm/Box-of-five-5-Quantum-LTO-4-tapes-UNUSED-800GB-1600GB-All-Sequential-numbers-/322553932039?hash=item4b19b66907:g:CbYAAOSwr~lYoR1l

tapes(lto-4, 1.6tb)

 

http://www.ebay.com/itm/Tyco-EMC-038-003-787-Cable-Mini-SAS-SFF-8088-to-SFF-8088-2M-/371488174209?epid=1120536505&hash=item567e6bb081:g:QFEAAOSwBLlVPo~p

cable to connect tape drive to pc

 

http://www.ebay.com/itm/LSI-Logic-SAS3801E-Dual-SFF-8088-SAS-SATA-PCIe-Storage-Controller-HBA-Card-/182197136207?epid=1577543291&hash=item2a6bcbab4f:g:AFIAAOSwdGFYuaeQ

Hba to use the card

 

I have never looked into tape storage before. If I add more tape drives to the system will it be treated as a completely separate drive or more total storage?

[Electronics Wizardy]

Just now, xl3b4n0nx said:

I have never looked into tape storage before. If I add more tape drives to the system will it be treated as a completely separate drive or more total storage?

You have to use a program to put it on the tape, you can't just copy files to it like a normal drive.

 

You can use tar on linux for this.

[xl3b4n0nx]

54 minutes ago, Electronics Wizardy said:

You have to use a program to put it on the tape, you can't just copy files to it like a normal drive.

 

You can use tar on linux for this.

Ok. Will this program send the data to multiple drives if one single drive is not large enough? Is this a program I download or do I have to make one?

[Mikensan]

RAID is only a stopgap for uptime, not meant to protect your data. Backing up to externals are fine so long as it's an actual backup and not archive (meaning a copy exists on the internal and external). You can use FlexRAID or Windows Storage Spaces to pool your disks to make life easier, however you're on ubuntu.

 

The nice thing about storage spaces and flexraid is they do not stripe your data but still offer parity. In the linux world, I'm not sure what is out there that does something similar. I'm only slightly familiar with mdadm for raid. Your school likely either offers Windows at a discount or for free. If you have an .edu email address you can get Microsoft Server 2016 for free (dreamspark).

 

Also crashplan is $5/m, so for about the tape drive setup that would cover you for about 3 years (unlimited storage).

[AshleyAshes]

2 hours ago, Electronics Wizardy said:

A raid card, tape drive and 8tb(tapes are compressed, storage could be as low as 4tb if your data can't be compressed) 

Well, if we're talking about media files, the data can't be compressed  This thread is primarily about storage and redundancy/backups for a Plex server.  Compressed media files by design have minimized all redundant data, if you use lossless compression of any kind compressed media, your compression ratio will be negligible.  So if it's for consumable media, you might as well just say that any tapes wold be 4TB, not 8TB.

[Electronics Wizardy]

Just now, xl3b4n0nx said:

Ok. Will this program send the data to multiple drives if one single drive is not large enough? Is this a program I download or do I have to make one?

you can make a tar archive across multiple tapes. Tar is preinstalled in most all linux distros.

[xl3b4n0nx]

43 minutes ago, Mikensan said:

RAID is only a stopgap for uptime, not meant to protect your data. Backing up to externals are fine so long as it's an actual backup and not archive (meaning a copy exists on the internal and external). You can use FlexRAID or Windows Storage Spaces to pool your disks to make life easier, however you're on ubuntu.

 

The nice thing about storage spaces and flexraid is they do not stripe your data but still offer parity. In the linux world, I'm not sure what is out there that does something similar. I'm only slightly familiar with mdadm for raid. Your school likely either offers Windows at a discount or for free. If you have an .edu email address you can get Microsoft Server 2016 for free (dreamspark).

 

Also crashplan is $5/m, so for about the tape drive setup that would cover you for about 3 years (unlimited storage).

I'm not a fan of cloud storage. I would rather keep my data with me. Also, I enjoy projects like this so the experience of setting it up and mainting it is fun for me.

[xl3b4n0nx]

33 minutes ago, AshleyAshes said:

Well, if we're talking about media files, the data can't be compressed  This thread is primarily about storage and redundancy/backups for a Plex server.  Compressed media files by design have minimized all redundant data, if you use lossless compression of any kind compressed media, your compression ratio will be negligible.  So if it's for consumable media, you might as well just say that any tapes wold be 4TB, not 8TB.

Most of the data would be media files of some kind. I want to make sure this is expandable. What are the benefits of Tape vs another set of HDDs(Internal or external) for backup?

[AshleyAshes]

Just now, xl3b4n0nx said:

Most of the data would be media files of some kind. I want to make sure this is expandable. What are the benefits of Tape vs another set of HDDs(Internal or external) for backup?

Well, I think the real question is: How important is the media to you?

Tape backups aren't typically seen for, well, let's admit it here, what is likely to be a hoard of pirated media.  Tapes involve the effort of backing up the data to the tapes, storing the tapes, ensuring you're producing more tapes at the right times.  It's more an 'Important buisness stuff and life's work' kinda deal.

This is why I use FlexRAID.  My system can survive a single drive failure and I can have as many storage drives as I want, but the parity drive just can't be smaller than any of the storage drives.  Even if two drives fail, all OTHER drives remain intact so I only lose two drives worth of content.  Anything more than that and honestly, I'd rather just 'require' the media that I'd lose.  It's just not important enough to invest more into it.

[xl3b4n0nx]

13 minutes ago, AshleyAshes said:

Well, I think the real question is: How important is the media to you?

Tape backups aren't typically seen for, well, let's admit it here, what is likely to be a hoard of pirated media.  Tapes involve the effort of backing up the data to the tapes, storing the tapes, ensuring you're producing more tapes at the right times.  It's more an 'Important buisness stuff and life's work' kinda deal.

This is why I use FlexRAID.  My system can survive a single drive failure and I can have as many storage drives as I want, but the parity drive just can't be smaller than any of the storage drives.  Even if two drives fail, all OTHER drives remain intact so I only lose two drives worth of content.  Anything more than that and honestly, I'd rather just 'require' the media that I'd lose.  It's just not important enough to invest more into it.

Isn't there a yearly rate for FlexRAID? After reading some of the comments and my own research tape seems a little over kill. I will be keeping multiple copies on different machines of the most important data. Which is only images and documents. 

[dalekphalm]

12 minutes ago, AshleyAshes said:

Well, I think the real question is: How important is the media to you?

Tape backups aren't typically seen for, well, let's admit it here, what is likely to be a hoard of pirated media.  Tapes involve the effort of backing up the data to the tapes, storing the tapes, ensuring you're producing more tapes at the right times.  It's more an 'Important buisness stuff and life's work' kinda deal.

This is why I use FlexRAID.  My system can survive a single drive failure and I can have as many storage drives as I want, but the parity drive just can't be smaller than any of the storage drives.  Even if two drives fail, all OTHER drives remain intact so I only lose two drives worth of content.  Anything more than that and honestly, I'd rather just 'require' the media that I'd lose.  It's just not important enough to invest more into it.

The other awesome thing about FlexRAID, which other RAID or RAID-like systems tend not to have: Unlimited number of parity disks.

 

Want RAID6-like functionality? Create two PPU's (Parity disks). Want 3-drive failure protection? Create another PPU, etc. So using this method, you can have pretty large disk arrays (Say, a 24-disk array), and not be too worried, because you can add, say, 4 or 5 PPU parity disks.

[AshleyAshes]

Just now, xl3b4n0nx said:

Isn't there a yearly rate for FlexRAID? After reading some of the comments and my own research tape seems a little over kill. I will be keeping multiple copies on different machines of the most important data. Which is only images and documents. 

No, FlexRAID is a fixed single cost.  They do however have SUPPORT subscriptions.

[dalekphalm]

Just now, xl3b4n0nx said:

Isn't there a yearly rate for FlexRAID? After reading some of the comments and my own research tape seems a little over kill. I will be keeping multiple copies on different machines of the most important data. Which is only images and documents. 

FlexRAID is a one-time fee. I bought FlexRAID like 6 years ago. I don't use it anymore (Currently using ESXi + Virtualized FreeNAS) but I could re-install it anytime I want. There is a license transfer process, but that process is free if you already own a license.

[AshleyAshes]

1 minute ago, dalekphalm said:

The other awesome thing about FlexRAID, which other RAID or RAID-like systems tend not to have: Unlimited number of parity disks.

 

Want RAID6-like functionality? Create two PPU's (Parity disks). Want 3-drive failure protection? Create another PPU, etc. So using this method, you can have pretty large disk arrays (Say, a 24-disk array), and not be too worried, because you can add, say, 4 or 5 PPU parity disks.

Have you ever done a drive replace with FlexRAID yet?  I have a spare 500GB and 1TB in my pool on the basis of 'Well, I had nothing better to do with those drives' but I'm thinking of swapping them out for 8TB drives over time rather than simply adding MORE drives.

[dalekphalm]

Just now, AshleyAshes said:

Have you ever done a drive replace with FlexRAID yet?  I have a spare 500GB and 1TB in my pool on the basis of 'Well, I had nothing better to do with those drives' but I'm thinking of swapping them out for 8TB drives over time rather than simply adding MORE drives.

I had to rebuilt a drive once - a few years ago, but I don't recall the exact process. It was pretty easy to do, but the rebuild of the data itself of course took some time.

 

With FlexRAID though, you could just add in the new 8TB drives, and then once they're added in, remove the existing drives - that's personally the process I would follow. Just copy the data from the old drives to the 8TB drive once it's added to the pool, then simply remove the old drives from the pool.

 

I would start by adding an 8TB drive as the PPU (especially if you plan on adding two or more 8TB drives in the first go).

 

You CAN add the 8TB drive as a DDU without expanding the PPU yet, as long as the 8TB drive doesn't fill up past the size of your PPU - I actually did that for a while (I had 3TB PPU, but a 4TB (2x 2TB) DDU - I just made sure the 4TB DDU never went above 3TB.

[xl3b4n0nx]

46 minutes ago, AshleyAshes said:

No, FlexRAID is a fixed single cost.  They do however have SUPPORT subscriptions.

Awesome. I think I may do that. I am going to get a crashplan subscription for my main back up and then make a RAID array using using FlexRAID.

[Mikensan]

You can encrypt data before putting it onto the cloud too. I also agree, the data I would actually like to backup I feel uneasy about being out of my control. 

 

Tape has a longer shelf life in theory compared to a disk, which is why most businesses like to use it. I've also looked at going to tape but I wouldn't know where I would keep an "offsite" copy and the initial cost (used equipment, we'll just say $200) isn't worth it to me. I have a second volume used only for backups, my worse case is a house fire or the system in which both volumes reside outright commits seppuku.

 

You could also use a friend/parent's computer as a "backup" server and rsync the data. 

 

I believe you can download a trial of flexraid just to see how you like it. Also keep in mind, @dalekphalm or @AshleyAshes might be able to speak more on, but you may need to wipe the disks when you create a new volume. (Is this correct?)

[dalekphalm]

32 minutes ago, Mikensan said:

You can encrypt data before putting it onto the cloud too. I also agree, the data I would actually like to backup I feel uneasy about being out of my control. 

 

Tape has a longer shelf life in theory compared to a disk, which is why most businesses like to use it. I've also looked at going to tape but I wouldn't know where I would keep an "offsite" copy and the initial cost (used equipment, we'll just say $200) isn't worth it to me. I have a second volume used only for backups, my worse case is a house fire or the system in which both volumes reside outright commits seppuku.

 

You could also use a friend/parent's computer as a "backup" server and rsync the data. 

 

I believe you can download a trial of flexraid just to see how you like it. Also keep in mind, @dalekphalm or @AshleyAshes might be able to speak more on, but you may need to wipe the disks when you create a new volume. (Is this correct?)

FlexRAID has I believe a 30-day trial. Also, no, you do not need to wipe disks when adding them to FreeNAS FlexRAID. You can add disks that already have data on them. The only major exception I can think of would be adding a Parity drive (PPU) - I don't believe you can have other data on a Parity drive.

[xl3b4n0nx]

1 minute ago, dalekphalm said:

FlexRAID has I believe a 30-day trial. Also, no, you do not need to wipe disks when adding them to FreeNAS. You can add disks that already have data on them. The only major exception I can think of would be adding a Parity drive (PPU) - I don't believe you can have other data on a Parity drive.

That works out because one of my 4TB drives is already empty. Both drives are fairly new. I plan to expand later. I adopted "early" into the 4TB and up capacity so I should be able to get more in the next few years pretty cheap. I just like having the redundancy as well as one storage location. How will Ubuntu see this location I create with FlexRAID? Will it be one drive? One folder?

[dalekphalm]

1 minute ago, xl3b4n0nx said:

That works out because one of my 4TB drives is already empty. Both drives are fairly new. I plan to expand later. I adopted "early" into the 4TB and up capacity so I should be able to get more in the next few years pretty cheap. I just like having the redundancy as well as one storage location. How will Ubuntu see this location I create with FlexRAID? Will it be one drive? One folder?

Basically, all of the individual HDD's will still show up as mounted drives. FlexRAID will then create a "virtual" drive that is mounted, that is the pool. I can't say exactly how Linux will "show" the pool, but I assume it'll show up as if it were just another mounted drive.

 

In Windows, it shows up just as another drive w/ a drive letter (Windows also removes the drive letters from the individual HDD's - Still accessible, but no drive letter). I assume Linux does something similar.

[Mikensan]

11 minutes ago, dalekphalm said:

FlexRAID has I believe a 30-day trial. Also, no, you do not need to wipe disks when adding them to FreeNAS. You can add disks that already have data on them. The only major exception I can think of would be adding a Parity drive (PPU) - I don't believe you can have other data on a Parity drive.

Did you mean FlexRAID instead of FreeNAS?

 

Ok that was my other thought, that at the minimum you'd need to wipe the parity disk. Cool beans.

[Mikensan]

I would also assume you'd see everything as sda01, sda02 etc... and under /mnt you'd see the pool created from flexraid. Disks don't show up under /mnt unless you mount them, so flexraid may just unmount the disks individually.

 

I wonder, could you use ZFS with flexraid?