Best way to back up 8TB of data? Looking for advice.

[chilicheeseburger]

I have 8TB of available storage in a software RAID5 array on a home server I've built and now I'm looking for a good way to back up all that data regularly.

 

What I have an mind is some external enclosure with it's own RAID array of hard drives that would be connected to the server via USB 3.0 that backs up the data regularly (let's say every 24h), of course skipping unchanged files. I'm running Ubuntu on the home server.

 

Is this possible , has anyone done this or something similar?

 

Of course if you have a different idea on how to back up large amounts of data, don't hesitate to bring that up.

 

[AidanJMessenger]

 

Would some form of online backup be suitable for you, or do you want it to be a physical backup solution?

Seeing as you're considering an external raid enclosure, I assume this backup is just to protect in case the RAID fails or something like that, as if there's a fire then it won't do anything to help that.

[chilicheeseburger]

 

 

Would some form of online backup be suitable for you, or do you want it to be a physical backup solution?

Seeing as you're considering an external raid enclosure, I assume this backup is just to protect in case the RAID fails or something like that, as if there's a fire then it won't do anything to help that.

 

 

I've thought about online solution vs on-site solution.

 

The problem with online is that it'll take ages to upload (I have ~8.5Mbps up) and every service I found works with monthly subscription fees. It would be very expensive for 8TB of online storage.

 

The risk of natural disasters like earth quakes, hurricanes and the like is pretty much non-existent here. A flood is also very unlikely. I thought about the fire possibility but I figured if the house burns down, the loss of my movies and music is probably my smallest problem.

[AidanJMessenger]

I've thought about online solution vs on-site solution.

 

The problem with online is that it'll take ages to upload (I have ~8.5Mbps up) and every service I found works with monthly subscription fees. It would be very expensive for 8TB of online storage.

 

The risk of natural disasters like earth quakes, hurricanes and the like is pretty much non-existent here. A flood is also very unlikely. I thought about the fire possibility but I figured if the house burns down, the loss of my movies and music is probably my smallest problem.

Fair enough.

I too am currently looking into some form of backup solution for my freenas. I was considering crashplan, because they're quite cheap and have freenas integration. My upload speed is only 10mbps (soon to be 20mbps when I upgrade), however I noticed crashplan does this cool thing where they'll actually send you some hard drives, you transfer your data onto them and send them back, and they put it on the cloud. Far faster. You can also do this to restore too. Pretty cool feature really. 

 

It's too bad they don't make a single drive that would fit all your data. You have to choice but to do some sort of RAID. Either like a RAID 0 of 2x4TB drives or, 3x4tb drives or 3x3tb drives. The lack of security when it comes to RAID 0 doesn't really matter too much when it's just a backup.

 

You could just take the RAID 5 and make it a RAID6 for the extra security.... but it still won't help if the raid itself corrupts. There is always the option of just buying perhaps 2x4TB external USB drives and instead of RAIDing them, just set certain things to go on each drive. I don't know how you have everything set up so I'm not sure how much of a Hassle that sort of set up would be. 

 

Why is it that backing up a large (but not too large) amount of data is so hard  

[chilicheeseburger]

Fair enough.

I too am currently looking into some form of backup solution for my freenas. I was considering crashplan, because they're quite cheap and have freenas integration. My upload speed is only 10mbps (soon to be 20mbps when I upgrade), however I noticed crashplan does this cool thing where they'll actually send you some hard drives, you transfer your data onto them and send them back, and they put it on the cloud. Far faster. You can also do this to restore too. Pretty cool feature really.

It's too bad they don't make a single drive that would fit all your data. You have to choice but to do some sort of RAID. Either like a RAID 0 of 2x4TB drives or, 3x4tb drives or 3x3tb drives. The lack of security when it comes to RAID 0 doesn't really matter too much when it's just a backup.

You could just take the RAID 5 and make it a RAID6 for the extra security.... but it still won't help if the raid itself corrupts. There is always the option of just buying perhaps 2x4TB external USB drives and instead of RAIDing them, just set certain things to go on each drive. I don't know how you have everything set up so I'm not sure how much of a Hassle that sort of set up would be.

Why is it that backing up a large (but not too large) amount of data is so hard

Well, 10TB HDDs are supposed to be coming soon right? I could use those but I'll have to wait and see what they'll cost

[dadasmithywinkle]

Well, 10TB HDDs are supposed to be coming soon right? I could use those but I'll have to wait and see what they'll cost

Yep! And 100 of those will get you a 1PB server...

[chilicheeseburger]

Yep! And 100 of those will get you a 1PB server...

Would be quite risky to build a 1PB server with new drives where you can't do much research on failure rates.

[ictdude1]

best bet is to get 2 4tb hard drives and an enclosure that allows them to be paired as a raid....if you have the money of course, doing it though and online backup source will take alot longer, you just wana hope that what ever enclosure you are using supports usb 3 same with the computer its stored on

[swifty7]

Fair enough.

I too am currently looking into some form of backup solution for my freenas. I was considering crashplan, because they're quite cheap and have freenas integration. My upload speed is only 10mbps (soon to be 20mbps when I upgrade), however I noticed crashplan does this cool thing where they'll actually send you some hard drives, you transfer your data onto them and send them back, and they put it on the cloud. Far faster. You can also do this to restore too. Pretty cool feature really. 

 

It's too bad they don't make a single drive that would fit all your data. You have to choice but to do some sort of RAID. Either like a RAID 0 of 2x4TB drives or, 3x4tb drives or 3x3tb drives. The lack of security when it comes to RAID 0 doesn't really matter too much when it's just a backup.

 

You could just take the RAID 5 and make it a RAID6 for the extra security.... but it still won't help if the raid itself corrupts. There is always the option of just buying perhaps 2x4TB external USB drives and instead of RAIDing them, just set certain things to go on each drive. I don't know how you have everything set up so I'm not sure how much of a Hassle that sort of set up would be. 

 

Why is it that backing up a large (but not too large) amount of data is so hard  

that's a terrible idea!!!  what if you have 8TB worth of pron?  Will crashplan go through every single file and laugh at your expense? Or maybe they will keep some pron of interest for themselves..LOL!!

 

thanks but no thanks!!!

 

yeah...yeah!!  I'm sure they will tell you that they have all kinds of privacy....yeah! right!!  like I'm so gullible  to believe that.

 

on a serious note,  I don't trust cloud storage in general let alone one that offers hdd's to be sent to you to backup your important data. 

[MoonSpot]

http://www.drobo.com/storage-products/5n/ Or something like it.  So long as it's backing up your active array and isn't taxed beyond backing up and regular piece of mind confirmations, it should be fairly bulletproof.  I ignore mine until the "oh $#!7" moments occur.

[chilicheeseburger]

Thank you guys for all the input!

best bet is to get 2 4tb hard drives and an enclosure that allows them to be paired as a raid....if you have the money of course, doing it though and online backup source will take alot longer, you just wana hope that what ever enclosure you are using supports usb 3 same with the computer its stored on

That's probably the solution I'm going with. But the harder part is how can I create an automatic backup process (in ubuntu)?

[ictdude1]

Thank you guys for all the input!

That's probably the solution I'm going with. But the harder part is how can I create an automatic backup process (in ubuntu)?

https://help.ubuntu.com/community/BackupYourSystem

[chilicheeseburger]

https://help.ubuntu.com/community/BackupYourSystem

 

easier than I thought, this helps a lot. Thanks!

[unijab]

You're using that incorrectly.

 

RAID does not give you security. It provides "redundancy" ; hence the "R" in RAID (except level 0).

 

 

 

 

The lack of security when it comes to RAID 0 doesn't really matter too much when it's just a backup.

 

You could just take the RAID 5 and make it a RAID6 for the extra security.... but it still won't help if the raid itself corrupts. 

 

[wpirobotbuilder]

If you have 8TB of data you genuinely need backed up, you're either a business or a very enthusiastic home user.

 

Either way, you're starting to get into the realm of solutions that businesses use to back up data. It's expensive to back up data, so you should only do it for the data you really care about.

 

The cheapest way is to get two 4TB drives and store them offline, syncing them once in a while. If you want to avoid data corruption you'll also want enterprise drives, since reading 8TB of data carries a significant risk of encountering an unrecoverable error. The only way to prevent that is to buy better quality drives.

[laxer]

You're using that incorrectly.

 

RAID does not give you security. It provides "redundancy" ; hence the "R" in RAID (except level 0).

 

No need to get your panties in a bunch, it gives security against data loss through redundancy. Security is very abstract so open your mind and stop acting like a 13 year old troll.

 

@OP, I would set up a RAID1/5/6 NAS somewhere on your local network away from your desktop.

 

My guess is that if you are trying to set up another level of redundancy it is from a physical attack. (Fire, burglary, etc)

 

While I would encourage offsite storage for something like this with 8TB of data and a relatively low upload speed it just is not feasible.

 

Running a cable across the house to a NAS/Server will give you some protection from physical attacks but, not a crazy amount. IMO, this is your best bet. 

[Captain_WD]

I have 8TB of available storage in a software RAID5 array on a home server I've built and now I'm looking for a good way to back up all that data regularly.

 

What I have an mind is some external enclosure with it's own RAID array of hard drives that would be connected to the server via USB 3.0 that backs up the data regularly (let's say every 24h), of course skipping unchanged files. I'm running Ubuntu on the home server.

 

Is this possible , has anyone done this or something similar?

 

Of course if you have a different idea on how to back up large amounts of data, don't hesitate to bring that up.

 

 

Hey chilicheeseburger,

 

Your plan seems pretty good and using a software to do that is a good idea. There are a number of good ones out there. I could suggest checking out WD SmartWare: http://products.wdc.com/support/kb.ashx?id=HyC6x0

 

Another option is to configure your system to do backups on another computer on your network or one or more external drives connected to a router. Do have in mind that a secure backup is considered one that is not connected to your system in order to prevent any damage during power failure or power shortage.

 

Captain_WD.