Recommendations for a good backup system? feat. 'Backblaze restore isnt great...'

[japers]

Hi All,

 

Ive recently set up my own company as a concept artist in film and entertainment and i have about 6 terabytes of project files and assets that i need to keep very safe. I currently have backblaze PLUS a windows backup on an external drive thats always connected. I recently discovered that backblaze has actually got really terrible restore options and its either a zip file (my data is way too large for that) or a drive shipped to my address which given i live in the UK and backblaze is US, probably isnt economical.

 

I feel like i should probably find another cloud provider as well as start taking disk images twice a month too.

 

Can anyone recommend another cloud platform or anything else i should be doing? Given that the data is sort of my livelihood and money maker i need to make sure that i have quick access to it if something dies and i cant spend a week resyncing it all from a cloud which is why i feel i need another local copy.  3-2-1 And all that jazz!

 

Any suggestions would be great!

 

[Wictorian]

18 minutes ago, japers said:

Hi All,

 

Ive recently set up my own company as a concept artist in film and entertainment and i have about 6 terabytes of project files and assets that i need to keep very safe. I currently have backblaze PLUS a windows backup on an external drive thats always connected. I recently discovered that backblaze has actually got really terrible restore options and its either a zip file (my data is way too large for that) or a drive shipped to my address which given i live in the UK and backblaze is US, probably isnt economical.

 

I feel like i should probably find another cloud provider as well as start taking disk images twice a month too.

 

Can anyone recommend another cloud platform or anything else i should be doing? Given that the data is sort of my livelihood and money maker i need to make sure that i have quick access to it if something dies and i cant spend a week resyncing it all from a cloud which is why i feel i need another local copy.  3-2-1 And all that jazz!

 

Any suggestions would be great!

 

Google Drive is probably the best choice, at least I know of.

But recently I downloaded some files and they were in a zip file. I don't think that's mandatory or will cause issues though.

[Brooksie359]

A good raid setup is always a great way to minimize the risk of losing your data. 

[japers]

Just now, Brooksie359 said:

A good raid setup is always a great way to minimize the risk of losing your data. 

I would use a RAID but i dont know much about the benefits given that the raid setup would be a significant investment for the storage needed for the overheads. Does it introduce any read latency when set up in an effective way that actually gives some redundancy?

 

Parsing times for my renders are something i need to monitor and i dont want to add any additional read time for larger texture files (we are talking dozens of TIFFs in the hundreds of MB each being loaded from disk every time i do a test render which could be 10 times per hour.

[japers]

18 minutes ago, Wictorian said:

Google Drive is probably the best choice, at least I know of.

But recently I downloaded some files and they were in a zip file. I don't think that's mandatory or will cause issues though.

I thought about it but the idea of posting all of the work some of which is NDA'd, to google drive, doesnt seem right.

 

Im actually quite annoyed at backblaze because their desktop app offers no restore functionality and it all has to be done through the web interface where the only options are download as zips which doesnt work for large amounts of data, or get a drive shipped to you. And given i live in the UK and the only have US offices isnt really an option as the shipping cost would be insane to return the drive.

 

It would be great to see that glaring issue addressed considering they advertise unlimited storage. Yes its unlimited. But good luck getting a significant amount of data back quickly.

[Guest]

2 hours ago, japers said:

Hi All,

 

Ive recently set up my own company as a concept artist in film and entertainment and i have about 6 terabytes of project files and assets that i need to keep very safe. I currently have backblaze PLUS a windows backup on an external drive thats always connected. I recently discovered that backblaze has actually got really terrible restore options and its either a zip file (my data is way too large for that) or a drive shipped to my address which given i live in the UK and backblaze is US, probably isnt economical.

 

I feel like i should probably find another cloud provider as well as start taking disk images twice a month too.

 

Can anyone recommend another cloud platform or anything else i should be doing? Given that the data is sort of my livelihood and money maker i need to make sure that i have quick access to it if something dies and i cant spend a week resyncing it all from a cloud which is why i feel i need another local copy.  3-2-1 And all that jazz!

 

Any suggestions would be great!

 

Where is the data sitting? I'd be looking at using Azure Files as a destination for backup (or AWS if that's your preference), it's just a case of how you get it there. If it's on a NAS (e.g. Synology, QNAP), they have native Azure & AWS backup options while with Windows Server, you could look at something like Veeam to replicate to Azure. 

[LAwLz]

9 hours ago, Brooksie359 said:

A good raid setup is always a great way to minimize the risk of losing your data. 

Just remember that RAID is not a backup solution. RAID is nice, but only if you use it in addition to a proper backup.

[Donut417]

12 hours ago, japers said:

Can anyone recommend another cloud platform or anything else i should be doing?

Id replace the external drive with a NAS. Something that has multiple bays that uses RAID or ZFS just to protect your onsite backup with al layer of redundancy. As far as cloud providers go, Google, Microsoft, Apple and drop box are a few options. Maybe see if you can find a provider who's in the UK or EU. Another option is if you have a secondary place that you can place a NAS, such as a friends or family members house you can just buy a second NAS and have them sync, the only issue with this option is its all on you. 

[IntMD]

This is from Backblazes website. Sounds like your data is probably in the Amsterdam data centre, so I imagine a disk backup would be sent from there and not the US.

 

Where are my files stored?

Backblaze Help

January 27, 2020 19:25

Follow

Backblaze currently has data centers in Sacramento California, Phoenix Arizona, and Amsterdam Netherlands. 

The region for an account, selected at the time of account creation, will determine what data centers may be used to store data for the account. The region for an account is available via the "My Settings" page of the Backblaze website, once signed in. 

Accounts in our US-West region will store data in both Sacramento and Phoenix data centers. Accounts in our EU-Central region will store data in the Amsterdam datacenter. 

[Helpful Tech Witch]

16 hours ago, Wictorian said:

Google Drive is probably the best choice, at least I know of.

But recently I downloaded some files and they were in a zip file. I don't think that's mandatory or will cause issues though.

Gdrive will zip the files into a zip if you download multiple files.

but if you download one folder, or one item your good

[Wictorian]

On 7/22/2021 at 1:31 AM, japers said:

I thought about it but the idea of posting all of the work some of which is NDA'd, to google drive, doesnt seem right.

Idk what you mean but I guess there will be no issues and really google drive is probably the best şf not perfectly adequate in all aspects.

 

On 7/22/2021 at 1:31 AM, japers said:

Im actually quite annoyed at backblaze because their desktop app offers no restore functionality and it all has to be done through the web interface where the only options are download as zips which doesnt work for large amounts of data, or get a drive shipped to you. And given i live in the UK and the only have US offices isnt really an option as the shipping cost would be insane to return the drive.

 

It would be great to see that glaring issue addressed considering they advertise unlimited storage. Yes its unlimited. But good luck getting a significant amount of data back quickly.

Do they not sell the drive? I think that is part of the program because then it would make it harder to proivde unlimited storage. I would say you can also give the drive to a friend/family or store it in a vault like in a bank or somewhere but syncinc would be difficult then. 

[12345678]

there are services that do offer unlimited data storage for free but you should check their tos

 

such as telegram

[Alan G]

That's a lot of data you have to store.  My experience may or may not be of much help.  I use Nova Backup to both a local drive set up and Amazon Web Service.  I've not had to do any restoration aside from just making sure things work OK (they do).  The nice thing with AWS, is that you can use other software to check the status of your storage.  I think that I am storing about 800 GB of data on AWS.  Uploading does take some time.  I pay about $4/month for storage (this also includes my wife's data).  My only experience with Azure is using them to host my small website.  I think you can get good pricing from them as well.

[tkitch]

First off, how much raw data are you needng to backup?

 

that's one of the most important questions

[WkdPaul]

Since this is your income source, I would go with what @Windspeed36 proposed ; on-site NAS with Azure or AWS backups.

 

The on-site will make sure you'll have quick access to all your files, and in the event of catastrophic failures, you'll have everything on the cloud (though rebuilding your on-premise backup will then be dependent on your internet connection).

 

While you currently have 6TB, that will not stay that size and will grow. So making sure everything is safe will cost money, and while you can appreciate the savings you're now getting with affordable or free solutions (though I doubt there are free solutions for 6TB+), you could regret it when something goes wrong.

 

Synology or QNAP have good solutions, but if you want to try to save money and like to tinker, building yourself a NAS and then using FreeNAS as the OS is also an option, you can also connect FreeNAS to AWS to have cloud backups (haven't looked if Azure is currently an option, but last I checked it relied on 3rd party options).

[GDRRiley]

On 7/21/2021 at 2:53 PM, japers said:

a drive shipped to my address which given i live in the UK and backblaze is US, probably isnt economical

backblaze pays for the shipping, and if you send the drive back you get refunded

you also are likely stored in the EU not the USA data centers

Edited July 23, 2021 by GDRRiley

[Garak]

You can use Google Workspace (Formally Gsuite) and upload using rclone with encryption if you're worried about NDAs.

[japers]

Thanks for all the ideas everyone!

 

TL:DR; I have chosen to go with the 10tb google drive plan after putting the monthly cost into the context of how my daily rate pays for multiple months of storage with them and that the loss of one or two days of revenue would pay for the storage for a year. I may also put together an offsite backup system with a NUC and another drive so i have a local but quickly accessible complete backup.

 

On 7/22/2021 at 11:42 AM, Donut417 said:

Id replace the external drive with a NAS. Something that has multiple bays that uses RAID or ZFS just to protect your onsite backup with al layer of redundancy. As far as cloud providers go, Google, Microsoft, Apple and drop box are a few options. Maybe see if you can find a provider who's in the UK or EU. Another option is if you have a secondary place that you can place a NAS, such as a friends or family members house you can just buy a second NAS and have them sync, the only issue with this option is its all on you. 

 A NAS or alternatively an offsite download-only backup at a friends using my NUC and a big drive might be a decent alternative to ensure i have a locally accessible but offsite backup.

 

On 7/22/2021 at 3:24 PM, IntMD said:

This is from Backblazes website. Sounds like your data is probably in the Amsterdam data centre, so I imagine a disk backup would be sent from there and not the US.

 

Where are my files stored?

Backblaze Help

January 27, 2020 19:25

Follow

Backblaze currently has data centers in Sacramento California, Phoenix Arizona, and Amsterdam Netherlands. 

The region for an account, selected at the time of account creation, will determine what data centers may be used to store data for the account. The region for an account is available via the "My Settings" page of the Backblaze website, once signed in. 

Accounts in our US-West region will store data in both Sacramento and Phoenix data centers. Accounts in our EU-Central region will store data in the Amsterdam datacenter. 

I did ask about this but was told that drives are only shipped from their california and arizona locations.

 

 

On 7/22/2021 at 3:43 PM, HelpfulTechWizard said:

Gdrive will zip the files into a zip if you download multiple files.

but if you download one folder, or one item your good

The good thing with Gdrive is having the ability to retain a constant local sync, something which the backblaze client doesnt have the features to do.

If my drives crap out entirely and i needed to download everything again i would be able to just redownload google drive sync and have it completely redownload everything with no caps on size. with backblaze it was either manually downloading thousands of zips, or having them mail me a drive halfway across the world.

 

 

On 7/23/2021 at 12:12 AM, Wictorian said:

Idk what you mean but I guess there will be no issues and really google drive is probably the best şf not perfectly adequate in all aspects.

 

Do they not sell the drive? I think that is part of the program because then it would make it harder to proivde unlimited storage. I would say you can also give the drive to a friend/family or store it in a vault like in a bank or somewhere but syncinc would be difficult then. 

Google drive is of course great, but im very aware that my google account is tied to alot of services that could in future have data breaches but also backblaze encrypts everything which i dont know that google drive does or at least its not immediately apparent from their marketing.

 

On 7/23/2021 at 3:16 AM, 12345678 said:

there are services that do offer unlimited data storage for free but you should check their tos

 

such as telegram

I would rather have something thats designed for data storage and has policies about backing up and being able to restore versions like google drive has but thanks.

 

On 7/23/2021 at 12:44 PM, Alan G said:

That's a lot of data you have to store.  My experience may or may not be of much help.  I use Nova Backup to both a local drive set up and Amazon Web Service.  I've not had to do any restoration aside from just making sure things work OK (they do).  The nice thing with AWS, is that you can use other software to check the status of your storage.  I think that I am storing about 800 GB of data on AWS.  Uploading does take some time.  I pay about $4/month for storage (this also includes my wife's data).  My only experience with Azure is using them to host my small website.  I think you can get good pricing from them as well.

I did consider setting up a custom AWS system but 'roll your own' was more of a last resort. Good to know their storage rates are reasonable though!

 

On 7/23/2021 at 3:36 PM, tkitch said:

First off, how much raw data are you needng to backup?

 

that's one of the most important questions

Its going to be around 4 terabytes required ant 2tb nice to have but choosing the 10tb from google drive as i have done gives me plenty of overhead.

 

On 7/23/2021 at 3:45 PM, wkdpaul said:

Since this is your income source, I would go with what @Windspeed36 proposed ; on-site NAS with Azure or AWS backups.

 

The on-site will make sure you'll have quick access to all your files, and in the event of catastrophic failures, you'll have everything on the cloud (though rebuilding your on-premise backup will then be dependent on your internet connection).

 

While you currently have 6TB, that will not stay that size and will grow. So making sure everything is safe will cost money, and while you can appreciate the savings you're now getting with affordable or free solutions (though I doubt there are free solutions for 6TB+), you could regret it when something goes wrong.

 

Synology or QNAP have good solutions, but if you want to try to save money and like to tinker, building yourself a NAS and then using FreeNAS as the OS is also an option, you can also connect FreeNAS to AWS to have cloud backups (haven't looked if Azure is currently an option, but last I checked it relied on 3rd party options).

Thanks for the thoughts. Youre right as soon as i put it into context that the price of a months 10tb on google drive is paid for many times over by my daily rate and that if i lose all my data id be out alot more money than a years worth of gdrive backups, it made total sense to start actually paying for backups now that im doing alright in my career!

 

Im considering using a spare NUC i have in a drawer to run an offsite download-only backup of the entire google drive at a friend or relatives house instead of a NAS and then i have a local but offsite backup that i can grab in 30 minutes if the worst happens and i need to actually resync everything from the cloud i wont lose any work time or revenue.

 

On 7/23/2021 at 5:16 PM, GDRRiley said:

backblaze pays for the shipping, and if you send the drive back you get refunded

you also are likely stored in the EU not the USA data centers

It seems they only ship drives from their US locations according to the email conversation i had with them. I t just doesnt suit my needs that theres no way for me to get all of my a chunk of data back overnight  through some form of restore and sync option and my only option would be to lose out on about a weeks revenue waiting for a drive in the mail.

 

On 7/23/2021 at 5:48 PM, Garak said:

You can use Google Workspace (Formally Gsuite) and upload using rclone with encryption if you're worried about NDAs.

Im looking into if google uses any sort of encryption in google drive at the moment. Im hoping they do.

[Wictorian]

On 7/24/2021 at 10:21 PM, japers said:

Thanks for all the ideas everyone!

 

TL:DR; I have chosen to go with the 10tb google drive plan after putting the monthly cost into the context of how my daily rate pays for multiple months of storage with them and that the loss of one or two days of revenue would pay for the storage for a year. I may also put together an offsite backup system with a NUC and another drive so i have a local but quickly accessible complete backup.

 

 A NAS or alternatively an offsite download-only backup at a friends using my NUC and a big drive might be a decent alternative to ensure i have a locally accessible but offsite backup.

 

I did ask about this but was told that drives are only shipped from their california and arizona locations.

 

 

The good thing with Gdrive is having the ability to retain a constant local sync, something which the backblaze client doesnt have the features to do.

If my drives crap out entirely and i needed to download everything again i would be able to just redownload google drive sync and have it completely redownload everything with no caps on size. with backblaze it was either manually downloading thousands of zips, or having them mail me a drive halfway across the world.

 

 

Google drive is of course great, but im very aware that my google account is tied to alot of services that could in future have data breaches but also backblaze encrypts everything which i dont know that google drive does or at least its not immediately apparent from their marketing.

 

I would rather have something thats designed for data storage and has policies about backing up and being able to restore versions like google drive has but thanks.

 

I did consider setting up a custom AWS system but 'roll your own' was more of a last resort. Good to know their storage rates are reasonable though!

 

Its going to be around 4 terabytes required ant 2tb nice to have but choosing the 10tb from google drive as i have done gives me plenty of overhead.

 

Thanks for the thoughts. Youre right as soon as i put it into context that the price of a months 10tb on google drive is paid for many times over by my daily rate and that if i lose all my data id be out alot more money than a years worth of gdrive backups, it made total sense to start actually paying for backups now that im doing alright in my career!

 

Im considering using a spare NUC i have in a drawer to run an offsite download-only backup of the entire google drive at a friend or relatives house instead of a NAS and then i have a local but offsite backup that i can grab in 30 minutes if the worst happens and i need to actually resync everything from the cloud i wont lose any work time or revenue.

 

It seems they only ship drives from their US locations according to the email conversation i had with them. I t just doesnt suit my needs that theres no way for me to get all of my a chunk of data back overnight  through some form of restore and sync option and my only option would be to lose out on about a weeks revenue waiting for a drive in the mail.

 

Im looking into if google uses any sort of encryption in google drive at the moment. Im hoping they do.

You can make another account. I'm pretty sure they encrypt it and they check for malware if the file size isn't too big