NAS System with huge amount of files needed to be synced to a cloud

[HiroKhan]

Heyahey,

 

I'm a techie working in a small design agency with 3 employees and we want to upgrade & secure our storage system...

So currently, we have a Synology DS213+ with 6TB of storage running in RAID... (So, 3TB usable storage) We work from this NAS (live) and store all our work files/PC backups/... on it. It works pretty basic but well. Nonetheless, we need to advance our small storage.

 

The problem: We have 898,286 files in 141,236 folders and we need to Backup those files GDPR conform to a cloud service. We tried out CloudSync by Synology from the package center but we ran into trouble with sync errors, because our NAS system is only designed for about 50,000 files being synced. (I guess it's a index issue, anyway files on the cloud backup are missing, wrong and not updated, it's a known issue declared in the synology community)

 

With that in mind, we decided to upgrade our storage system.

 

Our wish: We want to backup all 898,286+ files using OneDrive or HiDrive (we prefer HiDrive, because it's a German service running under German laws) and therefore we need a more powerful solution that is capable of syncing that many files without problems. We have 5 PCs connected to and working from the NAS constantly uploading and updating our work files to the Diskstation. My dream would be an 45Drives NAS, but that is totally overkill and I guess not really needed for our work scenario. We just need a simple, future proof and low-maintenance solution that works great with that many files and doesn't run into problems with syncing. It should be under 600€ and should be capable of uploading/downloading data from a cloud service without problems.

 

I guess my problem is pretty basic, but due to the amount of files that needed to be synced, I didn't found a NAS that works flawlessly... So I hope you have some advice or a piece of hardware you can suggest

 

 

Thank you,

-Alex

[Electronics Wizardy]

First thing, is there any way to cut down on the number of files? That would make everything better.

 

The hige end sysnology nas boxes handle up to around 400k files I think, but your still over that limit now.

 

You probably want something like S3 here for this. There made to handle many more files and speed than a normal cloud drive for consumers.

 

Id look into moving to a full serer instead of a nas, id probably go with something like windows server here if your running windows systems. Then you can use something like veeam to backup the system to s3 or similar. Also those backups are normally block based so the number of files won't really matter.

[Radium_Angel]

13 minutes ago, HiroKhan said:

It should be under 600€

 

13 minutes ago, HiroKhan said:

 898,286 files in 141,236 folders

 

13 minutes ago, HiroKhan said:

cloud service

 

14 minutes ago, HiroKhan said:

future proof and low-maintenance solution that works great

Good luck.

I'd start adding zeros to that price tag....

[mariushm]

Is it possible to make archives out of very old folders you don't frequency access?  You can zip, you can use 7zip (free open source archiver) and make 7z archives for better compression, you can use WinRAR if you want to add extra error recovery information ... and a side effect is you also reduce the amount of disk space if the files are compressible.

 

 

[eldiabolo]

These file numbers shouldnt really be a problem anymore in 2020. It might take a while (days or weeks) for the inital upload, but generally this is possible. I don't know about any of your suggested hosting solutions, but I'd search for something that can be mounted in a linux and then let rsync run.

Maybe the NAS can host a linux vm/ container with rsync? Mount the filesystem of the data in that, mount the external backup-provider and rsync away.