Network Storage Design

[mike911]

Forum Post

 

Hi, I am deciding between possibility of opting for DIY, Off-Shelf NAS or stay with my existing 2-bay NAS. Seeking advise about how to re-design my existing setup that I have or the way I manage my data. I have been spending alot of time to manage, re-sort and re-sync my data across all disks as the 2 main drives are constantly full. Which is exhaustive as it can take days to get the sync-ing done.

 

Existing Setup

[EXT4] 2x2tb on a Dlink DNS-320 [service Drives]

[EXT4] 2x1tb and 2x3tb Spare Drives [backup Duplicate of data]

[New] And recent additional of 2x3tb Spare Drives

 

There is no RAID 0/1 deployed or scheduled backup done. What I usually do is to find some spare time every 3-4 mths and manually copy/sync the disks across through another desktop as I need the sata ports with . How I know which files to be synced are stored as an additional duplicate copy across the 2x2tb service disks. (Which was the primary cause of disk usage). However as manual file movement is involved. There are still files being missed out.

 

I tend to over backup (1:1 or 1:2, as main:backup), as I have past experience of losing data. My data consist largely on Video Media and some smaller data (School, Work, Personal)

 

I drafted a few plans but unsure which path to choose (i.e. budget, green operation, cost effectiveness, easier management)

1. DIY NAS with 5 Bay (Without RAID), 1tb for scratch disk, 2x3tb as video disk, 1x2tb as data disk

Risk : Low

Spares : Sufficient Spares for 1:1 backup

Effort : Large Effort required for syncing data

Cost : Medium Build Cost, Mid-High Operating Cost

2. DIY NAS with 5 Bay (With 4 Bay RAID5), 1tb for scratch, 4x3tb as video and data [RAID 5]

Risk :  Low-Medium

Spares : In-Sufficient Spares for 1:1 backup

Effort : Medium Effort required for syncing Critical Data

Cost : Medium Build Cost, Mid-High Operating Cost

3. Off-Shelf 4 Bay (1 Spare), 1tb for scratch, 3tb for video, 2tb for data

Risk :  Low

Spares : Sufficient Spares for 1:1, or 1:2 backup with Extra Spare Storage for additional data

Effort : Large Effort required for syncing Critical Data

Cost : Mid-High Build Cost, Low-Mid Operating Cost

4. Continue existing 2-Bay NAS, (Use my desktop as disk management), DISK1 as scratch, video and DISK2 for backup and data

Risk :  Low

Spares : Sufficient Spares for 1:1, or 1:2 backup with Extra Spare Storage for additional data

Effort : Largest Effort required for syncing Critical Data

Cost : Lowest Build Cost (Already Have), Lowest Operating Cost

 

Although my data is in a mess now, but I dont want to rush the planning hence I hope to find some better off solution in terms of management or to redesign another new setup.

[Enderman]

Raid 5 or raid 10, like linus did in his home NAS video is the best, safest, and easiest option

[mike911]

Okays, I'll take a look at that video.

[alpenwasser]

@mike911:

When you say "effort for syncing", do you mean the actual

work, or just the time required until it's done?

[mike911]

oops. I was referring to the effort required to sync the data and backup across all disks

[alpenwasser]

oops. I was referring to the effort required to sync the data and backup across all disks

I'd say in that case the primary issue would be to find a syncing

tool which is easy to use and which you can just start and let

run for however long it takes to finish the job. If you have the

right tool, syncing should become pretty painless, regardless of

which of your choices you end up going with.

For instance, I have to keep my dad's business data synced across

three computers, and I use rsync to do that. One command will

sync two machines, so to sync all three it's just two commands

and the rest is done automatically. And it does incremental backups,

so it will only update what has been changed since the last sync,

so after the initial sync it's actually pretty quick.

If you're not comfortable with terminal tools, there are GUI options

as well, though I don't use any at the moment.

Wikipedia has a pretty expansive list of backup software, if you're

looking for inspiration.

I've heard good things about BackupPC, though I've never personally

tried it. But if I were to switch to something other than rsync, I'd

probably give that one a try.

[infamous_last]

SyncBack (windows based) has served me well for years, free version has all the options you need.

I use a HP Microserver as a NAS/HTPC.

The barebones model is a great base and dirt cheap (N54L) for what it is.

[mike911]

I'd say in that case the primary issue would be to find a syncing

tool which is easy to use and which you can just start and let

run for however long it takes to finish the job. If you have the

right tool, syncing should become pretty painless, regardless of

which of your choices you end up going with.

For instance, I have to keep my dad's business data synced across

three computers, and I use rsync to do that. One command will

sync two machines, so to sync all three it's just two commands

and the rest is done automatically. And it does incremental backups,

so it will only update what has been changed since the last sync,

so after the initial sync it's actually pretty quick.

If you're not comfortable with terminal tools, there are GUI options

as well, though I don't use any at the moment.

Wikipedia has a pretty expansive list of backup software, if you're

looking for inspiration.

I've heard good things about BackupPC, though I've never personally

tried it. But if I were to switch to something other than rsync, I'd

probably give that one a try.

 

To reflect back, its quite true that sync-ing backups is part of the primary issue.

I heard before about rsync, do let me read up more about the backup solutions.

Wont say that i am very proficient with terminal tools, but if there are sufficient guides, i wont mind giving it a try since the underlying OS of the NAS is linux.

 

However I noticed that the NAS CPU is bottlenecked, as it does take up quite large amount of time during file transfers between my computer (through Gigabit network) or within internal disks. Which was also why I am exploring if the possibility of changing or migrating hardware is required.

 

 

SyncBack (windows based) has served me well for years, free version has all the options you need.

I use a HP Microserver as a NAS/HTPC.

The barebones model is a great base and dirt cheap (N54L) for what it is.

 

 

I see, let me look up on it too. As most of my disks are in ext4 format. I usually manage them using ubuntu (i.e. disconnect my computer disks, and rewire to my spares) instead of using 3rd party windows drivers (http://www.ext2fsd.com/)

 

The HP Microserver looks cool, but to convert to Singapore would be approx SGD$550 (Based on Amazon USD$388). Somewhere close to a Synology/Asustor 4-Bay NAS.

[alpenwasser]

To reflect back, its quite true that sync-ing backups is part of the primary issue.

I heard before about rsync, do let me read up more about the backup solutions.

Wont say that i am very proficient with terminal tools, but if there are sufficient guides, i wont mind giving it a try since the underlying OS of the NAS is linux.

rsync has a ton of options, so getting what you need out of it

can be a bit overwhelming for beginners. Personally I pretty

much use only one command:

rsync -arv --progress source/ target/

There are however some GUI wrappers for it which could make it

easier to use if you're not that comfortable with the CLI, for

example grsync. There's also duplicity, which is a GUI tool

that uses the rsync algorithm as well.

[wpirobotbuilder]

I'd recommend going with a solution that suits your level of hassle required. DIY involves much more work and troubleshooting, but you can customize your solution much more than a prebuilt. Also if you like tinkering it's a good option.

 

If you want a solution that just works, prebuilt really is the way to go.

[mike911]

rsync has a ton of options, so getting what you need out of it

can be a bit overwhelming for beginners. Personally I pretty

much use only one command:

 

rsync -arv --progress source/ target/

There are however some GUI wrappers for it which could make it

easier to use if you're not that comfortable with the CLI, for

example grsync. There's also duplicity, which is a GUI tool

that uses the rsync algorithm as well.

 

 

 

I've did a quick youtube for rysnc tutorials, and I'm surprised to see how it functions. (Initially i thought it will only pull data incrementally and bulking in data size without removing "deleted" files). I will spend more time to understand its functionality. Thanks for sharing about rsync.

In this case, I think this command is to be executed from my computer (where the spare disks are plugged to) instead from my NAS. So ideally I'm managing from my computer and pulling/syncing data through the NAS.

 

 

I'd recommend going with a solution that suits your level of hassle required. DIY involves much more work and troubleshooting, but you can customize your solution much more than a prebuilt. Also if you like tinkering it's a good option.

 

If you want a solution that just works, prebuilt really is the way to go.

 

I'd like to explore DIY solution if it is feasible in comparison with pre-built. Of course there are pros and cons of each (i.e. power consumption, price, etc.) Although i wont deny that going for DIY, I'll definitely need to spend some quality time to learn the OS, like freenas, or some form of Linux based OS.

[mike911]

Hi guys, does any one have any comments about asustor NAS (in terms of plugins, flexibility in terms of data management, reliability)?

e.g. An Intel Atom AS-304T vs a Marvel DS414 cost similar. but an intel Synology variant (DS412+) cost way more