So why a NAS over a roll-your-own?

[kiddsupreme]

I had a question I was hoping you gents/ladies could help me answer. In recent years, it has become apparent that NAS systems have been increasingly popular for home use (i.e. Synology, etc). However, I always wondered why someone would prefer one of the pre-built solutions over a purpose-built home server with hard drives you provided yourself. For example, I have a Linux box (A Ivy Bridge i5, with 16GB of RAM, 128GB SSD for host OS duties, and a total of 20TB of available HDD space). 

 

I have been able to run multiple applications concurrently without noticing any performance degradation (Plex Server, sabNZBD+, Sickbeard, Couchpotato, Headphones, Mylar, Samba, etc). However, I have had friends who own Synology devices and have tried to run a fraction of these programs, and have their systems slow to a crawl. 

 

So I ask the simple question: Taking out the convenience factor of these pre-built NAS devices (design, hardware support, warranty, dimensions, etc), why would you select the NAS over the homegrown Linux box? 

[Fulgrim]

It's the same with with prebuilt PC's.

 

People either can't be arsed putting one together themselves, or they lack the know-how.

[Darkstar101]

I prefer pre-built NAS because usually they're ARM and eat little power.

I have a D-Link DNS-325 2 bay NAS with 2x 4Tb WD Red, and I'm planning to upgrade to a D-Link DNS-345 4 bay NAS with 4x 8Tb WD Red.

[kiddsupreme]

I prefer pre-built NAS because usually they're ARM and eat little power.

I have a D-Link DNS-325 2 bay NAS with 2x 4Tb WD Red, and I'm planning to upgrade to a D-Link DNS-345 4 bay NAS with 4x 8Tb WD Red.

 

But what if:

 

1. Power consumption isn't a concern for you
2. Because you plan on having this system run 24/7, and you want the processing power an Intel or AMD processor has over an ARM processor

 

Would there be any other reasons you'd select the NAS?

[kiddsupreme]

For NAS setup I would choose Arch > mdadm > LVM > Btrfs > LXC > ownCloud, Transmission, etc. Or alternatively zfs-git with ECC RAM. Other options with a web GUI include FreeNAS, NAS4Free or OpenMediaVault. A web server manager called Cockpit for systemd is being developed for Fedora which is also available in Arch.

 

Thanks for the post. I utilize Fedora (mainly because I've been using Red Hat since 1995), and I'm use to it, and I appreciate it being updated every 6 months.  Are you saying that people like yourself prefer NAS devices because of the web GUI interface over what setting up your own Linux distro would require?

[SMURG]

Rolling your own is always an option, but using an i5 to do it is beyond overkill. I make my own NAS systems but I use Atoms - more than powerful enough, low-power and completely silent because they're passively cooled. 16GB of RAM is also overkill, unless you're running a massive ZFS array with deduplication and whatnot I suppose. I run ZFS on 4GB just fine, although I haven't tried dedupe yet.

[kiddsupreme]

Rolling your own is always an option, but using an i5 to do it is beyond overkill. I make my own NAS systems but I use Atoms - more than powerful enough, low-power and completely silent because they're passively cooled. 16GB of RAM is also overkill, unless you're running a massive ZFS array with deduplication and whatnot I suppose.

 

Thanks for the response SMURG... I guess I should have been more forthcoming with the information in the beginning. Yeah, the i5 and the 16GB of RAM are most likely overkill for the job. However, there are times when I utilize my Linux box for things beside what was already mentioned (processing blu-rays, dvds, etc.) that benefit from faster processors and sometimes more RAM.

[kiddsupreme]

I'm saying I prefer to set up my own NAS with Linux using only the command line.

Ahhh! Ok, my mistake -- I misinterpreted what you were trying to say. I too prefer utilizing the command line for the majority of my tasks. 

[kiddsupreme]

I've switched from Ext4 to Btrfs for a while now mainly due to the transparent compression feature for my VMs. I started experimenting with its multi-driver feature and was surprised how easy it was to convert from a single drive > RAID 0 > RAID 1 > RAID 5 all while online and mounted.

 

I have to admit that I do not run RAID on my Linux box... I'm more concerned with capacity than reliability (crazy talk, I know). Of the 20TB of data, I'd say 700GB of that is stuff I can't afford to ever lose (home photos, video, etc.) I have Crashplan, and a Amazon EC2 for backing up the important memories. Everything else would be highly inconvenient if I lost it, but it wouldn't kill me. 

 

The reason I was asking about NAS setups and everything was I was thinking I would get something modest in terms of power: The NAS device's primary job would be to backup data from the Linux box. Not sure if I still want to go this route or not, because every time I think about buying the storage required to backup 20TB, I start thinking that I'd rather add an ADDITIONAL 20TB of available storage in my Linux box... 

[kiddsupreme]

Here is a script I made to setup ownCloud in Arch.

 

http://linustechtips.com/main/topic/184184-how-to-self-host-your-own-cloud-storage-with-owncloud/?p=2545747

 

I have never heard of ownCloud before. I take it its like having your own Cloud provider? Does it provide encryption services and things like that?

[Darkstar101]

But what if:

 

1. Power consumption isn't a concern for you
2. Because you plan on having this system run 24/7, and you want the processing power an Intel or AMD processor has over an ARM processor

 

Would there be any other reasons you'd select the NAS?

No, I only care about power consumption, check my computers in my signature.

[kiddsupreme]

No, I only care about power consumption, check my computers in my signature.

 

Well played sir, well played... 

[kiddsupreme]

http://doc.owncloud.org/server/6.0/admin_manual/apps/files_encryption/index.html

 

in addition you can also encrypt your entire raid array or drive with LUKS. 

 

So what are you suggesting? An online backup solution with ownCloud? Or a local solution utilizing ownCloud?

[kiddsupreme]

If you're just storing data then rsync or Filezilla over SSH is probably the easiest and most secure.

 

I'm basically trying to decide between:

 

1. a local backup solution
2. a internet backup solution

For 20TB. Once I have that information, the next piece I would like to determine is the best way to go about accomplishing either one of these...

[kiddsupreme]

Do you plan on having redundancy on your backup machine or continue keeping the drives separate? I've heard of 8 and 10TB drives coming later this year/next year so those would be the cheapest $/TB. Google Drive have an unlimited storage for $10 per month plan.

 

I was thinking on keeping it separate. Something where I could back it up, then take it offsite just in case. I'm not sure about the feasability of backing up to the cloud... Could you imagine having to restore 20TB over the Internet?

 

I also was hearing whispers about 8 & 10TB hard drives... But knowing myself, I'd just upgrade the current storage in my Linux Box, and do something stupid like 100TB JBODs, lol

[alpenwasser]

Most common arguments (convenience, power consumption) have

already been mentioned, tech support could also be one (Synology

for example seems to be rather excellent at that).

Personally I'm running my server with Arch Linux and ZFS on it,

so far my experience has been good.

Aside from that I just wanted to pop in for two thing:

1. @kiddsupreme: Sounds like your box may be eligible for

our Storage Showoff thread, which can be found here.

2. 

I run ZFS on 4GB just fine, although I haven't tried dedupe yet.

 

Just a warning: I've read of two people on [H] who lost their entire

ZFS pool (non-recoverable) when they tried out dedupe and did not have

enough RAM. So unless you're just doing it to play around, or really

have a ton of RAM, I would highly advise against enabling dedupe.

You may already know this of course, but considering the rather severe

consequences this can apparently have, I thought I'd mention it.

[threadysparrow]

Because I'm not paying $600 for a Linux Box when I can pay $300 for what I need. Not everyone needs to use all of those programs. The majority of people only need them for storage.

[SMURG]

Just a warning: I've read of two people on [H] who lost their entire

ZFS pool (non-recoverable) when they tried out dedupe and did not have

enough RAM. So unless you're just doing it to play around, or really

have a ton of RAM, I would highly advise against enabling dedupe.

You may already know this of course, but considering the rather severe

consequences this can apparently have, I thought I'd mention it.

Oh yeah, if I was gonna enable dedupe I'd max out all the RAM slots first. S'why I said I could understand the need for 16GB if he was rolling with dedupe.

[alpenwasser]

Oh yeah, if I was gonna enable dedupe I'd max out all the RAM slots first. S'why I said I could understand the need for 16GB if he was rolling with dedupe.

Just thought I'd add this link here about the subject.

For those who don't with to read the entire thing: How much

space the dedupe table takes up depends on the number of allocated

blocks in your ZFS pool. Since the block size in ZFS can be

variable, this can be a bit tricky to calculate (more on it here).

However, assuming an average block size of 64K (which is a pretty

reasonable guess for most cases, otherwise, adjust calculations

accordingly), it would look something like this:

 

Amount of storage: 1 TB (1,073,741,824K)Number of blocks: 1,073,741,824K/64K = 16,777,216Dedupe table size: 16,777,777 blocks * 320 bytes/block = 5,368,709,120 bytes ≅ 5 GB

Now, naturally ZFS does not exclusively keep ZFS dedupe tables in

its RAM, but also lots of other data (other metadata, cached block

data). The total amount of the ARC which metadata is allowed to

take up (dedupe tables count as metadata apparently) is is 25%, so

if you want to keep 5 GB worth of dedupe table in your memory,

you'll need about 20 GB of RAM per TB of storage in your ZFS pool.

Now, it is also mentioned that one need not necessarily keep all

dedupe table data in memory, in which case RAM requirements can

be reduced, however the performance of your pool will likely severly

degrade in such a situation, and as said, there have apparently

been cases of data loss in such scenarios (although that might

get fixed at some point as it obviously was buggy behavior from

what I can tell).

So yeah, bottom line:

For the ZFS dedupe table to reside completely in memory, provision

about 20 GB of RAM per 1 TB of ZFS pool capacity (blimey, that

would make 580 GB of RAM for my build ).

EDIT

As said though, this is just a rough calculation based on average

block size. If you primarily have large files which mostly use

the maximum block size, this could go down significantly, if you

primarily have small files which use smaller blocks, it could go

much higher still.

/EDIT

For completeness' sake, here's also the dedupe page from Oracle's

docs, although the metadata/ARC thing isn't mentioned on there

it seems.

[techswede]

Why would you "roll your own" when the store bought ones work just as well?

 

I just realized that this is the root of all evil in this world. A cigarette that has micro usb

[kiddsupreme]

Wow, thanks folks for the additional information contributed to this thread. I have to admit I don't have any first hand knowledge or experience using zfs, I've just mainly using ext4 as my file system of choice. Believe it or not, I've been looking at tape storage as well. LTO tape storage to be exact. Considering LTO-4 tapes can be procured for as little as $20 a cartridge, and can hold anywhere from 800gb to 1.6tb depending on compression, it would seem to be more cost effective as far as the amount of data to be stored. Of course, getting your hands on a LTO-4 drive for a reasonable price is a challenge in itself. But I may have a friend who works in a data center who might be able to let me use one they are planning on mothballing. If that's the case, it's just a matter of getting a SCSI card to install on the linux box and the appropriate drivers.

I guess we will see going forward what is the better long term storage option.

[IdeaStormer]

Wow, thanks folks for the additional information contributed to this thread. I have to admit I don't have any first hand knowledge or experience using zfs, I've just mainly using ext4 as my file system of choice. Believe it or not, I've been looking at tape storage as well. LTO tape storage to be exact. Considering LTO-4 tapes can be procured for as little as $20 a cartridge, and can hold anywhere from 800gb to 1.6tb depending on compression, it would seem to be more cost effective as far as the amount of data to be stored. Of course, getting your hands on a LTO-4 drive for a reasonable price is a challenge in itself. But I may have a friend who works in a data center who might be able to let me use one they are planning on mothballing. If that's the case, it's just a matter of getting a SCSI card to install on the linux box and the appropriate drivers.

I guess we will see going forward what is the better long term storage option.

 

Not sure if you know but most LTO tape drives use Fiber Channel so you'll also need a fiber channel switch to get data off of it if not directly attached to a fiber channel card. You can't just buy the drive on its own and use it. I haven't seen one on SCSI unless you're talking a DLT tape drive which is less capacity, depending on its version. Then there's the whole how are you saving the data on to the tape, tar? proprietary software? How are you going to document what is where so you know which tape to load if not using some king of journaling tape backup software?

[kiddsupreme]

Not sure if you know but most LTO tape drives use Fiber Channel so you'll also need a fiber channel switch to get data off of it if not directly attached to a fiber channel card. You can't just buy the drive on its own and use it. I haven't seen one on SCSI unless you're talking a DLT tape drive which is less capacity, depending on its version. Then there's the whole how are you saving the data on to the tape, tar? proprietary software? How are you going to document what is where so you know which tape to load if not using some king of journaling tape backup software?

 

Sorry I misspoke. Yes, it is Fibre Channel. In regards to what software I would plan on utilizing, I would imagine something in the open source space (if I had to buy something like Symantec Netbackup, I would imagine the costs would get out of hand quickly). As far as keeping track of what data is on which tape, I'd imagine some manual system (separate it out by physical hard drive). I would imagine it shouldn't take more than 10-12 physical LTO-4 tapes to backup everything. Of course I could be mistaken, but it is definitely a project that sounds promising if it can be executed properly.