Building a NAS at home? Don't use unRaid or hardware RAID

[Miklos]

I love watching LTT videos but with the recent influx of NAS/Storage videos I often have to behave like an old man yelling at the TV because I see silly mistakes.

 

Building your own NAS? Here are some basic rules of thumb:

 

1) Filesystem has to be ZFS

2) No hardware RAID cards

3) You can never have too much RAM (aim for 1GB RAM per TB of usable storage)

4) FILESYSTEM HAS TO BE ZFS

 

If you stick to those few rules, you will have a great time, but this excludes unRaid which LTT likes to use for dedicated NAS builds. unRaid uses an inferior filesystem and has strange limitations while costing money in some situations. This is nuts, as there is a superior free alternative that is just as easy to use, FreeNAS. Personally I have a FreeNAS at home but I use FreeBSD professionally to run all our NAS solutions both in Europe and in Asia.

 

New to FreeNAS? There are many blog posts, guides and videos on the subject but I always recommend checking out the excellent official documentation: http://doc.freenas.org/9.3/freenas.html (current version)

 

And I'm sure there are a lot of people here that are more than willing to aid you in case you need help finding hardware.

[TheGamingBarrel]

You can use FlexRAID, @looney is practically a representative for them.

[Miklos]

You can use FlexRAID, @looney is practically a representative for them.

True, but it's not really free and has some serious limitations: http://www.flexraid.com/2013/10/04/table-comparison-of-transparent-raid-vs-raid-over-file-system/

 

I like the idea of what FlexRAID is trying to do, but right now for a home NAS, FreeNAS is far superior.

[NinjaJc01]

True, but it's not really free and has some serious limitations: http://www.flexraid.com/2013/10/04/table-comparison-of-transparent-raid-vs-raid-over-file-system/

 

I like the idea of what FlexRAID is trying to do, but right now for a home NAS, FreeNAS is far superior.

FreeNAS won't run on older/lower end hardware,. where as you can just use Ubuntu and Samba to share the drives with high performance even on low end hardware.

[Miklos]

FreeNAS won't run on older/lower end hardware,. where as you can just use Ubuntu and Samba to share the drives with high performance even on low end hardware.

Not true, FreeNAS runs fine on old hardware, and has samba also.

Another misconception about FreeNAS.

[NinjaJc01]

Not true, FreeNAS runs fine on old hardware, and has samba also.

Another misconception about FreeNAS.

Have you seen the recommended specs? I was checking them out yesterday.

[leadeater]

2) No hardware RAID cards

 

Well I would still trust my LSI 9361-8i or IBM M5110 RAID card with 1GB flash cache upgrade over any cheaply built FreeNAS system. Traditional RAID setups are not dead and all the issues that are thrown around on this forum are over exaggerated hype, based on a real technical problem but seriously misunderstood.

 

Also ZFS is not the be all and end all, extremely good but doesn't suite every situation.

 

While I'm not saying anyone should go out and by as expensive RAID cards as the ones I have, especially with the feature upgrades, but blanket ruling out every storage system and technology in existence for ZFS is rather extreme statement to make. 

 

But yes my default response to anyone wanting to build a NAS would be to use FreeNAS, after I ask what they currently have and want to do as this may make me say something else.

[unijab]

Well I would still trust my LSI 9361-8i or IBM M5110 RAID card with 1GB flash cache upgrade over any cheaply built FreeNAS system. Traditional RAID setups are not dead and all the issues that are thrown around on this forum are over exaggerated hype, based on a real technical problem but seriously misunderstood.

 

Also ZFS is not the be all and end all, extremely good but doesn't suite every situation.

 

While I'm not saying anyone should go out and by as expensive RAID cards as the ones I have, especially with the feature upgrades, but blanket ruling out every storage system and technology in existence for ZFS is rather extreme statement to make. 

 

But yes my default response to anyone wanting to build a NAS would be to use FreeNAS, after I ask what they currently have and want to do as this may make me say something else.

 

If you tried to use a raid controller with ZFS you'd be in for some trouble after a while.

ZFS is a very good filesystem.

You can still build a relatively cheap machine to run FreeNAS. New i3's have ECC support for memory and older server hardware is becoming quite cheap. YOu dont need 6 cores or more unless you are planning on 10Gb or faster speeds.. You can easily build a machine that will saturate GbE speeds with a dual core if you have a few drives in your array.

 

 

I'd recommend watching this 3 part video series....

Here is the first

[Miklos]

Have you seen the recommended specs? I was checking them out yesterday.

http://www.freenas.org/hardware-requirements/

Latest FreeNAS has dropped 32bit CPU support as 4GB is max addressable RAM, but you shouldn't run a NAS on 10+ year old hardware anyway.

[Miklos]

Well I would still trust my LSI 9361-8i or IBM M5110 RAID card with 1GB flash cache upgrade over any cheaply built FreeNAS system. Traditional RAID setups are not dead and all the issues that are thrown around on this forum are over exaggerated hype, based on a real technical problem but seriously misunderstood.

Also ZFS is not the be all and end all, extremely good but doesn't suite every situation.

While I'm not saying anyone should go out and by as expensive RAID cards as the ones I have, especially with the feature upgrades, but blanket ruling out every storage system and technology in existence for ZFS is rather extreme statement to make.

But yes my default response to anyone wanting to build a NAS would be to use FreeNAS, after I ask what they currently have and want to do as this may make me say something else.

You assume that people only use FreeNAS on cheaply built hardware, which is far from the case. Regarding trust, since I started using exclusively ZFS and non hardware RAID controllers in 2008 I've had zero data loss over both private NAS setups and many PB of enterprise NAS', but I'm not saying hardware RAID doesn't have it's uses they are just few and far between, especially when it comes to NAS.

[Miklos]

If you tried to use a raid controller with ZFS you'd be in for some trouble after a while.

ZFS is a very good filesystem.

You can still build a relatively cheap machine to run FreeNAS. New i3's have ECC support for memory and older server hardware is becoming quite cheap. YOu dont need 6 cores or more unless you are planning on 10Gb or faster speeds.. You can easily build a machine that will saturate GbE speeds with a dual core if you have a few drives in your array.

I'd recommend watching this 3 part video series....

Here is the first

https://www.youtube.com/watch?v=NRoUC9P1PmA

Exactly, my home NAS easily maxes it's 2Gbit link with a cheap AMD 4core CPU and 32GB RAM.

I just built a cheap ZFS solution for a company using refurbished SuperMicro hardware and new drives. It has 8 3GHz Xeon cores and 64GB ECC Registered RAM and saturates its two 10GbE links with ease, granted it has SSD L2ARC and ZIL but that is cheap nowadays.

[leadeater]

If you tried to use a raid controller with ZFS you'd be in for some trouble after a while.

 

Well yes of course, I would use hardware RAID only in that case. Every software storage solution that offers redundancy etc requires true access to the disks, this is well known and is not exclusive to FreeNAS. 

[leadeater]

You assume that people only use FreeNAS on cheaply built hardware, which is far from the case. Regarding trust, since I started using exclusively ZFS and non hardware RAID controllers in 2008 I've had zero data loss over both private NAS setups and many PB of enterprise NAS', but I'm not saying hardware RAID doesn't have it's uses they are just few and far between, especially when it comes to NAS.

 

Well no I don't assume people only use cheap hardware to build FreeNAS computers but that wasn't really my point. A good RAID card is extremely reliable and easily replaced if it does fail, in my opinion slightly easier than software solutions, but it is more about fit for purpose. If it comes down to either using a consumer motherboard and FreeNAS versus a decent RAID card I know which is my pick.

 

There are plenty of cases where some who has existing hardware could more cheaply and perfectly safely go down the route of hardware RAID than FreeNAS, due to hardware recommendations etc.

[leadeater]

-snip-

 

Actually I do come from quiet a strong enterprise storage background and have used a wide variety of storage technologies since the 90's. One of the big reasons as to why I said ZFS is not the be all and end all is I work largely in highly virtualized (VMware) Windows focused networks and in a lot of cases FreeNAS/ZFS just won't meet the requirements. Yes I've had FreeNAS servers running and integrating nicely but only really for network scratch space storage and SQL backup dumps for application support department to use for dev/test.

 

Over time I've come to expect certain levels of support and response time for storage systems that I deploy and interact with. This comes from using products from the likes of Netapp FAS, EMC, HP EVA, IBM V7000 and Dell Compellent who all offer better support than I can possibly get from a FreeNAS build using HP or Supermicro hardware. Also the data can be extremely important such as Medical information, University research data, Financial to just general data which you may not know how important it actually is.

 

One of the other important factors for me also is integration with backup software like CommVault or Veeam, with NetApp being the absolute best in that regard.

 

I totally respect @Miklos knowledge in storage, clearly knows his stuff. I just suspect our requirements are quite often different and lead us in different directions which leads to the difference in opinions about when to call it quits on RAID. Also it's not uncommon for me to be presented with an HP server with 40TB-80TB of DAS and asked to get it setup for use for a department in the University who want to store files on it but also want to run some applications and a MS SQL database on it, there is little choice but to run hardware RAID.

[scottyseng]

-snip-

 

Kind of curious, what do you recommend if you need to run Windows on the NAS though? My NAS is half NAS / half rendering machine (It has a far more powerful CPU than my main PC...14 core, 28 thread Xeon), so I run Windows Server 2012 R2 on it because quite of the few of the applications I use are Windows only. My NAS runs hardware RAID, and I've never had a single issue with it over the past 3 years. You would say build a separate FreeNAS machine in addition to the Windows Server above?

 

Also, curious, I don't know much about FreeNAS at all, but what happens if the NAS gets turned off suddenly? Is there any data loss? I've been saved countless times by my RAID card's on card battery backup (Though I'm buying a UPS very soon) when my computer PSU went nuts / sudden blackout.

[leadeater]

Kind of curious, what do you recommend if you need to run Windows on the NAS though? My NAS is half NAS / half rendering machine (It has a far more powerful CPU than my main PC...14 core, 28 thread Xeon), so I run Windows Server 2012 R2 on it because quite of the few of the applications I use are Windows only. My NAS runs hardware RAID, and I've never had a single issue with it over the past 3 years. You would say build a separate FreeNAS machine in addition to the Windows Server above?

 

If you want to stick with the single box you could use Storage Spaces. Since you are using Windows server you can also use auto tiering and SSD cache the disk pool, this is what I do with one of my servers at home with 6 Samsung 512GB Pros and a bunch on NAS HDDs. The nice thing about Storage Spaces and auto tiering is the SSD disks add to the total usable space on the disk pool, not just cache. Does a much better job than LSI cachecade too.

 

Storage Spaces isn't as mature of a product as FreeNAS and only offers raw disk pooling abilities, doesn't really have to do more since it runs on Windows. But the Microsoft ZFS alternative is pretty damn rubbish in comparison, ReFS.

 

As for a FreeNAS box losing power, the only issue I've had is the Active Directory integration breaking. @Miklos can talk more about this than I could since I'm really not a big FreeNAS user.

[scottyseng]

-snip-

 

Well, I have a hardware RAID card in the box already, a LSI MegaRAID 9260-8i CV, but interesting bit about storage spaces. I did not know you could use it as a SSD cache. I was planning on buying CacheCade for my LSI card, but you bring up a interesting idea to try. At the moment though, the server is more of a NAS (I haven't had time to give it anything to render yet...it just laughs and idles at 1% CPU usage most of its life at the moment).

 

Yeah, when I went to start collecting parts for my NAS, I honestly never heard of FreeNAS or ZFS until I joined the Linus Tech Tips forum last year. I only knew that if I wanted to do RAID, I needed to get a proper hardware card with a on card battery backup (I wasn't aware third party software RAID options like FlexRAID existed either). So I'm kind going to keep using the RAID card for now (It's worked fine for the past 3 years) since I spent so much money on it.

[leadeater]

 

 

Yea there is way too many things out there to know them all let alone have actually been able to use them and become proficient in it. I've only just started looking at storage spaces recently and I'm very impressed.

 

At work we are about to do a proof of concept for the new features in Server 2016, which includes looking at storage spaces, as part of the Microsoft TAP and RDP program. I think we were also the first production install of Skype for Business too but don't quote me on that, we had to get special permission to do it. If you are interested in what TAP/RDP is this guy explains it rather well, plus I have no idea what is and is not ok to talk about so saying nothing is best

 

http://www.jamesserra.com/archive/2011/10/what-is-microsoft-tap-and-rdp/

[Miklos]

If you want to stick with the single box you could use Storage Spaces. Since you are using Windows server you can also use auto tiering and SSD cache the disk pool, this is what I do with one of my servers at home with 6 Samsung 512GB Pros and a bunch on NAS HDDs. The nice thing about Storage Spaces and auto tiering is the SSD disks add to the total usable space on the disk pool, not just cache. Does a much better job than LSI cachecade too.

Storage Spaces isn't as mature of a product as FreeNAS and only offers raw disk pooling abilities, doesn't really have to do more since it runs on Windows. But the Microsoft ZFS alternative is pretty damn rubbish in comparison, ReFS.

As for a FreeNAS box losing power, the only issue I've had is the Active Directory integration breaking. @Miklos can talk more about this than I could since I'm really not a big FreeNAS user.

I'm glad you mentioned NetApp and EMC as their solutions are built on top of FreeBSD and ZFS as with FreeNAS and TrueNAS.

I do not run Windows server on anything as it has nothing to offer my solutions I can't do in FreeBSD so I can't really offer any help in that Regards other than you can run Windows in a VM on FreeBSD/FreeNAS and use a ZFS volume as your drive.

In regards to loss of data, a UPS will provide same and more insurance towards data loss and ZFS is copy on write so no loss of data once it's written, Ofc the ZIL is in memory but a UPS takes care of that easily.

But saying hardware raid is easier to replace than a software layer is nonsense, also ZFS is resistant to even HDD controller malfunctions, solar flares and what not, something that you don't get with regular hardware RAID.

As I mentioned, hardware RAID has its places, just not in most NAS setups. When we talk DAS it's an entirely different conversation :-)

[leadeater]

I'm glad you mentioned NetApp and EMC as their solutions are built on top of FreeBSD and ZFS as with FreeNAS and TrueNAS.

I do not run Windows server on anything as it has nothing to offer my solutions I can't do in FreeBSD so I can't really offer any help in that Regards other than you can run Windows in a VM on FreeBSD/FreeNAS and use a ZFS volume as your drive.

In regards to loss of data, a UPS will provide same and more insurance towards data loss and ZFS is copy on write so no loss of data once it's written, Ofc the ZIL is in memory but a UPS takes care of that easily.

But saying hardware raid is easier to replace than a software layer is nonsense, also ZFS is resistant to even HDD controller malfunctions, solar flares and what not, something that you don't get with regular hardware RAID.

As I mentioned, hardware RAID has its places, just not in most NAS setups. When we talk DAS it's an entirely different conversation :-)

 

Netapp actually have their own file system, WAFL, which yea is very much like ZFS and is also why there has been some legal battles over it. Also I really did mean only slightly easier, ZFS you just replace the HBA or if total system failure re-import the pools etc. Both are easy, just hardware RAID is less tied to the OS and the type of OS. ZFS offers much more as you know but 2 maybe 3 features really only get used in home setups: integrity, cache and dedup. 

 

Anyway I am in agreement that RAID is not best suited to NAS setup compared to other options available, which I also said in my first post. I just disagree that when building a NAS it must always be ZFS at the exclusion of anything else.

 

That example of DAS btw was being used as a NAS and application server and database server. Horrible I know and happens more often than it should, but when the parts have been purchased without any IT involvement your already shit out of luck

[Miklos]

Netapp actually have their own file system, WAFL, which yea is very much like ZFS and is also why there has been some legal battles over it. Also I really did mean only slightly easier, ZFS you just replace the HBA or if total system failure re-import the pools etc. Both are easy, just hardware RAID is less tied to the OS and the type of OS. ZFS offers much more as you know but 2 maybe 3 features really only get used in home setups: integrity, cache and dedup.

Anyway I am in agreement that RAID is not best suited to NAS setup compared to other options available, which I also said in my first post. I just disagree that when building a NAS it must always be ZFS at the exclusion of anything else.

That example of DAS btw was being used as a NAS and application server and database server. Horrible I know and happens more often than it should, but when the parts have been purchased without any IT involvement your already shit out of luck [emoji14]

Haha I know that situation all to well, work with what you got sometimes :-)

Yes ZFS is not one size fits all, but it covers pretty broadly I think we can agree on ;-)

Also most home users who doesn't know or want to learn and just need something to work out of the box should probably go with Synology or the likes anyway. I just get the sense that most here likes to tinker and learn, and they should definitely try ZFS.

[kermur]

On 1/1/2016 at 7:21 AM, Miklos said:

 

 

1) Filesystem has to be ZFS

...

3) You can never have too much RAM (aim for 1GB RAM per TB of usable storage)

4) FILESYSTEM HAS TO BE ZFS

 

This is a very old post, but I want to post to set the record straight. For advanced features such as deduplication etc, there is no doubt that zfs is superior, but your statements are myopic and don't take into consideration the different use-cases that people may have for building a NAS.

 

If I am creating a pool for heavy database usage, I use zfs/FreeNas every time. For other use-cases such as media storage, zfs is total overkill. A RAID5 array on Ubuntu or OpenMediaVault is perfect for "write-once"/ read-only purposes and has extremely low RAM requirements.

 

Why would I want a z-pool to store TV shows? I can build a system with a crappy Celeron CPU and 2GB of ram and no ecc memory- how would zfs benefit me in this case? It wouldn't. It would cost me more $ and offer no tangible benefit.

 

I encourage people to use what makes sense for their needs. Yes, Un-Raid is not free (for most setups) and that may be a deal breaker. But, if you are running a plex server or something similar, un-raid, with its out-of-the-box docker integration may make alot more sense than z-pools. If you don't want to pay for UnRaid, vanilla Ubuntu (or any other linux distro) or a storage specifric distro such as OpenMediaVault will work just fine.

 

zfs is an amazing filesystem and freenas is an amazing product. That doesn't mean that it is the tool for every situation. On the contrary, linux based systems with more traditional raid configurations can save you money on hardware and are easier to manage for for people who don't know BSD/Unix.

 

Just my $0.02.

 

 

kermur