High end Home NAS

[Crek]

Hi guys

 

I have a project for a high end home NAS, making my research and trying to see how I'll proceed

 

my usage scenario is one where on a daily basis considerable amounts of data (about 1.5 - 3TB of large files total) will be going in and out of this NAS and updated with no internet, no  VMs or PLEX or anything running on it, it'll be serving 3 workstations total, however should one day's work overlap another, I'll be in BIG trouble, speed is of the essence, reliability under load as well, and ease of maintenance (regardless of possible RAID rebuild times) however I will have a standalone computer with 1 mission: back up for this NAS's RAW data, should I lose the raid, I will have all the data at hand, pretty fast too even if not updated to the very last minute of use.

note that this NAS will be in an all 10Gbe network

 

my main concern here is built VS bought, and no rack mounting for both

 

As far as bought is concerned, I've been considering Qnap TVS 1282T which comes with a built in 540T2 dual 10Gbe NIC installed and Thunderbolt 2 for volume expansion, say I get this, fill it with RAM, Noctua the exhaust fans and CPU fan and change the PSU to something more ear friendly, begin with 6 HDDs, either 8 or 10TB drives in RAID 1+0 ...

after some research I found that this NAS uses what seems to be an IBM Serveraid M1015 for connecting all the drives.

Now Qnap's QTS IS Linux based but I'm still trying to see how reliable this setup would be

 

I did look at Synology NASs but it would appear like Qnap are generally higher quality builds, fixable noisy cases and more stable ... ?

 

on the other hand there's me building a NAS, and regardless of case, mobo, ram, and 10Gbe NICS, I've been looking at raid cards, preferably non ROC ones or simply just dual SAS cards with direct access to HDDs, I guess a 4-6 core Xeon would be better than ROC ... ? but also looking at OS options, what worries me are a few things, primarily that I don't even know a single Linux thing , never used the OS, but also that some of the most common OSs used, like FreeNas seems to have trouble playing nice with this or that controller, may it be on board or PCI-E, I'm considering using Windows Server to try and not have those kinds of issues with good drivers but also since I have a license, note that I do have some time to build this NAS but once up and running I can't have it fail on me too regularly, I am a tinkerer, and prefer a built NAS because it's a lot easier to maintain but I really want to have a reliable NAS, the same HDDs are gonna go in this NAS as the bought one if ever, up to a total of 12 or 16 HDDs however.

 

looking at some core options I see that ZFS is a Wholly Grail to a sizable chunk of home NAS builders however there's no proof of the actual exclusivity of it's features and reliability, then again I can't figure out how will an eventual Raid Z2 or 3 be performing with 6-12 or 18 drives, on another note I seem to understand that 128-256 GB ram is barely adequate for it should I not have any cache SSD ?, also an NVME SSD or 2 is very probable for me however not necessary, I mean where am I gonna cache 850GB if that happens to be happening in a single copy ... I've seen some NVME issues mentioned with Qnap's QTS, and seems to be a Linux thing, not just QTS, any thoughts on this ?

 

considering caching, please note that "hot" data being cahced in an SSD for frequent use is a terrible idea in my case especially that the amount of data that may be labeled equally hot can be enormous and all of it change drastically in a very short time frame, what's way more useful for me is a network cache, copying the actual data being written to SSD first , then to the actual physical share and that's it, but, again, not necessary if troublesome

 

I know I have't really ventured that much into NAS before but in this case 28+ TB usable space is a must, any ppl out there with Windows Server experience with this ? because I think this thread is gonna be swarming with Linux OS based defenders, and for good reasons too I guess ...

 

Internet Could storage is ABSOLUTELY out of the question so please don't even think about it

 

 

thanks for u'r time and attention

[samiscool51]

ummm... 

custom may be out of the question as it might be too expensive....

maybe a OEM server might be usable but i'm not sure with your requirements

also thats a lot of text....

[leadeater]

Firstly I'm glad to see that you are already aware that read caching is not effective for the data change rate you are expecting, write caching can still help though but needs to be both big enough and actually faster than the capacity pool. If your reading/writing large streams of sequential data 10+ HDD will actually be faster than 2 SSD in a mirror set to cache.

 

Also I would advise against RAID 10 as it is not any safer or any faster for sequential reading or writing, in fact slower. RAID 10 is for high queue depth random write I/O so if that isn't what you are doing then don't use it. There is a caveat to this recommendation but I'll mention it when I get to it. 

 

On to FreeNAS/FreeBSD. Not having ever used a Linux/BSD OS before isn't going to be too much of an issue, it is all driven by the web interface for normal operation. If something does go horribly wrong then Linux/BSD knowledge would help but you can just reinstall FreeNAS and re-import the ZFS pool rather than fix it. Shouldn't ever be an issue anyway.

 

The 128GB-256GB ram you mention really is not required. Those large ram requirements per TB is for when you are using the deduplication feature of ZFS which you won't be using for performance reasons. Start with 64GB and increase only if performance is an issue and can be conclusively proved that it is due to insufficient ram.

 

Should you actually use FreeNAS versus say a QNAP or Windows Server well that depends on a few things like what client OS you are using. If the clients accessing the NAS are Windows then I would be leaning more towards Windows for better SMB3 compatibility since performance is a focus of yours. Remember that both FreeNAS and QNAP use SAMBA for creating Windows shares so suffer the same fate in that respect, not that it's slow or anything but not as fast as a Windows computer with comparable hardware.

 

Now there are two options you could go for hardware wise for Windows Server

• Hardware RAID (Perfectly fine don't listen to the internet doom train esp since you are going to back the system up)
• Windows Server 2012 R2/2016 Storage Spaces

Hardware RAID:

Using a good hardware RAID controller such as an LSI 9361-8i with 1G/2G flash cache will give extremely good RAID 6 performance in both read and write, easily expandable (much more so than ZFS), and OS agnostic.

 

Windows Storage Spaces:

The first thing is to decide on the type of configuration you want to use as this will determine if Windows Server 2016 is required. Another thing to keep in mind which I eluded to at the start is with Storage Space if high performance is required then mirror configurations on all storage tiers is required, meaning like RAID 10.

 

You could do a pure HDD Windows Storage Spaces configuration which means you could use either Server 2012 R2 or Server 2016. If you want to do a tier configuration then Server 2016 is a must.

 

The reason for why Server 2016 is a must for tier configurations is there are important improvements that are only in this OS. The primary one is that ReFS has been made much smarter and will always write to the fastest tier and disks, even for modifications. So if you have 4-6 SSDs and 10 HDDs in the pool writes will always go to the SSDs ensuring fast performance, especially if the HDD tier is configured in dual parity which you won't be doing.

https://blogs.technet.microsoft.com/clausjor/2015/11/19/storage-spaces-direct-in-technical-preview-4/

 

As a bit of performance data for you I use Windows Server 2016 Storage Spaces with 6 Samsung Pro SSDs with larger HDDs. I have my SSD tier configured in two-way mirror and HDD tier in dual parity and I get 2.5GB/s read and 1.5GB/s write. It is also important to know that in Storage Spaces the SSD tier is usable capacity, it is NOT just a cache.

 

 

2 hours ago, Crek said:

I did look at Synology NASs but it would appear like Qnap are generally higher quality builds, fixable noisy cases and more stable ... ?

Yes I also prefer QNAP over Synology but for this a prebuilt NAS would be able to do the job but at a much higher cost.

[samiscool51]

@leadeater windows server 2016 is not stable at this current point (i tried it it's not that great at this point in time, needs more work, that's why it's a preview)

i would advise against it if you need a stable server

[leadeater]

Just now, samiscool51 said:

@leadeater windows server 2016 is not stable at this current point (i tried it it's not that great at this point in time, needs more work, that's why it's a preview)

i would advise against it if you need a stable server

Windows Server 2016 is stable and is as of now GA.

[samiscool51]

1 minute ago, leadeater said:

Windows Server 2016 is stable and is as of now GA.

like right as you posted? wow word goes round quick...

hmm...

[leadeater]

16 minutes ago, samiscool51 said:

like right as you posted? wow word goes round quick...

hmm...

2 Days ago for general public, for VLSC customers a little earlier (2 weeks from memory) and people part of vNEXT/TAP/RDP that info is NDA.

[samiscool51]

36 minutes ago, leadeater said:

2 Days ago for general public, for VLSC customers a little earlier (2 weeks from memory) and people part of vNEXT/TAP/RDP that info is NDA.

ah technicalitys... what i bitch you are..... (i can say that because i'm a girl, so that makes it OK!(i think))

[Crek]

Thanks @leadeater for this more than valuable and welcome info, indeed I wasn't stuck on raid 10 but somehow feel like eventually fixing it is more foreseeable , given that parity is 1 on 1, it may be silly actually.

all 3 machines it will be serving as well as the backup computer are windows 10 indeed so SMB 3, I was considering windows server 2016 and shared samiscool51 on the idea that the OS has just been released and prone to the usual bugs but now that u say it's been doing good with u, that's a good sign, and after all it can only get better

 

I'm now reading about Queue depth and what this dictates in raid scenarios

 

I'll be checking out the raid card u generously suggested, although I do already have a windows server 2012 license for the moment, I'm also looking into tiers,what they are and how they operate, so far I may not go into them because as far as I'm reading, I may not need to, but that may change, one thing I'm sure of however, is that I'll be building me a NAS after all and use Windows Server for OS

 

 

Things on the internet go in all directions, without necessarily a good enough explanation for either

thanks again for all the spot on info, and the great pointers, I certainly did appreciate u sharing a stats of a system setup u've used

now at least I have more focus and can move forwards in research, some things are totally new to me, and for those also thanks for the right track

[scottyseng]

10 minutes ago, Crek said:

-snip-

Ah, just to add a bit more, make sure you get a good UPS battery back up unit for whatever NAS you plan to get for safety reasons. You shouldn't risk a power outage. Also, the RAID card mentioned before, make sure you get the battery for the RAID card as well (It runs slower without it...I can confirm when my RAID card battery died vs after I replaced it).

 

I personally run Windows Server 2012 R2 in my home NAS (waiting for Dreamspark to get Windows Server 2016 so I can get it for free) with hardware RAID, no issues, though I have the older RAID card model, the LSI 9260-8i with two RAID6 arrays.

 

A lot of what RAID level would be better for you matters on what kind of data you need to move around, like for me, RAID6 has better sequential speeds (large files) vs RAID10, but RAID10 as mentioned above, is good for lot of random IO or smaller files.

 

Oh, and if you're swapping fans, be careful, if you pack a NAS full of drives, the silent fans might not move enough air to keep the drives cool. If you're using WD Reds or cooler running drives, you should be fine, but with hot enterprise 7200 RPM drives, server with high RPM fans have them for a reason. I had issues with overheating when I changed out the fans (I used a stack of BeQuiet fans) so I had to move my drives around to space them apart.

[Crek]

thanks @scottyseng , a 5KVA UPS is already installed, giving me up to an hour of extra time with the load it has during brownouts, kinda common here, so awareness is pretty much a must, and seems like I'll be building my own NAS , that means total freedom in cooling solutions and yes it opens a lot of doors for me as opposed to modifying a pre-conceived NAS, I have a now storage pc running 5 7200 and 3 10.000 HDDs, so yup, I'm aware of the heat, and I may actually start with seagate 10 Gb drives and now re-assessing raid level that I may use, as for case since I have some space I have a LianLi PC-P80 laying around, 3 140mm fans at the front, surplus Rosewill HDD cases at my friend's also have 120mm fans of their own at their back for every 4 HDDs, that's push-pull for HDDs CPU will be jealous, those will not hold any SSDs anyway so Sata 2 is more than ample speed for each HDD.

 

still have a lot to read but I think I'm good

[leadeater]

@Crek You should be able to do a bit of validation and performance confirmation as most of the core parts to the system are common across multiple different configurations: FreeNAS, Server 2012 or Server 2016.

 

You could order the motherboard, cpu, ram, 4-6 HDDs, 1 SSD/HDD (or use current spare) for OS and maybe the 10Gb NIC. Then you can install an OS and do benchmarking, HDD failure and recovery, expansion etc. How does say 4 HDDs on Server 2012 R2 Storage Spaces perform, now expand to 5-6 and test again. Using that data how many do you need to get the throughput you want?

 

Now try FreeNAS and do the same tests over again, same for Server 2016.

[Crek]

exactly , I've ordered a few components already, those should reach me between 2 to 4 weeks, generally I order via proxy from the States, then ship here, u can imagine RMA then , will have to wait for them to be able to do even tests, perhaps FreeNAS in a VM to see how I get along with it

[leadeater]

25 minutes ago, Crek said:

exactly , I've ordered a few components already, those should reach me between 2 to 4 weeks, generally I order via proxy from the States, then ship here, u can imagine RMA then , will have to wait for them to be able to do even tests, perhaps FreeNAS in a VM to see how I get along with it

Will be a lot easier to install FreeNAS on to a usb drive and use that, if you go with the VM then you need an HBA to pass through to the VM with the HDD attached to it.

[TechXero]

@leadeater

 

Hey man, I love your reply to my friend @Crek it's full of very useful information. And like my friend I am looking to expand my storage server, but not as high end as he mentioned, I like to keep it simple and just add an HBA non SAS card to add more drives to my current Storage PC. I just don't know which brand is best, with best transfer speeds and stability. 

 

config below :

 

- Core i7 3770k CPU

- 32GB HyperX Beast DDR3 RAM (I know it's overkill but nowhere else to use them LoL)

- GTX 650 Ti Boost OC GPU (Maybe Overkill but meh)

- 8 HDDs Saturating all 8 SATA Ports...

 

I created a thread asking for help here 

Would appreciate your input.

 

Thanks