Home File Server Advice

[aubryscully]

Hi, so I currently have an old HP Pavilion tower with an i5-2300 running Server 2016 acting as a file server/playground for me to learn more about the OS. Due to Dell's pricing error on the 250GB WD Blue SSDs, I now own 6 of them. So all in all, my storage setup consists of:

1x old Intel 160GB SSD acting as OS drive

2x WD Red 3TB in motherboard RAID-1 for data storage

6x 250GB WD Blue SSD

 

I plan on using these SSDs for video work in Premiere and After Effects. In order to access files quickly, I will soon be buying 2x 20GB NICs and an SFP+ cable to connect my workstation and the server.

 

What is the recommended setup for these SSDs to get good performance, but preferably also a parity? I don't want to buy a hardware RAID card, so is a RAID-Z array a good idea? Would I be able to run some linux distro as a VM and patch the drives into it for the array? While I am alright in navigating around Server 2016, I am a total noob at Linux. However, I am willing to learn in order to get it all setup and working.

 

Thanks for the help.

[Mikensan]

SFP+ is 10gb I believe, and I'm not too familiar with 20gb NICs so just be sure of compatibility. 

 

With pure speed in mind and little in redundancy, Raid 10 is the way to go. You can certainly do a triple mirror stripe to give you the warm and fuzzies of a 2 disk failure tolerance.

 

Your biggest challenge will be whatever you chose to handle your raid should be capable of taking care of SSDs as I highly suspect those SSDs do not self-trim. Older raid cards generally have no tech to take care of SSDs.

 

@leadeater might have some input on using Storage Spaces as opposed to pure Raid for your SSDs.

[leadeater]

I would stick with Windows Server 2016 and Storage Spaces.

 

Get another 3TB WD Red then add the 6 SSDs and 3 HDD to the same storage pool then create a tiered virtual disk with the SSDs using Mirror configuration and the HDDs using Parity configuration, this will give you a virtual disk with the performance of 6 SSDs but with the capacity of HDDs. This will mean all writes will go to SSD and frequently accessed data will also get moved to the SSD tier for fast reads.

 

The SFP+ NICs you plan to buy will ensure you get fast access to that storage from your workstation.

[aubryscully]

@Mikensan Sorry, I meant 10GB NICs. Thanks for the input! Would Storage Spaces handle TRIM well?

 

@leadeaterThat seems like a really neat option! How much further down could Storage Spaces scale if I were to get rid of a few of the SSDs? Also, would it be fine for me to only link my server to my PC, then bridge the 10gb connection and the gigabit lan on my PC to give the server access to the rest of the lan as well as the internet? My workstation is the only machine that needs fast access, and the server doesn't do much online at all. While I know this configuration isn't ideal, due to the sucky networking layout of my house, it wouldn't be feasible to get a cat5e wired to the server. While I know I could just use a small gigabit switch to wire my main machine and server to the rest of the network, I wouldn't like to as it would add extra cables and messiness to my desk setup.

 

Thanks for all of the help

[leadeater]

Just now, aubryscully said:

@Mikensan Sorry, I meant 10GB NICs. Thanks for the input! Would Storage Spaces handle TRIM well?

 

@leadeaterThat seems like a really neat option! How much further down could Storage Spaces scale if I were to get rid of a few of the SSDs? Also, would it be fine for me to only link my server to my PC, then bridge the 10gb connection and the gigabit lan on my PC to give the server access to the rest of the lan as well as the internet? My workstation is the only machine that needs fast access, and the server doesn't do much online at all. While I know this configuration isn't ideal, due to the sucky networking layout of my house, it wouldn't be feasible to get a cat5e wired to the server. While I know I could just use a small gigabit switch to wire my main machine and server to the rest of the network, I wouldn't like to as it would add extra cables and messiness to my desk setup.

 

Thanks for all of the help

2 SSDs (mirror) min since you can't use the SSDs to cache a protected HDD tier with an unprotected SSD tier (simple). Storage Spaces is actually extremely good at data management, 250GB will be likely more than double what you need.

 

Currently I use 6 512GB Samsung Pros and 3 3TB WD Reds in the same configuration I just mentioned, but since you can create multiple virtual disks from a pool I have an SSD only simple VD for my steam library which is essentially 6 SSD RAID 0 and another VD of that above configuration.

 

What I do is I have a 1Gbps connection to my main network and a separate 10Gb NIC using different IP subnet directly connected to my server for fast file server access. I'm not sure how well bridging will work, try it and see. A backup option could be a USB wireless NIC for the server.

[aubryscully]

5 minutes ago, leadeater said:

2 SSDs (mirror) min since you can't use the SSDs to cache a protected HDD tier with an unprotected SSD tier (simple). Storage Spaces is actually extremely good at data management, 250GB will be likely more than double what you need.

 

Currently I use 6 512GB Samsung Pros and 3 3TB WD Reds in the same configuration I just mentioned, but since you can create multiple virtual disks from a pool I have an SSD only simple VD for my steam library which is essentially 6 SSD RAID 0 and another VD of that above configuration.

 

What I do is I have a 1Gbps connection to my main network and a separate 10Gb NIC using different IP subnet directly connected to my server for fast file server access. I'm not sure how well bridging will work, try it and see. A backup option could be a USB wireless NIC for the server.

Wow, that's a really nice config that you have!

 

So am I correct in assuming that the 3x HDDs would (kind of) be in a RAID 5, but through Windows? If one were to die, could you add another and let it rebuild itself like in RAID?

[leadeater]

12 minutes ago, aubryscully said:

Wow, that's a really nice config that you have!

 

So am I correct in assuming that the 3x HDDs would (kind of) be in a RAID 5, but through Windows? If one were to die, could you add another and let it rebuild itself like in RAID?

Yes that is correct, you can do single parity (RAID 5) and dual parity (RAID 6). Dual parity requires 4 disks minimum.

 

Parity performance itself is rather terrible on Storage Spaces so you can't really use it without SSDs, not if you want more than 100MB/s speed. You can also do both tiers as mirror configurations, it really just comes down to how much capacity you need and how much extra your prepared to pay for speed vs usable capacity.

 

So as far as maturity goes ZFS is better but in certain ways has stricter hardware compatibility and requirements, that is if you want good performance. ZFS is also BSD and Linux so if your not comfortable with those platforms usability is worse, this is user dependent. Windows is also far superior at serving up SMB/Windows network shares.

 

FreeNAS solves the usability issue quite nicely since it's all web interface driven, you can drop to CLI if need.

 

Pretty much there is no perfect answer just one that best fits your needs/criteria.

 

Edit:

Wow fixed tons of mistakes in that, must be tired.

[aubryscully]

@leadeater I do like the idea of keeping all within the Server 2016 environment. With the SSDs as a cache, would I be seeing full SSD performance of 500+ seq read/writes given that the SSDs can handle it and it all gets configured right?

 

Also, out of curiosity, if you were to have 2-3 server nodes with all the same drives on each, could you have a VM running storage spaces and have failover clustering enabled to keep all files accessible even if one node goes down?

[leadeater]

22 minutes ago, aubryscully said:

@leadeater I do like the idea of keeping all within the Server 2016 environment. With the SSDs as a cache, would I be seeing full SSD performance of 500+ seq read/writes given that the SSDs can handle it and it all gets configured right?

 

Also, out of curiosity, if you were to have 2-3 server nodes with all the same drives on each, could you have a VM running storage spaces and have failover clustering enabled to keep all files accessible even if one node goes down?

Most of the time yes you will get full SSD speed, benchmark when you have configured it to check performance.

 

Yes this is called Storage Spaces Direct in a hyper-converged configuration, where the server is both a storage node and also VM host.

https://technet.microsoft.com/en-us/windows-server-docs/storage/storage-spaces/storage-spaces-direct-overview

https://technet.microsoft.com/en-us/windows-server-docs/storage/storage-spaces/hyper-converged-solution-using-storage-spaces-direct

https://technet.microsoft.com/en-us/windows-server-docs/storage/storage-spaces/storage-spaces-direct-hardware-requirements

[leadeater]

Here's the kind of performance you can get with SATA SSD and HDD, storage spaces also supports NVMe.

 

I forget what the configurations were for these two since it was done some time ago.

 

 

Edit:

Think the first is a 6 SSD mirror and the second is a 4 SSD simple.

Edited January 6, 2017 by leadeater

[aubryscully]

@leadeater Wow those numbers look nice. Thanks again for all of the help! I will update once I have it all setup and working

[Mikensan]

@aubryscully since windows can see the disks directly, they will be taken care of by windows in regards to trim. Windows just needs to know it's working with SSDs and the rest is magic.

 

Thanks @leadeater, knew you were the guy to go to for storage spaces :-) Something I hope to play with one day soon.

[Dark]

On 1/5/2017 at 3:35 PM, Mikensan said:

SFP+ is 10gb I believe, and I'm not too familiar with 20gb NICs so just be sure of compatibility. 

 

With pure speed in mind and little in redundancy, Raid 10 is the way to go. You can certainly do a triple mirror stripe to give you the warm and fuzzies of a 2 disk failure tolerance.

 

Your biggest challenge will be whatever you chose to handle your raid should be capable of taking care of SSDs as I highly suspect those SSDs do not self-trim. Older raid cards generally have no tech to take care of SSDs.

 

@leadeater might have some input on using Storage Spaces as opposed to pure Raid for your SSDs.

TRIM is OS dependent.

 

Garbage collection is handled within the SSD itself (TRIM facilitates this process if present).  The majority of SSDs these days have a garbage collection that is sufficient enough to keep the drive operating properly without TRIM. 

[leadeater]

12 minutes ago, Dark said:

TRIM is OS dependent.

 

Garbage collection is handled within the SSD itself (TRIM facilitates this process if present).  The majority of SSDs these days have a garbage collection that is sufficient enough to keep the drive operating properly without TRIM. 

Larger SSDs handle GC quite well, smaller ones tend to drop in performance rather quickly without TRIM. My 512GB Samsung Pros barely survive properly under ESXi, the fix is easy just OP by about 60GB and you're all good.

 

Pretty much the only difference between a server SSD and a consumer SSD is internal OP, power loss protection and slightly better optimized GC. For example server SSD 480GB vs consumer SSD 512GB, both have the same amount of NAND.

 

Anyway nice bonus with Windows Storage Spaces is TRIM is supported and works .

[Dark]

9 hours ago, leadeater said:

Larger SSDs handle GC quite well, smaller ones tend to drop in performance rather quickly without TRIM. My 512GB Samsung Pros barely survive properly under ESXi, the fix is easy just OP by about 60GB and you're all good.

 

Pretty much the only difference between a server SSD and a consumer SSD is internal OP, power loss protection and slightly better optimized GC. For example server SSD 480GB vs consumer SSD 512GB, both have the same amount of NAND.

 

Anyway nice bonus with Windows Storage Spaces is TRIM is supported and works .

Makes perfect sense!

 

I finally started digging in to Storage Spaces information and I'm tempted to create an HA Storage Spaces 'san/nas' to serve iscsi and NFS to a vmware cluster.

 

Right now we have lots of Netapp storage but it consumes a very large amount of budget and we've since started deploying 'enterprise' QNAPs with only solid state storage to handle our dev environments, downside is they are single controller (their dual controller no longer supports SATA due to LSI dropping the interposer card).

 

So now I'm going to spec out two dell 1U boxes and disk shelf to test it out and see if it would support our use case.

[leadeater]

5 minutes ago, Dark said:

Makes perfect sense!

 

I finally started digging in to Storage Spaces information and I'm tempted to create an HA Storage Spaces 'san/nas' to serve iscsi and NFS to a vmware cluster.

 

Right now we have lots of Netapp storage but it consumes a very large amount of budget and we've since started deploying 'enterprise' QNAPs with only solid state storage to handle our dev environments, downside is they are single controller (their dual controller no longer supports SATA due to LSI dropping the interposer card).

 

So now I'm going to spec out two dell 1U boxes and disk shelf to test it out and see if it would support our use case.

I currently use Storage Spaces for my home ESXi env, tried both iSCSI and NFS which both work well.

 

We also use Netapp at work, 6 node 8060 per campus for production storage (AFF, FC+SAS, SAS, SATA aggregates) and 4 node 3220 for backup storage, which yea is expensive but performs extremely well plus has excellent support.

 

My advice would be to also look in to Storage Spaces Direct (S2D), either in a hyper-converged configuration (Host+Storage) or storage only or even a mixed of node types.

 

Other thing you should have a look at is Nutanix which we have already moved our development VM environment to, ESXi has the hypervisor. Our production VM environment is slowly moving to all Nutanix. However personally I am not totally sold on Nutanix, you have to be very careful in your storage configurations else you'll end up having massive amounts of unused storage like we do. I'm also not fully convinced that the local SSD caching is better than Netapp FCSAS or AFF would be.

 

If Nutanix I personally would only go with all flash configurations else you really aren't making a real architecture improvement.

 

I like the idea of Nutanix and it offers a lot more than just cross host pool storage and VM hosting aka hyper-converged (stupid buzz words), in that technical regard I my personal opinion Storage Spaces Direct does a better and higher performing job especially if you go with a NVMe + SSD or NVMe + SSD + SATA design.

[Lurick]

2 minutes ago, leadeater said:

I currently use Storage Spaces for my home ESXi env, tried both iSCSI and NFS which both work well.

 

We also use Netapp at work, 6 node 8060 per campus for production storage (AFF, FC+SAS, SAS, SATA aggregates) and 4 node 3220 for backup storage, which yea is expensive but performs extremely well plus has excellent support.

 

My advice would be to also look in to Storage Spaces Direct (S2D), either in a hyper-converged configuration (Host+Storage) or storage only or even a mixed of node types.

 

Other thing you should have a look at is Nutanix which we have already moved our development VM environment to, ESXi has the hypervisor. Our production VM environment is slowly moving to all Nutanix. However personally I am not totally sold on Nutanix, you have to be very careful in your storage configurations else you'll end up having massive amounts of unused storage like we do. I'm also not fully convinced that the local SSD caching is better than Netapp FCSAS or AFF would be.

 

If Nutanix I personally would only go with all flash configurations else you really aren't making a real architecture improvement.

 

I like the idea of Nutanix and it offers a lot more than just cross host pool storage and VM hosting aka hyper-converged (stupid buzz words), in that technical regard I my personal opinion Storage Spaces Direct does a better and higher performing job especially if you go with a NVMe + SSD or NVMe + SSD + SATA design.

 

If you want to go with all flash, have you looked at Nimble, not sure how their pricing is but I've been looking at them just out of curiosity lately.

[Dark]

@leadeater Thank you for the information and I'll check out Nutanix.

 

I've been reading up on S2D but we won't be doing anything with Hyper-V (we have a fairly large amount of ESXi hosts).

 

Have you implemented/observed deduplication in Storage Spaces?

 

We have a few Netapp clusters with our most recent being an 8080 all-flash.  We really like our netapp storage, just not the cost.

[leadeater]

Just now, Lurick said:

 

If you want to go with all flash, have you looked at Nimble, not sure how their pricing is but I've been looking at them just out of curiosity lately.

Yea we had a brief look at Nimble, only issue for us is it's block storage only and we use SMB and NFS directly off the Netapps for general network shares and for our Linux servers, not just for ESXi storage. Currently we would go with Netapp SolidFire over Nimble due to the same limitations and SolidFire has extremely awesome features, really top notch storage QoS.

[leadeater]

1 minute ago, Dark said:

@leadeater Thank you for the information and I'll check out Nutanix.

 

I've been reading up on S2D but we won't be doing anything with Hyper-V (we have a fairly large amount of ESXi hosts).

 

Have you implemented/observed deduplication in Storage Spaces?

 

We have a few Netapp clusters with our most recent being an 8080 all-flash.  We really like our netapp storage, just not the cost.

You can still do S2D in storage node only configuration so you have scale out storage and a more resilient design. Down side is S2D requires more servers than shared JBOD and really only starts to pull ahead in cost and performance in large enough configurations.

 

Currently in Server 2016 Storage Spaces + ReFS does not support deduplication, however recently I think they allowed you to create NTFS virtual disks again. When I was using Storage Spaces in Server 2012 R2 at home I was getting around 25%-35% deduplication.

[Dark]

3 minutes ago, Lurick said:

 

If you want to go with all flash, have you looked at Nimble, not sure how their pricing is but I've been looking at them just out of curiosity lately.

We've looked at almost every growing storage solution out, including Nimble and Pure.  Unfortunately, the software side of them is still a bit immature and we're worried that if they do pick up popularity that we'll soon be in the same place as we are with netapp (crazy support fees).

[leadeater]

14 minutes ago, Dark said:

We've looked at almost every growing storage solution out, including Nimble and Pure.  Unfortunately, the software side of them is still a bit immature and we're worried that if they do pick up popularity that we'll soon be in the same place as we are with netapp (crazy support fees).

And to make a switch there is a lot of cost in resource time, training, environment optimization, process integration etc. It can often be more cost effective to deal with the high cost of what ever your current storage is if it is performing well and the support is excellent. We've had controllers fail in the night and have been replaced before 7am.

 

Basically every Netapp customer has the same complaint, cost. ONTAP 9 is sort of nice how they have removed all the separate licenses for each storage protocol type (SMB, NFS, iSCSI, FC) plus snapmirror/snapvault and license on storage capacity, cheaper for some customers and more expensive for others. Also being a Commvault customer I'm very suspicious of capacity based licensing, Netapp has got nothing on Commvault when it comes to cost OMFG.

 

Edit:

Also on Storage Spaces deduplication it's not inline it's scheduled which is actually nice since it has little performance impact at all, so long as you have quiet times to actually run the deduplication job.

[leadeater]

@Dark @Lurick

Oh yea interesting information Weta Digital is switching from Netapp storage to Oracle ZFS appliances, not everyone is as extreme as Weta though .

 

http://medianetwork.oracle.com/video/player/5135849400001

http://www.oracle.com/us/products/servers-storage/storage/nas/creating-digital-innov-culture-3045215.pdf

[Lurick]

16 minutes ago, leadeater said:

@Dark @Lurick

Oh yea interesting information Weta Digital is switching from Netapp storage to Oracle ZFS appliances, not everyone is as extreme as Weta though .

 

http://medianetwork.oracle.com/video/player/5135849400001

http://www.oracle.com/us/products/servers-storage/storage/nas/creating-digital-innov-culture-3045215.pdf

Interesting, although they need to change their network around with a different vendor >.>