What raid/ZFS to use? & FreeNAS SMB Multichannel

[Inrix]

I'm planning on making a 16TB redundant nas using 8TB WD Red HDDS. What's the best raid type/ZFS (I don't use ZFS much but I'm up for trying it with something like FreeNAS) to use for the best redundancy? I don't really care about the amount of drives I need to buy as long as its not more than 2x drives per 1 drive storage capacity. 

 

Also does FreeNAS support SMB Multichannel for giving higher bandwidth to single transfers using multiple network adapters?

Also, 

https://nz.pcpartpicker.com/user/Inrix/saved/#view=VDFnQ7 Here is the current build ish planned for it.... If there is anything you see that could be better then please do tell me

[79wjd]

My vote goes for Raid Z2 (Raid 6), so you'd need four 8tb drives to get 16tb, or alternatively, six 4tb drives. 

[Electronics Wizardy]

9 minutes ago, Inrix said:

Also does FreeNAS support SMB Multichannel for giving higher bandwidth to single transfers using multiple network adapters?

nope, if you want that, use server 2016.

 

 

[Inrix]

14 minutes ago, Electronics Wizardy said:

nope, if you want that, use server 2016.

 

 

K I probably will then...

 

19 minutes ago, djdwosk97 said:

My vote goes for Raid Z2 (Raid 6), so you'd need four 8tb drives to get 16tb, or alternatively, six 4tb drives. 

 

Looks to be pretty good, How easy is Raid 6 to expand? Id really like to be able to just put more drives in the system and have the volume expand if that's possible. Also would I need a separate raid card to do that or is it possible to do it motherboard/Server 2016 level (Would it be better to have a dedicated raid card with a battery for write offload)?

 

What raid card would you recommend from this https://www.pbtech.co.nz/ site if you think I need one?

[Electronics Wizardy]

13 minutes ago, Inrix said:

K I probably will then...

Id take a look at storage spaces then. Its the windows server eqv of zfs and works pretty well and supports most of the same features.

 

You don't want a raid card, you want a hba, look at a used dell h200

[Inrix]

3 minutes ago, Electronics Wizardy said:

Id take a look at storage spaces then. Its the windows server eqv of zfs and works pretty well and supports most of the same features.

 

You don't want a raid card, you want a hba, look at a used dell h200

So I would be best off running with a raid card then? Id rather stay away from servers as they are (most of them) noisy and the point of this is also to be quiet.

[Electronics Wizardy]

Just now, Inrix said:

So I would be best off running with a raid card then? Id rather stay away from servers as they are (most of them) noisy and the point of this is also to be quiet.

don't get a raid card, its makes it so you can't really use zfs or storage spaces well. You can't also use raidz with a raid card.

 

You want a hba. A dell h200 is just a hba(search it on ebay) thats just a rebranded lsi 9207 8i.

[Inrix]

What HBA card would you recommend I use from this site then? https://www.pbtech.co.nz/

 

And sorry I didn't quite understand your previous comment.

 

Basically a HBA card has proper integration for Server 2016's RaidZ stuff right? Sorry, I haven't done a lot of stuff in the area of server 2016 so I'm still new to it

[leadeater]

3 hours ago, Inrix said:

Also does FreeNAS support SMB Multichannel for giving higher bandwidth to single transfers using multiple network adapters?

You need the very latest version of SAMBA for multichannel support and it's still considered experimental.

[leadeater]

10 hours ago, Inrix said:

What HBA card would you recommend I use from this site then? https://www.pbtech.co.nz/

 

And sorry I didn't quite understand your previous comment.

 

Basically a HBA card has proper integration for Server 2016's RaidZ stuff right? Sorry, I haven't done a lot of stuff in the area of server 2016 so I'm still new to it

I wouldn't buy a new HBA just get one from ebay, there's thousands of them around the $80-$100 NZD mark. I personally go for the IBM M1015 (LSI 9211).

[Eniqmatic]

Yep as above, multichannel support is coming in an upcoming release, the past few SAMBA versions have had it in as "experimental" and FreeNAS currently does have the ability to do it, you just won't find it in the GUI, you will need to tweak with the backend. Weather or not it works, I will leave that for you to discover!

[brandishwar]

22 hours ago, Inrix said:

How easy is Raid 6 to expand? Id really like to be able to just put more drives in the system and have the volume expand if that's possible.

Since this hasn't been addressed yet, I'll address it here. Note this assumes you're going with FreeNAS and not Windows Server 2016.

 

In most instances you cannot expand a RAID 6 setup. Not even a RAID-Z2 setup (ZFS RAID 6) on FreeNAS can be expanded, not without destroying the array and rebuilding it to include the new drive. It's way too problematic to support this. While some RAID cards do support being able to expand a RAID array, it is extremely time consuming.

 

If you want to be able to just "add drives", then you'll need to use mirrored pairs. I'll use my NAS as an example. I have three mirrored pairs in my NAS: two pairs of 4TB drives, and a pair of 6TB drives, giving an effective raw storage space of 14TB minus file system overhead. I started with a pair of 4TB in a mirrored pair, then added a second mirrored pair of 4TB as a second v-dev in FreeNAS. Then expanded the pool to use it. And did the same when I added a pair of 6TB drives.

 

While I lose out on space doing this, the main advantage is the fact that I could add a pair of 6TB drives and not lose anything to it. The next addition is likely to be a pair of 8TB drives when I feel the need for it and prices at the time.

[leadeater]

5 hours ago, brandishwar said:

While some RAID cards do support being able to expand a RAID array, it is extremely time consuming.

It may be time consuming but it is extremely easy and can be done when the system is online and you are using it, in the case of every LSi and Adaptec card that is. Hardware RAID is one of the most flexible and delivers the best performance consistency in every configuration when using a BBU compared to everything else, it's only let down is data integrity.

 

Far as FreeNAS goes you can do that same as you do using RAIDZ but you have to expand using a lot more disks at a time which for me is a rather big drawback to FreeNAS or more correctly ZFS.

 

 

[Jarsky]

It's hardly time consuming. 

 

I have an LSI 9271-8i for my home NAS. I just replaced 2 smaller drives yesterday, expanded the virtual drive, recalculated parity and extended the partition.

It took about 14hrs total and about 30 minutes of interactive time. The other awesome thing about good hardware controllers, is I can expand from RAID 5 to 6 no issue, and add additional drives with basically no interaction. 

 

As stated with ZFS, while its awesome, if you want to extend the zpool you have to create your vdevs in one go which makes expanding a much more difficult process. 

Thats where something like Rockstor is a good alternative to FreeNAS. It uses btrfs, which is a next-gen filesystem like zfs and refs, but gives you the flexibility to extend/expand easily. Another alternative, is using Windows Storage Spaces - while it doesn't officially support on the fly resizing, you *CAN* overprovision your pool to start with and add drives as you run out of space - e.g you can create a 100TB pool, with 4 x 8TB drives in a 2 disk parity, and continually keep adding 8TB drives as you need to. 

 

As for HBA's I've had great success with the AOC-SAS2LP-MV8, but also would recommend the LSI 9211-8i 

[leadeater]

2 hours ago, Jarsky said:

Windows Storage Spaces - while it doesn't officially support on the fly resizing

What do you mean by that? Storage Spaces supports adding disks to existing pools and resizing existing storage tiers and virtual disks. Depending on configuration you have to use powershell to do the resizing, multiple storage tiers, and the cmdlet for it is Resize-StorageTier.

https://technet.microsoft.com/en-us/library/dn454978(v=wps.630).aspx

 

Once you've increased the size of a storage teir you can extend the virtual disk that uses it. Not everything can be done using the Server Manager GUI which isn't made clear without going in to the technet documentation and spending some time using it.

[Jarsky]

5 hours ago, leadeater said:

What do you mean by that? Storage Spaces supports adding disks to existing pools and resizing existing storage tiers and virtual disks. Depending on configuration you have to use powershell to do the resizing, multiple storage tiers, and the cmdlet for it is Resize-StorageTier.

https://technet.microsoft.com/en-us/library/dn454978(v=wps.630).aspx

 

Once you've increased the size of a storage teir you can extend the virtual disk that uses it. Not everything can be done using the Server Manager GUI which isn't made clear without going in to the technet documentation and spending some time using it.

Ah OK wasn't aware of that cmdlet, I haven't used it since 2012, possibly added in 2012 R2? But yeah was aware a lot of the functionality is cmdlet only, but when I was using storage spaces last there was virtually nil support documentation either which is a shame given how good it is, often getting underrated as an option 

[leadeater]

8 minutes ago, Jarsky said:

Ah OK wasn't aware of that cmdlet, I haven't used it since 2012, possibly added in 2012 R2? But yeah was aware a lot of the functionality is cmdlet only, but when I was using storage spaces last there was virtually nil support documentation either which is a shame given how good it is, often getting underrated as an option 

Yep documentation is still the biggest problem with Storage Spaces, so much of how it works is poorly explained or buried in cmdlet documentation and never really explains what it actually does. Much of it is trial and error.

 

Think it was added in Server 2012 R2, I think that was when storage tiers were added. Was that a thing in Server 2012? I've never really used straight 2012 and never with Storage Spaces, only used Server 2012 R2 and Server 2016 Storage Spaces.

[Jarsky]

45 minutes ago, leadeater said:

Yep documentation is still the biggest problem with Storage Spaces, so much of how it works is poorly explained or buried in cmdlet documentation and never really explains what it actually does. Much of it is trial and error.

 

Think it was added in Server 2012 R2, I think that was when storage tiers were added. Was that a thing in Server 2012? I've never really used straight 2012 and never with Storage Spaces, only used Server 2012 R2 and Server 2016 Storage Spaces.

There were storage tiers in 2012, but I don't believe there was a way to resize them then - which is why I used to use the above workaround I posted about over provisioning. Besides my NAS now though, all my servers are virtualised (and at work we strictly work on enterprise, so 99% virtualisation with SAN LUN's) so have very little need to play with storage spaces since I moved to 2012 R2 & 2016!