Storage Servers, the hell, what am i doing wrong here?

[TehWardy]

Hi all, i'm new here so go easy on me please.

So I'd just like to say, I've been in cloud solutions as an architect for a few years now and am "familiar" with the software side of dealing with scale but this has truly whittled me down to the point of no return.

 

I have 2 existing 5 bay qnap nas boxes, and my plan was simple, build a new server and migrate all my data from those nas boxes on to it, hopefully if I use the right hardware I can run a few other things as VM's too.

So I went out and bought from some local guy a pretty beefy dual Xeon server with 24 bays worth of storage slots and a set of 10x4TB WD Red HDDs.

I did some digging and settled on disk sizes of 4TB since one of my existing nas boxes already runs 5 of them I figured once I was setup I could add those back in and continue to use them too. 

 

Then came the day of the build. 

Since i'm a windows guy I figured I'd start with Windows server 2019 datacenter edition (really go all out).

 

After the relatively complex learning curve of dealing with rack mounted Supermicro boards over the ME ethernet port to "remote in to the OS installation" (super cool) I started to realise the error of my ways.

My plan was to have a single Raid 6 volume consisting of the first 10 drives, and once all the data was over, add a few more of the ones from my existing nas as both more data drives and have one or two hot spares (i figured I would play that by ear).

 

First problem ... Windows apparently doesn't support the creation of Raid 6 volumes.

Second Problem ... The Motherboard has a single SAS port on it, running to a backplane that runs 4 SAS / SATA drives, then there's an add in card with 2 more SAS ports doing the same thing.

 

So I had a total of 12 SAS based drives that I could plugin, and I needed 10 to get going.

 

Having plugged in my drives to the first 10 bays I had only a few of them show up in windows, this perplexed me, hours of digging later I found out I had to "flash the card" in to "IT mode" to allow it to act as a sort of pass through provider of the 8 drives it could support, without doing so the card was expected to have raid configured on it.

That in itself wasn't a problem though, since I had figured out that windows can't give me a raid 6 volume I had to use the hardware / compatible tooling to make what i wanted happen.

 

Having installed the drivers for my SAS card and the relevant tools, I then hit problem 3 ... Raid arrays could only be defined on drives connected to a single controller.

There had to be a solution for this ... well it turns out there is ... 

 

Enter FreeNas, I set about installing that since a Raid-Z volume can be defined across the entire system, the OS just needed to see the drives and I had already done the tricky bit of putting the SAS card in to it's required IT mode state.

Being second hand the server then spent the next 5 hours of my time presenting me with dodge cables, what I thought was failed power rails on the cpu turned out to be splitters that weren't needed, and a bunch of "small bios, SAS bios and Os related settings" to get everything to correctly show up.

 

So I create my Raid-Z2 volume (which is the FreeNas version of a Raid 6 volume ... 
I then have the pain of "unix based systems not being quite the same as a Linux based system" or my linux friends telling me "oh that's my distro, can't really help you there", and other such issues. 

And what was my problem ... permissions.

 

I found the most trivial guides (even on youtube e.g. this one https://www.youtube.com/watch?v=sMZ-s8wHkHw) and for some reason, whatever I did, it just didn't work.

So ... I FINALLY solve the permissions issues and a get a share to show up on my local machine from the newly configured Raid 6 volume on what is basically a super-computer for what i'm about to ask it to do and then ....

 

I begin the file copy from my old NAS boxes.

Within minutes, drives start dropping off the array, all hell starts breaking loose ... yet more "broken splitters" are to blame, not a bad test of volume recovery though I suppose, panic mode reduces a little, and then I realise the speed of my transfer ... 80MB/s oh dear.

 

I have dual 1gbit ethernet going from each nas, and the same going from the server all directly attached to a Netgear GS316  switch.
Surely this can't be it?

 

I've transferred at  over 120MB/s from my local machine to one of the nas boxes before. 

Then it hit me! ... I'm doing a drag and drop from box to box on my local machine, I'm the bottleneck!

 

So I hit up the freenas shell ... 

Hundreds of "attempted" combinations of using rsync, scp, ftp, ssh, ect ,ect later I give up.

I simply can't get a nas / the new freenas box to directly talk to each properly.

It either out right fails, or the transfer speed is <30MB/s.

 

At this point ... I'm sitting here thinking, was I right to ditch windows? 

Internally on my LAN I really only use windows anyway ... but I simply couldn't get a volume setup as I could in FreeNas so this was the next best thing right?

 

... I scream at the screen, pet the dog, get some consolation token from the wife and sit down to think about this logically ...

Why am I doing this on my own! 

 

So here I am ...

 

So I have tons of questions, but really don't know where to start, I guess the obvious start point is ...
 - Should I continue to use a Unix based storage solution when i'm clearly out of my depth or is there some sort of software solution I can add to windows that would achieve the same solution on top of windows allowing me to manage this in a way that i'm used to?

  - What do people think of FreeNas? 
I have already figured out that adding the other 5 drives later is going to be a problem, i'm likely going to need to get hold another 5 so that the same number of drives can be added, my understanding is that ZFS builds a sort of Raid 10 config on top of 2 Raid 6 (RaidZ2) volumes in that situation.

  - Am I looking storage wrong here, what should I do when i'm looking at 40+ TB of disks and I want it to be extensible for future capacity needs?

  - Are raid volumes even some thing that should be extended? 

  - How do people manage say ... 100TB of storage?
  - Can anyone help me with either a FreeNas or Window Server based setup?

 

I hope my story gives other noobs like me some insight in to the experience of moving from a "pre-built" nas like a qnap / synology solution, but hopefully the comments will help improve this for others to help avoid the pitfalls.

I thought I was prepared, clearly I wasn't, I made some silly mistakes, but despite all that, i'm not convinced this is the end as I still have performance issue to resolve.

[JustAsh]

Also interested here as "the Linux friend"

[Windows7ge]

22 hours ago, TehWardy said:

Should I continue to use a Unix based storage solution when i'm clearly out of my depth or is there some sort of software solution I can add to windows that would achieve the same solution on top of windows allowing me to manage this in a way that i'm used to?

 

Personally I'm a proponent for free and/or open source software. If you can learn how to use it it's great when you need to set up several boxes. If you are insistent to use Windows Server you should be able to create a software RAID6 using Windows Storage Spaces regardless of the controllers the drives are connected to. Do be wary of the column width though.

 

22 hours ago, TehWardy said:

What do people think of FreeNas? 

It's a great little OS with a lot of functionality with the exception of you've mentioned the desire to run virtual machines. FreeNAS uses Bhyve as a hypervisor and it is just dreadful. If you want ZFS+a good hypervisor look into PROXMOX or Debian+QEMU. Significantly better performance with support for hardware pass-though.

 

22 hours ago, TehWardy said:

- Am I looking storage wrong here, what should I do when i'm looking at 40+ TB of disks and I want it to be extensible for future capacity needs?

  - Are raid volumes even some thing that should be extended? 

  - How do people manage say ... 100TB of storage?
  - Can anyone help me with either a FreeNas or Window Server based setup?

ZFS can easily handle large amounts of data. Exabytes even. The way you expand a ZFS pool is by appending vdevs. In theory your only limitation is how many interfaces you have to connect more drives. Otherwise ZFS is great at managing very large pools.

[TehWardy]

I do have a separate server I've been using as a hypervisor, that uses HyperV which is pretty awesome for what I need.
TBH I only need to run a few bits like Plex, web services, which I can probably just download plugins and deploy them to a jail (from what I understand).
 

There's definitely a lot to learn, i'm still a bit worried about how i'm going to setup the users / groups for it as trying to set myself up took a while.

 

Right now my biggest problem is getting the 20TB of existing data on there, it seems to be pushing about 1TB per hour at the moment and not having a LACP capable switch slows things down a lot.

 

Windows storage spaces confused the hell out of me, and I couldn't see a way to configure a single raid 6 volume so I figured, for ease of configuration FreeNas did (on face value at least) appear to offer a "cleaner" solution.

 

Might be something worth virtualising at some point and testing out.

Thanks for the feedback.

[Nick7]

In Storage Spaces, RAID5 is called 'parity' and RAID6 ' dual parity'.

However, you do NOT want to use that without SSD's for write back cache, as performance is abysmal.

 

As for your initial issues - was scrubbing happening when testing performance?

Did you try locally performance of you ZFS pool?

(you can time dd or cp commands to test throughput)

 

10 drives should have decent performance, though RAID6 is not really best for writes.

[unijab]

20 hours ago, Windows7ge said:

The way you expand a ZFS pool is by adding more vdevs.

 

fixed

[Windows7ge]

Just now, unijab said:

 

fixed

Oh wait, yeah you're right.

[Windows7ge]

20 hours ago, TehWardy said:

not having a LACP capable switch slows things down a lot.

This is only useful when you have more than one client using the aggregated link at a time. No one client can exceed the speed of a single link as LA operates as a Load Balancing feature rather than a true aggregator.

 

Now a perk of Windows (the features hit or miss on Linux) is SMB3.0 Multi-channel. This allows the full throughput of multiple network ports for a single user/session.

 

2 hours ago, Nick7 said:

However, you do NOT want to use that without SSD's for write back cache, as performance is abysmal.

According to what I've been told at least with Windows Server 2019 is the parity performance with Storage Spaces has improved over previous versions. 2019 is the version of Windows Server he has which is why I've mentioned it as an option.

[TehWardy]

5 hours ago, Nick7 said:

In Storage Spaces, RAID5 is called 'parity' and RAID6 ' dual parity'.

However, you do NOT want to use that without SSD's for write back cache, as performance is abysmal.

 

As for your initial issues - was scrubbing happening when testing performance?

Did you try locally performance of you ZFS pool?

(you can time dd or cp commands to test throughput)

 

10 drives should have decent performance, though RAID6 is not really best for writes.

Agreed 10 drives should well outstretch the performance of a single 1gbit ethernet pipe when each drive is indiviudally capable of a write rate around 150MB/s.
Even with the computation time (which seems to be swallowing a mere 0.3% of my total available compute power).

 

Which is why I don't think ssd's are really needed here.

I doubt the pool was scrubbing since it's a new pool, although I did see a 20MB/s improvement the following morning in the ongoing migration copy process, so maybe!

I haven't tested the local perf ... figured there was no gain to that since it's how fast i can get data on and off it that I care about.

I do have plenty of space for expansions, maybe a 10gig ethernet card is the way to go and a LACP supporting switch.

 

 

3 hours ago, Windows7ge said:

This is only useful when you have more than one client using the aggregated link at a time. No one client can exceed the speed of a single link as LA operates as a Load Balancing feature rather than a true aggregator.

 

Now a perk of Windows (the features hit or miss on Linux) is SMB3.0 Multi-channel. This allows the full throughput of multiple network ports for a single user/session.

 

According to what I've been told at least with Windows Server 2019 is the parity performance with Storage Spaces has improved over previous versions. 2019 is the version of Windows Server he has which is why I've mentioned it as an option.

I hadn't clicked that aggregation wouldn't work as a single 2 gig pipe, nice catch!

 

The SMB 3.0 idea is interesting though ... One of my friends suggested that I finish the data migration then install windows on the metal, pass through the physical discks to freenas running on a HyperV VM ... this would give me smarter networking on to the raw hardware and I could "configure a virtual switch" for passing the data in to freenas on a single pipe at the new higher speed.

 

 

3 hours ago, unijab said:

The way you expand a ZFS pool is by adding more vdevs.

 

fixed

Looking at the documentation I got the impression this wasn't "expanding" the existing pool but instead running a second pool along side it. 

Is that not the case?

Weird that the UI pushes you to add the same number of drives in the second vdev ...

Until you realise that what it's doing is basically turning 2x RAID6 pools in to a single striped RAID 10 pool when you do that.

 

 

[Windows7ge]

26 minutes ago, TehWardy said:

I hadn't clicked that aggregation wouldn't work as a single 2 gig pipe, nice catch!

 

The SMB 3.0 idea is interesting though ... One of my friends suggested that I finish the data migration then install windows on the metal, pass through the physical discks to freenas running on a HyperV VM ... this would give me smarter networking on to the raw hardware and I could "configure a virtual switch" for passing the data in to freenas on a single pipe at the new higher speed.

Only problem there is the virtual pipe ends at the virtual switch. You'd still have to assign FreeNAS 2 NICs on different networks and enable SMB3 Multi-channel in the config file. Without those your connection would still max out at Gigabit or whatever else is the slowest link.

[Jarsky]

On 1/13/2020 at 8:14 AM, TehWardy said:

So I have tons of questions, but really don't know where to start, I guess the obvious start point is ...
 - Should I continue to use a Unix based storage solution when i'm clearly out of my depth or is there some sort of software solution I can add to windows that would achieve the same solution on top of windows allowing me to manage this in a way that i'm used to?

If you want to use Windows, it does have "RAID6" in the form of Storage Spaces. But honestly, if you wanted to use Windows i'd probably go with a hardware RAID setup for what you're trying to achieve.

 

What is the card you have? 

It sounds like the card is dual port, you can increase the capacity by using a SAS Expander like the Intel res2sv240. I have one of these paired with a LSI 9271-8i which provides me 20 ports + 4 from the controllers second SAS port, gives me 24 ports total. 

The advantage to having a LSI hardware setup would also be that you *can* add additional drives to your array and expand it. 

 

Quote

  - What do people think of FreeNas? 
I have already figured out that adding the other 5 drives later is going to be a problem, i'm likely going to need to get hold another 5 so that the same number of drives can be added, my understanding is that ZFS builds a sort of Raid 10 config on top of 2 Raid 6 (RaidZ2) volumes in that situation.

FreeNAS is fantastic as storage only. It is great at handling ZFS and doing SMB, NFS & iSCSi, but its not really great if you're looking for an all-in-one solution. The Plugins, VM & Docker features are quite limited and being based on FreeBSD its quite hard to do work arounds like you can in Linux.

 

Quote

  - Am I looking storage wrong here, what should I do when i'm looking at 40+ TB of disks and I want it to be extensible for future capacity needs?

No I think you have a good grasp on the concepts. When you're working with 8+ disks you definately want at least 2 redundancy disks like you're trying to achieve. Just each way of approaching the storage array concept has its positives & negatives. 

 

Quote

  - Are raid volumes even some thing that should be extended? 

Theres always risk involved when extending RAID's that are distributed. I can only tell you that my hardware array has been the only one that i've had 100% success with, with a 60TB array

 

Quote

  - How do people manage say ... 100TB of storage?

ZFS is amazing for this. Wether you use FreeNAS, Proxmox, or any other Linux distro with ZFSonLinux. 

Essentially with ZFS, one of its powers is that you can create multiple VDEV's (e.g RAID5 or RAID6's) and combine them together in a single zpool so they look like a single logical volume (what ZFS calls a dataset)

You could for example with your 10 disks, create 2 x RAID5's (RAIDZ1) with 5 disks each. Then with your 4 spare disks later, if you get 1 more then you can add those 5 to create a 3rd RAIDZ1 vdev. 

   

Quote

  - Can anyone help me with either a FreeNas or Window Server based setup?

You just need to decide what base you want to use and we can help from there. 

Personally rather than FreeNAS I would go Debian or Ubuntu with ZFSonLinux & Webmin (it gives you a web gui interface that includes a ZFS manager).

You can then install Docker, KVM, QEMU, etc...as you need to for doing docker or VM management. 

Also if you really don't want the hassle of Linux, then Windows Storage Spaces can do parity RAID's, but id probably lean more into going Hardware RAID + SAS Expander card to connect all the drives to your controller.