Hi all, i'm new here so go easy on me please.
So I'd just like to say, I've been in cloud solutions as an architect for a few years now and am "familiar" with the software side of dealing with scale but this has truly whittled me down to the point of no return.
I have 2 existing 5 bay qnap nas boxes, and my plan was simple, build a new server and migrate all my data from those nas boxes on to it, hopefully if I use the right hardware I can run a few other things as VM's too.
So I went out and bought from some local guy a pretty beefy dual Xeon server with 24 bays worth of storage slots and a set of 10x4TB WD Red HDDs.
I did some digging and settled on disk sizes of 4TB since one of my existing nas boxes already runs 5 of them I figured once I was setup I could add those back in and continue to use them too.
Then came the day of the build.
Since i'm a windows guy I figured I'd start with Windows server 2019 datacenter edition (really go all out).
After the relatively complex learning curve of dealing with rack mounted Supermicro boards over the ME ethernet port to "remote in to the OS installation" (super cool) I started to realise the error of my ways.
My plan was to have a single Raid 6 volume consisting of the first 10 drives, and once all the data was over, add a few more of the ones from my existing nas as both more data drives and have one or two hot spares (i figured I would play that by ear).
First problem ... Windows apparently doesn't support the creation of Raid 6 volumes.
Second Problem ... The Motherboard has a single SAS port on it, running to a backplane that runs 4 SAS / SATA drives, then there's an add in card with 2 more SAS ports doing the same thing.
So I had a total of 12 SAS based drives that I could plugin, and I needed 10 to get going.
Having plugged in my drives to the first 10 bays I had only a few of them show up in windows, this perplexed me, hours of digging later I found out I had to "flash the card" in to "IT mode" to allow it to act as a sort of pass through provider of the 8 drives it could support, without doing so the card was expected to have raid configured on it.
That in itself wasn't a problem though, since I had figured out that windows can't give me a raid 6 volume I had to use the hardware / compatible tooling to make what i wanted happen.
Having installed the drivers for my SAS card and the relevant tools, I then hit problem 3 ... Raid arrays could only be defined on drives connected to a single controller.
There had to be a solution for this ... well it turns out there is ...
Enter FreeNas, I set about installing that since a Raid-Z volume can be defined across the entire system, the OS just needed to see the drives and I had already done the tricky bit of putting the SAS card in to it's required IT mode state.
Being second hand the server then spent the next 5 hours of my time presenting me with dodge cables, what I thought was failed power rails on the cpu turned out to be splitters that weren't needed, and a bunch of "small bios, SAS bios and Os related settings" to get everything to correctly show up.
So I create my Raid-Z2 volume (which is the FreeNas version of a Raid 6 volume ...
I then have the pain of "unix based systems not being quite the same as a Linux based system" or my linux friends telling me "oh that's my distro, can't really help you there", and other such issues.
And what was my problem ... permissions.
I found the most trivial guides (even on youtube e.g. this one https://www.youtube.com/watch?v=sMZ-s8wHkHw) and for some reason, whatever I did, it just didn't work.
So ... I FINALLY solve the permissions issues and a get a share to show up on my local machine from the newly configured Raid 6 volume on what is basically a super-computer for what i'm about to ask it to do and then ....
I begin the file copy from my old NAS boxes.
Within minutes, drives start dropping off the array, all hell starts breaking loose ... yet more "broken splitters" are to blame, not a bad test of volume recovery though I suppose, panic mode reduces a little, and then I realise the speed of my transfer ... 80MB/s oh dear.
I have dual 1gbit ethernet going from each nas, and the same going from the server all directly attached to a Netgear GS316 switch.
Surely this can't be it?
I've transferred at over 120MB/s from my local machine to one of the nas boxes before.
Then it hit me! ... I'm doing a drag and drop from box to box on my local machine, I'm the bottleneck!
So I hit up the freenas shell ...
Hundreds of "attempted" combinations of using rsync, scp, ftp, ssh, ect ,ect later I give up.
I simply can't get a nas / the new freenas box to directly talk to each properly.
It either out right fails, or the transfer speed is <30MB/s.
At this point ... I'm sitting here thinking, was I right to ditch windows?
Internally on my LAN I really only use windows anyway ... but I simply couldn't get a volume setup as I could in FreeNas so this was the next best thing right?
... I scream at the screen, pet the dog, get some consolation token from the wife and sit down to think about this logically ...
Why am I doing this on my own!
So here I am ...
So I have tons of questions, but really don't know where to start, I guess the obvious start point is ...
- Should I continue to use a Unix based storage solution when i'm clearly out of my depth or is there some sort of software solution I can add to windows that would achieve the same solution on top of windows allowing me to manage this in a way that i'm used to?
- What do people think of FreeNas?
I have already figured out that adding the other 5 drives later is going to be a problem, i'm likely going to need to get hold another 5 so that the same number of drives can be added, my understanding is that ZFS builds a sort of Raid 10 config on top of 2 Raid 6 (RaidZ2) volumes in that situation.
- Am I looking storage wrong here, what should I do when i'm looking at 40+ TB of disks and I want it to be extensible for future capacity needs?
- Are raid volumes even some thing that should be extended?
- How do people manage say ... 100TB of storage?
- Can anyone help me with either a FreeNas or Window Server based setup?
I hope my story gives other noobs like me some insight in to the experience of moving from a "pre-built" nas like a qnap / synology solution, but hopefully the comments will help improve this for others to help avoid the pitfalls.
I thought I was prepared, clearly I wasn't, I made some silly mistakes, but despite all that, i'm not convinced this is the end as I still have performance issue to resolve.