Baby's First Server Build

[blumpy]

I've had several servers through out the years but I've always gone with dedicated file servers for backups and archives i.e. Drobo, QNAP, and Synology but I outgrow these system almost annually.  So I've decided to build a custom rack server that I can manually expand as needed and use on projects instead of just slow access offline storage.  I've decided to use Unraid for the file server and possible a virtualization of windows running Vienna Ensemble Pro for orchestral samples and a tiered file system with HDDs, SSD and NVME over 10gbe.

 

Questions in logical order:

Can a virtualization of windows take down an Unraid server?  I'd assume there are protections for this and the virtualization might crash but it is sandboxed and won't take out Unraid?  If that is true then I will have the system run both, otherwise I'll run VEP on a separate system.

 

I'm assuming ECC ram is necessary for a fileserver, true? Should I avoid Ryzen for a Unraid server and if Ryzen works well will Unraid make use of ECC on AMD?  If Ryzen is okay, has anyone had 4x32gb UDIMMs work with a Ryzen MB?  If not, I'll consider a Threadripper so I can have 8x16gb RAM of ECC UDIMM.

 

If Ryzen/Threadripper are bad ideas or if ECC is not going to work with Unraid, I'll do a Xeon server. 

 

What file system works best for workstation tiered performance?  btrfs, ,xfs, zfs, ext4 something else? I'm assuming btrfs because of the cache drives I'd be using but maybe I'm misinformed.

 

Am I out of my mind and should continue using local NVME storage for the workstations archived and backed up to a pre-built file server and leave the pro-IT to the pros?

 

All typographical errors ™ 2018

[conman2305]

I'm not terribly familiar with Unraid in particular, but assuming it's built on one of the standard hypervisors like KVM or Xen, each VM has dedicated memory and paging tables so that one VM cannot "take out" another. This is the primary purpose of a (good) hypervisor - to make sure that even if you infest one VM with WannaCry, the other VMs on the system are totally and completely unaware.

 

ECC RAM is not strictly required to run a file server, but consider the penalties of not doing so. If a small bit of data loss should the memory encounter a correctable error (separate from uncorrectable errors which even ECC RAM can't save you from) is acceptable to you, you can save a buck.

 

I am very partial to ZFS for any sort of heavy duty cycle storage solution. So again, its going to depend on your exact use case. I personally use FreeNAS both at work and at home, but as I understand it Unraid has several built in storage options that will probably make the most sense. Connecting this storage to your workstations can be a bit tricky, so I personally advise SMB shares for Windows. 

 

Hopefully that helped a bit. Let me know if you want me to explain any of my ramblings more.

[Salv8 (sam)]

unless you need to have two systems in one, it's not worrth it

get a Linux server distro and install some virtualization software on it, it's cheeper and won't make you pull your hair out because it's confusing as fuck

unRAID is meant for HUGE virtualization workloads (Microsoft's azure is an example of huge virtualization workloads) and since you are running one virtual machine, it's not worth it, also it's expensive and requires a licence depending on how many storage devices you have attached to the system

pricing: https://lime-technology.com/pricing/

just get a server on eBay and buy some HDD's that fill your storage needs and install Linux onto the system

here i will get you started

this server is good for your needs:

https://www.ebay.com/itm/SuperMicro-Server-10-Bay-2-5-SATA-SAS-SSD-HDs-CSE-116-X9SRH-7F-16GB-E5-2650L/322951268974?hash=item4b31654a6e:g:cIsAAOSwZlZaO~MR

a lot of drive bays, at least 16 GB of RAM for virtualization and 8 cores for those virtual machines and redundant PSU's!! (very useful if the server has to stay running)

there is no HDD's included but it's always a good idea to buy new HDD's anyways as the old ones could be on their last legs and could fail at anytime...

(my server is like that and i fear i will lose my data, but i keep two copy's, one on my server and one on my main desktop)

EDIT: COMPLEATLY FORGOT, install a GUI onto the server distro if it doesn't have one (most don't), google it if you don't know how to do it

Edited March 2, 2018 by Salv8 (sam)
forgot something

[blumpy]

19 hours ago, conman2305 said:

I'm not terribly familiar with Unraid in particular, but assuming it's built on one of the standard hypervisors like KVM or Xen, each VM has dedicated memory and paging tables so that one VM cannot "take out" another. This is the primary purpose of a (good) hypervisor - to make sure that even if you infest one VM with WannaCry, the other VMs on the system are totally and completely unaware.

 

ECC RAM is not strictly required to run a file server, but consider the penalties of not doing so. If a small bit of data loss should the memory encounter a correctable error (separate from uncorrectable errors which even ECC RAM can't save you from) is acceptable to you, you can save a buck.

 

I am very partial to ZFS for any sort of heavy duty cycle storage solution. So again, its going to depend on your exact use case. I personally use FreeNAS both at work and at home, but as I understand it Unraid has several built in storage options that will probably make the most sense. Connecting this storage to your workstations can be a bit tricky, so I personally advise SMB shares for Windows. 

 

Hopefully that helped a bit. Let me know if you want me to explain any of my ramblings more.

Thank you for your feedback. 

 

What is your experience with FreeNas?  For some reason I was under the impression that setting up unraid would be relativity easy but perhaps I am mistaken.  I loosely looked into FreeNas but was a little confused by their hardware product line vs build your own solutions, i.e. does it work well for cached storage on build-in NVME, sata ssds and HDD or do I have to use their pcie cards? 

 

Ultimately all I really need out of this is a intelligent caching with expandable storage/archive.  Currently each workstation has NVME and SSD for ready quick access to 5-6TB of data.  The rest is stored in a RAID 6 NAS which is as slow as one could imagine. What I'd like to do is have a RAID, resistant to bit rot, that acts as one big cloud with the cache so that recently access files are transferred from the RAID to the cache with the bottleneck being the maximum speed of 10GBE.  As video content increases with size, I'd likely want the ability to run multiple tiers of caching to be future proof for at least 5-6 years or more if possible.

 

Thoughts?

[blumpy]

19 hours ago, Salv8 (sam) said:

unless you need to have two systems in one, it's not worrth it

get a Linux server distro and install some virtualization software on it, it's cheeper and won't make you pull your hair out because it's confusing as fuck

unRAID is meant for HUGE virtualization workloads (Microsoft's azure is an example of huge virtualization workloads) and since you are running one virtual machine, it's not worth it, also it's expensive and requires a licence depending on how many storage devices you have attached to the system

pricing: https://lime-technology.com/pricing/

just get a server on eBay and buy some HDD's that fill your storage needs and install Linux onto the system

here i will get you started

this server is good for your needs:

https://www.ebay.com/itm/SuperMicro-Server-10-Bay-2-5-SATA-SAS-SSD-HDs-CSE-116-X9SRH-7F-16GB-E5-2650L/322951268974?hash=item4b31654a6e:g:cIsAAOSwZlZaO~MR

a lot of drive bays, at least 16 GB of RAM for virtualization and 8 cores for those virtual machines and redundant PSU's!! (very useful if the server has to stay running)

there is no HDD's included but it's always a good idea to buy new HDD's anyways as the old ones could be on their last legs and could fail at anytime...

(my server is like that and i fear i will lose my data, but i keep two copy's, one on my server and one on my main desktop)

EDIT: COMPLEATLY FORGOT, install a GUI onto the server distro if it doesn't have one (most don't), google it if you don't know how to do it

In the past I've usually preferred one device per function.  This is the first time I've considered a device to pull double duty, as a file server and as a Vienna Ensemble Server but the two are highly linked.  The ebay solution definitely wouldn't work for VEP as 64GB of ram for each of the two virtualization would be needed.  I'd prefer to have the ability to expand to 3-4 virtualization. Lost of ram is needed for VEP.  So I'd either need separate VEP servers, or one huge server with lots of cores and ram.  I'd prefer an all in one solution because if I go with separate VEP servers it will eat away at the network throughput to workstations; i.e. The VEP server grabbing data via a virtual port in the same CPU then sending it to the workstation is preferred vs a separate VEP server(s) grabbing the data over the network, processing it then sending it to the workstations over the network. 

 

The other issue is noise.  I need this to be quiet.  Which is why I'm going with a custom build.  The price range would be awesome but I need 5-6 TB of as fast as possible storage as a starting point with 40-50TB of storage that is accessible but not ultrafast to replace local SSDs, NVMEs and RAIDs for workstations.... Maybe I can plop in some SAS ssds as a cache....I'm not sure.

 

I probably need a pro for this but I'm trying to do it on my own.

 

Thoughts?

[Salv8 (sam)]

54 minutes ago, blumpy said:

-snip-

if you have a friend or dude you know that does this stuff. ask him for some help

since it's your first time it's better to learn as much as you can from a pro as if you screw up, it's your fault and probably will cost you a lot...

there is no shame in asking for help or learning from your mistakes, (unless you are stubborn then you do what you want) that is how we grow and learn

many a IT pro has made a mistake and learned from it, you think Linus became a know-it-all over night? (well... i wouldn't be surprised, this is Linus we are talking about)

no! he tried making his own router and some many problems arised that well...

watch it if you want to see Linus really annoyed

my point is if you have a friend ask him for some help in this... trust me it's better then going into the rabbit hole that is frustration and a rabbit wearing a suit and holding a pocket watch

[blumpy]

4 hours ago, Salv8 (sam) said:

if you have a friend or dude you know that does this stuff. ask him for some help

since it's your first time it's better to learn as much as you can from a pro as if you screw up, it's your fault and probably will cost you a lot...

there is no shame in asking for help or learning from your mistakes, (unless you are stubborn then you do what you want) that is how we grow and learn

many a IT pro has made a mistake and learned from it, you think Linus became a know-it-all over night? (well... i wouldn't be surprised, this is Linus we are talking about)

no! he tried making his own router and some many problems arised that well...

watch it if you want to see Linus really annoyed

my point is if you have a friend ask him for some help in this... trust me it's better then going into the rabbit hole that is frustration and a rabbit wearing a suit and holding a pocket watch

I totally agree and have no issue asking for help, which is why I'm here.  That said, while I'm aware there are people who specialize in these sort of things, I do not know anyone....or of those that I know, they'd be more likely to come to me for advice.  Which is why I'm here.

 

While this is my first high speed file server build, I've built other servers in the past, email servers, simple stuff. 

[sturgis]

There are definitely benefits to unraid, though I really like freenas too. Due to low income flow though, I ended up settling on unraid.  Freenas is great. Its FAST if you set it up right.  For me the biggest negative had to do with the path to upgrade storage.  If you have the budget to really plan out your needs in advance, then its awesome.  You can't just pop in a larger drive at a later date, so pre-planning is key.  Well ok you can replace a single drive with a larger one, but you won't get any additional space until all drives in the array are upgraded to the same size.  (you could of course add a second array)  

 

I like unraid because the biggest requirement is that the parity drive be larger than any of the other drives, and you can replace and/or add a drive to your array at any time.   A few days ago I added a 4tb drive without a hiccup.  The parity works well also. (I bought a used server and had 1 bad cable that ended up failing.  Drive fell out of the array, and the system went into emulated mode. I replaced the flaky cable, fired up again, made sure settings were right, started the array and voila'  Rebuit and runs great)   Unraid does support 2 parity drives so you can have 2 drive failures and lose nothing. (But.. um. Backup backup backup!)   Also, since the drives are not striped, if things were to go catastrophic, you still have all the data on any drives that didn't fail.  

 

Then there is the cache pool.  You can add multiple ssds to a pool (same size drives) with btrfs and have various choices as to how they're pooled.  

 

I haven't set up my windows vm yet, I've been messing with many other things (quite a few linux vms, as well as a bunch of dockers) but you might check out this video for a windows vm setup guide.  Spaceinvader one has quite a few really good videos.  

 

As for hardware, I also recommend used server/workstation grade hardware.  I ended up with a z800 with 24g ECC and duel xeons.  I did a processor swap for x5670s giving me a pretty nice 12 core 24 thread system.  I have yet to make it break a sweat.   (I like the linked supermicro too, but cost per TB of storage is cheaper with 3.5" drives, so that's my preference rather than the 2.5s in that particular server.  8 drives plus cache possible in my system)   (my first effort was an old athlon xp 2 core system with 8G memory and it ran surprisingly well)

 

As mentioned, there is a cost involved for the unraid key, but its a lifetime thing, so for me the cost wasn't much of a factor. 

[blumpy]

On 3/17/2018 at 2:37 PM, sturgis said:

There are definitely benefits to unraid, though I really like freenas too. Due to low income flow though, I ended up settling on unraid.  Freenas is great. Its FAST if you set it up right.  For me the biggest negative had to do with the path to upgrade storage.  If you have the budget to really plan out your needs in advance, then its awesome.  You can't just pop in a larger drive at a later date, so pre-planning is key.  Well ok you can replace a single drive with a larger one, but you won't get any additional space until all drives in the array are upgraded to the same size.  (you could of course add a second array)  

 

I like unraid because the biggest requirement is that the parity drive be larger than any of the other drives, and you can replace and/or add a drive to your array at any time.   A few days ago I added a 4tb drive without a hiccup.  The parity works well also. (I bought a used server and had 1 bad cable that ended up failing.  Drive fell out of the array, and the system went into emulated mode. I replaced the flaky cable, fired up again, made sure settings were right, started the array and voila'  Rebuit and runs great)   Unraid does support 2 parity drives so you can have 2 drive failures and lose nothing. (But.. um. Backup backup backup!)   Also, since the drives are not striped, if things were to go catastrophic, you still have all the data on any drives that didn't fail.  

 

Then there is the cache pool.  You can add multiple ssds to a pool (same size drives) with btrfs and have various choices as to how they're pooled.  

 

I haven't set up my windows vm yet, I've been messing with many other things (quite a few linux vms, as well as a bunch of dockers) but you might check out this video for a windows vm setup guide.  Spaceinvader one has quite a few really good videos.  

 

As for hardware, I also recommend used server/workstation grade hardware.  I ended up with a z800 with 24g ECC and duel xeons.  I did a processor swap for x5670s giving me a pretty nice 12 core 24 thread system.  I have yet to make it break a sweat.   (I like the linked supermicro too, but cost per TB of storage is cheaper with 3.5" drives, so that's my preference rather than the 2.5s in that particular server.  8 drives plus cache possible in my system)   (my first effort was an old athlon xp 2 core system with 8G memory and it ran surprisingly well)

 

As mentioned, there is a cost involved for the unraid key, but its a lifetime thing, so for me the cost wasn't much of a factor. 

Thank you for all the advise.  Very helpful!  I'll be backing up to LTO and bluray.

 

I'll probably go with Unraid, but with FreeNas, let's say I have 10 x 10TB drives (2 parity) and tiered cache.  If I want to add more 10TB drives that would not be possible or difficult?