Unraid/FreeNas/Storage Spaces Build planning

bimmerman · August 26, 2019

Hi all,

I just built a Ryzen 3600 based system, which has left my old trusty X58 rig pretty much collecting dust. I'm debating repurposing it for an Unraid/Freenas style build, as I think it'd at least have some utility there. But, before I do, I have a few questions where I'm uncertain of what the right route to go is. I'd appreciate any insights! Even if the answer is 'buy a Synology'.

My goal with this system is to primarily be a NAS, but to also play around with virtualization and maybe Plex too. The NAS functionality is key, the rest are nice to haves but it's why I'm primarily leaning towards Unraid. Budget is ~$5-600 or so, ideally for drives.

Format below is organized somewhat by section, each bullet point is something I'm curious about for planning.

Current (idle, unused) System:

Gigabyte GA-X58-UD3R rev2.0 + Xeon X5675 + 24 GB rams + Coolermaster AIO

R9 290 GPU

2x 1TB + 1x 2TB storage HDD

1x 950 Pro NVMe as PCIe boot drive

850W power supply

HAF 932 case

Desired changes to use as NAS/VM/Server box

Revert to stock clocks from current 4.5GHz overclock

add 4-5 NAS HDDs for an array

add SSDs for cache drive(s)

use Unraid for array managment, virtualize Win 10 at least one instance

Anyway. Here are some questions! I'd appreciate any insight you all can provide.

Hardware bottleneck questions.

Specifically, Sata II vs III. My motherboard does not have the best storage controllers-- all the Sata ports are behind the chipset, which definitely isn't modern-system-fast. I have a bunch of Sata 2, and two Sata 3 ports, all sharing the chipset link bandwidth to the CPU.
Is this an issue for HDD-based Raid 5?
While I have a bunch of excess PCIe lanes, PCIe 2.0 is all I have on the board, and 16x of it will likely be for GPU. That leaves 16x remaining since the other slot sharing bandwidth with the GPU is blocked by it.

Network bottleneck questions.

The motherboard has a single onboard 1Gb nic. I don't have fancy networking equipment (currently), but do have a multiport switch.
Would it be a benefit to add in a PCIe network card for teaming/link aggregation?
Or am I unlikely to saturate the 1GbE link with maybe 2-3 concurrent device requests?

Storage setup questions.

I'm looking at adding 4-5 4TB NAS HDD drives in a Raid 5 or whatever ZFS/BTRFS does, maybe paired with a couple small SSDs for caching.
The system currently has 1x 2TB HDD + 2x 1TB HDD (old!) + 1x 256GB 950 Pro NVMe boot drive (yep!). I am not tied to reusing these for the main storage array due to age, but maybe the NVMe for Cache and the others for backup/spares?
Is it better to add another PCIe card for M.2 cache, or to run all drives via SATA? Cheaper, certainly, to run Sata drives.
Is there a benefit to running 4 vs 5 drives in the storage array?
- Reason I ask is that, should this system fail AND the drives will be transferable, most (Ryzen) ITX systems I've seen are hard-capped at 2x M.2 and 4x SATA, and that would likely be the replacement build's form factor.
- Why Ryzen ITX? Because my current 3600 is likely to be replaced with a 3950X in the next few months, so....will have a CPU floating around. Not tied to this idea, if there's a better solution though, for the replacement NAS CPU/Mobo, but it IS the easy button currently.
If the array is built with 5 drives at first, can the array be shrunk (assuming it's not full of data) to 4 drives should the underlying system fail and the new system doesn't have enough SATA? (see above question)
Does Unraid/Freenas play well with PCIe SATA cards (to get more ports) when the array is split between the mobo-SATA and PCIe-SATA?

Failure tolerance questions.

Let's say I put 4-5 drives in the rig for the storage array, two cache drives, and am running Unraid, when the old Xeon and Mobo decide to let out the magic smoke.
Can I transfer the drives to a new platform (likely this Ryzen 5) without losing stored data?
Or will losing the CPU/Mobo mean the array is unreadable and needs to be completely started over from wiped/blank drives? This is by far my biggest concern.

Virtualization questions.

How does virtualization on Unraid work with respect to storage drives? Does each VM need its own drive that is separate from the cache/storage arrays?
Do the array and VM-drives, if separate, need to be on different controllers?

Expanding existing system feasibility

I'm not sure if this is doable, but do external enclosures exist for 5-7 SATA/M.2 drives that aren't a full blown NAS? My new case (Meshify C) doesn't have room to transfer the drives from this hypothetical build (HAF 932) in case of total failure. Being able to connect the drives to some sort of break out enclosure would be great, which could connect via USB-C/A/ PCIe / who knows.

Hardware age -- old stuff is old.

The core of my X58 system has stayed with me since buying in 2010. Mobo, Power supply, case, 2x 1TB drivers are since 2010. CPU/RAM/cooler/GPU/storage has changed over time.
I am moderately concerned that the age of my components will cause an eventual failure-- while there's no way to prevent this, I'm flagging it as a concern that may mean it's not wise to do this, depending on fault tolerance.

This system won't be up 24/7 due to hilariously bad power consumption from old tech. This'll be more a backup server and media when I want it device-- will that pose any issues w/r/t Unraid/ Unraid-VMs?

Thanks all! I tried to organize it, but....long post.

TL;DR: lots of questions in the build planning stages for repurposing an old computer for NAS/VM/Server duty.

AbydosOne · August 26, 2019

3 hours ago, bimmerman said:

R9 290 GPU

I'd drop the GPU down to something simple like a GT1030 (unless you really want to virtualize a gaming machine, which is a bad idea (and can't be done in FreeNAS yet).

3 hours ago, bimmerman said:

I have a bunch of Sata 2, and two Sata 3 ports, all sharing the chipset link bandwidth to the CPU.

Is this an issue for HDD-based Raid 5?

Basically no chance of affecting HDD performance on SATA II. Before I put an LBA in my server, the whole array was running off SATA II ports without issue.

3 hours ago, bimmerman said:

That leaves 16x remaining since the other slot sharing bandwidth with the GPU is blocked by it.

See my first suggestion.

3 hours ago, bimmerman said:

Would it be a benefit to add in a PCIe network card for teaming/link aggregation?

Or am I unlikely to saturate the 1GbE link with maybe 2-3 concurrent device requests?

In my (relatively non-expert) experience, link aggregation requires managed switches and seperate VLANs, and even then it may not work properly, depending on the server OS. For streaming media or sharing files, gigabit will probably suffice. If you're looking for more, you could do a 10G point-to-point link to another computer for relatively (~$50) cheap (which is what I did, pretty easily maxing out the speed of my HDD array somewhere north of 400MBps).

3 hours ago, bimmerman said:

maybe paired with a couple small SSDs for caching

[My answers are going to be based on ZFS and FreeNAS, but should hold fairly well across any NAS OS]

Mostly pointless for the average user. With 24GB of RAM, the ZFS ARC will be enough. L2ARC will only begin to help when you have multiple users accessing the same data a lot (e.g. reading from a database).

3 hours ago, bimmerman said:

If the array is built with 5 drives at first, can the array be shrunk (assuming it's not full of data) to 4 drives should the underlying system fail and the new system doesn't have enough SATA?

Generally, growing/shrinking is hard/dangerous, if not outright impossible (Unraid maybe being an exception). ZFS VDEVs can be added to a pool, but can't be removed. Drives can added to a VDEV, but can't be removed, as well. Best practice is to aim high and not have to worry about it later.

3 hours ago, bimmerman said:

Does Unraid/Freenas play well with PCIe SATA cards (to get more ports) when the array is split between the mobo-SATA and PCIe-SATA?

ZFS: actually, yes. Because each drive is enumerated by GUID, as long as the system can communicate with the drive, it can be put in the array, basically regardless of controlling device. Drive controllers can be changed even after the array is built. Pretty neat, eh?

3 hours ago, bimmerman said:

Can I transfer the drives to a new platform (likely this Ryzen 5) without losing stored data?

ZFS can be transferred between host systems, as long as one copy of all data is available. FreeNAS will import arrays that it finds with basically a few clicks.

3 hours ago, bimmerman said:

hilariously bad power consumption from old tech

When idling, most PCs use negligible power, honestly. At last calculation, I think my server uses ~$5 a month.

3 hours ago, bimmerman said:

This'll be more a backup server and media when I want it device-- will that pose any issues w/r/t Unraid/ Unraid-VMs?

Servers don't like being shut down and restarted. FreeNAS takes the better part of five minutes to boot for me, and then any VMs would have to start, too, so it's not exactly an "on demand" glorified external hard drive.

bimmerman · August 27, 2019

On 8/26/2019 at 2:56 PM, AbydosOne said:

I'd drop the GPU down to something simple like a GT1030 (unless you really want to virtualize a gaming machine, which is a bad idea (and can't be done in FreeNAS yet).

Basically no chance of affecting HDD performance on SATA II. Before I put an LBA in my server, the whole array was running off SATA II ports without issue.

See my first suggestion.

Good to know re: SATA II. On GPU, agreed, with the caveat that the 290 (and an HD 5870) are what I have lying around the house.

On 8/26/2019 at 2:56 PM, AbydosOne said:

In my (relatively non-expert) experience, link aggregation requires managed switches and seperate VLANs, and even then it may not work properly, depending on the server OS. For streaming media or sharing files, gigabit will probably suffice. If you're looking for more, you could do a 10G point-to-point link to another computer for relatively (~$50) cheap (which is what I did, pretty easily maxing out the speed of my HDD array somewhere north of 400MBps).

Hm, 10GbE direct would be sufficient. Gigabit for the majority of (wifi) devices, 10 gig direct to the computer that would benefit. I like this idea.

On 8/26/2019 at 2:56 PM, AbydosOne said:

[My answers are going to be based on ZFS and FreeNAS, but should hold fairly well across any NAS OS]

Mostly pointless for the average user. With 24GB of RAM, the ZFS ARC will be enough. L2ARC will only begin to help when you have multiple users accessing the same data a lot (e.g. reading from a database).

My ignorance is showing, but what's ARC?

I did some reading last night on caching and I agree--I'm not sure I would benefit. My understanding is the array, be it Storage Spaces or ZFS or Unraid would dynamically move frequent files to faster drives while slower drives will have the 'cold' data. For my use....I think you're right and there's little benefit to be had. Frequently used stuff will already be on the computer, the NAS is more for backups and media which won't be accessed predictably.

On 8/26/2019 at 2:56 PM, AbydosOne said:

Generally, growing/shrinking is hard/dangerous, if not outright impossible (Unraid maybe being an exception). ZFS VDEVs can be added to a pool, but can't be removed. Drives can added to a VDEV, but can't be removed, as well. Best practice is to aim high and not have to worry about it later.

Roger that.

On 8/26/2019 at 2:56 PM, AbydosOne said:

ZFS: actually, yes. Because each drive is enumerated by GUID, as long as the system can communicate with the drive, it can be put in the array, basically regardless of controlling device. Drive controllers can be changed even after the array is built. Pretty neat, eh?

ZFS can be transferred between host systems, as long as one copy of all data is available. FreeNAS will import arrays that it finds with basically a few clicks.

That's awesome and good to know, this may be the winner.

Thanks so much for your help! This is super informative.

Salacious B Crum · August 29, 2019

unRAID drives can be pulled from the server and read on another computer if a drive or hardware fails all other data is still obtainable with the proper procedures followed actually unRAID likes sata II for at least the boot drive and is recommended over sata III you will not be off to the races with unRAID but properly configured you should saturate HDD's and there are options for improving transfer speeds. I have a peer to peer 10Gbe network between my unRAID box and my PC no problems there, your CPU has a passmark score of 8132 so theoretically you should get 4 1080p streams in plex maybe more downclocking your CPU makes sense as your not going to see any appreciable gain with an overclock on a NAS I do recommend a UPS for any NAS as a power failure at the right moment will inevitably cause corruption when you least expect it

bimmerman · September 5, 2019

Thanks everyone! I've been doing a LOT of reading on Unraid vs FreeNas vs Storage Spaces, and am only getting more confused. This list might help someone in the future, who knows. I'd really appreciate feedback from those who know more than I do! I've summarized what I do and don't want from this build, as well as summarized my reading and am not sure how best to proceed. Please advise! I am not doing IO intensive stuff. Mostly backup and archival, sometimes media streaming and file accessing.

What I'm after is:

A (easy!) backup server solution first and foremost, augmented with external drive backups. Must be able to do the external backup thing.
A media center streamer eg Plex, mostly for music and photos with some video, because currently all this is fragmented and disorganized between devices.
A NAS for my various devices for documents and files, same reason as the above. Bonus if ipad/android can access files outside of Plex-type app.
Bonus points if it is a functional Windows machine as well, because I can do more with the box, plus I know I can shut a windows machine down when not in use without issue.

What I'm not after is:

a command-line hellhole of weekends lost to trying to get device A to talk to device B, translating terminal output into actual words to google
any variant of the above. Simpler is better; I CAN do that if I need (e.g., Powershell), I don't WANT to HAVE to do that to use or modify or diagnose simple things on the machine. I know I can't get away from it entirely, but I want to minimize command line and in-the-guts settings as much as possible.
a Personal Cloud. It'd be cool, but it's not needed and I don't really want to deal with vulnerabilities from allowing outside file transfers. I can dropbox that stuff.

From my reading it seems like (my opinions, open to changing):

Unraid is fairly user friendly, great at being a Plex/media center and NAS and can also do VMs, but isn't necessarily a good option for backups due to its data array parity scheme;
FreeNas is much more finicky but is the best NAS and Backup solution that can do Plex also, but without really the ability to virtualize a Windows machine (eg for ingesting optical media content locally) AND doesn't seem to play nice with external backup drives;
a Synology-type NAS is like a FreeNas that's more user friendly, less upgradeable, and costs more, BUT is a one stop shop for data storage and mild server-ing without fuss;
Windows Storage Spaces seems (???) to tick all the boxes-- it has solid data protection with parity, it can function as a NAS by sharing the volume as a network drive, it can run Plex (??), and it is a native windows box so no virtualization needed-- I can still do work tasks, I can ingest media, I can sort/organize/etc....it's more usable IMO.
- Albeit, it seems to have major limitations w/r/t speeds, plus windows being windows doing dumb windows update things sometimes.

Am I wrong there? Am I overlooking or misunderstanding something? What else should I be looking at or what questions should I be asking? I am leaning towards doing a Storage Spaces after reading all this, but if it's really going to be atrociously slow OR not protect my data.....well then that's not the right way to go.

Hardware-wise, plan is the same for all that involve repurposing the computer: NAS/Backup 5+ disk array built on an HBA for sure, maybe 10GbE, maybe SSD write caching if it makes sense (only useful for Storage Spaces?). It is also on a UPS, though power in my area is solid.

Anyway, here's my more detailed list of pros/cons and some lingering questions. Spoiler tags really just to reduce post size. Everything down here is summarized from a 40,000 ft view up above.

Unraid
- Questions:
  - How is Unraid as a backup solution, realistically?
  - How does array speed compare with something like Storage Spaces?
  - How well does Unraid work with external hard drives and hot plugging?
- Spoiler
  - Pros:
    Can easily expand storage arrays
    
    Virtualization support
    
    Docker support (eg Plex)
    
    Community support
    
    Linux, if you know Linux
  - Cons:
    Linux, if you don't know Linux
    
    array read/writes aren't any better than single disks, since parity isn't striped.
    
    array redundancy isn't as strong as on other options here, not sure it's a good option for a backup server first and media server second.
  - Cost (est):
    $50-100 for a license

FreeNas
- Spoiler
  - Pros:
    Free!
    
    ZFS file system
    
    seems to be regarded highly if NAS functionality is the most important
    
    considered to be very data secure
  - Cons:
    internet suggests it is not that user friendly if Linux isn't a known quantity
    
    command line is a thing
    
    much more difficult to back up, e.g. with an external drive. Internet basically says 'use another ZFS box to back up' which is a no-go.
    
    Not really a windows virtualization option, so the machine basically would become a NAS and that's it.
  - Cost (est):
    $free!
- Questions:
  - Is backing up really a pain? Is there an easy solution that doesn't involve another ZFS system?
  - Can a Freenas system be set up relatively painlessly and quickly with good performance?
  - Can Windows be virtualized well enough to actually use? Or is Freenas better suited to sticking the tower in a closet and leaving it alone?

Windows Storage Spaces
- Spoiler
  - Pros:
    It's windows. Zero re-learning how to use and interact with the machine
    
    Can expand the storage space volume easily
    
    If computer or drives fail, can plug into any windows machine and they are readable / array is repairable
    
    Can use the machine as a backup windows machine or a workstation without any need for virtualization, because it's a windows box
  - Cons:
    Need to buy Win 10 Pro for Workstation for ReFS system
    
    Consensus on Read/Write speeds are that performance is low, relative to other options
    
    It's windows, so who knows if an update will brick it or if it's secure relative to other systems
  - Cost (est):
    Either $0, or ~$200 to upgrade Win10 Home license on old comp if ReFS matters
- Questions:
  - Is ReFS a must-have?
  - How are the read/write speeds in a parity setup, from those who have used them?
  - From my reading, mirror-accelerated parity, especially with SSDs, will greatly improve R/W speeds. Does this require ReFS?
  - How does parity, mirror-accelerated or otherwise, compare with ZFS RaidZ2 from a usability, data protection, and speed standpoint?

Synology NAS (or other brands)
- Spoiler
  - Pros:
    Set it up once, done.
    
    Can be used for media streaming depending on model
    
    Some docker/app support
    
    If drive fails or if box fails, can plug into Linux to recover data
    
    Doesn't repurpose old computer
  - Cons:
    Cost is highest
    
    Doesn't repurpose old computer
    
    No 10GbE without spending a ton depending on brand/model.
  - Cost (est):
    $650 for the 5 bay DS1019+, before drives. (Synology, others obv vary)

Option 5: Forget all this server stuff and just use an external backup drive / drive enclosure with existing PC.
- Spoiler
  - Pros:
    Can move between systems
    
    anything can read it
    
    automated software for backups makes it easy
    
    Cheapest and easiest option by far
    
    Can easily compliment most of the above solutions
  - Cons:
    Minimal/zero server functionality
    
    single point of failure if backup drive dies
    
    dual/multi-drive backup drive begins to cost a lot
  - Cost (est): ~$100-150 for something in the 4-10 TB range

Salacious B Crum · September 5, 2019

unRAID is first and foremost a NAS os I get a solid 113 -130's MB/s below is a full parity check run when the system is idle but it has a very strong VM component as well as Docker so it makes a perfect NAS server/Plex media server, drives of any size/brand/speed/formfactor can be removed and replaced with a simple procedure you can run it with 1 or 2 parity drives for protection against 1 or 2 drive failures, it has an incredible user support group on their forum it is extremely user friendly to setup and maintain many options for automation for backing up and the like, I have not used and external drive but I see no reason for it not to play nice with a usb drive I would not recommend a usb drive as part of the array quite honestly I don't see the benefit just shuck the drive and place it in the array I currently have my mac's setup to backup to my unRAID box on a routine schedule you have access to any shares via AFP, SMB or NFS