UnRaid - All in 1 Storage solution - Raid 10 vs zfs

[Fantos]

Sup guys

I have some problems with zfs on unraid.

 

Plan A
I wanted an all in 1 Solution with zfs on unraid 3x4tb raidz1. Plex, Steamlink/Moonlight, Vms, Docker, Backups and few game server + Ts3 in one pool.

But that’s not how things work on unraid right?

Plan B

Instead of the raid Controller (9211 8i)

i could buy another 4tb Wd red and make it a raid 10. So I won’t blow my budget and have good performance.

Would That solve my problem with zfs.

 

I need decent read/write speeds for 4K streams up to 50-70 MB/ps.

To not bottleneck my other vm/docker etc, I would like some fast raid/zfs solution with backup.

Does anyone have any ideas on how I can do this.

 

-Im  completely new to server stuff, unraid and everything around it.

 

My Parts

-2x Intel Xeon e5 2670v1 

-Supermicro X9DRL-iF

-4x 8gb 12800e

-Lsi 9211 8i

-3x4tb wd red efrx

-Gtx 770

[leadeater]

6 minutes ago, Fantos said:

I wanted an all in 1 Solution with zfs on unraid 3x4tb raidz1. Plex, Steamlink/Moonlight, Vms, Docker, Backups and few game server + Ts3 in one pool.

But that’s not how things work on unraid right?

You can use ZFS with unRAID, question I have for you is why? Why not use unRAID's native data parity/distribution? Seems to me you've discounted the most reliable/supported option if you are going to use unRAID.

[jj9987]

How many streams?

 

If you want ZFS, you might want to consider FreeNAS instead. unRAID does have some support for ZFS, but it is not their primary target (not sure if it is community or officially supported). If you want RAID 10, unRAID is not the way to go at all. In addition to that, with RAID 10 you lose quite a bit more space for redundancy as compared to ZFS or unRAID's own parity.

[Archer42]

IMO you expect far too much from what you have. I mean 3 hdd-s in parity "raid" will not have great performance, no matter which software you use. If you expect to be able to read, or especially write 50-70MB/s and be able to run stuff like vm-s or game servers you'll need SSD-s, and not the cheapest ones either. Such small hdd array will not be able to handle it, just like single hdd or even worse. And getting one more hdd / switching to raid 10 will not solve the issues either.

[Nick7]

First and foremost - why you want to use ZFS over unRaid's native RAID solution?

As far as performance goes, ZFS is far from being best performant FS. It's robust, it's reliable, but it's not top-performant.

With only 3HDD's, no matter what FS, your VM's performance will not be that good really. Yes, it'll work.. but nothing to write home to.

Depending how good you are in Linux:

1) Easiest and simplest is unRaid, with it's native RAID. But, with just 3HDD's, you will not be too happy with performance.

2) My option would be custom Linux, ZFS or other FS, but adding an SSD to speed up things (caching). Docker's as native under Linux, etc... all this would require some knowledge of Linux.

[Archer42]

8 minutes ago, Nick7 said:

As far as performance goes, ZFS is far from being best performant FS. It's robust, it's reliable, but it's not top-performant.

ZFS can be actually pretty good in terms of performance when configured appropriately.

Obviously not with raidzN, because parity arrays are inherently slow and are essentially a way to create efficient bulk storage.

Adding a bunch of simple mirror pairs as separate vdevs into a pool would be one example of relatively fast configuration, but it will definitely need to be more than 1-2 vdevs to be noticeably effective.

[Fantos]

6 minutes ago, leadeater said:

You can use ZFS with unRAID, question I have for you is why? Why not use unRAID's native data parity/distribution?

I read its safe, fast and you would get good capacity. Out of your drives.

 

 

5 minutes ago, jj9987 said:

How many streams?

Probably just 1 maybe 2.

 

36 minutes ago, jj9987 said:

you might want to consider FreeNAS instead.

I did too but unraid‘s AppStore, easier gpu passthrough and more simple gui keeps me on unraid

 

[Nick7]

9 minutes ago, Archer42 said:

ZFS can be actually pretty good in terms of performance when configured appropriately.

Obviously not with raidzN, because parity arrays are inherently slow and are essentially a way to create efficient bulk storage.

Adding a bunch of simple mirror pairs as separate vdevs into a pool would be one example of relatively fast configuration, but it will definitely need to be more than 1-2 vdevs to be noticeably effective.

I did not say ZFS is *slow*, but it's slower compared to other non-COW filesystems.

If you have low count of drives - like 3-4 drives, and you want max performance, ZFS is poor choice.

ZFS definitely has it's reasons and place to use (well, I use it in my NAS with 3 drives in RAIDZ1), but performance is not it's best suite.

ZFS does get 'better' with lots of RAM, when lots of stuff you need is in ARC, but similar can be said for other FS's when stuff you use is in RAM cache.

Don't get me wrong, I love ZFS, and really prefer it myself. But if performance is 1st, and reliability/features 2nd, ZFS is not way to go.

[Aragorn-]

I dont like FreeNAS. Seems to have far too many "oh you cant do that" type inconveniences, and i find the interface awful.

 

ZFS on Linux on the other hand, does exactly what i want, so thats what i run on my home server.

 

Never used unRAID, i guess though, theres sorta two questions intertwined, one, feature-set, and two usability of the GUI/Interface. If you happen to really like unRAID's GUI but want to use ZFS, then it probably makes sense?

 

As for the performance stuff, come on guys, its a single user running a few VM's, and a home NAS. Performance is just not an issue.

 

Its not like your going to see orders of magnitude differences switching from ZFS to anything else. Running VM's from spinning disks is never going to be performant.

 

Personally i have a ZFS mirror on SSD which stores my VM's, and then the main datastore is ZFS on spinning disks (in my case a pair of mirrors, as i've got 2x6tb and 2x8tb disks)

 

[leadeater]

50 minutes ago, Fantos said:

I read its safe, fast and you would get good capacity. Out of your drives.

Well so is unRAID's native capability so I don't actually understand why you want to use ZFS instead. While you can it's not a good idea unless you actually are an experienced system administrator and configurator because it's well outside the standard support use case for unRAID. That doesn't make it bad or anything but it does make it totally on you to do it correctly where as if you just use the unRAID GUI and assigned drives to it's pool configuration and select a parity drive you're good to go in minutes and also ben protected from drive failure.

 

I'm just not seeing a reason to use ZFS instead of unRAID here? This would be a different conversation if you had 24 disks but you only have 3 and for such a small deployment there is little to gain with ZFS, in fact potentially worse in a few ways.

[Aragorn-]

I know nothing of unRAID's capabilities nor the OP's exact reasoning... For me some of ZFS's most useful features include block level compression and its built in "logical volume manager" where you can seamlessly create different file systems and zvols within a pool. Its replication and snapshotting functionality is also great. I create regular snapshots, and sync them over the internet with a remote backup machine. I dont need lots of disks to find these features useful!

 

I'm always a bit wary of proprietary solutions. I know i can spin up any Linux or BSD instance and it will read my ZFS disks. Same reason i always used Linux mdraid over proprietary motherboard/controller based RAID systems in the past.

[Fantos]

Ok thanks for all the answers and ideas. 
I would like to change the question.

 

What would the best and most efficient storage solution for my usecase.

 

• 1 Plex 4K stream up to 70 MB/ps  
• Some Game and Ts3 Server 
• Couple vms including Steamlink/Moonlight 
• Dockers for home automation, WireGuard, Pihole, etc
• Mac backup 

What would be the best storage setup including hdd and raid type /redundancy on unraid.

I already own 3x 4tb drives 

So my idea would be to buy another 1 4tb hdd and combine them to an 8tb array with another 8 tb for redundancy.

And buy 1 500gb ssd for vms, game server, docker, Steamlink and back it up to an old drive like every night to reduce utilization.

 Would that be even possible?

 

You guys are just the best! Props to the community!

[Electronics Wizardy]

3 hours ago, Fantos said:

What would be the best storage setup including hdd and raid type /redundancy on unraid.

juse use the incldued parity. It will be plenty for your needs, and is what works best and what unraid is designed to use. Unraid really isn't made to be used with zfs or hardware raid. Id go proxmox if you want to use zfs or hardware raid here.

 

3 hours ago, Fantos said:

And buy 1 500gb ssd for vms, game server, docker, Steamlink and back it up to an old drive like every night to reduce utilization.

Unraid will use this as a cache drive and works well. Run your vms off the ssd. 

[kirkyg]

On 2/15/2021 at 4:31 PM, Fantos said:

Ok thanks for all the answers and ideas. 
I would like to change the question.

 

What would the best and most efficient storage solution for my usecase.

 

• 1 Plex 4K stream up to 70 MB/ps  

You keep saying you need 70 MB or Megabytes / second to stream 4k.  That is simply not true.  A remux 4k HDR movie might be in the 50-70 megaBITS / second bitrate.  you need to multiply your MegaBYTES by .125 as the conversion is 1 bit to .125 bytes.  So that means you need a read speed of 6.25 to 8.75MB/s in terms of storage capability.

 

Also note, if you are using media thats already encoded for you like WebDL's much of those are encoded at a lower bitrate so it would require even less.

[Jarsky]

On 2/15/2021 at 10:45 PM, Fantos said:

 

Plan A
I wanted an all in 1 Solution with zfs on unraid 3x4tb raidz1. Plex, Steamlink/Moonlight, Vms, Docker, Backups and few game server + Ts3 in one pool.

But that’s not how things work on unraid right?

 

 

This is exactly what I used ZFS on UnRAID for (NVMe's in a RAIDZ Mirror) originally before they brought in multi cache pool support. 

Then I would mount the paths for the VM's and Dockers. I used the primary UnRAID pool for mass storage with my large 8TB disks

 

Theyre working on introducing ZFS support into the next major UnRAID I believe. 

[voyager_]

If the performance with dockers and vms are the issue you can have them use the cache drive only in unraid. I have two unraid servers running. One handles Plex, Home Assistant, VPN, and a couple of minecraft servers all are running off of the cache drive (other then plex transcoding, that is set to Ram). The second server has a windows VM running off the cache drive and has no issues with read/write performance. Then I let unraid do what it does and handle the spinning disks. I have no issues with read and write speeds with 4k files managing it this way. 

[kirkyg]

I'm curious if you ran unraid in a VM under proxmox what the read/write capabilities would be for an array with one parity drive but not in RAID or ZFS.  Passed through HW vs not.  Anyone have experience with that configuration?

[jde3]

On 2/15/2021 at 1:45 AM, Fantos said:

Sup guys

I have some problems with zfs on unraid.

 

Plan A
I wanted an all in 1 Solution with zfs on unraid 3x4tb raidz1. Plex, Steamlink/Moonlight, Vms, Docker, Backups and few game server + Ts3 in one pool.

But that’s not how things work on unraid right?

Plan B

Instead of the raid Controller (9211 8i)

i could buy another 4tb Wd red and make it a raid 10. So I won’t blow my budget and have good performance.

Would That solve my problem with zfs.

 

You can do both plans with ZFS. You don't need separate pools unless you want to have that, the concept design for ZFS is "pooled storage" so you can add everything in one unless you don't want that for some reason. Separate pools are not a backup solution because a software error or malicious program can delete both pools available to the system. To be a proper backup you need to have the data somewhere else. (separate system or cloud or something like that)

ZFS can run in Raidz-1 (like a raid1) or with mirrored vdevs in a stripe (like a raid 10). ZFS has better caching algorithms than native Linux does. If performance is your primary concern you want Plan B. The reason for this is to allow the filesystem (both filesystem types) to be able to access more than one device at a time in read operations. You have to sacrifice capacity for performance.. and that is true with any filesystem.

The choice on Unraid or other platform is up to you. I don't like paying for free software and having artificial limits on how I can use it but you know.. you can if you find it easier. The below is simply Ubuntu Server.

Example mirrored vdevs (like raid 10) in zfs with a large/fast cache device:
 

  pool: tank
 state: ONLINE
config:

	NAME                                    STATE     READ WRITE CKSUM
	tank                                    ONLINE       0     0     0
	  mirror-0                              ONLINE       0     0     0
	    sda                                 ONLINE       0     0     0
	    sde                                 ONLINE       0     0     0
	  mirror-1                              ONLINE       0     0     0
	    sdb                                 ONLINE       0     0     0
	    sdf                                 ONLINE       0     0     0
	  mirror-2                              ONLINE       0     0     0
	    sdc                                 ONLINE       0     0     0
	    sdg                                 ONLINE       0     0     0
	  mirror-3                              ONLINE       0     0     0
	    sdd                                 ONLINE       0     0     0
	    sdh                                 ONLINE       0     0     0
	  mirror-4                              ONLINE       0     0     0
	    sdi                                 ONLINE       0     0     0
	    sdj                                 ONLINE       0     0     0
	logs	
	  nvme0n1p1                             ONLINE       0     0     0
	cache
	  nvme0n1p2                             ONLINE       0     0     0

Read:
dd if=./testfile of=/dev/null bs=1M
1693+1 records in
1693+1 records out
1775712629 bytes (1.8 GB, 1.7 GiB) copied, 2.29246 s, 775 MB/s

Subsequent read (from cache):
dd if=./testfile of=/dev/null bs=1M
1693+1 records in
1693+1 records out
1775712629 bytes (1.8 GB, 1.7 GiB) copied, 0.314652 s, 5.6 GB/s

^ Solid performance from ZFS on spinning rust.

[Nick7]

If solely performance is your concern, stay away from ZFS.

ZFS is good, solid FS, but performance ain't it's niche.

I do use ZFS primarily due to integrity (like checksum checking of all data) and reliable RAIDZ1.

However, if you need raw performance, mdadm+XFS, or even adding SSD/NVMe cache using bcache will give quite better performance than ZFS - especially after some time in use.