How does expansion with ZFS work?

[Zmimgo]

Hello there!

 

Continuing on my last thread (where I think I talked about different CPU options for a future NAS), I have an idea of how the system itself is going to be like. Next thing on the bucket list is the storage configuration.

 

For the storage configuration, I want two parity drives and one hot-swap drive - which won't just sit there and do nothing, will use it as a downloads folder, which I can just erase when I need to swap out a drive which has failed in the RAID. The case configuration I'm going for is a normal ATX case, using those 120mm fan to 3.5 inch drive adapters, giving me a total of nine drives of the same kind. But here is the "special" part:

 

I want to start off with 3 drives (2 in mirror, and an extra for hot-swap). The hot-swap drive is going to be used for downloads, and is going to be erased when a drive in the RAID 1 fails. If nothing breaks, the next upgrade is going to be adding two drives, erasing the download drive, create a RAID-Z2 VDEV with the three now empty drives and one of the drives used in the mirror. After creating the RAID-Z2 VDEV, I will copy the files from the drive not used in the RAID-Z2 VDEV, erase the drive, and use it as the new hot-swap/download folder. (bonus question: can you create a RAID-Z2 VDEV without the data on one of the drives used for the RAID-Z2 VDEV being erased?) After that, I will be gradually adding more drives to the RAID-Z2 VDEV, simply for more storage.

 

And since I will be gradually adding more drives to the system, I'll want an easy, secure and fast way to expand the VDEV without the usage of external drives. I have heard of ZFS supporting expansion, but after an hour of Google Searches, I have given up hope on finding information. Instead, I am going to ask the question here.

 

How does expansion with ZFS work? How long time does it take? What are the ways of doing it, and how risky are they for the data? If it isn't released yet, when will it be? (Earliest I see myself needing this would be around Q3 2024)

 

And another question, what OS do I want to use? I'll be using it mainly as a NAS, but also as a "do everything"-server, where I will have VMs running for users because faster boot times and only needing one motherboard, chassis, PSU etc. I want great support for software I can use to access the NAS outside of my home network (also, what software do you recommend for that? perferably open source), and also great support for virtualization, including GPU passthrough (asking this because have heard TrueNAS VM isn't the greatest). I also want an usable GUI and support for ZFS, and the lighter the host OS is, the better. 

 

I know I may be asking for a lot here, but yeah, sorry I guess haha

 

Thanks for your time!

[RONOTHAN##]

14 minutes ago, Zmimgo said:

I have heard of ZFS supporting expansion,

I mean it does, but not in the way that you're describing it. 

 

15 minutes ago, Zmimgo said:

How does expansion with ZFS work? How long time does it take?

You expand in vDevs, adding them at a time. You want the vDevs to be as similar to each other as possible, so same drives, same capacity, same number of drives, and same layout. So if you have two drives mirrored, you need to add drives to the system in mirrored pairs, preferably with the same capacity. Adding vDevs to a zpool takes next to no time, a couple minutes at the longest, and is very safe to do. 

 

What you were thinking about was expanding a vDev, a feature ZFS doesn't currently support. It is rumored to be on the roadmap, but it's not currently there, and since ZFS is mainly developed for enterprise customers who wouldn't even consider doing that, I wouldn't expect it to be available for quite a while. This method does have inherent risks, since the drives are going to be reading and writing 24/7 for a few days, maybe even a few weeks depending on the drive capacity. Also, last I checked the rumors said that they aren't gonna allow for a vDev to change type, so if you have mirrored drives, you can't go to RAID-Z2 by adding in 2 more drives. 

 

26 minutes ago, Zmimgo said:

And another question, what OS do I want to use?

The three biggest ones are TrueNAS Core, UnRAID, and ProxMox. TrueNAS Core is free and has the best web interface, but VMs for it kinda suck and don't really support any sort of hardware passthrough. UnRAID costs money, but it has basically the best interface when dealing with VMs, though it doesn't natively use ZFS and using it requires setting it up through the command line. ProxMox is somewhere in the middle. It supports ZFS out of the box, and has nearly as good VM support as UnRAID does. NAS functionality is a bit more complicated, since you have to setup SAMBA yourself with the command line, or deal with a Turnkey Linux container which would require you to dedicate a certain amount of storage to NAS functionality at the start. This can be expanded in the future when you add more storage, but it's a bit of a pain. If you're fine with waiting a bit, TrueNAS Scale is looking like it will be the best one out of the box, since it will have great VM support (the same backend the UnRAID and ProxMox uses) with the rest of their awesome web GUI. It's currently in beta so I wouldn't recommend using it for your main system, but if you're not planning on getting this server up and running for ~6 months it will be a great option once it comes out of Beta. Also, if you don't need the VMs right away, you could use TrueNAS Core and then easily port all your settings over and migrate the OS once TrueNAS Scale comes out of Beta. 

 

37 minutes ago, Zmimgo said:

I want great support for software I can use to access the NAS outside of my home network

Nextcloud. It's basically Google Drive (including a lot of the extra bits like a Calendar) for free, and it works pretty easily on every one of the previously mentioned operating systems.

[Zmimgo]

17 minutes ago, RONOTHAN## said:

I mean it does, but not in the way that you're describing it. 

Found this online and that's what I am referencing. Meant OpenZFS, sorry if it confused you.

 

17 minutes ago, RONOTHAN## said:

You expand in vDevs, adding them at a time. You want the vDevs to be as similar to each other as possible, so same drives, same capacity, same number of drives, and same layout. So if you have two drives mirrored, you need to add drives to the system in mirrored pairs, preferably with the same capacity. Adding vDevs to a zpool takes next to no time, a couple minutes at the longest, and is very safe to do. 

 

What you were thinking about was expanding a vDev, a feature ZFS doesn't currently support. It is rumored to be on the roadmap, but it's not currently there, and since ZFS is mainly developed for enterprise customers who wouldn't even consider doing that, I wouldn't expect it to be available for quite a while. This method does have inherent risks, since the drives are going to be reading and writing 24/7 for a few days, maybe even a few weeks depending on the drive capacity. Also, last I checked the rumors said that they aren't gonna allow for a vDev to change type, so if you have mirrored drives, you can't go to RAID-Z2 by adding in 2 more drives. 

The main problem for VDEVs I have is that even though they allow for the same amount of drives that can fail, it must be on the right VDEVs. Also, once I reach, lets say three VDEVS, I'll lose one drive compared to if I just used RAID-Z2 on all instead of RAID-Z1 on each VDEV. I do understand that the amount of data that doesn't get lost if a VDEV fails is much greater, but still. May do that, but for now, probably won't. And if I did, I would probably just mirror em, since I will only be adding one or two drives at a time.

 

17 minutes ago, RONOTHAN## said:

The three biggest ones are TrueNAS Core, UnRAID, and ProxMox. TrueNAS Core is free and has the best web interface, but VMs for it kinda suck and don't really support any sort of hardware passthrough. UnRAID costs money, but it has basically the best interface when dealing with VMs, though it doesn't natively use ZFS and using it requires setting it up through the command line. ProxMox is somewhere in the middle. It supports ZFS out of the box, and has nearly as good VM support as UnRAID does. NAS functionality is a bit more complicated, since you have to setup SAMBA yourself with the command line, or deal with a Turnkey Linux container which would require you to dedicate a certain amount of storage to NAS functionality at the start. This can be expanded in the future when you add more storage, but it's a bit of a pain. If you're fine with waiting a bit, TrueNAS Scale is looking like it will be the best one out of the box, since it will have great VM support (the same backend the UnRAID and ProxMox uses) with the rest of their awesome web GUI. It's currently in beta so I wouldn't recommend using it for your main system, but if you're not planning on getting this server up and running for ~6 months it will be a great option once it comes out of Beta. Also, if you don't need the VMs right away, you could use TrueNAS Core and then easily port all your settings over and migrate the OS once TrueNAS Scale comes out of Beta. 

Have heard of all but honestly, it was so confusing haha. TrueNAS Scale looks promising, probably will go for that.

 

17 minutes ago, RONOTHAN## said:

Nextcloud. It's basically Google Drive (including a lot of the extra bits like a Calendar) for free, and it works pretty easily on every one of the previously mentioned operating systems.

Have used Nextcloud before, but it was such a bad experience. Probably an edge case with my super old laptop used as an server, but still a little bit sour. Will probably give it another chance.

 

Thanks for your help!

 

EDIT: Yes I do know waht RAID-Z3 is, I just used RAID-Z2 to compare against the multiple VDEVs because I'll probably end up using that.

[RONOTHAN##]

5 hours ago, Zmimgo said:

Found this online and that's what I am referencing. Meant OpenZFS, sorry if it confused you.

 

That's what I was referring to as well. That article is being a bit ambitious with the time frames IMO. I wouldn't hold your breath waiting for it.

 

5 hours ago, Zmimgo said:

The main problem for VDEVs I have is that even though they allow for the same amount of drives that can fail, it must be on the right VDEVs. Also, once I reach, lets say three VDEVS, I'll lose one drive compared to if I just used RAID-Z2 on all instead of RAID-Z1 on each VDEV. I do understand that the amount of data that doesn't get lost if a VDEV fails is much greater, but still. May do that, but for now, probably won't. And if I did, I would probably just mirror em, since I will only be adding one or two drives at a time.

I get that, but that's the way ZFS was designed. Adding drives one at a time has inherent risks with something going wrong, and your data will be inaccessible for a few days while the array rebuilds itself. In the business world where you rely on those servers, any more downtime than an hour or two is basically unacceptable, especially when it comes to data, and the vDev approach allows for quick and reliable expansion, just for the cost of more drives up front. It doesn't make as much sense for general consumers, but that's not really who ZFS is designed around. For consumers, not having access to your files for a few days usually isn't that big a deal, so single drive expansion does make more sense for them, but if that's the feature you want there are alternatives that have that. ZFS is designed for reliability with high numbers of drives, and in that scenario the vDev model makes a lot more sense. ZFS still works well with lower numbers of drives, but a lot of its advantages and optimizations don't translate quite as well. 

 

If you want easy drive expansion, use UnRAID and its default (IIRC) XFS to allow for single drive expansion like you were talking about. With lower numbers of drives (<=8) it's more than sufficient and will work better for what you want.

5 hours ago, Zmimgo said:

Have heard of all but honestly, it was so confusing haha. TrueNAS Scale looks promising, probably will go for that.

Just wait for it to get out of beta first. Running beta software on a server isn't really a great idea unless you really know what you're doing.

[Mihle]

Benefit of the standard drive system in Unraid, aka JBOD with parity is:

• Very easely expandable. 
• Supports all kinds of drive sizes, if you have a 2 TB, a 4 TB and two 6 TB drives, you can use them, you put one of the 6TB ones as parity and then you got 12 TB usable storage.
• If you have 5 disk where 1 is parity, if somehow 2 disks fail, data on the 3 other drives is still readable if it was parity drive and a data drive that failed. I'm RAID, all data would have been lost. (true with 2 parity and 3 disks failing, the data disks that didn't file is still fine) Maybe not the best argument for Unraid as that is an situation you shouldn't need to be in either way tho. 

Negatives:

• If you read files that main data drive is only one of them, you will only get the speed of one disk. 
• Unless you let it wait with parity to you are finished writing to it(I think it's called turbo write or something), or have cache drives, write speed is limited to the speed of the slowest of the the data drive you are writing to and the parity drives.

You hit the limit of 1 gbit Internet before you reach the max speed of one disk tho. 

 

Those are that I can remember right now, might have missed something.

 

As for Unraid itself, it's not too hard to set up, not sure compared to Truenas, but def easier than what freenas was quite a few years ago and and the other option is. 

 

It works well with VMs. 

[Zmimgo]

3 hours ago, RONOTHAN## said:

I get that, but that's the way ZFS was designed. Adding drives one at a time has inherent risks with something going wrong, and your data will be inaccessible for a few days while the array rebuilds itself. In the business world where you rely on those servers, any more downtime than an hour or two is basically unacceptable, especially when it comes to data, and the vDev approach allows for quick and reliable expansion, just for the cost of more drives up front. It doesn't make as much sense for general consumers, but that's not really who ZFS is designed around. For consumers, not having access to your files for a few days usually isn't that big a deal, so single drive expansion does make more sense for them, but if that's the feature you want there are alternatives that have that. ZFS is designed for reliability with high numbers of drives, and in that scenario the vDev model makes a lot more sense. ZFS still works well with lower numbers of drives, but a lot of its advantages and optimizations don't translate quite as well. 

Also understood that. Thought it would be a good idea to familiarize myself a little bit with ZFS, and upfront cost is something I would be able, but not want to deal with. Thanks for your explanation though, made so much more sense than the other ones.

 

25 minutes ago, Mihle said:

Negatives:

• If you read files that main data drive is only one of them, you will only get the speed of one disk. 
• Unless you let it wait with parity to you are finished writing to it(I think it's called turbo write or something), or have cache drives, write speed is limited to the speed of the slowest of the the data drive you are writing to and the parity drives.

A bit of downtime is ok for home use, and I don't really care about speed on the storage array (will have a cache drive).

 

28 minutes ago, Mihle said:

You hit the limit of 1 gbit Internet before you reach the max speed of one disk tho. 

Going 2.5G, maybe even SFP+ if I can find a cheap switch that has SFP+ and 2.5G ports that also is cooled passivly.

 

May considor going UnRaid, but 119$ sure does sound like a little bit much for me haha, although it is a lifetime license. 

[Mihle]

As for Nextcloud, I use it and I like it, even tho it could be better in some ways. 

 

It works great to sync with a PC, might be little slow when you start syncing it on a new PC, but so is Google Drive for example. Slower with many small files rather than few bug ones. 

 

When viewing photos on it in browser, it's little slow for me at showing the photo previews. I think it's because it creates new ones every time instead of using already generated ones, I have an plug in that is supposed to generate the previews when it runs , but I think they might not get used. This might be an issue with my system or the Docker I use, but I don't know. If your CPU is faster it would be faster than mine even if it happens. 

 

Uploading photos from phone to nextcloud automatically works great for me currently, but downloading from Nextcloud to your phone via the sync folder function require you to restart the phone for them to show in the standard gallery app. Also, can be little weird at syncing to the standard photo folder on your phone rather than to the folder location on your phone it chooses. (from Nextcloud to phone, upload from phone to nextcloud work fine) My experience is with android tho, can't comment on iOS. 

 

Other than that it has been nice. 

[Mihle]

8 minutes ago, Zmimgo said:

A bit of downtime is ok for home use, and I don't really care about speed on the storage array (will have a cache drive).

 

Going 2.5G, maybe even SFP+ if I can find a cheap switch that has SFP+ and 2.5G ports that also is cooled passivly.

 

May considor going UnRaid, but 119$ sure does sound like a little bit much for me haha, although it is a lifetime license. 

Worth noting that Unraid doesn't have read cache as TrueNAS with ZFS do. Only write cache. That is one of the other negatives. You can choose to let some things stay on the cache all the time tho if you want to, or stay on it as long as it's space on it and then move it to the array when it comes full.

But if automatic read cache is something you require then Unraid with standard JBOD with parity is not for you, don't know how it works if you install ZFS plug in on Unraid. 

 

Can't comment on 2,5 G it as I don't have that personally. 

[RONOTHAN##]

1 hour ago, Zmimgo said:

Going 2.5G, maybe even SFP+ if I can find a cheap switch that has SFP+ and 2.5G ports that also is cooled passivly.

 

If you're gonna go TrueNAS, avoid 2.5G networking. TrueNAS is based off FreeBSD, and 2.5G networking cards have next to no driver support with FreeBSD. Unless you wanna write your own drivers and deal with rebuilding the kernel, stick to the officially supported network cards on FreeBSD or go to something Linux based (Basically any NAS OS other than TrueNAS). That does also apply to SFP+ cards, though to a lesser degree.