ZFS upgrades..

darkfeder111 · December 30, 2023

Hey everyone hope everyone had the best holidays without too many system issues!
I was asked by my boss to increase the capacity of one of our client servers, they are running truenas scale with a topology of 8x4TB ironwolfs in raid-z2 and a single ssd as a metadata special vdev.
They already bought 8x8TB ironwolfs and a couple of Crucial MX500s ssd.
First of all i would like replace that single ssd with a raid of ssd because i’ve read that it’s a single point of failure because it’s a special vdev another option is to remove the ssd entilrely because from what i understand it’s not being utilzed. This is the output of zpool iostat -v
special - - - - - -
sdh1 18.3G 446G 0 11 120 204K
What would be the best approach to solve all this issues?
To create a new pool with the new disks and two new ssds as a metadata vdev? and then transferinng the data between pools?
Is it worth it to have a metadata vdev? The client mostly stores big video files.
Another option i tought is to add the new disks as a vdev to the existing pool and then transfering the data between vdev? is it even possible?

Thank you very much!

Gorgon · December 30, 2023

First make sure there is at least two backups (3-2-1 - three copies of the data in 2 locations, one off-site) before you start.

Normally I'd recommend replace the existing 4TB disks one at a time with an 8TB disk, wait until the vdev is re-built then repeat until all are replaced then you should be able to expand the vdev from 24TB to 48TB. This may take longer than just creating a new RAIDZ2 and transferring the data but the system will still be usable during the 8 rebuilds.

With respect to the MetaData vDev. Is it useful? - you'd have to do some testing. At any rate having just one SSD as the MetaData vDev is a single point of failure. When the MetaData SSD fails all the data in the Pool will be gone.

If it's using a Consumer SSD (like the MX500s) rather than enterprise SSDs (Optane, Kyoxia ...) then that would be even more sketchy as the MetaData vDev will burn through their limited write lifetime pretty quickly.

See this thread and others at TrueNAS.

For video files increasing the recordsize to a larger value (1MB) as mentioned in the thread above might help both MetaData and pool performance.

So, if you want to use a MetaData mirror then I'd get some Enterprise SSDs and create a new vDev with the new drives with the MetaData vDev then transfer the files over (which will create the new MetaData on the fly) but you'd have to change all existing fileshares to point to the new vDev.

Now would also be a good time to asses if the existing memory is adequate for the new configuration and add more ECC Memory if required.

LIGISTX · December 31, 2023

On 12/30/2023 at 8:17 AM, darkfeder111 said:

single ssd as a metadata special vdev.

This is extremely bad…

If that 1 SSD dies, all data is lost. Special metadata vdevs should have the same parity as the main array… you need this fix this immediately.

On 12/30/2023 at 8:17 AM, darkfeder111 said:

remove the ssd entilrely because from what i understand it’s not being utilzed.

I don’t believe you can do this. There may be a way I am not familiar with, but once you enable metadata to be on its own vdev, you can’t undo that, to be fair, I’m not even sure if you can alter the vdev once created, as in I’m not sure you can make the special metadata device a mirror. May need to use the new 8TB drives as a new vdev, copy all data over solve it that way.

darkfeder111 · January 2

On 12/31/2023 at 7:40 PM, LIGISTX said:

This is extremely bad…

If that 1 SSD dies, all data is lost. Special metadata vdevs should have the same parity as the main array… you need this fix this immediately.

I know! I was speechless when i found out! it'll be for sure the first thing to address!

I did some research and all i have to do is buy another ssd of similar size and the zfs attach, zfs will then create a mirror and copy all the data to the new drive! I'm debating even adding two ssd so that is a three way mirror.

Thank you everyone for your contribution i'll get on with this project and let you know!