Proxmox install using zfs with hot spare?

[ianm_ozzy]

Hi.

 

I  have a proxmox 7 server that the boot & virtual machine drive failed.  It was quite an inconvenience.

No data was lost & the server is not related to my livelihood.

 

I used another SSD and re-instaled proxmox &  the virtual machines from a cifs backup on my main machine (linux/windows dual boot).

What I want to do is install the proxmox OS and virtual machines on a zfs setup, which is possible as I understand.

So it is a hone machine for entertainment and are not looking to spend  a lot of cash.

Presently a 256GB for boot & virtual machines.  A 1/2 TB  SSD used as a lancache store (possibly n it way out).

 

So the Team GX2 drives are very cheap. I am aware they are not the best ssds out there & are not expecting miracles from them.

So my idea:

 Buy 3 of the 512GB SSDs.

Install in raid z1 with 2 of them and the SSD I have already - as the proxmox boot drive(s). The one I have already will probably fail first, but want to get my moneys worth from it!!

This I understand is possible.

What I want is the third to be a hot spare.  I am unsure how to setup this at install. There seems to be little option for this at proxmox setup.

So about 1TB of space will be available for virtual machines. The lancache will be the biggest hog.

 

When I drive dies on there, buy another and install at my convenience - well that is the idea.

 

Help is appreciated.

 

Thank you

 

 

 

[brwainer]

So ZFS boot is a little bit of a lie…. Yes, your OS root will live on ZFS and be mirrored, but your motherboard doesn’t understand ZFS. What do we do?

 

The answer is to install the bootloader and kernel to a separate partition that uses a conventional filesystem. This partition is set up the same way on every drive, and when an update happens (new kernel install) that gets done on every drive as well.

 

Therefore, it is possible at the ZFS layer to assign that third drive as a hot spare, but if it ever gets used the boot partition won’t be automatically made and kept in sync. I would just keep the drive around, make sure email alerts from the system are working (SMART errors and ZFS-ZED alerts are automatically sent to the email address provided during install) and when needed follow the Proxmox documentation for how to replace a boot drive in a ZFS mirror. 

[ianm_ozzy]

10 hours ago, brwainer said:

So ZFS boot is a little bit of a lie…. Yes, your OS root will live on ZFS and be mirrored, but your motherboard doesn’t understand ZFS. What do we do?

 

The answer is to install the bootloader and kernel to a separate partition that uses a conventional filesystem. This partition is set up the same way on every drive, and when an update happens (new kernel install) that gets done on every drive as well.

 

Therefore, it is possible at the ZFS layer to assign that third drive as a hot spare, but if it ever gets used the boot partition won’t be automatically made and kept in sync. I would just keep the drive around, make sure email alerts from the system are working (SMART errors and ZFS-ZED alerts are automatically sent to the email address provided during install) and when needed follow the Proxmox documentation for how to replace a boot drive in a ZFS mirror. 

 

I have no idea how to setup those seperate partitons partitions in zfs,. Is there an option in the GUI?

 

Also  the documentation of replacing a ZFS mirror is a problem as I want to use  RAID Z1, unless it is exactly the same process.

 

[Electronics Wizardy]

13 hours ago, ianm_ozzy said:

Install in raid z1 with 2 of them and the SSD I have already - as the proxmox boot drive(s). The one I have already will probably fail first, but want to get my moneys worth from it!!

 

Riad z1 with 2 drives makes no sense. With 2 drives you might as well use a mirror.

 

13 hours ago, ianm_ozzy said:

What I want is the third to be a hot spare.  I am unsure how to setup this at install. There seems to be little option for this at proxmox setup.

 

Why not just make a triple mirror then? Then any 2 drives can fail and you won't lose any data.

 

13 hours ago, ianm_ozzy said:

What I want is the third to be a hot spare.  I am unsure how to setup this at install. There seems to be little option for this at proxmox setup.

 

You can setup the mirror and hot spares after install, I'd do that here.

 

24 minutes ago, ianm_ozzy said:

 

I have no idea how to setup those seperate partitons partitions in zfs,. Is there an option in the GUI?

 

Also  the documentation of replacing a ZFS mirror is a problem as I want to use  RAID Z1, unless it is exactly the same process.

 

Look at the ZFS wiki page under changing a failed device for how to setup a mirror or replace deat zfs devices in proxmox https://pve.proxmox.com/wiki/ZFS_on_Linux

 

 

[brwainer]

31 minutes ago, ianm_ozzy said:

 

I have no idea how to setup those seperate partitons partitions in zfs,. Is there an option in the GUI?

 

Also  the documentation of replacing a ZFS mirror is a problem as I want to use  RAID Z1, unless it is exactly the same process.

 

Setting up the partitions is handled by the installer when you select a ZFS boot type. For replacing a failed drive, the documentation steps direct you to use sgdisk to copy the partition table from a remaining good drive. Yes, the steps should be the same with a RAIDZ1.

 

https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#_zfs_administration -> Changing a failed bootable device

Or https://pve.proxmox.com/wiki/ZFS_on_Linux#_zfs_administration - looks to have the same content

[brwainer]

4 minutes ago, Electronics Wizardy said:

Riad z1 with 2 drives makes no sense. With 2 drives you might as well use a mirror.

 

Why not just make a triple mirror then? Then any 2 drives can fail and you won't lose any data.

 

You can setup the mirror and hot spares after install, I'd do that here.

 

Look at the ZFS wiki page under changing a failed device for how to setup a mirror or replace deat zfs devices in proxmox https://pve.proxmox.com/wiki/ZFS_on_Linux

 

 

They have 1 512GB SSD already and are thinking about buying 3 more.

[Electronics Wizardy]

1 minute ago, brwainer said:

They have 1 512GB SSD already and are thinking about buying 3 more.

Wow I read that wrong. 

 

IN that case, I'd just do raidz2 instead of a hot spare then.

[ianm_ozzy]

4 hours ago, Electronics Wizardy said:

Wow I read that wrong. 

 

IN that case, I'd just do raidz2 instead of a hot spare then.

 It is not what I wish to do.  One present drive has not much time left.

I want raidz1  (3 drives) and when one fails, one automatically takes over.

I do not wish to spend time doing very low level stuff in the command line.

 

At the proxmox install, there seems no option for a 'hot spare'.

 

How is this done?

 

 

[ianm_ozzy]

4 hours ago, Electronics Wizardy said:

Riad z1 with 2 drives makes no sense. With 2 drives you might as well use a mirror.

 

Why not just make a triple mirror then? Then any 2 drives can fail and you won't lose any data.

 

You can setup the mirror and hot spares after install, I'd do that here.

 

Look at the ZFS wiki page under changing a failed device for how to setup a mirror or replace deat zfs devices in proxmox https://pve.proxmox.com/wiki/ZFS_on_Linux

 

 

 You misunderstand.

 I have 1 1/2 TB drive already - I expect it will not last much much longer.

 

Buy three more  new cheap ones.

Two of the new ones + the old one in a raidz1 - as boot and for virtual machines.

The other new one as hot spare.

I want to set this up and  if a drive fails, is replaced automatically.

How exactly is this done?

 

So 4  1/2 TB drives in total

.

 

 

 

 

[Electronics Wizardy]

1 hour ago, ianm_ozzy said:

 It is not what I wish to do.  One present drive has not much time left.

I want raidz1  (3 drives) and when one fails, one automatically takes over.

I do not wish to spend time doing very low level stuff in the command line.

 

At the proxmox install, there seems no option for a 'hot spare'.

 

How is this done?

 

 

You gotta set this up on the command line later on. Set it up the same way would replace a failed drive in those links, but set up the drive as a hot spare instead of a data drive.

 

But might as well go raidz2 then. You can just let it fail and leeave the spot open.

[brwainer]

7 hours ago, ianm_ozzy said:

 It is not what I wish to do.  One present drive has not much time left.

I want raidz1  (3 drives) and when one fails, one automatically takes over.

I do not wish to spend time doing very low level stuff in the command line.

 

At the proxmox install, there seems no option for a 'hot spare'.

 

How is this done?

 

 

You can have the drive be a hot spare at the ZFS level, but not at the bootloader level - there will always be command line work in setting up the replacement drive to be bootable.

[ianm_ozzy]

Well went to get the SSDs, & could get two at short notice.

I am not looking to wait for deliveries or hunting round for more.

So RAIDZ1  with the two I bought and one I have already.

 

So if a drive fails make backups of the virtual machines  in a hurry, then  go buy 1 other & replace at my convenience.

 

it is annoying but are not looking to spend much time on this.

 

[brwainer]

7 hours ago, ianm_ozzy said:

Well went to get the SSDs, & could get two at short notice.

I am not looking to wait for deliveries or hunting round for more.

So RAIDZ1  with the two I bought and one I have already.

 

So if a drive fails make backups of the virtual machines  in a hurry, then  go buy 1 other & replace at my convenience.

 

it is annoying but are not looking to spend much time on this.

 

If you care about the data at all, you should have regular backups at all times.

[ianm_ozzy]

18 hours ago, brwainer said:

If you care about the data at all, you should have regular backups at all times.

Agree.

So:

Least important data - games on my main server as an ISCSI share with regular snapshots.  Backup up/synced from time to time with an external drive.  Steam, origin & epic are the online  backups.

Media - audio  & video - on the main server.  2 external hard drive backups done weekly.  So no media there that cannot be put on there again.  It will be a little hassle.

Important data: 1- 2TB on server. Backup up typically daily to another machine - an old one dual boot linux (media play) & truenas scale for the backups. Also to 4 external drives synced weekly.

 

Presently a RAIDZ1 for most storage on the server (a truenas scale VM). I think buying anther drive for raidZ2 could be smart, or maybe a hot spare.

 

I have two machines running proxmox, always on.

I think I will install a proxmox backup server on each, to back each other up - the virtual machines that is.

 

Any other tips that are not costly (ideally free)  & not involving more machines on permanently are appreciated!