Jump to content

Best boot and vm drive configuration for Proxmox

glitch452
 Share
Posted

Hi Everyone,

 

I’m putting together a new home-lab build and I want to try out Proxmox VE for the first time.

I’m looking for some advice on storage configurations for the boot drive and a drive to run the VMs from.

 

  • What is your preferred configuration?
  • Do you use NVMe or SATA SSDs for both, or strictly enterprise grade SSDs?
  • How do you handle redundancy & Backus?
  • Do you mirror either of those drives? If so, how? Is there a good software RAID (ZFS/Motherboard RAID) way to provide redundancy that also works for the boot drive?

 

Let me know your approach, and importantly, why you made the decisions you did.

 

Thank you!

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
38 minutes ago, glitch452 said:

Hi Everyone,

 

I’m putting together a new home-lab build and I want to try out Proxmox VE for the first time.

I’m looking for some advice on storage configurations for the boot drive and a drive to run the VMs from.

 

  • What is your preferred configuration?
  • Do you use NVMe or SATA SSDs for both, or strictly enterprise grade SSDs?
  • How do you handle redundancy & Backus?
  • Do you mirror either of those drives? If so, how? Is there a good software RAID (ZFS/Motherboard RAID) way to provide redundancy that also works for the boot drive?

 

Let me know your approach, and importantly, why you made the decisions you did.

 

Thank you!

Proxmox supports ZFS mirroring for the boot drive. I run a mirror of nvme drives, 500GB each for boot + VM’s. I just keep my VM’s on the boot drive, no real reason not to for a homelab. 

Rig: i7 13700k - - Asus Z790-P Wifi - - RTX 4080 - - 4x16GB 6000MHz - - Samsung 990 Pro 2TB NVMe Boot + Main Programs - - Assorted SATA SSD's for Photo Work - - Corsair RM850x - - Sound BlasterX EA-5 - - Corsair XC8 JTC Edition - - Corsair GPU Full Cover GPU Block - - XT45 X-Flow 420 + UT60 280 rads - - EK XRES RGB PWM - - Fractal Define S2 - - Acer Predator X34 -- Logitech G502 - - Logitech G710+ - - Logitech Z5500 - - LTT Deskpad

 

Headphones/amp/dac: Schiit Lyr 3 - - Fostex TR-X00 - - Sennheiser HD 6xx

 

Homelab/ Media Server: Proxmox VE host - - 512 NVMe Samsung 980 RAID Z1 for VM's/Proxmox boot - - Xeon e5 2660 V4- - Supermicro X10SRF-i - - 128 GB ECC 2133 - - 10x4 TB WD Red RAID Z2 - - Corsair 750D - - Corsair RM650i - - Dell H310 6Gbps SAS HBA - - Intel RES2SC240 SAS Expander - - TreuNAS + many other VM’s

 

iPhone 14 Pro - 2018 MacBook Air

Link to comment
Share on other sites
Link to post
Share on other sites
  • 2 months later...
Posted
On 1/5/2024 at 2:24 AM, LIGISTX said:

Proxmox supports ZFS mirroring for the boot drive. I run a mirror of nvme drives, 500GB each for boot + VM’s. I just keep my VM’s on the boot drive, no real reason not to for a homelab. 

Are you using consumer SSDs?  If so have you had any issues with SSD wear out?  I've read that some people having issues with this.  

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
1 hour ago, tekdmon said:

Are you using consumer SSDs?  If so have you had any issues with SSD wear out?  I've read that some people having issues with this.  

I am, and seemingly haven’t yet. I have been running a pair of 980’s (non pro) for over a year). They are wearing a bit fast, last I looked maybe a month ago they were reporting 20% life lost which I do admit seems a bit crazy… but if I can get 4-5 year out of them which that suggests I would, that seems acceptable to me. 

Rig: i7 13700k - - Asus Z790-P Wifi - - RTX 4080 - - 4x16GB 6000MHz - - Samsung 990 Pro 2TB NVMe Boot + Main Programs - - Assorted SATA SSD's for Photo Work - - Corsair RM850x - - Sound BlasterX EA-5 - - Corsair XC8 JTC Edition - - Corsair GPU Full Cover GPU Block - - XT45 X-Flow 420 + UT60 280 rads - - EK XRES RGB PWM - - Fractal Define S2 - - Acer Predator X34 -- Logitech G502 - - Logitech G710+ - - Logitech Z5500 - - LTT Deskpad

 

Headphones/amp/dac: Schiit Lyr 3 - - Fostex TR-X00 - - Sennheiser HD 6xx

 

Homelab/ Media Server: Proxmox VE host - - 512 NVMe Samsung 980 RAID Z1 for VM's/Proxmox boot - - Xeon e5 2660 V4- - Supermicro X10SRF-i - - 128 GB ECC 2133 - - 10x4 TB WD Red RAID Z2 - - Corsair 750D - - Corsair RM650i - - Dell H310 6Gbps SAS HBA - - Intel RES2SC240 SAS Expander - - TreuNAS + many other VM’s

 

iPhone 14 Pro - 2018 MacBook Air

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
2 hours ago, LIGISTX said:

I am, and seemingly haven’t yet. I have been running a pair of 980’s (non pro) for over a year). They are wearing a bit fast, last I looked maybe a month ago they were reporting 20% life lost which I do admit seems a bit crazy… but if I can get 4-5 year out of them which that suggests I would, that seems acceptable to me. 

Have you disabled any of the HA services?   I saw a post on Reddit recommending to disable these, stating it can reduce logging.  https://www.reddit.com/r/homelab/s/sbGseqkmUA

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
33 minutes ago, tekdmon said:

Have you disabled any of the HA services?   I saw a post on Reddit recommending to disable these, stating it can reduce logging.  https://www.reddit.com/r/homelab/s/sbGseqkmUA

I did do a bit of digging this morning after posting my previous post, and I did end up setting one of the logs to write to RAM instead of disc. I have a threat on proxmox forum and level1techs asking a few questions, but I may end up doing more of what’s suggested in that reddit thread as well as potentially log2RAM. 
 

Thanks for the link!

Rig: i7 13700k - - Asus Z790-P Wifi - - RTX 4080 - - 4x16GB 6000MHz - - Samsung 990 Pro 2TB NVMe Boot + Main Programs - - Assorted SATA SSD's for Photo Work - - Corsair RM850x - - Sound BlasterX EA-5 - - Corsair XC8 JTC Edition - - Corsair GPU Full Cover GPU Block - - XT45 X-Flow 420 + UT60 280 rads - - EK XRES RGB PWM - - Fractal Define S2 - - Acer Predator X34 -- Logitech G502 - - Logitech G710+ - - Logitech Z5500 - - LTT Deskpad

 

Headphones/amp/dac: Schiit Lyr 3 - - Fostex TR-X00 - - Sennheiser HD 6xx

 

Homelab/ Media Server: Proxmox VE host - - 512 NVMe Samsung 980 RAID Z1 for VM's/Proxmox boot - - Xeon e5 2660 V4- - Supermicro X10SRF-i - - 128 GB ECC 2133 - - 10x4 TB WD Red RAID Z2 - - Corsair 750D - - Corsair RM650i - - Dell H310 6Gbps SAS HBA - - Intel RES2SC240 SAS Expander - - TreuNAS + many other VM’s

 

iPhone 14 Pro - 2018 MacBook Air

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Consumer SSDs can be a problem.

Unless you wish to pay up, then its 4 bits/cell, no RAM & fairly sad TBW values.

 

Presently I have  3 x sata SSDs in RAIDZ1 for boot & VMs.  That is not ideal.

 

So on the cheap have ordered  pcie x1 to nvme adapters.

Also some 16GB optane. also inexpensive.

 

My soon to be setup:

 

2 x 16GB mirrored optane as boot, with spares just in case. Probably mirrored ZFS

 

3 x  512GB sata SSDs in RAIDZ1  for VMs.

If there are any failures,  I expect  it to be one of the sata SSDs, which are physically easier to replace.

 

 

 

 

 

Main Machine: CPU: 5800X3D  RAM: 32GB  GPU: RTX 3080  M/B: ASUS B550-E Storage: 2 x 256GB NVME boot, 1/2 TB NVME OS: Windows 10, Ubuntu 22.04

Server1:  M92p micro  CPU: i5-3470T  RAM: 8GB OS: Proxmox  Virtual Machines: Opnsense router, LXC containers: netboot server, download manager

Server2: CPU: 3600X  RAM: 64GB M/B MSI B450 Tomahawk  OS: Proxmox  Virtual machines: Windows 10, 3 x Ubuntu Linux, Truenas scale (16TB logical storage)

Link to comment
Share on other sites
Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing Servers, NAS, and Home Lab

×