Jump to content

A Beginners Guide to PROXMOX

4 hours ago, Bendy_BSD said:

So far it's awesome and it's exactly as you described it with the usb driver i mean. ^u^

are xeon processors capable of nested virtualization?

I'm not certain nested virtualization is a hardware limitation or something simply imposed by the host to stop you from setting something up with degraded performance.

 

So long as the CPU has VT-x support I believe nested virtualization is an OS level limitation. LCX Containers can be set with nesting via:

LXC Container -> container_name -> Options -> Features -> Tick "Nesting"

I don't know about the Virtual Machines. I'm not seeing the same option.

Link to comment
Share on other sites

Link to post
Share on other sites

1 hour ago, Windows7ge said:

I'm not certain nested virtualization is a hardware limitation or something simply imposed by the host to stop you from setting something up with degraded performance.

 

So long as the CPU has VT-x support I believe nested virtualization is an OS level limitation. LCX Containers can be set with nesting via:


LXC Container -> container_name -> Options -> Features -> Tick "Nesting"

I don't know about the Virtual Machines. I'm not seeing the same option.

gotcha.  
 

 

What kinda sucks atm for me is that I can't play R6S within the windows vm because of battle eye. 😢

Link to comment
Share on other sites

Link to post
Share on other sites

6 minutes ago, Bendy_BSD said:

gotcha.  
 

 

What kinda sucks atm for me is that I can't play R6S within the windows vm because of battle eye. 😢

The game may have an issue with virtual environments. In my VFIO QEMU/KVM tutorial I outlined how to fix NVIDIA error code 43 where-in you trick Windows into thinking it's not inside a VM. Maybe doing the same think will trick Battle Eye?

 

The only problem is I found this tweak for virt-manager. It adds a parameter to the VM's .XML file. I don't know how or where this may exist for PROXMOX but you can give it a shot.

<features>
	<hyperv>
		...
		<vendor_id state='on' value='linustech'/>
		...
	</hyperv>
	...
	<kvm>
	<hidden state='on'/>
	</kvm>
</features>

 

Link to comment
Share on other sites

Link to post
Share on other sites

  • 2 years later...

Hello and thank you for the excellent resource.

I'm a super noob and way out of my comfort zone... but I will persist. 

 

I need to pass through some of my NIC ports, a GPU, on board sata and a HBA

 

The HBA and the GPU are easy as I can block the driver using 9.4.1

But the others get messy as dont want to pass through everything using a particular driver.

 

Using the method in 9.4.3 I get stuck... seems like the module vfio-pc isn't where I need it, I have zero I idea what I'm doing.

root@pve:~# update-initramfs -u -k all
update-initramfs: Generating /boot/initrd.img-5.15.102-1-pve
/etc/initramfs-tools/scripts/init-top/vfio-driver-override.sh: 5: cannot create /sys/bus/pci/devices/0000:0a:00.0/driver_override: Directory nonexistent
/etc/initramfs-tools/scripts/init-top/vfio-driver-override.sh: 5: cannot create /sys/bus/pci/devices/0000:0a:00.1/driver_override: Directory nonexistent
modprobe: FATAL: Module vfio-pc not found in directory /lib/modules/5.15.102-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.

 

Super thanks for any help, suggestion, pointer... sorry I know its an old thread.

Link to comment
Share on other sites

Link to post
Share on other sites

3 hours ago, thesteamy said:

Hello and thank you for the excellent resource.

I'm a super noob and way out of my comfort zone... but I will persist. 

 

I need to pass through some of my NIC ports, a GPU, on board sata and a HBA

 

The HBA and the GPU are easy as I can block the driver using 9.4.1

But the others get messy as dont want to pass through everything using a particular driver.

 

Using the method in 9.4.3 I get stuck... seems like the module vfio-pc isn't where I need it, I have zero I idea what I'm doing.

root@pve:~# update-initramfs -u -k all
update-initramfs: Generating /boot/initrd.img-5.15.102-1-pve
/etc/initramfs-tools/scripts/init-top/vfio-driver-override.sh: 5: cannot create /sys/bus/pci/devices/0000:0a:00.0/driver_override: Directory nonexistent
/etc/initramfs-tools/scripts/init-top/vfio-driver-override.sh: 5: cannot create /sys/bus/pci/devices/0000:0a:00.1/driver_override: Directory nonexistent
modprobe: FATAL: Module vfio-pc not found in directory /lib/modules/5.15.102-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.

 

Super thanks for any help, suggestion, pointer... sorry I know its an old thread.

I do need to update this tutorial. Some things don't corrispond to the latest PROXMOX version anymore.

 

Possible mistype?

Module vfio-pc not found in directory

Driver is "vfio-pci".

 

You are going to have issues with the onboard devices if they aren't in their own IOMMU group or can't pass everything in the group.

 

0000:0a:00.0

0000:0a:00.1

 

What do these two devices show up as in your lspci command and can I see a copy of your vfio-driver-override.sh script?

 

What version of Proxmox are you running?

Link to comment
Share on other sites

Link to post
Share on other sites

Hello and thank you Windows7ge,

Im trying to get this running on proxmox 7.2-3, sorry I should have known to put that in original post.

10 hours ago, Windows7ge said:

Possible mistype?

 good catch, my dyslexia would have never seen that... but i still get the "FATAL:"

 

root@ProxMoxBox:~# chmod 755 /etc/initramfs-tools/scripts/init-top/vfio-driver-override.sh
chown root:root /etc/initramfs-tools/scripts/init-top/vfio-driver-override.sh
update-initramfs -u -k all
update-initramfs: Generating /boot/initrd.img-5.15.30-2-pve
modprobe: FATAL: Module vfio-pci not found in directory /lib/modules/5.15.102-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
root@ProxMoxBox:~# 

 

10 hours ago, Windows7ge said:

0000:0a:00.0

0000:0a:00.1

 

What do these two devices show up as in your lspci command

Results from

lspci -nn

0a:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 [8086:1528] (rev 01)
0a:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 [8086:1528] (rev 01)

 

10 hours ago, Windows7ge said:

can I see a copy of your vfio-driver-override.sh script?

 

  GNU nano 5.4                                                     /etc/initramfs-tools/scripts/init-top/vfio-driver-override.sh                                                               
#!/bin/sh
PREREQS=""
DEVS="0000:00:1f.2 0000:0a:00.0 0000:0a:00.1"
for DEV in $DEVS;
  do echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
done

modprobe -i vfio-pci

#00:1f.2 SATA controller [0106]: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] - IOMMU Group 27
#03:00.0 3D controller [0302]: NVIDIA Corporation GP104GL [Tesla P4] [10de:1bb3] - IOMMU Group 33
#04:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] - IOMMU Group 34
#0a:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 [8086:1528] (rev 01) - IOMMU Group 39
#0a:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 [8086:1528] (rev 01) - IOMMU Group 40



 

 

I can't tell you how happy I was to see your response this morning. it was like a lighthouse on a stormy night. I'm so grateful for your guidance and help. Thank you.

 

p.s I hope I'm doing the quotes and codes right in my forum post... I'm so noob I had to google that too.😞

 

Link to comment
Share on other sites

Link to post
Share on other sites

Let's see. I have pass-through working on 7.1-7. Same methods.

 

2 hours ago, thesteamy said:

good catch, my dyslexia would have never seen that... but i still get the "FATAL:"

Did you update-initramfs after adding all the vfio drivers to /etc/modules then restart the server?

 

You can verify if the driver should load at next boot with:

lsinitramfs /boot/initrd.img-5.15.30-2-pve | grep vfio

If nothing shows up the driver isn't going to load. I imagine the driver might show the unavailable error if it hasn't been loaded into RAM by the kernel.

 

3 hours ago, thesteamy said:

0a:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 [8086:1528] (rev 01)
0a:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 [8086:1528] (rev 01)

Ah, onboard Intel X540. That might be in it's own IOMMU group if there's no 0a:00:02. At the same time the paravirtualization driver works very well on Linux/UNIX and the windows downloadable driver is good too if you end up not being able to pass this through.

 

I assume you copy/pasted the script and just modded where you needed it.

 

Alternativly there are the other two options to perform pass-through. The vfio-pci driver helps in certain circumstances but just loading no driver into the device can work as well. If you don't have another X540 you could just blacklist the Device ID 8086:1528, or if you know that driver is only being used on that device you could blacklist the driver system-wide.

 

So you can mix & match methods to get the result you need.

Link to comment
Share on other sites

Link to post
Share on other sites

 

 

Thanks again Windows7ge,

Sorry for the long feedback loop, I'm down under and can only work on this in the mornings.

 

I couldn't get it working, so I swapped in a fresh m.2 and went for a clean install.

 

Proxmox 7.4-3 now installed.

9.2 - Double checked Vt-d is enabled

9.3 - 

./ls-iommu.sh

 - reports a huge list of IOMMU,  below are the devices of interest

 

IOMMU Group 26 00:1f.2 SATA controller [0106]: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] (rev 05)

IOMMU Group 27 01:00.0 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)

IOMMU Group 28 01:00.1 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)

IOMMU Group 29 01:00.2 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)

IOMMU Group 30 01:00.3 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)

IOMMU Group 32 03:00.0 3D controller [0302]: NVIDIA Corporation GP104GL [Tesla P4] [10de:1bb3] (rev a1)

IOMMU Group 33 04:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)

IOMMU Group 36 08:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 12)

IOMMU Group 37 09:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 [8086:1528] (rev 01)

IOMMU Group 38 09:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 [8086:1528] (rev 01)

 

Does that look right so far?

 9.4.1 - Here is where things get wonky

sudo nano /etc/modprobe.d/blacklist.conf

reports 

-bash: sudo: command not found

Works if I drop the sudo and just do a straight nano.... but is this the root of my problems?

 

blacklist ixgbe
blacklist mpt3sas
blacklist nouveau
blacklist nvidiafb

restart

 

nano /etc/modules
  GNU nano 5.4                                                                              /etc/modules                                                                                       
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

 

update-initramfs -u -k all

then I hit another road block, the vfio driver will isn't listed.

root@rack:~# lsinitramfs /boot/initrd.img-5.15.102-1-pve | grep vfio
root@rack:~# 

 

I really hope I'm just doing something super noob... and it sins a hardware limitation.

 

Thank you so much for your time, it really is amazing that people with so much knowledge, like yourself, also have the patience to help people way out of their depth, like me. 

 

 

 

Link to comment
Share on other sites

Link to post
Share on other sites

5 hours ago, thesteamy said:

Works if I drop the sudo and just do a straight nano.... but is this the root of my problems?

Sudo is for when you want to run a command with elevated privledges. Ie as if you were root. In Proxmox the only usable accout is root so sudo isn't needed. Don't know if I included that...

 

5 hours ago, thesteamy said:

then I hit another road block, the vfio driver will isn't listed.

Make sure you have the correct kernel version in the command "initrd.img-5.15.102-1-pve". Proxmox sometimes has multiple kervel versions installed. When you run update-initramfs -u -k -all the output should list the kernel it's using. Here it doesn't hurt to try rebooting and seeing if that makes it work.

 

5 hours ago, thesteamy said:

Thank you so much for your time, it really is amazing that people with so much knowledge, like yourself, also have the patience to help people way out of their depth, like me.

The ironic thing is I could say the exact same thing to others myself. There are a lot of special topics I don't understand that I have no-one to turn to for answers. That's why I write tutorials. To share my findings to help the next person.

Link to comment
Share on other sites

Link to post
Share on other sites

Thanks again.

 

I think I have the right command but still not finding the right path?...

root@rack:~# nano /etc/modules
root@rack:~# update-initramfs -u -k all
update-initramfs: Generating /boot/initrd.img-5.15.102-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.

Would this have anything to do with my boot disk being EXT4 rather than ZFS?

Link to comment
Share on other sites

Link to post
Share on other sites

@Windows7geI am noticing some weird RAM management behavior on Windows guests with ballooning memory enabled. 16G max, 4G minimum ram = 16G ram usage on the VM all the time. 32G max, 8G minimum ram = 2G ram usage. Weirdest part is, the latter results in apps crashing due to lack of ram.

 

Noticed anything like it before? All guest tools are installed too.

mY sYsTeM iS Not pErfoRmInG aS gOOd As I sAW oN yOuTuBe. WhA t IS a GoOd FaN CuRVe??!!? wHat aRe tEh GoOd OvERclok SeTTinGS FoR My CaRd??  HoW CaN I foRcE my GpU to uSe 1o0%? BuT WiLL i HaVE Bo0tllEnEcKs? RyZEN dOeS NoT peRfORm BetTer wItH HiGhER sPEED RaM!!dId i WiN teH SiLiCON LotTerrYyOu ShoUlD dEsHrOuD uR GPUmy SYstEm iS UNDerPerforMiNg iN WarzONEcan mY Pc Run WiNdOwS 11 ?woUld BaKInG MY GRaPHics card fIX it? MultimETeR TeSTiNG!! aMd'S GpU DrIvErS aRe as goOD aS NviDia's YOU SHoUlD oVERCloCk yOUR ramS To 5000C18

 

Link to comment
Share on other sites

Link to post
Share on other sites

6 hours ago, thesteamy said:

Would this have anything to do with my boot disk being EXT4 rather than ZFS?

I would expect no but the behavior is strange. Tell you what, Saturday I can setup a spare box I have and we can try replicating the error you're getting on your version of PROXMOX. It's possible the procedure has changed and might require different operations. That happened to using QEMU/KVM on Ubuntu.

 

3 hours ago, Levent said:

@Windows7geI am noticing some weird RAM management behavior on Windows guests with ballooning memory enabled. 16G max, 4G minimum ram = 16G ram usage on the VM all the time. 32G max, 8G minimum ram = 2G ram usage. Weirdest part is, the latter results in apps crashing due to lack of ram.

 

Noticed anything like it before? All guest tools are installed too.

I haven't played with Balloning/dynamic RAM allocation but what I have noticed is Windows creates some file system in RAM and fills what you give it which means if the hypervisor gives it 16GB it's going to swollow 16GB but in the VM RAM usage is like...2.5GB. Very annoying. Haven't replicated it on Linux or UNIX. It's just a bad OS design for virtualization.

 

I generally static my RAM allocation for all my VMs only changing it as needed but I have dumb amounts of RAM to go around so conserving it is not really a neccesity. I have to help thesteamy I might be able to test the problem you're having at the same time. I assume Windows 10 22H2? I should be able to make Windows 11 happen. You can add a TPM module in Proxmox.

Link to comment
Share on other sites

Link to post
Share on other sites

17 minutes ago, Windows7ge said:

I would expect no but the behavior is strange. Tell you what, Saturday I can setup a spare box I have and we can try replicating the error you're getting on your version of PROXMOX. It's possible the procedure has changed and might require different operations. That happened to using QEMU/KVM on Ubuntu.

 

I haven't played with Balloning/dynamic RAM allocation but what I have noticed is Windows creates some file system in RAM and fills what you give it which means if the hypervisor gives it 16GB it's going to swollow 16GB but in the VM RAM usage is like...2.5GB. Very annoying. Haven't replicated it on Linux or UNIX. It's just a bad OS design for virtualization.

 

I generally static my RAM allocation for all my VMs only changing it as needed but I have dumb amounts of RAM to go around so conserving it is not really a neccesity. I have to help thesteamy I might be able to test the problem you're having at the same time. I assume Windows 10 22H2? I should be able to make Windows 11 happen. You can add a TPM module in Proxmox.

Yeah W10 22H2. Just typical windows bs installed (IIS, msbuild) and qemu guest tools. Weird thing is this seem to happen at random intervals too. One boot it’s totally fine and the next it’s all messed up. I am certain this is a windows thing but haven’t really had a chance to replicate it elsewhere. 

mY sYsTeM iS Not pErfoRmInG aS gOOd As I sAW oN yOuTuBe. WhA t IS a GoOd FaN CuRVe??!!? wHat aRe tEh GoOd OvERclok SeTTinGS FoR My CaRd??  HoW CaN I foRcE my GpU to uSe 1o0%? BuT WiLL i HaVE Bo0tllEnEcKs? RyZEN dOeS NoT peRfORm BetTer wItH HiGhER sPEED RaM!!dId i WiN teH SiLiCON LotTerrYyOu ShoUlD dEsHrOuD uR GPUmy SYstEm iS UNDerPerforMiNg iN WarzONEcan mY Pc Run WiNdOwS 11 ?woUld BaKInG MY GRaPHics card fIX it? MultimETeR TeSTiNG!! aMd'S GpU DrIvErS aRe as goOD aS NviDia's YOU SHoUlD oVERCloCk yOUR ramS To 5000C18

 

Link to comment
Share on other sites

Link to post
Share on other sites

Hello Windows7ge,

Thanks again for your help. I don't want to make a mess of your thread.

 

I have managed to get my GPU,HBA and the SATA controller working with a clean install of proxmox VE 8.0.3 and a ZFS raid 0 (single) boot drive.

Maybe the version? Maybe ZFS? Probably just me being "special".

 

I still can't get individual ports on a NiC to be passed through using 9.4.2 or 9.4.3

If I blacklist the driver (igb) then follow 9.4.3 > lspci -vnn will show the defined ports as using vfio and the remaining ports showing the default driver

unfortunately the default driver ports don't seem to work... and I need to console into the server and either remove the blacklist (igb) or take the device addresses out of the script. Same result with the onboard x540 and (ixgbe)

 

Is this the expected behaviour and I've missed the point?

 

 

A few times the process strays from your excellent tutorial, might just be the new version?

 

9.2 my grub file doesn't have "splash"  is it ok just to add the (intel_iommu=on) after quite?

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

 

using

update grub 

 

I get promoted to use the below

proxmox-boot-tool refresh

 

9.4.1

sudo nano /etc/modprobe.d/blacklist.conf

Should it be ?

nano /etc/modprobe.d/blacklist.conf

 

There were 2 other gotchas but I'll need to go step by step again tomorrow and make better notes.

 

Thanks again.

 

Link to comment
Share on other sites

Link to post
Share on other sites

51 minutes ago, thesteamy said:

Thanks again for your help. I don't want to make a mess of your thread.

My intention behind the thread was to help people in your position. You're not deviating from why I made it. I'm surprised you found a thread no-one posted on in almost 3 years though. Search engines are wild or you've been scouring for a while to stumble across this.

 

56 minutes ago, thesteamy said:

I still can't get individual ports on a NiC to be passed through using 9.4.2 or 9.4.3

Things get tricky with NIC's and hardware pass-thourgh. A lot of server grade NICs are pre-configured in a way which enables each port to be in it's own IOMMU group which is important to pass it though. Either the hardware device has to be alone in an IOMMU group or you have to pass every device in that group to the VM.

 

Theoretically there are ways to break groups apart but I've never gone down that rabbit hole...

 

Personally I wouldn't pass-though NIC ports. Chances are you're not doing anything that would make the benefit worthwhile. Let the hypervisor manage them, ports on the host don't require IP's or any additional configuration, think of it like a network switch but with one physical port, just use the paravirtualization model not the E1000, and for your windows guests install the virtio driver to get the virtual adapter going. Windows doesn't have a driver.

 

1 hour ago, thesteamy said:

If I blacklist the driver (igb) then follow 9.4.3 > lspci -vnn will show the defined ports as using vfio and the remaining ports showing the default driver

unfortunately the default driver ports don't seem to work... and I need to console into the server and either remove the blacklist (igb) or take the device addresses out of the script. Same result with the onboard x540 and (ixgbe)

When you blacklist a driver it blocks the kernel from using it on all applicable hardware devices at startup. You stop that driver system-wide. Not the best option when you have like-devices that need the driver for the hypervisor. Next option would be to look at Device ID but I would expect the Device ID but I think that's only on a per controller basis, pretty sure it won't work on a per port basis.

 

1 hour ago, thesteamy said:

I get promoted to use the below

Proxmox does have it's own command for Update Grub. I ran into this myself writing the guide those years ago.

 

I may write this guide again from scratch with the latest version. I'll just link it at the top of the article so people go to the updated relevant version.

 

1 hour ago, thesteamy said:

Should it be ?

Again, sudo is a command you use in Linux and UNIX-like environments when you are not the user "root". If you have a user account on a Linux system and that user exists in the sudoer file or part of the "wheel" group you can run commands that require elevated privileges by pre-ceeding the command with sudo and entering the administrator password.

 

In PROXMOX the only account that exists by default is root. "sudo" is not needed at all. Maybe I put sudo in my tutorial instructions, I don't remember...

 

Tomarrow we can go over more of the specifics. I'll whip a box together and I'll test all three methods to passing hardware through to a VM. We can document changes to the OS where need be.

Link to comment
Share on other sites

Link to post
Share on other sites

On 6/22/2023 at 1:14 AM, thesteamy said:

Would this have anything to do with my boot disk being EXT4 rather than ZFS?

I"m currently testing PROXMOX 8.0-3 the latest release and I was able to replicate almost the exact same error you got. A little data is suggesting it may have something to do with using EXT4 instead of ZFS as you guessed. The issue potentially stemming from EXT4 not putting the proper files in the / directory as ZFS does by default but I need to test this to verify it. It might not change anything.

 

Edit:

 

Can you show me the output of the following command?

lsblk

 

Link to comment
Share on other sites

Link to post
Share on other sites

@thesteamy It looks like I'm going to have to post a thread on the PROXMOX forums. I figured out one problem but now I hit one of the first issues you were having.

 

So for reference, MAKE SURE you install the OS in UEFI mode. Go into your BIOS and select UEFI Only and disable CSM. PROXMOX needs EFI (part of UEFI boot) to run the functions for this process. Furthurmore use ZFS instead of EXT4 as ZFS doesn't try to create an LVM which is a whole other can of worms.

 

Secondly. None of my prior methods for loading drivers into hardware devices are working anymore. update-initramfs -u -k all seems to do nothing even though I've gotten rid of the errors. I'll post a thread on their forums and update you if I get responses.

Link to comment
Share on other sites

Link to post
Share on other sites

Hi Win

10 hours ago, Windows7ge said:

Can you show me the output of the following command?

root@pve:~# lsblk
NAME                                MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda                                   8:0    1    0B  0 disk 
zd0                                 230:0    0    8G  0 disk 
├─zd0p1                             230:1    0    1M  0 part 
├─zd0p2                             230:2    0  512M  0 part 
└─zd0p3                             230:3    0  7.5G  0 part 
nvme0n1                             259:0    0 13.4G  0 disk 
├─nvme0n1p1                         259:1    0 1007K  0 part 
├─nvme0n1p2                         259:2    0  512M  0 part 
└─nvme0n1p3                         259:3    0 12.9G  0 part 
nvme1n1                             259:4    0  3.6T  0 disk 
├─nvme1n1p1                         259:5    0  128G  0 part 
├─nvme1n1p2                         259:6    0  128G  0 part 
└─nvme1n1p3                         259:7    0  3.4T  0 part 
  ├─nvme--thin-nvme--thin_tmeta     253:0    0 15.8G  0 lvm  
  │ └─nvme--thin-nvme--thin-tpool   253:2    0  3.4T  0 lvm  
  │   ├─nvme--thin-nvme--thin       253:3    0  3.4T  1 lvm  
  │   ├─nvme--thin-vm--100--disk--0 253:4    0    4M  0 lvm  
  │   ├─nvme--thin-vm--100--disk--1 253:5    0   16G  0 lvm  
  │   ├─nvme--thin-vm--101--disk--0 253:6    0    8G  0 lvm  
  │   ├─nvme--thin-vm--103--disk--0 253:7    0   32G  0 lvm  
  │   ├─nvme--thin-vm--104--disk--0 253:8    0    4M  0 lvm  
  │   ├─nvme--thin-vm--104--disk--1 253:9    0   64G  0 lvm  
  │   ├─nvme--thin-vm--105--disk--0 253:10   0    8G  0 lvm  
  │   ├─nvme--thin-vm--106--disk--0 253:11   0    4M  0 lvm  
  │   ├─nvme--thin-vm--106--disk--1 253:12   0   64G  0 lvm  
  │   ├─nvme--thin-vm--107--disk--0 253:13   0   50G  0 lvm  
  │   ├─nvme--thin-vm--108--disk--0 253:14   0    4M  0 lvm  
  │   ├─nvme--thin-vm--108--disk--1 253:15   0   64G  0 lvm  
  │   ├─nvme--thin-vm--109--disk--0 253:16   0   16G  0 lvm  
  │   ├─nvme--thin-vm--110--disk--0 253:17   0    4M  0 lvm  
  │   ├─nvme--thin-vm--110--disk--1 253:18   0   64G  0 lvm  
  │   ├─nvme--thin-vm--102--disk--0 253:19   0   64G  0 lvm  
  │   └─nvme--thin-vm--111--disk--0 253:20   0   64G  0 lvm  
  └─nvme--thin-nvme--thin_tdata     253:1    0  3.4T  0 lvm  
    └─nvme--thin-nvme--thin-tpool   253:2    0  3.4T  0 lvm  
      ├─nvme--thin-nvme--thin       253:3    0  3.4T  1 lvm  
      ├─nvme--thin-vm--100--disk--0 253:4    0    4M  0 lvm  
      ├─nvme--thin-vm--100--disk--1 253:5    0   16G  0 lvm  
      ├─nvme--thin-vm--101--disk--0 253:6    0    8G  0 lvm  
      ├─nvme--thin-vm--103--disk--0 253:7    0   32G  0 lvm  
      ├─nvme--thin-vm--104--disk--0 253:8    0    4M  0 lvm  
      ├─nvme--thin-vm--104--disk--1 253:9    0   64G  0 lvm  
      ├─nvme--thin-vm--105--disk--0 253:10   0    8G  0 lvm  
      ├─nvme--thin-vm--106--disk--0 253:11   0    4M  0 lvm  
      ├─nvme--thin-vm--106--disk--1 253:12   0   64G  0 lvm  
      ├─nvme--thin-vm--107--disk--0 253:13   0   50G  0 lvm  
      ├─nvme--thin-vm--108--disk--0 253:14   0    4M  0 lvm  
      ├─nvme--thin-vm--108--disk--1 253:15   0   64G  0 lvm  
      ├─nvme--thin-vm--109--disk--0 253:16   0   16G  0 lvm  
      ├─nvme--thin-vm--110--disk--0 253:17   0    4M  0 lvm  
      ├─nvme--thin-vm--110--disk--1 253:18   0   64G  0 lvm  
      ├─nvme--thin-vm--102--disk--0 253:19   0   64G  0 lvm  
      └─nvme--thin-vm--111--disk--0 253:20   0   64G  0 lvm  

 

 

I'm currently installed on a ZFS volume, that other drive (nvme1n1) had been set up and populated from the old ext4 install. I have been struggling to find a way to remount the different partitions.... but thats a problem for another day. 

 

7 hours ago, Windows7ge said:

I'll post a thread on their forums and update you if I get responses.

This is really next level of you, this stranger admires your commitment to strangers 🙂 

 

I'm some what relived that its not just me, bummed that it's not just me either.

 

Thanks again.

 

 

UPDATE: I've done another clean install on ZFS after changing my bios as suggested... now I dont get any pass through. Yesterday with legacy enabled I could get GPU, HBA and SATA controller but the NiC's were still being cranky... I wish I could find a copy of Proxmox 7.1 and follow your tutorial and never update...

 

Edited by thesteamy
update
Link to comment
Share on other sites

Link to post
Share on other sites

7 hours ago, thesteamy said:

UPDATE: I've done another clean install on ZFS after changing my bios as suggested... now I dont get any pass through. Yesterday with legacy enabled I could get GPU, HBA and SATA controller but the NiC's were still being cranky... I wish I could find a copy of Proxmox 7.1 and follow your tutorial and never update...

Ideally you're going to want something that's sustainable not EOL. When I get home from work today I can try installing 7.4-1 and see if these functions still work.

 

I do have a couple older .ISO's kicking around. I need to double-check, it's also possible Proxmox might have an online archive. We can try to find the latest version where this trick still works.

Link to comment
Share on other sites

Link to post
Share on other sites

@thesteamy It looks like for the time being your best option is to try just blacklisting the driver if only that device is using it or any other device isn't being used. I just tested 7.4-1 and everything is hung-up on adding vfio-pci to /etc/modules. It's just not working. The VFIO modules aren't being loaded least as far as I can tell.

 

Blacklisting the driver and passing-through that way can work. The Device Address method is just the best of both worlds but I can't figure out what's going on.

Link to comment
Share on other sites

Link to post
Share on other sites

Thanks Win,

Looks like proxmox only has the latest version of 7 and 8 to download, no archive... If I get a support subscription they "might" be able to find it... haha.

 

I'll try agin with 7.4 next weekend.

 

On a curious note the console shows the "L1TF CPU bug present and SMT" warning now, didn't when I had legacy bios enabled seems strange.

 

Thanks for your on going help.

Link to comment
Share on other sites

Link to post
Share on other sites

12 hours ago, thesteamy said:

On a curious note the console shows the "L1TF CPU bug present and SMT" warning now, didn't when I had legacy bios enabled seems strange.

I don't know what would make it specific to UEFI but from a retired proxmox staff members mouth:

Quote

It's not an error message, it's a warning. It just tells you that you have Hyper-Threading (SMT) enabled, which means that a malicious guest VM could theoretically access confidential data on the host.

This is a hardware flaw on intel systems, and can (as far as I'm aware) only be mitigated by disabling Hyper-Threading (SMT), usually done in BIOS. Keep in mind that this incurs a (potentially hefty) performance penalty.

This exploit was discovered a few years ago. These were codenamed spectre & meltdown. It has the potential for a VM to learn what the host OS is doing. Not necessarily explicitly what it's doing but clues gathered over time as it reads memory locations the VM was not mean't to have access to.

 

I will look into if I have an old .ISO or if I can find an archive but it looks like blacklisting the driver is still possible. It's just not ideal.

Link to comment
Share on other sites

Link to post
Share on other sites

  • 2 weeks later...

Hello Windows7ge,

I have persisted and think I have found a solution, following on from your suggestion of disabling the legacy boot option in bios.

https://www.servethehome.com/how-to-pass-through-pcie-nics-with-proxmox-ve-on-intel-and-amd/

This article suggests that some installs dont use grub anymore but systemd instead. With that first new clue I continued to test. I dont think blacklisting works until you make the script for device ID. I can now isolate individual ports on my NiC.... little wins. Still got a lot to learn. but getting there.

Thanks for your help so far. I wonder with if this new info makes some sense to you.

 

Link to comment
Share on other sites

Link to post
Share on other sites

32 minutes ago, thesteamy said:

Hello Windows7ge,

I have persisted and think I have found a solution, following on from your suggestion of disabling the legacy boot option in bios.

https://www.servethehome.com/how-to-pass-through-pcie-nics-with-proxmox-ve-on-intel-and-amd/

This article suggests that some installs dont use grub anymore but systemd instead. With that first new clue I continued to test. I dont think blacklisting works until you make the script for device ID. I can now isolate individual ports on my NiC.... little wins. Still got a lot to learn. but getting there.

Thanks for your help so far. I wonder with if this new info makes some sense to you.

 

It's been so long I don't remeber the specifics but that would make sense. If GRUB isn't being used then updating grub isn't going to influence changes. It might explain the error I still see with update-initramfs about EFI.

 

I will have to read into it more. I presumed GRUB was a perminant replacement to systemd.

Link to comment
Share on other sites

Link to post
Share on other sites

Another thing I've discovered... seems I need to blacklist the driver before I can blacklist via device id. so for my NiC that is proxmox management meant I had to do it via console CLI, Blacklist drive > blackilist device ID > un black list driver + just the device ID post on the NiC stay vfio

 

And not sure if a quirk of IPMI but rather than reboot I need to power cycle the whole box

 

So I think I have made it. Hope my random updates help someone.

 

Thanks,

 

Link to comment
Share on other sites

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


×