iommu How to ensure, that IOMMU is working propperly?

[LinuxTux23]

Hey there,

 

My Server is using Proxmox as Hypervisor (KVM backend) to spawn VMs.

I wanna use the HAB (specs) to passthrough my 4HDDs to a TrueNAS Core VM.

 

In order to do that, I first enabled the IOMMU support in the BIOS/UEFI. After that, I booted into Proxmox and checked whether IOMMU is detected. It spitted out the following output

[    1.332679] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    1.344441] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.345378] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

which in my mind means that IOMMU is already enabled, even if I never configured it in the GRUB cmdline.

 

I also checked the IOMMU groups, to validate the HBA is in its own separate group. 

[...]
IOMMU Group 25 2d:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 [1000:0087] (rev 05)
[...]

And in fact, it is in its own dedicated IOMMU group.

 

Did someone encounter this before, or is my understanding of IOMMU at some point wrong ?

 

Server Specs:

- AsRockRack X570D4U Mainboard

- AMD Ryzen 5 3600

- x2 16GB Crucial Micron ECC RAM

- x2 Seagate Ironwolf 4TB

- x2 WD RED 4TB

- LSI Logic / Symbios Logic SAS2308 HBA

[leadeater]

27 minutes ago, LinuxTux23 said:

And in fact, it is in its own dedicated IOMMU group.

 

Did someone encounter this before, or is my understanding of IOMMU at some point wrong ?

You don't normally have to configure or touch IOMMU/VT-d/AMD-Vi unless something is wrong or you need to override something. If you have a properly configured motherboard that has these features in mind the only thing you have to do is enable it then in your chosen hypervisor pass the device through to a VM, that's it. Extra configuration is only required when something dumb has been done like the PCIe device and onboard NIC being group together for example.

[LinuxTux23]

2 hours ago, leadeater said:

You don't normally have to configure or touch IOMMU/VT-d/AMD-Vi unless something is wrong or you need to override something. If you have a properly configured motherboard that has these features in mind the only thing you have to do is enable it then in your chosen hypervisor pass the device through to a VM, that's it. Extra configuration is only required when something dumb has been done like the PCIe device and onboard NIC being group together for example.

Sounds really comprehensible to me, but why does the Proxmox documentation list steps to "enable" IOMMU ?

 

https://pve.proxmox.com/wiki/Pci_passthrough

https://pve.proxmox.com/wiki/PCI(e)_Passthrough

[Electronics Wizardy]

In this case I wouldn't use passthough at all. Id let proxmox handle zfs and storage, and then just use a container or vitrual disk for the nas vm. That way things like ram caching is handled better, and its easier to share storage between vms.

 

What happens when you try to pass the hba to a vm?

[LinuxTux23]

10 hours ago, Electronics Wizardy said:

In this case I wouldn't use passthough at all. Id let proxmox handle zfs and storage, and then just use a container or vitrual disk for the nas vm. That way things like ram caching is handled better, and its easier to share storage between vms.

 

What happens when you try to pass the hba to a vm?

Hey, when I try to pass through the HBA to the VM it gets detecter by the OS and works perfectly fine.

[GamingIsGambling]

I found this and had a similar issue with passing through a graphics card. This page in general helped, posting here for if anyone else finds this forum late like me.

 

https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Physical_disk/partition