Enabling ACS results in GPU hotswapping being broken?

Levent · November 9, 2023

Long story short, I had to enable ACS and use following kernel parameters in order to be able to passthrough HBAs and onboard sata controller.

My kernel parameters in full are as follows: "quiet amd_iommu=on pcie_acs_override=downstream,multifunction" from what I am seeing, "multifunction" parameter should have grouped GP104 GPU Core and GP104 Audio controller in the same IOMMU group but I can clearly see that it didnt (I also removed that to see if anything changed, it didnt??).

I am trying to passthrough a GTX1080 to a VM and once vm is shut down, device should be bonded back to host so that I can get a working KVM without having to restart the host. I previously managed to get this to work with proxmox hookscripts and basic perl and bash script combination. Gist of the script where it did the unbinding and binding is listed below ( I actually had to edit device IDs few minutes ago as they changed but rest of is identical)

#!/bin/bash
echo "0000:0b:00.0" > "/sys/bus/pci/devices/0000:0b:00.0/driver/unbind" && echo "0000:0b:00.0" > "/sys/bus/pci/drivers/nouveau/bind"
echo "0000:0b:00.1" > "/sys/bus/pci/devices/0000:0b:00.1/driver/unbind" && echo "0000:0b:00.1" > "/sys/bus/pci/drivers/snd_hda_intel/bind"
exit

GPU passthrough works fine still however, when I turn the VM off and hookscript is triggered, this is the output.

/usr/local/bin/unbind_and_bind.sh: line 3: /sys/bus/pci/devices/0000:0b:00.0/driver/unbind: No such file or directory

Funny enough this is NOT the case if I runt the script before I launch the VM and GPU is passed to the VM.

When I run lspci -v 0b:00.0 I get an interesting output which I did not get when I had ACS disabled however I am not sure if it is related or not.

root@homeserver:~# lspci -v -s 0b:00.0
0b:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1080] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: ASUSTeK Computer Inc. GP104 [GeForce GTX 1080]
        !!! Unknown header type 7f
        Memory at fb000000 (32-bit, non-prefetchable) [size=16M]
        Memory at 7fe0000000 (64-bit, prefetchable) [size=256M]
        Memory at 7ff0000000 (64-bit, prefetchable) [size=32M]
        I/O ports at f000 [size=128]
        Expansion ROM at fc000000 [disabled] [size=512K]
        Kernel modules: nvidiafb, nouveau

Any ideas? I have to have this feature working because I turn VMs on and off that use the same GPU and I occasionally need KVM access.