pci-e passthrough help

[Ashleyyyy]

so i just installed my R9 290X for use with pci-e passthrough. i followed this guide:

https://forum.level1techs.com/t/ubuntu-17-04-vfio-pcie-passthrough-kernel-update-4-14-rc1/119639

 

the problem i'm having is it uses the radeon kernel driver, not amdgpu. how can i fix that?

[Electronics Wizardy]

If you passing through a gpu you want it using the vfio driver. so disable the amdgpu driver and it will work fine. The amd gpu driver has replaced radeon for newer cards.

[Ashleyyyy]

yeah i did that, but it uses radeon, not amdgpu. 

 

output from lspci -nnv:

04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT / Grenada XT [Radeon R9 290X/390X] [1002:67b0]
    Subsystem: Micro-Star International Co., Ltd. [MSI] R9 290X Gaming [1462:3080]
    Kernel driver in use: radeon
    Kernel modules: radeon, amdgpu

 

i edited out some irrelevant stuff

[Ashleyyyy]

3 minutes ago, Electronics Wizardy said:

If you passing through a gpu you want it using the vfio driver. so disable the amdgpu driver and it will work fine. The amd gpu driver has replaced radeon for newer cards.

forgot to quote sorry

[Electronics Wizardy]

5 minutes ago, firelighter487 said:

yeah i did that, but it uses radeon, not amdgpu. 

 

output from lspci -nnv:

04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT / Grenada XT [Radeon R9 290X/390X] [1002:67b0]
    Subsystem: Micro-Star International Co., Ltd. [MSI] R9 290X Gaming [1462:3080]
    Kernel driver in use: radeon
    Kernel modules: radeon, amdgpu

 

i edited out some irrelevant stuff

what distro?

 

What does your grub options look like? Id blacklist both.

[Ashleyyyy]

Just now, Electronics Wizardy said:

what distro?

 

What does your grub options look like? Id blacklist both.

ubuntu 18.04. 

 

i modified the line in grub like the guide said

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=1 intel_iommu=on"

 

[Electronics Wizardy]

Just now, firelighter487 said:

ubuntu 18.04. 

 

i modified the line in grub like the guide said

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=1 intel_iommu=on"

 

add rd.blacklist=radeon and rd.blacklist=amdgpu. Then reboot, That justed worked for me on fedora.

[Ashleyyyy]

4 minutes ago, Electronics Wizardy said:

add rd.blacklist=radeon and rd.blacklist=amdgpu. Then reboot, That justed worked for me on fedora.

doesn't work. 

 

i also noticed my 780ti doesn't work anymore. it's detected in lspci but doesn't output anything... 

[Electronics Wizardy]

Just now, firelighter487 said:

doesn't work. 

 

i also noticed my 780ti doesn't work anymore. it's detected in lspci but doesn't output anything... 

what gpu's do you have and what drivers?

 

Does your system support passthough(have you had it working before) Some just won't do it.

[Ashleyyyy]

1 minute ago, Electronics Wizardy said:

what gpu's do you have and what drivers?

 

Does your system support passthough(have you had it working before) Some just won't do it.

i have a gtx 780Ti, and a R9 290X. 

 

i have had passthrough working with a different motherboard and an RX460, but i changed motherboard and i swapped to the R9 and now it won't work. 

[Electronics Wizardy]

1 minute ago, firelighter487 said:

i have a gtx 780Ti, and a R9 290X. 

 

i have had passthrough working with a different motherboard and an RX460, but i changed motherboard and i swapped to the R9 and now it won't work. 

so after than config change what driver was the r9 running?

 

 

[Ashleyyyy]

Just now, Electronics Wizardy said:

so after than config change what driver was the r9 running?

radeon... 

 

i'm gonna try fedora then. i think my ubuntu install is broken anyway. it's doing some weird things... 

[Electronics Wizardy]

Just now, firelighter487 said:

radeon... 

 

i'm gonna try fedora then. i think my ubuntu install is broken anyway. it's doing some weird things... 

check the blacklist wording that may be wrong. Also you may need to edit the inti ram image

[Ashleyyyy]

Just now, Electronics Wizardy said:

check the blacklist wording that may be wrong. Also you may need to edit the inti ram image

how do i do that? i'm gonna install fedora in a bit because i can't update the kernel of my ubuntu install anymore for some reason, and my 780Ti doesn't work... 

[Electronics Wizardy]

Just now, firelighter487 said:

how do i do that? i'm gonna install fedora in a bit because i can't update the kernel of my ubuntu install anymore for some reason, and my 780Ti doesn't work... 

I think you need to make the image with dracut in fedora or mkinitramfs(wrong google it) in other distros. You edit the settings in /etc/modprobe.d/(and make a config file here)

[Ashleyyyy]

4 minutes ago, Electronics Wizardy said:

I think you need to make the image with dracut in fedora or mkinitramfs(wrong google it) in other distros. You edit the settings in /etc/modprobe.d/(and make a config file here)

well i'll see when i get fedora installed. 

[Ashleyyyy]

1 hour ago, Electronics Wizardy said:

I think you need to make the image with dracut in fedora or mkinitramfs(wrong google it) in other distros. You edit the settings in /etc/modprobe.d/(and make a config file here)

ok i got fedora installed, and found this guide:

https://forum.level1techs.com/t/ryzen-virtualization-success-gtx-1080ti-passthrough-with-windows-10-client/117390

 

but that's for passing through a nvidia card. is the process basically the same for an amd card?

[Electronics Wizardy]

Just now, firelighter487 said:

ok i got fedora installed, and found this guide:

https://forum.level1techs.com/t/ryzen-virtualization-success-gtx-1080ti-passthrough-with-windows-10-client/117390

 

but that's for passing through a nvidia card. is the process basically the same for an amd card?

yea basically the same, you need intel iommu instead of amd, replace nouveau with radeon and amdgpu.

 

Not all of this is needed, like the nested virt

 

For me all i had to to was

-enable iommu in bios(i was on amd am3 with a fx 8350)

-install fedora and virt-manager and make a win 10 vm (all default)

-black list nouvau(also tried it with a 5450 and it worked fine)

-enable iommu in grub.

-add pci devices to the vm.

 

You can also test with a network card/sound card/raid card or anouther pcie device as there normally easier to get passed though

[Ashleyyyy]

@Electronics Wizardy ok i installed fedora and followed the guide. i swapped pci-e slots so my POST and stuff is on my 780Ti. now the R9 doesn't have any kernel drivers in use. no radeon or amdgpu, but no vfio too. because of that i can't pass it through yet... 

[Ashleyyyy]

@Electronics Wizardy ok turns out i'm just an idiot, i didn't get the id's right. 

 

now i have another issue. whenever i want to boot up the vm it says failed to set iommu for container operation not permitted....

[Electronics Wizardy]

1 hour ago, firelighter487 said:

@Electronics Wizardy ok turns out i'm just an idiot, i didn't get the id's right. 

 

now i have another issue. whenever i want to boot up the vm it says failed to set iommu for container operation not permitted....

screenshot? Are the iommu groups good?

[Ashleyyyy]

 

6 minutes ago, Electronics Wizardy said:

screenshot? Are the iommu groups good?

to me all that looks good. as you can see, both gpu's have their own IOMMU group, and vfio has control of the R9 290X

[Electronics Wizardy]

40 minutes ago, firelighter487 said:

 

to me all that looks good. as you can see, both gpu's have their own IOMMU group, and vfio has control of the R9 290X

what board do you have?

 

you probably want this setting in the initramfs config file 

options vfio_iommu_type1 allow_unsafe_interrupts=1

[Ashleyyyy]

Just now, Electronics Wizardy said:

what board do you have?

 

you probably want this setting in the initramfs config file 

options vfio_iommu_type1 allow_unsafe_interrupts=1

ASUS Z8PE-D18

 

where is the initramfs config file?

[Electronics Wizardy]

Just now, firelighter487 said:

ASUS Z8PE-D18

 

where is the initramfs config file?

make a file in /etc/modprobe.d

 

call it like vfio.conf

 

 

Since you haven't gotten passthrough working, and I don't see any other reports of it, there is a chance this board just won't do it, but there are a few more tweaks to try first.