VFIO GPU Pass-though w/ Looking Glass KVM on Ubuntu 19.04

[Windows7ge]

Just now, SPCQC said:

I will do it in the next hours as I deleted them earlier

Desktop should have a trashcan. Not certain if deletion from the CLI is permanent deletion though. Might be.

[SPCQC]

1 minute ago, Windows7ge said:

Desktop should have a trashcan. Not certain if deletion from the CLI is permanent deletion though. Might be.

It's permanent.

[Windows7ge]

1 minute ago, SPCQC said:

It's permanent.

Did you copy/paste my examples for 5.2 & 5.3 or did you write them out manually?

[SPCQC]

7 minutes ago, Windows7ge said:

For all intensive purposes it should work but it is expected that some steps may be different.

 

Can you show me the contents of both the file you created using Device ID and the file you created using Device Address?

Also, can it be because I have in my boot folder these two?

/boot/initrd.img-5.3.0-28-generic

/boot/initrd.img-5.3.0-40-generic

 

[SPCQC]

Just now, Windows7ge said:

Did you copy/paste my examples for 5.2 & 5.3 or did you write them out manually?

Copy paste

[Windows7ge]

1 minute ago, SPCQC said:

Also, can it be because I have in my boot folder these two?

/boot/initrd.img-5.3.0-28-generic

/boot/initrd.img-5.3.0-40-generic

 

This should not be the issue. When you run sudo update-initramfs -u it will only update the initramfs image that's currently in use. You can check which kernel is currently loaded with uname -r. It's the only one you have to worry about.

[Windows7ge]

9 minutes ago, SPCQC said:

Copy paste

Did you verify that you used the correct prefix for your system?

[SPCQC]

44 minutes ago, SPCQC said:

I will do it in the next hours as I deleted them earlier

5.2 File vfio-driver-override.conf was :

 

options vfio-pci ids=10de:1b82,10de:10f0

 

5.3 File vfio-driver-override.sh was :

 

#!/bin/sh
PREREQS=""
DEVS="0000:02:00.0 0000:02:00.1"
for DEV in $DEVS;
  do echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
done

modprobe -i vfio-pci

 

When I put the command to see the prefix, the only prefix I saw was 0000 si that's why I put 0000é

[Windows7ge]

15 minutes ago, SPCQC said:

5.2 File vfio-driver-override.conf was :

 

options vfio-pci ids=10de:1b82,10de:10f0

 

5.3 File vfio-driver-override.sh was :

 

#!/bin/sh
PREREQS=""
DEVS="0000:02:00.0 0000:02:00.1"
for DEV in $DEVS;
  do echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
done

modprobe -i vfio-pci

 

When I put the command to see the prefix, the only prefix I saw was 0000 si that's why I put 0000é

Both looks and sounds good. When you tried to override the device driver using Device Address did you enable the vfio-pci driver(s)? What output do you get with the following command?

lsinitramfs /boot/initrd.img-5.3.0-40-generic | grep vfio

 

[SPCQC]

25 minutes ago, Windows7ge said:

Both looks and sounds good. When you tried to override the device driver using Device Address did you enable the vfio-pci driver(s)? What output do you get with the following command?

lsinitramfs /boot/initrd.img-5.3.0-40-generic | grep vfio

 

I got this output

lib/modules/5.3.0-40-generic/kernel/drivers/vfio
lib/modules/5.3.0-40-generic/kernel/drivers/vfio/vfio_virqfd.ko
lib/modules/5.3.0-40-generic/kernel/drivers/vfio/vfio.ko
lib/modules/5.3.0-40-generic/kernel/drivers/vfio/vfio_iommu_type1.ko
lib/modules/5.3.0-40-generic/kernel/drivers/vfio/pci
lib/modules/5.3.0-40-generic/kernel/drivers/vfio/pci/vfio-pci.ko
scripts/init-top/vfio-driver-override.sh

 

[Windows7ge]

5 minutes ago, SPCQC said:

I got this output

lib/modules/5.3.0-40-generic/kernel/drivers/vfio
lib/modules/5.3.0-40-generic/kernel/drivers/vfio/vfio_virqfd.ko
lib/modules/5.3.0-40-generic/kernel/drivers/vfio/vfio.ko
lib/modules/5.3.0-40-generic/kernel/drivers/vfio/vfio_iommu_type1.ko
lib/modules/5.3.0-40-generic/kernel/drivers/vfio/pci
lib/modules/5.3.0-40-generic/kernel/drivers/vfio/pci/vfio-pci.ko
scripts/init-top/vfio-driver-override.sh

 

So the drivers have been set to load on start-up. Everything is looking correct but it's still not working when you make the script and reboot.

 

You said when you use by Device Address that both your GPUs give-out?

[SPCQC]

5 minutes ago, Windows7ge said:

So the drivers have been set to load on start-up. Everything is looking correct but it's still not working when you make the script and reboot.

 

You said when you use by Device Address that both your GPUs give-out?

That is exact, symptoms are as written previously 

[Windows7ge]

4 minutes ago, SPCQC said:

That is exact, symptoms are as written previously 

Did you setup the system so you can remote in? Did you run lspci -vnn to check what device drivers got overridden?

[SPCQC]

3 minutes ago, Windows7ge said:

Did you setup the system so you can remote in? Did you run lspci -vnn to check what device drivers got overridden?

Yes I have rdp access and ssh access.

You want me to test for Device Adress or Device ID?

[Windows7ge]

11 minutes ago, SPCQC said:

Yes I have rdp access and ssh access.

You want me to test for Device Adress or Device ID?

Device Address would be preferred it's a method I have experience testing and know works. If both GPUs are dropping-out I wonder if it's a distribution issue, a version issue, or a configuration issue.

[SPCQC]

47 minutes ago, Windows7ge said:

Did you setup the system so you can remote in? Did you run lspci -vnn to check what device drivers got overridden?

All looks fine

 

[Windows7ge]

2 minutes ago, SPCQC said:

All looks fine

Yep. This looks entirely correct vfio-pci replace all components of the NVIDIA GPU and your AMD card is untouched.

 

You said you get a ~purple-ish or orange page on the output of the AMD card but nothing is on the screen yes? Do you use a login screen?

[SPCQC]

1 minute ago, Windows7ge said:

Yep. This looks entirely correct vfio-pci replace all components of the NVIDIA GPU and your AMD card is untouched.

 

You said you get a ~purple-ish or orange page on the output of the AMD card but nothing is on the screen yes? Do you use a login screen?

Hi. Yes I have a login screen.

Right now, on the NVIDIA, I have a purple-ish screen. On the AMD, blackscreen.

[Windows7ge]

1 minute ago, SPCQC said:

Hi. Yes I have a login screen.

Right now, on the NVIDIA, I have a purple-ish screen. On the AMD, blackscreen.

Doesn't make sense. Lets try this though. Assuming you're looking at these screens right now press Enter. Type your password. Then press Enter again. Does anything happen?

 

In case it says the password was wrong type it again (without pressing enter beforehand) then hit enter at the end.

 

Does anything happen on either output?

[SPCQC]

5 minutes ago, Windows7ge said:

Doesn't make sense. Lets try this though. Assuming you're looking at these screens right now press Enter. Type your password. Then press Enter again. Does anything happen?

 

In case it says the password was wrong type it again (without pressing enter beforehand) then hit enter at the end.

 

Does anything happen on either output?

I know this doesn't make sense... Nothing happens.

 

I'm thinking about switching AMD card for another one, or changing the order in the computer.

[Windows7ge]

Just now, SPCQC said:

I know this doesn't make sense... Nothing happens.

 

I'm think about switch AMD card for another one, or changing the order in the computer.

Well...it does a tiny bit because I've experienced something similar myself so I should ask before you do that what slots are these GPU's installed in? Is the NVIDIA Card in the top-most slot or is the AMD card?

[SPCQC]

Just now, Windows7ge said:

Well...it does a tiny bit because I've experienced something similar myself so I should ask before you do that what slots are these GPU's installed in? Is the NVIDIA Card in the top-most slot or is the AMD card?

As written in the lspci -vnn, the NVIDIA card is in slot 2 and the AMD card is in slot 5.

[Windows7ge]

2 minutes ago, SPCQC said:

As written in the lspci -vnn, the NVIDIA card is in slot 2 and the AMD card is in slot 5.

That doesn't really help universally explain which slots the cards are in. Sometimes the PCI_e slots count down>up and sometimes it's the other way around.

 

Which GPU is closer to the CPU physically.

[SPCQC]

1 minute ago, Windows7ge said:

That doesn't really help universally explain which slots the cards are in. Sometimes the PCI_e slots count down>up and sometimes it's the other way around.

 

Which GPU is closer to the CPU physically.

NVIDIA GPU is closer to CPU

If you want to go discord instead or something like that, don't hesitate.

[Windows7ge]

2 minutes ago, SPCQC said:

NVIDIA GPU is closer to CPU

If you want to go discord instead or something like that, don't hesitate.

That's alright. I don't mind this. I do feel bad for the people following this topic though but they have the ability to stop if this drives them nuts. Besides we're almost to the end of all the troubleshooting I can offer you. If we hit the end all I can blame it on is the motherboard as not being compatible with this type of setup.

 

Swap the cards positions but before you do that disable the Device Address script (I'd probably move it to the desktop or the home directory then update-initramfs again). What sounds like the problem right now is because you're using the GPU in the top slot the system is still trying to run it because it's the Default Video Output device. We can fix that by simply putting it in a different slot and moving the GPU intended for the Host into the top slot. Then Ubuntu should default output to the AMD GPU and the NVIDIA card will hopefully have a blackscreen. You will have to re-lookup the new Device Addresses and set them to the script then re-add it and run update-initramfs again.