VFIO GPU Pass-though w/ Looking Glass KVM on Ubuntu 19.04

[SPCQC]

10 minutes ago, Windows7ge said:

That's alright. I don't mind this. I do feel bad for the people following this topic though but they have the ability to stop if this drives them nuts. Besides we're almost to the end of all the troubleshooting I can offer you. If we hit the end all I can blame it on is the motherboard as not being compatible with this type of setup.

 

Swap the cards positions but before you do that disable the Device Address script (I'd probably move it to the desktop or the home directory then update-initramfs again). What sounds like the problem right now is because you're using the GPU in the top slot the system is still trying to run it because it's the Default Video Output device. We can fix that by simply putting it in a different slot and moving the GPU intended for the Host into the top slot. Then Ubuntu should default output to the AMD GPU and the NVIDIA card will hopefully have a blackscreen. You will have to re-lookup the new Device Addresses and set them to the script then re-add it and run update-initramfs again.

If you want to know something lame... I don't have the space to put my NVIDIA graphic in slot five. The cooler is too big...

 

I'll check if there's a wat to default the other GPU.

[Windows7ge]

12 minutes ago, SPCQC said:

If you want to know something lame... I don't have the space to put my NVIDIA graphic in slot five. The cooler is too big...

 

I'll check if there's a wat to default the other GPU.

Well that's a problem. This could be the fix. PCI_e riser?

 

I have a strong feeling the answer is going to be that there isn't one.

[SPCQC]

Hi!

 

This message is just to tell you that I have forfeited for now.

 

Thanks for your help.

[Windows7ge]

8 minutes ago, SPCQC said:

for now.

I'll be here to help you out when you decide to take another crack at it.

[Alex Smargle]

So I'm following along with this guide and having finished the 8.1 section, upon attempting to restart the VM I'm getting an error that says

Quote

Error starting domain: internal error: Process exited prior to exec: libvirt: error: unable to set AppArmor profile 'libvirt-5d9750a7-d888-4c9b-a061-a48a373cac52' for '/usr/bin/qemu-system-x86_64': No such file or directory

I've looked into the /usr/bin/ folder though and the qemu-system-x86_64 is there so I'm not sure where the issue is exactly or why it can't seem to find it?

I have the full error details still open in case it helps someone out. Or other things but I'm not sure what would actually be helpful.

[Windows7ge]

7 minutes ago, Alex Smargle said:

So I'm following along with this guide and having finished the 8.1 section, upon attempting to restart the VM I'm getting an error that says

I've looked into the /usr/bin/ folder though and the qemu-system-x86_64 is there so I'm not sure where the issue is exactly or why it can't seem to find it?

I have the full error details still open in case it helps someone out. Or other things but I'm not sure what would actually be helpful.

What distro are you running? Did you restart AppArmor? You may also need to restart the computer.

 

If it doesn't help temporarily remove

<shmem name='looking-glass'>
  <model type='ivshmem-plain'/>
  <size unit='M'>32</size>
</shmem>

from your libvirt configuration. Does the VM successfully boot again? If not the issue may come from a step even further back.

 

Adding Looking Glass to the VM can be a little tricky. If we have to we can test disabling SE Linux outright.

[Alex Smargle]

I'm running Ubuntu 19.10 on host and Windows 10 Home on guest.
I did restart AppArmor.

I have not restarted the host but I'll try that and the shmem thing if it doesn't work and get back to you in a few.

[Alex Smargle]

2 minutes ago, Alex Smargle said:

I'm running Ubuntu 19.10 on host and Windows 10 Home on guest.
I did restart AppArmor.

I have not restarted the host but I'll try that and the shmem thing if it doesn't work and get back to you in a few.

Neither a restart of host or removing shmem worked. Same error message.

[Windows7ge]

8 minutes ago, Alex Smargle said:

Neither a restart of host or removing shmem worked. Same error message.

I wanted to test this on 19.10 but my desktop didn't want to run 19.10 for some reason. Hopefully we can get this working so I can tell people it works.

 

Make sure there are no errors in the AppArmor file for now. I should have more time tomorrow to research this issue with you. I would test disabling SE Linux. It's possible the AppArmor exceptions that work in 19.04 don't in 19.10. If you remove the changes you made to AppArmor and restart the service, does the VM start or the error message change?

[Alex Smargle]

5 minutes ago, Windows7ge said:

I wanted to test this on 19.10 but my desktop didn't want to run 19.10 for some reason. Hopefully we can get this working so I can tell people it works.

 

Make sure there are no errors in the AppArmor file for now. I should have more time tomorrow to research this issue with you. I would test disabling SE Linux. It's possible the AppArmor exceptions that work in 19.04 don't in 19.10. If you remove the changes you made to AppArmor and restart the service, does the VM start or the error message change?

Backtracking and removing the changes to /etc/apparmor.d/abstractions/libvirt-qemu does actually let the VM boot as normal. It also reminded me of something else that didn't happen quite like normal.

At the end of 8.1, it says to run
sudo touch /dev/shm/looking-glass && sudo chown vfio:kvm /dev/shm/looking-glass && sudo chmod 660 /dev/shm/looking-glass

is that supposed to be run exactly as is because when I try that I get a notice about invalid user: 'vfio:kvm', it said user so I assumed it meant...uh, me and changed the name? Is that actually not what I was supposed to do and possibly indicative of a larger issue?

 

As for disabling SE Linux, I'm not sure off hand how I'd go about doing that for the VM as I don't recall seeing anything that mentioned SE Linux.

 

[Windows7ge]

10 hours ago, Alex Smargle said:

Backtracking and removing the changes to /etc/apparmor.d/abstractions/libvirt-qemu does actually let the VM boot as normal. It also reminded me of something else that didn't happen quite like normal.

At the end of 8.1, it says to run
sudo touch /dev/shm/looking-glass && sudo chown vfio:kvm /dev/shm/looking-glass && sudo chmod 660 /dev/shm/looking-glass

is that supposed to be run exactly as is because when I try that I get a notice about invalid user: 'vfio:kvm', it said user so I assumed it meant...uh, me and changed the name? Is that actually not what I was supposed to do and possibly indicative of a larger issue?

 

As for disabling SE Linux, I'm not sure off hand how I'd go about doing that for the VM as I don't recall seeing anything that mentioned SE Linux.

 

If you didn't copy/paste then I suspect you may have had a syntax error.

 

I have a number of "note"s that you need to change the user in vfio:kvm to the user you plan to use. The would usually be the primary user of the computer. Group KVM should stay the same.

 

The better/proper way is to make an AppArmor exception. I did not include SE Linux instructions because it's quick'n'dirty. You basically disable ALL the security. I only recommend it for testing purposes.

 

If it comes to it you can disable SE Linux by going to:

sudo nano /etc/libvirt/qemu.conf

Locate and change:

security_driver = "selinux"

to

security_driver = "none"

Save/exit, then run the command:

sudo systemctl disable apparmor.service

And restart the computer.

[Alex Smargle]

I did copy and paste but going back through the terminal history shows me that when I changed the username, I must have deleted the colon by accident so it was trying to add a user named alexkvm which, of course, doesn't exist.

I've tried readding the apparmor changes now that I've actually run that command with a real existing user and group the error is gone, even with the apparmor changes.

Thanks for the speedy help (even if the problem was me!). I'll continue down through the guide and let you know how it all works out on 19.10.

[Alex Smargle]

Hello, I'm here to bother you again!

I've gotten down to 8.5.2 but inside device manager I don't have 'Microsoft Basic Display Adapter' under display adapters. Just my graphics card and a Red Hat QXL controller. I did try out the next step, in case it was just a hardware difference, and while the script does launch Looking Glass, it doesn't prompt for password and it doesn't show the screen.

[Nuzzles]

On 4/11/2020 at 12:49 PM, Alex Smargle said:

Hello, I'm here to bother you again!

I've gotten down to 8.5.2 but inside device manager I don't have 'Microsoft Basic Display Adapter' under display adapters. Just my graphics card and a Red Hat QXL controller. I did try out the next step, in case it was just a hardware difference, and while the script does launch Looking Glass, it doesn't prompt for password and it doesn't show the screen.

I'm sorry to post in such an old thread, but I'm experiencing this as well. I disabled the Red Hat QXL display driver and rebooted. I can no longer see the login screen on Windows and Looking Glass opens up with just a default screen.

 

The guest VM window:

Spoiler

Looking Glass client window:

Spoiler

It may be worth noting that I am on Fedora 35 right now.

[Windows7ge]

28 minutes ago, Nuzzles said:

I'm sorry to post in such an old thread, but I'm experiencing this as well. I disabled the Red Hat QXL display driver and rebooted. I can no longer see the login screen on Windows and Looking Glass opens up with just a default screen.

 

The guest VM window:

  Hide contents

Looking Glass client window:

  Hide contents

It may be worth noting that I am on Fedora 35 right now.

I never thought anybody would comment on this again.

 

You've made good progress but I'll admit my memory is a little fuzzy. Haven't done this on Ubuntu in a long time and definitely never attempted it on Fedora/RHEL.

 

From what I can remember you disable the Windows Basic Display Adapter (or in your case the Red Hat QXL Display) and as a result you are going to lose the output on virt-manager. Unless things have changed since I posted this topic virt-manager cannot display it's KVM and Looking Glass run at the same time. Virt-manager will however continue to act as a keyboard/mouse input so if you connect a display to the passed through GPU you can use the frozen screen to control the VM temporarily.

 

IIRC you need to install Red Hat drivers into Device Manager. Then Looking Glass has a windows executable that needs to be ran at startup. Once it runs the Looking Glass display will become active. From there you can choose to explore enabling the clipboard between host/VM and/or enabling audio from the VM to the host.

 

When I have a little more time I'll quote sections of the tutorial to narrow down the search for you.

[Windows7ge]

1 hour ago, Nuzzles said:

I can no longer see the login screen on Windows

Just remembered something important. Because Windows needs to launch the looking-glass.exe file before Looking Glass actually works I personally could not find a way to get Windows to run the file before logging in. You could very well type in your password on the Looking Glass window and if you have the .exe set to run it may work once windows actually logs in.

 

Outside of this you're going to want to look at section 8. Looking Glass Configuration. In particular sections:

8.1 - Modify libvirt Config

8.2 - Making AppArmor Exceptions

8.3 - Installing the IVSHMEM Driver

8.5.1 - Installing & Auto-Launching the Looking Glass Host

 

Assuming you've already performed 8.1, 8.2 (your directories may differ), & 8.3 I'd investigate 8.5.1 by connecting a monitor to the GPU, opening VM Manager, and using the frozen display for keyboard/mouse and launch the Looking Glass windows executable. See if the Linux window starts working.