VFIO GPU Pass-though w/ Looking Glass KVM on Ubuntu 19.04

[WereCatf]

28 minutes ago, Windows7ge said:

I would like to ask is this a utilization reduction that Windows (the VM) reports or does this reduce the Linux hosts utilization keeping the VM running?

On the host.

[bimmerman]

Looking forward to tinkering this weekend, just read through the guide and been doing some google research.

 

If I'm not looking to use looking glass, and instead just want to use linux to spin up the VMs, is vfio still the right hypervisor? I.e., no need for linux once the VMs are up, and preferably no gpu dedicated to linux either. If so, how does blacklisting work if you have multiple gpus that all need to get passed to the VMs, some identical some not?

[Windows7ge]

2 minutes ago, bimmerman said:

Looking forward to tinkering this weekend, just read through the guide and been doing some google research.

 

If I'm not looking to use looking glass, and instead just want to use linux to spin up the VMs, is vfio still the right hypervisor? I.e., no need for linux once the VMs are up, and preferably no gpu dedicated to linux either. If so, how does blacklisting work if you have multiple gpus that all need to get passed to the VMs, some identical some not?

QEMU is the hypervisor. The back-end to be more specific. VFIO is simply a virtualization driver that can be assigned to hardware devices which enables them to be passed through to a VM.

 

Blacklisting a driver is an alternative to the vfio-pci driver where the kernel will effectively ignore the device(s) and any VM can scoop it up when it needs it. No extra driver needed. The problem with this solution is it blacklists the driver system wide. If any other device relies on that driver it will also not be loaded.

 

If the board had onboard video or you can connect at least a temporary gpu then you can blacklist the driver components for the GPUs. Setup OpenSSH-Server so you can remote in and un-blacklist drivers if necessary.

 

The guide is going to be receiving a couple of updates very soon. Performance Optomization and an Audio device that streams over the virtio bridge so not only do you control both systems with one keyboard & mouse. You have audio from both machines playing simultaneously.

[shuri]

Instead of looking glass, I like using barrier (https://github.com/debauchee/barrier). You just switch the input on your monitor and move your cursor to the side to control the vm.

[Windows7ge]

13 minutes ago, shuri said:

Instead of looking glass, I like using barrier (https://github.com/debauchee/barrier). You just switch the input on your monitor and move your cursor to the side to control the vm.

Sounds like it's not a bad alternative if you only have one monitor. Personally I think Looking Glass is better when you have multiple displays at your disposal. This way you don't have to switch inputs and don't have to sacrifice a display from the host for the VM. You can have both at the same time.

[Windows7ge]

The tutorial has been updated with the following sections:

 

9. Scream Audio

    9.1 - Windows Setup

    9.2 - Linux Setup

    9.3 - Making Things Easy

Courtesy of @2FA

Since the tutorial takes advantage of virtio-pci-net for the primary network adapter using scream-over-LAN was still a better option than IVSHMEM. However the IVSNMEM option is still functional in the event the Windows VM is for pen-testing or executing untrustworthy code as it enables you to use NAT and still pass audio to the host. However that's not the intended demographic so it was not included.

 

10.3 - Hyperv

Courtesy of @WereCatf

I investigated virtio-scsi but could not figure it out. Since the intended File System is ext3 or ext4 I don't even know if this would be beneficial for most users. Ideally users would pass-through a dedicated storage device anyways.

 

Then there were some misc changes. Updating the Index, adding Git & Scream to the software list, changing the VirtIO NIC from optional to necessary, additional troubleshooting steps added, etc.)

[WereCatf]

6 minutes ago, Windows7ge said:

Since the intended File System is ext3 or ext4

I have no idea what you mean with that. The virtual drive is just an SCSI-drive, instead of a SATA-drive, to the guest and doesn't care what filesystem you use on it.

[Windows7ge]

22 minutes ago, WereCatf said:

I have no idea what you mean with that. The virtual drive is just an SCSI-drive, instead of a SATA-drive, to the guest and doesn't care what filesystem you use on it.

Now I'm confused. When you were talking about a supporting File System like ZFS or BTRFS were you referring to inside the VM or outside?

Quote

Personally, I use virtio-scsi instead of virtio-blk and set both the "Discard mode" to "unmap" and "Detect zeroes" as "unmap" -- this way, the HDD-image on the hypervisor only ever uses as much space as the data inside it (as long as the hypervisor filesystem supports it, like e.g. Btrfs or ZFS)

*confusion intensifies*

[2FA]

25 minutes ago, Windows7ge said:

Now I'm confused. When you were talking about a supporting File System like ZFS or BTRFS were you referring to inside the VM or outside?

*confusion intensifies*

I believe WereCatf is referring to thin provisioning on the host, which not all filesystems or storage setups will support. For example, LVM does not support thin provisioning regardless of filesystem used, but  EXT4/BTRFS/ZFS/etc by themselves do.

[Windows7ge]

1 minute ago, 2FA said:

I believe WereCatf is referring to thin provisioning on the host, which not all filesystems or storage setups will support. For example, LVM does not support thin provisioning regardless of filesystem used, but  EXT4/BTRFS/ZFS/etc by themselves do.

I may re-investigate it at some point. Right now though bigger problems are identifying why scrolling the mouse wheel up and down causes the page to scroll down and down.

 

I have also investigated the clipboard so people can copy/paste between the two but all of my testing and following the official WiKi I cannot get it working. It just refuses.

[WereCatf]

9 hours ago, 2FA said:

I believe WereCatf is referring to thin provisioning on the host, which not all filesystems or storage setups will support. For example, LVM does not support thin provisioning regardless of filesystem used, but  EXT4/BTRFS/ZFS/etc by themselves do.

Indeed.

[Windows7ge]

7 hours ago, WereCatf said:

Indeed.

Ah, OK. I did not setup LVM when I installed Ubuntu and I am using EXT4. If I can figure it out through virt-manager I'll add it but virt-manager has no virtio-scsi option it's "virtio" or "scsi". There's also nothing labeled virtio-blk and testing any of these directly in the .XML file results in an errors so currently I'm at a loss.

 

Unrelated to that by pure coincidence when @2FA suggested Scream and pointed me to a thread on Level1Techs the moderator talking to OP on that thread is actually the Author of Looking Glass. So that's cool. Might be able to fix the clipboard after all.

 

And figure out how to fix the scroll wheel. Maybe. I might be on my own there.

[WereCatf]

2 hours ago, Windows7ge said:

Ah, OK. I did not setup LVM when I installed Ubuntu and I am using EXT4. If I can figure it out through virt-manager I'll add it but virt-manager has no virtio-scsi option it's "virtio" or "scsi". There's also nothing labeled virtio-blk and testing any of these directly in the .XML file results in an errors so currently I'm at a loss.

I overestimated your knowledge of QEMU/KVM, sorry. In virt-manager, you click on the "Add new hardware", then select "Controller" of type "SCSI" and model of "Virtio SCSI." Then you change the bus-type of your HDD from SATA or IDE to SCSI. Also, change "Discard mode" and "Detect zeroes" for the HDD to "unmap" and you're all set.

 

Performance-wise there is no practical difference, it's just all about letting the image for the VM shrink and grow along with the data inside it and the hypervisor OS knowing to use TRIM as needed. Whether any of that matters to someone is up to their own tastes, but I like to keep the images as small as possible, so they're faster to clone/backup/whatnot and, since I use an SSD, allowing the hypervisor OS to use TRIM on it is beneficial.

[Windows7ge]

3 minutes ago, WereCatf said:

I overestimated your knowledge of...

You can apply this to many, many, many things that I decide to branch out and explore. My knowledge is like a tree where all the open air between the branches is everything I don't know. Which if you can imagine your average tree there's a lot more air than their are branches.

 

8 minutes ago, WereCatf said:

Performance-wise there is no practical difference, it's just all about letting the image for the VM shrink and grow along with the data inside it and the hypervisor OS knowing to use TRIM as needed. Whether any of that matters to someone is up to their own tastes, but I like to keep the images as small as possible, so they're faster to clone/backup/whatnot and, since I use an SSD, allowing the hypervisor OS to use TRIM on it is beneficial.

Personally I would find something like this useful for when running TimeShift. Does Windows have a driver for it or does it still rely on the virtio drivers?

 

Unrelated. Talking to the LG author we've gotten the Clipboard working enabling copy/paste between the two OS's. Coincidentally the fix for the clipboard also fixes the scroll wheel problem.

[WereCatf]

1 minute ago, Windows7ge said:

Does Windows have a driver for it or does it still rely on the virtio drivers?

Yes, Windows does rely on those for virtio-scsi as well.

[Windows7ge]

The tutorial has been updated again to include:

• 1.2.11 - SPICE Guest Tools
• 8.4 - Installing the SPICE Guest Tools

This enables Clipboard (Copy/Paste) functionality between the Host and the VM. It also works as a patch for the scroll-wheel bug.

[GotCritified]

Then me having an rtx 2060 card means id need to bind it to linux and buy second expensive gaming gpu from AMD specificaly to make it go without driver hacking . . . .thats a spoiled Linus viable . . not working class viable option.

Thats bad, i wanted to have stable system virtualising gaming windows with all its pc breaking update bullshit.

Last update ms was so amazing to actualy wipe out partition during update, i though its just lost a bootloader or switched boot priority at the uefi but no,

windows start up repair greeted me with " what would u like to do with this unallocated space".

On third try it even wiped a second drive with fresh and completely separate linux installation, made specificaly to combat windows update bullshit. Ms update wiped both drives

 

Thus me thinking virtualising gaming windows would stop ms from **** me up.

[Windows7ge]

6 hours ago, GotCritified said:

Then me having an rtx 2060 card means id need to bind it to linux and buy second expensive gaming gpu from AMD specificaly to make it go without driver hacking . . . .thats a spoiled Linus viable . . not working class viable option.

Thats bad, i wanted to have stable system virtualising gaming windows with all its pc breaking update bullshit.

Last update ms was so amazing to actualy wipe out partition during update, i though its just lost a bootloader or switched boot priority at the uefi but no,

windows start up repair greeted me with " what would u like to do with this unallocated space".

On third try it even wiped a second drive with fresh and completely separate linux installation, made specificaly to combat windows update bullshit. Ms update wiped both drives

 

Thus me thinking virtualising gaming windows would stop ms from **** me up.

There's a lot of benefits that come with virtualization Windows. You have a lot more control over what it has access to. The hack to make a NVIDIA GPU work isn't complicated to setup. If you have an iGPU or lesser card (assuming you don't need a high performance GPU for your GNU/Linux purposes) then you can still make this work without having to fork-out a large sum of money for a second dGPU.

[SPCQC]

Hi!

I am in the step of creating the VM and now it is stuck on Creating Domain.

 

After few researches, I am not able to pass this.

 

If someone could help me, that would be awesome.

 

Thanks!

[Windows7ge]

15 minutes ago, SPCQC said:

Hi!

I am in the step of creating the VM and now it is stuck on Creating Domain.

 

After few researches, I am not able to pass this.

 

If someone could help me, that would be awesome.

 

Thanks!

Step 6.1 - New Virtual Machine?

[SPCQC]

Just now, Windows7ge said:

Step 6.1 - New Virtual Machine?

Hi! After step 6.2.5.3 - Looking Glass Components . After I clicked on Begin Installation

 

[Windows7ge]

15 minutes ago, SPCQC said:

Hi! After step 6.2.5.3 - Looking Glass Components . After I clicked on Begin Installation

What method did you use to block the GPU driver? A common problem is when the kernel still has a driver loaded into the device.

[SPCQC]

1 minute ago, Windows7ge said:

What method did you use to block the GPU driver? A common problem is when the kernel still has a driver loaded into the device.

I used method 5.1 - Blacklist the Driver. Please note that my monitor is plugged into the GPU that is for passthrough.

 

Also, I have in the /boot folder this, maybe it can help you.

/boot/initrd.img-5.0.0-28-generic

/boot/initrd.img-5.0.0-40-generic

[Windows7ge]

4 minutes ago, SPCQC said:

I used method 5.1 - Blacklist the Driver. Please note that my monitor is plugged into the GPU that is for passthrough.

 

Also, I have in the /boot folder this, maybe it can help you.

/boot/initrd.img-5.0.0-28-generic

/boot/initrd.img-5.0.0-40-generic

What is the Device Address of the GPU?

[SPCQC]

12 minutes ago, Windows7ge said:

What is the Device Address of the GPU?

dGPU (1070ti) : 02:00.0 / 02:00.1

LittleAMDCard : 03:00.0 / 03:00.1