Alright. This was a right pain in the ass and I knew it was going to take forever to get working right (~2 weeks...). I now have a Windows 10 VM running on QEMU & virt-manager with GPU passthough working on Ubuntu 19.04.

 

The physical system:

 

THREADRIPPER 1950X 16C/32T

64GB DDR4 2400MHz

2x R9 290X

2x SFP+ 10Gbit NIC.

 

In addition to QEMU & virt-manager I got Looking Glass working which lets me view what the passed-though GPU is displaying at near 1:1 performance without plugging a display into the GPU. (I'll explain why the GPU has an HDMI in it in a bit.)

 

So this is what I see between my two monitors.

Windows VM hardware:

CPU: 8C/16T Threadripper (shows up as an EYPC - Something about the VM enabling CPU features and Threadripper doesn't exist so it defaults to EPYC since it's kind of based on it)

32GB of RAM

10Gbit NIC bridge (virtio)

R9 290X

 

The ports on the back of my desktop work differently now too.

 

The left stack of four USB3.0 ports belong to windows. They are connected to a AMD Family 17h Model 00h-0fh USB3.0 Host Controller which exists in it's own IOMMU group. The two beneath the Ethernet port and everything right belong to Linux on other controllers in other groups. In the first photo the bottom GPU belongs to Windows. The top GPU belongs to Linux. The BCM57810S has had one 10G port connected to a bridge and passed though to Windows. This means both clients can run full 10Gig with no bottleneck.

 

This is an accurate representation of how my experience went following the setup steps:

1. Disabling the GPU driver (incorporating vfio-pci in place)
2. Are you using two identical GPU's? Oh boy that's an extra step.
3. Enable IOMMU support in the system kernel. (amd_iommu=on)
4. Update grup
5. Make sure you use the right prefixes or else your GPU won't pass-though correctly. All you gotta do is check /sys/bus/pci/devices. OK, sounds easy enooly f***ing s***: 

   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:00.0 -> ../../../devices/pci0000:00/0000:00:00.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:00.2 -> ../../../devices/pci0000:00/0000:00:00.2
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:01.0 -> ../../../devices/pci0000:00/0000:00:01.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:01.1 -> ../../../devices/pci0000:00/0000:00:01.1
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:01.3 -> ../../../devices/pci0000:00/0000:00:01.3
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:02.0 -> ../../../devices/pci0000:00/0000:00:02.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:03.0 -> ../../../devices/pci0000:00/0000:00:03.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:03.1 -> ../../../devices/pci0000:00/0000:00:03.1
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:04.0 -> ../../../devices/pci0000:00/0000:00:04.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:07.0 -> ../../../devices/pci0000:00/0000:00:07.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:07.1 -> ../../../devices/pci0000:00/0000:00:07.1
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:08.0 -> ../../../devices/pci0000:00/0000:00:08.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:08.1 -> ../../../devices/pci0000:00/0000:00:08.1
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:14.0 -> ../../../devices/pci0000:00/0000:00:14.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:14.3 -> ../../../devices/pci0000:00/0000:00:14.3
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:18.0 -> ../../../devices/pci0000:00/0000:00:18.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:18.1 -> ../../../devices/pci0000:00/0000:00:18.1
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:18.2 -> ../../../devices/pci0000:00/0000:00:18.2
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:18.3 -> ../../../devices/pci0000:00/0000:00:18.3
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:18.4 -> ../../../devices/pci0000:00/0000:00:18.4
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:18.5 -> ../../../devices/pci0000:00/0000:00:18.5
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:18.6 -> ../../../devices/pci0000:00/0000:00:18.6
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:18.7 -> ../../../devices/pci0000:00/0000:00:18.7
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:19.0 -> ../../../devices/pci0000:00/0000:00:19.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:19.1 -> ../../../devices/pci0000:00/0000:00:19.1
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:19.2 -> ../../../devices/pci0000:00/0000:00:19.2
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:19.3 -> ../../../devices/pci0000:00/0000:00:19.3
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:19.4 -> ../../../devices/pci0000:00/0000:00:19.4
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:19.5 -> ../../../devices/pci0000:00/0000:00:19.5
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:19.6 -> ../../../devices/pci0000:00/0000:00:19.6
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:00:19.7 -> ../../../devices/pci0000:00/0000:00:19.7
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:01:00.0 -> ../../../devices/pci0000:00/0000:00:01.1/0000:01:00.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:01:00.1 -> ../../../devices/pci0000:00/0000:00:01.1/0000:01:00.1
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:01:00.2 -> ../../../devices/pci0000:00/0000:00:01.1/0000:01:00.2
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:02:00.0 -> ../../../devices/pci0000:00/0000:00:01.1/0000:01:00.2/0000:02:00.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:02:01.0 -> ../../../devices/pci0000:00/0000:00:01.1/0000:01:00.2/0000:02:01.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:02:02.0 -> ../../../devices/pci0000:00/0000:00:01.1/0000:01:00.2/0000:02:02.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:02:03.0 -> ../../../devices/pci0000:00/0000:00:01.1/0000:01:00.2/0000:02:03.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:02:04.0 -> ../../../devices/pci0000:00/0000:00:01.1/0000:01:00.2/0000:02:04.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:02:09.0 -> ../../../devices/pci0000:00/0000:00:01.1/0000:01:00.2/0000:02:09.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:05:00.0 -> ../../../devices/pci0000:00/0000:00:01.1/0000:01:00.2/0000:02:02.0/0000:05:00.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:08:00.0 -> ../../../devices/pci0000:00/0000:00:01.1/0000:01:00.2/0000:02:09.0/0000:08:00.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:09:00.0 -> ../../../devices/pci0000:00/0000:00:01.3/0000:09:00.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:09:00.1 -> ../../../devices/pci0000:00/0000:00:01.3/0000:09:00.1
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:0a:00.0 -> ../../../devices/pci0000:00/0000:00:03.1/0000:0a:00.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:0a:00.1 -> ../../../devices/pci0000:00/0000:00:03.1/0000:0a:00.1
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:0b:00.0 -> ../../../devices/pci0000:00/0000:00:07.1/0000:0b:00.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:0b:00.2 -> ../../../devices/pci0000:00/0000:00:07.1/0000:0b:00.2
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:0b:00.3 -> ../../../devices/pci0000:00/0000:00:07.1/0000:0b:00.3
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:0c:00.0 -> ../../../devices/pci0000:00/0000:00:08.1/0000:0c:00.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:0c:00.2 -> ../../../devices/pci0000:00/0000:00:08.1/0000:0c:00.2
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:0c:00.3 -> ../../../devices/pci0000:00/0000:00:08.1/0000:0c:00.3
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:40:00.0 -> ../../../devices/pci0000:40/0000:40:00.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:40:00.2 -> ../../../devices/pci0000:40/0000:40:00.2
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:40:01.0 -> ../../../devices/pci0000:40/0000:40:01.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:40:01.1 -> ../../../devices/pci0000:40/0000:40:01.1
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:40:02.0 -> ../../../devices/pci0000:40/0000:40:02.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:40:03.0 -> ../../../devices/pci0000:40/0000:40:03.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:40:03.1 -> ../../../devices/pci0000:40/0000:40:03.1
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:40:04.0 -> ../../../devices/pci0000:40/0000:40:04.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:40:07.0 -> ../../../devices/pci0000:40/0000:40:07.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:40:07.1 -> ../../../devices/pci0000:40/0000:40:07.1
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:40:08.0 -> ../../../devices/pci0000:40/0000:40:08.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:40:08.1 -> ../../../devices/pci0000:40/0000:40:08.1
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:41:00.0 -> ../../../devices/pci0000:40/0000:40:01.1/0000:41:00.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:42:00.0 -> ../../../devices/pci0000:40/0000:40:03.1/0000:42:00.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:42:00.1 -> ../../../devices/pci0000:40/0000:40:03.1/0000:42:00.1
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:43:00.0 -> ../../../devices/pci0000:40/0000:40:07.1/0000:43:00.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:43:00.2 -> ../../../devices/pci0000:40/0000:40:07.1/0000:43:00.2
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:43:00.3 -> ../../../devices/pci0000:40/0000:40:07.1/0000:43:00.3
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:44:00.0 -> ../../../devices/pci0000:40/0000:40:08.1/0000:44:00.0
   lrwxrwxrwx 1 root root 0 Aug  3 09:02 0000:44:00.2 -> ../../../devices/pci0000:40/0000:40:08.1/0000:44:00.2
   

   *autistic screeching*

6. Did you remember to over-ride that GPU driver?...no...Return to Step 1.

7. DO YOU EVEN HUGEPAGES!?!

8. Let's make a RAMDisk so no matter what you're doing your system is taking up >32GB of RAM at all times. Weather the VM is running or not...

9. Check your system kernel to make sure all of our driver over-rides are in place at boot. Are they? No? Have fun figuring out how to add them manually. 

10. Alright I'll let you use the GUI for 5 mins.

11. MORE CLI!!!!!!!!

12. I should mention at any point in this guide if you f*** up a system file you may render your desktop unusable and will be forced to re-install Linux. I told you that before we started right?

13. You should pin your vcpus to your real CPUs in association to the NUMA Node closest to the GPU you're passing though. How do I do that? Hmmmn...good question. Google it!

14. You know that thing you did in Step 3? Well, you f***ed it up. FIX IT!

15. I don't even know what to have you do now. You're suppose to be HERE but you're over THERE. When you do this THAT'S suppose to happen but THIS happens instead. You're on your own. Google is your friend

16. Did Google help? No? START COMPLETELY OVER! RE-INSTALL LINUX!
17. Time to setup Looking Glass yay!...I'm not going to tell you how to do that either get Googling.
18. Does it start? Yes. Does it say your GPU is there? Yes. Does it crash? Only a little. YOU DID IT!!!

So after all of that I have a near native performance Windows install in a QEMU VM with 1/2 the systems resources accross the board. What else is there to do with this much CPU/RAM? The reason the passed-though GPU has an HDMI cable is because (and I think this is stupid) on the AMD GPU side every GPU AMD made after the Radeon HD 7XXX series (beginning of R7/9) AMD changed how their GPUs respond when no monitor is connected. The older GPU's would keep outputting to the display even when one wasn't there. Now, post 7XXX series the GPU effectively shuts down some major functions unless a display is detected. This breaks Looking Glass. It relies on a stream being output to a "monitor" so one must be plugged in.

 

Now the dummy plugs you use to see AMD GPU crypto miners using should do the trick just as well but I don't know how to make one so I just sacrificed a port on my monitor. If Looking Glass ever stops working while I'm in the middle of something it'll help as an emergency backup.

 

Now even right now this isn't working entirely as it was meant to. Looking Glass is suppose to act as a KVM accepting mouse & keyboard input. Keyboard worked fine for me. Mouse didn't work for jacks***. This is why I passed though one of my USB3.0 controllers. I just so happen to have a 2 system KVM switch. All I have to do is press a button and it will switch me in-between the Linux & Windows systems. Not 100% what I wanted but it works well enough. This enables me to overlap Linux windows on top of Looking Glass in the event I need more Linux desktop at a given time. Makes it so I don't have to dedicate a display to Windows.

 

Fun project, like someone mentioned there were a lot of online resources. The most detailed guide I followed was designed for Pop OS 19.04 (debian) but it wasn't hard to adapt to Ubuntu 19.04 (debain). I'm going to use this for some applications that I just cant get away from. Wine is not great for most of the .exe Windows applications I want to run and I have yet to find Linux alternatives for all of them. This will let me have the best of both worlds.

 

@Twilight I did it!