Does anyone have a successful VFIO/IOMMU GPU Passthrough here?

[Cyberspirit]

I'm really interested in this concept but, I'm at the point where I'm frustrated with the setup so, I'd like to know if anyone here was able to set it up and use it properly as a daily driver.

What distro do you use and what guide did you follow to actually make it work?

What are your experiences with the setup of vfio?

[Electronics Wizardy]

Yep I have one currently working.

 

Fedora 28

 

 

amd fx 8350 and a gigabyte 990fx board.

 

Just disable the driver of gpu you want to pasthough

Enable iommu in grub.

then make the vm.

 

Not really much to it. No real issues here.

 

 

[Cyberspirit]

Just now, Electronics Wizardy said:

Yep I have one currently working.

 

Fedora 28

 

 

amd fx 8350 and a gigabyte 990fx board.

 

Just disable the driver of gpu you want to pasthough

Enable iommu in grub.

then make the vm.

 

Not really much to it. No real issues here.

 

 

Oh, nice. How does it run compared to a normal Windows install?

I personally got to the point where everything should be set up but it just doesn't work but, I'll try to figure that out later.

I just want to know if all this frustration is even worth it in the end.

[Electronics Wizardy]

1 minute ago, Cyberspirit said:

Oh, nice. How does it run compared to a normal Windows install?

I personally got to the point where everything should be set up but it just doesn't work but, I'll try to figure that out later.

I just want to know if all this frustration is even worth it in the end.

It runs fine, no real issues. Mainly for fun though, I have many other systems for windows and gaming.

 

What issues do you have?

[Cyberspirit]

Just now, Electronics Wizardy said:

It runs fine, no real issues. Mainly for fun though, I have many other systems for windows and gaming.

 

What issues do you have?

Well I'm running Manjaro and I set up iommu and I disabled the drivers so, the GPU doesn't output to my monitor which is good.

My first issue is that I couldn't set the VM to UEFI mode and most likely because of that my GPU is only partially detected.

[Ashleyyyy]

thanks to @Electronics Wizardy i do have it working. 

 

i have 2 Xeon X5650's - 12 core's (6 core's for the VM).

24GB of ram (12GB for the VM).

R9 290X for the VM.

 

i only play Overwatch on that VM and my fps only drops into the 50's and 40's if there's a lot on screen...

 

whenever there are a lot of characters on screen i get those fps drops, but that could be because of a load of different things. i've been tweaking this setup for a few weeks and i got it to run a lot better. when i first set it up Overwatch was practically unplayable. it would lag with a lot less stuff on screen. 

 

@Electronics Wizardy recommended that i pinned my cpu core's and i haven't figured out how to do that yet so it might be because i need to do that still...

 

 

[Electronics Wizardy]

4 hours ago, Cyberspirit said:

Well I'm running Manjaro and I set up iommu and I disabled the drivers so, the GPU doesn't output to my monitor which is good.

My first issue is that I couldn't set the VM to UEFI mode and most likely because of that my GPU is only partially detected.

what do you mean you can't set your vm to uefi mode? Thats a option at inital setup for a vm, any not related to passthrough at all.

 

Is the iommu groups on your board good?

 

[Cyberspirit]

1 hour ago, Electronics Wizardy said:

what do you mean you can't set your vm to uefi mode? Thats a option at inital setup for a vm, any not related to passthrough at all.

 

Is the iommu groups on your board good?

 

I just switched to Fedora so, we'll see if it works better there.

 

As for the UEFI mode, I've read online that newer GPUs need that. Maybe I misinterpreted it?

The groups were good and, I successfully isolated my GPU and I was able to add it to the VM but, it seemed like the VM was using a stock renderer instead of my card even though Windows did recognize the actual card.

[Electronics Wizardy]

9 minutes ago, Cyberspirit said:

I just switched to Fedora so, we'll see if it works better there.

 

As for the UEFI mode, I've read online that newer GPUs need that. Maybe I misinterpreted it?

The groups were good and, I successfully isolated my GPU and I was able to add it to the VM but, it seemed like the VM was using a stock renderer instead of my card even though Windows did recognize the actual card.

what are you using for vms? In fedora id use virt-manager.

 

What does your vm setup look like? Screenshot?

 

 

[Cyberspirit]

13 minutes ago, Electronics Wizardy said:

what are you using for vms? In fedora id use virt-manager.

 

What does your vm setup look like? Screenshot?

 

 

Sorry for wasting your time I'm reinstalling Fedora at the moment because something apparently went wrong with it and it kept crashing at startup.

[jde3]

I had one based on Gentoo without systemd.

 

What I wanted is I wanted a system that could play two copies of a game and stream one copy over Steam link to a system that was unable to play it.

 

Host

Ryzen 7 (first gen)

RX580 primary GPU

R9270x slave GPU

B350 Mobo

ZFS ZVOL pool backed storage.

 

I had to spend a long time tuning KVM's processor layout so that it would properly report the Ryzen 7 to the guest with 4 cores and 8 threads (half the CPU) and not show up as an opteron. I also set CPU pinning to IO threads. Using two AMD cards was a challenge but I was able to disable the slave cards PCI bus via a kernel boot option so that the AMDGPU module would only attach to the primary card. I used synergy to mouse between the two system.

 

This worked and accomplished the task just fine.. the game played perfectly or at least well enough but..

 

What I noticed about this is that processes on the host would stomp on the guest. The host still has access to all it's CPU's even tho some are shared to the guest and if the host decides to put a task on one of the guest cores the guest performance will tank till the host lets go of that core. The only way to fix this is with Linux cgroups and assign the remaining nonshared cores to your user but I gave up at that point because of the complexity of setting up cgroups.. I could have done this but it was A LOT more work than I wanted to do.. So in the end if you run a portage build on the host.. forget about using the guest till it's done basically... and windows would frequently go to the sad place every time a multi threaded process kicked off on the host.. and that is kind of the point of having a Ryzen so...

 

In the end I just nuked it because.. where as it's a fun toy, it's not really that useful. It is not the panacea of cross platform gaming people think it is.. and it still requires windows meaning it doesn't help support native linux game development.

[Ashleyyyy]

19 hours ago, Cyberspirit said:

I successfully isolated my GPU and I was able to add it to the VM but, it seemed like the VM was using a stock renderer instead of my card even though Windows did recognize the actual card.

you do realize you need to plug in a display to the GPU the VM has...? it will only work with a display connected to the card itself.

 

i recommend plugging in a cable from the card to your display and just changing the input on the display when you want to use Windows. 

 

EDIT: there is a thing called looking glass, which Wendell and Linus demonstrated in their video, but i haven't tried it so i can't tell you how difficult it is to make it work. 

link to the video:

 

[Cyberspirit]

20 hours ago, firelighter487 said:

you do realize you need to plug in a display to the GPU the VM has...? it will only work with a display connected to the card itself.

 

i recommend plugging in a cable from the card to your display and just changing the input on the display when you want to use Windows. 

 

EDIT: there is a thing called looking glass, which Wendell and Linus demonstrated in their video, but i haven't tried it so i can't tell you how difficult it is to make it work. 

link to the video:

 

Yeah, I was able to get it working partially but, I think I'll postpone the project until I get a new SSD. I'm tired of reinstalling Windows whenever something breaks.

Also, the GPU was connected to the monitor and, it did output to it but some config was probably messed up (definitely my fault) so, it didn't detect the card completely.

(Meaning: Windows saw the card's name and all that but, it couldn't detect how much VRAM it had and because of that benchmarks were crashing)

[Guest]

On 1/9/2018 at 7:47 PM, Cyberspirit said:

Well I'm running Manjaro and I set up iommu and I disabled the drivers so, the GPU doesn't output to my monitor which is good.

My first issue is that I couldn't set the VM to UEFI mode and most likely because of that my GPU is only partially detected.

Hi, don't worry it is not that hard. The main fact is that even the Arch Wiki is not totally complete and there is no clear and easy documentation about PCI pass through-ing about some issues that can occur.
You are probably missing the VBIOS for your GPU.
Also, using an UEFI bios is required...

[Cyberspirit]

6 hours ago, Lukyp said:

Hi, don't worry it is not that hard. The main fact is that even the Arch Wiki is not totally complete and there is no clear and easy documentation about PCI pass through-ing about some issues that can occur.
You are probably missing the VBIOS for your GPU.
Also, using an UEFI bios is required...

Huh? I had no idea that you needed the VBIOS. That might actually be the problem then. 

[Guest]

11 hours ago, Cyberspirit said:

Huh? I had no idea that you needed the VBIOS. That might actually be the problem then. 

It is required in some cases, with the same GPU (GTX 1060) and an intel CPU I had no problems, but when I switched to ryzen a VBIOS was required.

[Cyberspirit]

On 2018. 09. 07. at 4:41 PM, Lukyp said:

It is required in some cases, with the same GPU (GTX 1060) and an intel CPU I had no problems, but when I switched to ryzen a VBIOS was required.

I'll look into it more next month. Most likely will pick up a new SSD because dual booting had some weird issues. 

 

Also, I just noticed that you are new here.

Welcome to the forum!

Edited September 8, 2018 by Cyberspirit