Help with upgrading my Linux machine for VFIO

[SebiAi]

Budget (including currency): 1500€ - I am willing to go a little bit (max. 100€-150€) beyond that budget for more performance

Country: Austria

Games, programs or workloads that it will be used for: Programming/Game Dev, Web surfing, 1440p High Gaming (Minecraft, GTA V, Simulators like Euro Truck Simulator 2), VR (Beatsaber, Boneworks, ...) - I really like high definition gaming

Existing PC:

Quote

PCPartPicker Part List

CPU: Intel Core i7-9700K 3.6 GHz 8-Core Processor
CPU Cooler: Noctua NH-D14 64.95 CFM CPU Cooler
Motherboard: MSI MPG Z390 GAMING EDGE AC ATX LGA1151 Motherboard
Memory: G.Skill Ripjaws V 32 GB (2 x 16 GB) DDR4-3600 CL18 Memory
Storage: Samsung 970 Evo 500 GB M.2-2280 PCIe 3.0 X4 NVME Solid State Drive
Storage: SanDisk Ultra 3D 2 TB 2.5" Solid State Drive
Video Card: MSI GAMING GeForce GTX 1070 Ti 8 GB Video Card
Case: be quiet! Pure Base 600 ATX Mid Tower Case
Power Supply: Enermax EMG700AWT 700 W 80+ Gold Certified Fully Modular ATX Power Supply
Monitor: Samsung LC24FG73 23.5" 1920 x 1080 144 Hz Curved Monitor
Monitor: LG 27GN800-B 27.0" 2560 x 1440 144 Hz Monitor
Total: €0.00
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2023-04-27 15:35 CEST+0200

 

Hi, I want to upgrade my current PC setup and I would like your help with it but I have some "special" requirements:

• I switched from Windows 10 to Linux (Linux Mint with Cinnamon) as my primary OS a year ago. I am currently using a dual boot setup with Windows 10.
  I would like to change this setup: New AMD Card for Linux and my old NVIDIA for a Windows VM (VFIO style + Evdev for peripherals + PulseAudio for sound).
  => This means I need:
  • An AMD Graphics Card so I can finally use Wayland
  • A high wattage Power Supply with maybe room for a GPU upgrade for the VM down the line (1000W to 1200W?)
  • A Motherboard which can handle two Graphics Cards regarding the PCIe Slots (preferably at least Gen 4 x8 on both PCIe slots for the Cards if something like this exists)
  • [OPTIONAL] USB PCIe Card with separate USB Controller for the VM
• RGB, WIFI and a tempered glass window in the case is not needed
• I still would like to reuse some of my old parts:
  • Peripherals
  • Monitors
  • Storage
  • Fans
  • GPU for my Windows VM
  • Maybe the Case - I would prefer a Case with a built in GPU support bracket
  • CPU Cooler??? - Not sure about that and if I would need an upgrade package for mounting

 

I tried my stab at configuring an AMD build myself with my local prices sourced from https://geizhals.at/ (mylemon as vendor most of the time):

Quote

PCPartPicker Part List

CPU: AMD Ryzen 7 7700X 4.5 GHz 8-Core Processor  (€326.19)
CPU Cooler: Noctua NH-D15 82.5 CFM CPU Cooler  (€89.90)
Motherboard: MSI PRO B650-P WIFI ATX AM5 Motherboard  (€200.69)
Memory: Corsair Vengeance 32 GB (2 x 16 GB) DDR5-5600 CL36 Memory  (€109.80)
Storage: Samsung 970 Evo 500 GB M.2-2280 PCIe 3.0 X4 NVME Solid State Drive  (Purchased For €0.00)
Storage: SanDisk Ultra 3D 2 TB 2.5" Solid State Drive  (Purchased For €0.00)
Video Card: MSI GAMING GeForce GTX 1070 Ti 8 GB Video Card  (Purchased For €0.00)
Video Card: Sapphire PULSE Radeon RX 6800 16 GB Video Card  (€527.79)
Case: Corsair 4000D Airflow ATX Mid Tower Case  (€95.69)
Power Supply: Corsair RM1000x (2021) 1000 W 80+ Gold Certified Fully Modular ATX Power Supply  (€189.87)
Total: €1539.93
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2023-04-27 17:02 CEST+0200

Feel free to throw this build over board and start fresh.

 

I am very grateful for any contributions.

[Pesukarhu]

I have been using primarily linux for 2,5 years now, with virtualized windows 10 for gaming. My specs are:
3900X overclocked, 32GB DDR4 ram, ASUS PRIME X570-PRO mobo, custom loop water cooling for cpu,
1060 6gb for linux, 2060 super for windows, two usb 3.0 cards for windows, 10gbe nic for connecting to nas,
2TB nvme ssd (pcie 3.0), 1TB sata ssd, 750W seasonic PSU, custom case,
two 50" 4k TVs as monitors (i like big text, and do more work than gaming)

Things to note:

-I previously had a motherboard which only had 4 lanes to the second pcie x16 slot. This was noticeably limiting the performance of the 2060 super, since it only supports pcie 3.0. Current mobo is "SLI compatible" and runs the two pcie x16 slots in x8 + x8 which solves the problem.
So you will probably need a X670 motherboard, or a secondary gpu that supports pcie 4. (3.0 at x8 = 4.0 at x4)

-Rather annoyingly the card in the top slot is always the primary gpu, and while you can configure xorg to use the second card for linux the primary GPU wouldnt display anything when attached to a VM. It was detected, but windows could not use it. There may be ways to fix/workaround, i could not find them.

-The top GPU will likely be starved for air unless the bottom GPU is mounted vertically with a riser. I modified a rather large old case to support vertical mounting a second GPU while providing sufficient airflow to it. There is around 30mm between the side panel and the GPU, which seems to be enough. Airflow in the case is going bottom-to-top, with the cpu radiator outside on the top of the case.

-Highly recommend a dedicated usb controller for windows, the VR-compatible pcie-cards are less than 30€. I have two because the first one did not work well with my valve index and i had space for two cards. I got them from amazon.de

 

-I have 4 usbs on the front panel, 2 for linux and 2 for windows. This has been exactly enough in my experience. There are 6 more for windows on the back of the pc, and however many the mobo has for linux.

 

-A cheap usb "switch" is used for mouse+keyboard. The non-mechanical ones seem to work better. Cost direct from china is something like 10-15€. I am currently using a 2in, 4out model.

 

-I use two inputs on the TVs for switching displays. I fortunately have an older remote with a dedicated HDMI button, so i just need to push one button to switch between linux and windows. Looking glass is cool, but i was too lazy to try setting it up.

-There is something not-smooth about using VR in the virtualized windows. Not sure what it is. Using native windows is better for VR applications. Although i think i played ETS2 in vr on linux and it was running stably. Of course a 1060 isnt fast enough so it was potato quality visually.

-I could not get pulseaudio to cooperate with QEMU/KVM and used a usb audio card instead for windows. I use an external mixer anyways so this was not a problem for me.

-Pulseaudio in general has only given me problems, I have been using pipewire instead for over a year now and it has gone smoothly after initial configuration.

-I did measure the power consumption under full load, but i forget what it was. I think i was almost disappointed by how low it was, though.

-I use samba to share folders between linux and windows. This is not optimal and i wanted to try other things but got lazy when looking into it. Other options seemed rather complicated for simultaneous access.

This took a while to write, hopefully it is even partially useful
I can provide more details on the running config tomorrow if needed.
Your upgrade seems reasonable, i would recommend a motherboard with better suited pcie though.

[SebiAi]

1 hour ago, Pesukarhu said:

This took a while to write, hopefully it is even partially useful

I can provide more details on the running config tomorrow if needed.
Your upgrade seems reasonable, i would recommend a motherboard with better suited pcie though.

Thank you very much for your insight on this topic. I have realized that I this whole VFIO thing is not very beginner friendly but I will try it anyways and if everything goes wrong I can still revert back to dual booting.

Regarding the PCIe motherboard choice:

1 hour ago, Pesukarhu said:

-I previously had a motherboard which only had 4 lanes to the second pcie x16 slot. This was noticeably limiting the performance of the 2060 super, since it only supports pcie 3.0. Current mobo is "SLI compatible" and runs the two pcie x16 slots in x8 + x8 which solves the problem.
So you will probably need a X670 motherboard, or a secondary gpu that supports pcie 4. (3.0 at x8 = 4.0 at x4)

Because the 6800 is a PCIe 4.0 x16 and the 1070 TI is a PCIe 3.0 x16 that means I would need a PCIe 4.0 x16 and a PCIe 4.0 x8 slot on the motherboard? Do such motherboards exist or do I need to get one with two PCIe x8 slots and sacrifice a bit of performance of the 6800?

To ask my question more precisely: What exact PCIe configuration would I need to get the most performance out of both of my cards?
As you might have noticed I am not familiar with PCIe lanes .

 

As an example take the GIGABYTE X670 AORUS Elite AX:

According to the manual this motherboard is not suitable because I don't have a full PCIe 3.0 x16 slot for my 1070 TI (4.0 x4 => 3.0 x8). Is my brain braining right?

[SebiAi]

To address the other points you mentioned:

3 hours ago, Pesukarhu said:

-Rather annoyingly the card in the top slot is always the primary gpu, and while you can configure xorg to use the second card for linux the primary GPU wouldnt display anything when attached to a VM. It was detected, but windows could not use it. There may be ways to fix/workaround, i could not find them.

Should not be a problem for me because I plan to have the 6800 in the first slot for Linux and the 1070 TI in the second.

 

4 hours ago, Pesukarhu said:

-The top GPU will likely be starved for air unless the bottom GPU is mounted vertically with a riser. I modified a rather large old case to support vertical mounting a second GPU while providing sufficient airflow to it. There is around 30mm between the side panel and the GPU, which seems to be enough. Airflow in the case is going bottom-to-top, with the cpu radiator outside on the top of the case.

Yea, I'm going to try it normal (both horizontally) and if one of the GPUs suffocates I'll look into solutions then, I guess. Or I'll choose a case which has enough space to vertically mount the second GPU.

 

4 hours ago, Pesukarhu said:

-Highly recommend a dedicated usb controller for windows, the VR-compatible pcie-cards are less than 30€. I have two because the first one did not work well with my valve index and i had space for two cards. I got them from amazon.de

 

-I have 4 usbs on the front panel, 2 for linux and 2 for windows. This has been exactly enough in my experience. There are 6 more for windows on the back of the pc, and however many the mobo has for linux.

 

-A cheap usb "switch" is used for mouse+keyboard. The non-mechanical ones seem to work better. Cost direct from china is something like 10-15€. I am currently using a 2in, 4out model.

Yup, I'll get a dedicated PCIe USB controller card thingy - for the peripherals I'll try to use the method provided in the Arch guide: Passing keyboard/mouse via Evdev

If this does not work I'll use the USB switcher

 

4 hours ago, Pesukarhu said:

-I use two inputs on the TVs for switching displays. I fortunately have an older remote with a dedicated HDMI button, so i just need to push one button to switch between linux and windows. Looking glass is cool, but i was too lazy to try setting it up.

I've looked a little bit into looking glass but I does not support HDR (duh, both Xorg and Wayland don't even have support as far as I am aware). Sadly both my monitors only have one DisplayPort port => I'll use two DP Switch I guess (about 25€+ each) or a KVM switch box which also has usb and audio.

 

4 hours ago, Pesukarhu said:

-I could not get pulseaudio to cooperate with QEMU/KVM and used a usb audio card instead for windows. I use an external mixer anyways so this was not a problem for me.

-Pulseaudio in general has only given me problems, I have been using pipewire instead for over a year now and it has gone smoothly after initial configuration.

I personally only had a few minor problems with PulseAudio but I have heard that PipeWire is somewhat better than PulseAudio and also better supported by audio related applications (correct me if I'm wrong, please). I will need to do some distro hopping to find the perfect match for me with Wayland and PipeWire. Do you have any recommendations regarding distros @Pesukarhu (or anybody else)?

[Pesukarhu]

21 hours ago, SebiAi said:

To ask my question more precisely: What exact PCIe configuration would I need to get the most performance out of both of my cards?

The 6800 seems to be around the same performance as a 2080ti, which only had pcie 3.0 x16. Based on this, there shouldnt be any performance limitation when running the 6800 at pcie 4.0 x8. The speed doubles with each generation, so you only need half the lanes when using a newer generation.

The 1070ti should be just fine with 8 lanes, which it can use at 3.0 speeds.

So you want two x16 slots that are running at x8. The cheapest X670 board i found with them was 540€
For intel a Z790 with the slots the cheapest was 570€. For refence, my X570 board was on sale for 220€ one year ago, regular price was 250€.

I actually dont know what to recommend, maybe get a regular mobo and upgrade the windows gpu later to one that supports pcie 4.0? The 1070 will work and game even with 4 lanes, just wont reach max performance in all scenarios. It depends very much on the game if there even is any performance loss.

[SebiAi]

13 minutes ago, Pesukarhu said:

The 6800 seems to be around the same performance as a 2080ti, which only had pcie 3.0 x16. Based on this, there shouldnt be any performance limitation when running the 6800 at pcie 4.0 x8. The speed doubles with each generation, so you only need half the lanes when using a newer generation.

The 1070ti should be just fine with 8 lanes, which it can use at 3.0 speeds.

So you want two x16 slots that are running at x8. The cheapest X670 board i found with them was 540€
For intel a Z790 with the slots the cheapest was 570€. For refence, my X570 board was on sale for 220€ one year ago, regular price was 250€.

I actually dont know what to recommend, maybe get a regular mobo and upgrade the windows gpu later to one that supports pcie 4.0? The 1070 will work and game even with 4 lanes, just wont reach max performance in all scenarios. It depends very much on the game if there even is any performance loss.

Ok! Thank you very much for the explanation.

 

I don't really know how exactly to filter the motherboards to find one that supports PCIe 4.0 x8 on two PCIe slots. I only found one (312,73€): ASUS ProArt B650-Creator

• PCPartPicker
• Geizhals

Would this one work?

How do I filter/search for such a specific requirement on PCPartPicker/Geizhals?

[Pesukarhu]

18 hours ago, SebiAi said:

Do you have any recommendations regarding distros

I use arch linux at home, linux mint at work. Mint is way easier to get up and running, however installing niche software is easier on arch since someone will have made an AUR package for it...
I have also installed centos on some production machines at work. Less packages are easily available so i dont recommend it for home use. I dont even know why we use it. One of my coworkers uses gentoo so it could be "worse" i suppose.

Arch has broken a few times after updating, once even needing a live usb to fix. On the other hand, updates with bugfixes are available very quickly.
Mint has been reliable and less scary to install updates. At this point, installing windows updates is scarier

I use cinnamon as the desktop environment, and it has its problems. I still prefer it since it is quite windows-like and familiar. Sometimes certain apps cause everything except the mouse cursor to freeze. Changing to a console tty and killing the offending app usually fixes it without closing other apps.

Also, one of the reasons i changed to pipewire is easyeffects - having a functional audio compressor is very nice.

[Pesukarhu]

6 minutes ago, SebiAi said:

I don't really know how exactly to filter the motherboards to find one that supports PCIe 4.0 x8 on two PCIe slots. I only found one (312,73€): ASUS ProArt B650-Creator
 

Would this one work?

How do I filter/search for such a specific requirement on PCPartPicker/Geizhals?

I couldnt figure out if there even is an option to filter specifically enough - however asus has these really convenient "catalogues": https://www.asus.com/microsite/motherboard/AMD-AM5-X670-B650/b650/download/ASUS-B650-Series-Specs.pdf
You can see that the B650-creator is the only b650 board from asus with dual x8 slots. It does seem like a good option, and not stupidly expensive.

[SebiAi]

19 minutes ago, Pesukarhu said:

Also, one of the reasons i changed to pipewire is easyeffects - having a functional audio compressor is very nice.

How did you switch to pipewire? Did you follow any tutorial on how to switch to pipewire on Linux Mint and if so can you link it?

[SebiAi]

12 minutes ago, Pesukarhu said:

I couldnt figure out if there even is an option to filter specifically enough - however asus has these really convenient "catalogues": https://www.asus.com/microsite/motherboard/AMD-AM5-X670-B650/b650/download/ASUS-B650-Series-Specs.pdf
You can see that the B650-creator is the only b650 board from asus with dual x8 slots. It does seem like a good option, and not stupidly expensive.

Oh, that catalogue is really nice!

According to AMD the B650 chipset and up can support 2x8 => just need to find one from a vendor

[Pesukarhu]

53 minutes ago, SebiAi said:

How did you switch to pipewire? Did you follow any tutorial on how to switch to pipewire on Linux Mint and if so can you link it?

I still use pulseaudio on the work pc. Works fine for listening to internet radio.
For the home pc i followed the instructions on arch wiki, it was a drop-in replacement. Pavuaudio still works for volume control etc. qpwgraph is useful for routing audio in weird ways should that be needed.