Is Intels Virtualization (vt-d / vt-x) and PCI-passthrough faster than AMDs (V / Vi)?

[Sereiya]

After talking about my upcoming new PC build a friend told me that Intels Virtualization technology (Vt-d / Vt-X) is way faster than AMDs (V / Vi), and that if I want to pass a PCIe device through to a virtual Win10 guest system, I would get a hefty performance penalty on AMD because their Ryzens are not designed to directly access/control IOMMU groups and that I HAVE to get an Intel system to avoid those problems. There were some numbers during the conversation, ranging between "about 90% of possible fps on AMD, maybe less" and "at least 30% performance loss on AMD".

 

Though I find one forum post on tomshardware with the "best answer" claiming Intel is "better" with IOMMU, I can't find any reliable data on that topic, especially if it comes to PCI-passthrough. I never heard about any noticable performance differences between team Red and team Blue regarding virtualization myself, which is why I planed to go for a Ryzen 7 / X570 based system on Arch Linux, including a Win10 guest system with a 5700 XT passed through for games.

 

Hope you can help me out here, I really don't want to buy unsuitable hardware.

[mathew7]

Sorry, I don't know about AMD, but before I got my 1070, I played with VT-d on Radeons 5950 and 7850. GPU performance was within 1%, CPU performance was only affected because turbo-boost was not working (Xen 4.0-4.2). Oh...and guest was only W7.

I got to this post because I wait for my new 5700XT and will want to resume that setup on my 8700K. Just browsing if anybody posted a 5700 story.

 

PS: If you want to game under VM, be ready for other-than-GPU headaches:

 - USB: passing by software has it's issues (which I see constantly with non-VT-d virtual machines), but  passing controller with VT-d means you need 2 controllers (for guest and host). And good luck finding one that works good under VT-d or linux. Back when I did it, sandy/ivy-bridge chipsets had 2 USB2 controllers (for Win7 installation ?!?!) and 1 USB3, with USB3 taking over all devices if driver was loaded. I just disabled USB3 in BIOS, and could find each HW port to which USB2 controller went. Now (since 6th gen) they only have 1 USB3, thus an external controller is mandatory and it's hard finding a chip that works either in VT-d or on linux (tried some of those, and the best one would fail with racing wheel FFB).

 - sound card: again I've had big issues, best PCI(e) cards were VIA-based (Hercules Fortissimo IV) and CM-based (Xonar DGX). Creative drivers would lock the VM and host. But now I have a USB (Creative), thus I expect only USB problems to be had/solved.

 - LAN: this may be minor, but I've had problems with XENs virtual device...had to pass a PCI/PCIe card, which I bought an Intel (I don't recall Realtek results and why I went Intel)

My current setup is MiniITX, so the USB problem is major...have to try thunderbot VT-d on laptop to see if I should bother buying another MiniITX that has thuderbolt port.

[Sereiya]

Thanks for answering.

I actually already got my machine to work properly. It's based on Gigabytes X570 Aorus Pro, the Ryzen 7 3700X, a 5700 XT for passthrough, a GTX 960 for host ('cause I already had that one - have to use the nouveau driver though because of that pesky freeze- / stutterproblem caused by Nvidia's driver which still gave me bad performance after deleting the corner-option in my system settings) and plenty of RAM.

I either was incredibly lucky, AMDs driver support for Linux is way better than Intels (which isn't really much of a secret, is it?) or things improved a lot since you built your system. I don't want to assume Vt-d being worse than AMD-V at this point since I can't test it.

I'm passing through my (only) mouse and keyboard, making use of some qemu commandline options and the hardcoded CTRL + CTRL hotkey for switching from host to guest and vice versa. I also passthrough my sound this way and let qemu send it directly to pulseaudio almost without any latency (it shows up as just another program making sound). It has to match pulseaudio's standard settings, of course. (in case you wonder: using arguments (arg) instead of environment variables (env) is QEMUs new standard, you don't need to export those variables on your host anymore. Still worked great though.)

For the switchable passthrough of my input devices to work properly I had to add two VirtIO devices before the standard PS/2 devices so the guest gets something more reliable to work with.

<domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">
[...]
    <disk type="block" device="disk">
      <driver name="qemu" type="raw" cache="none" io="threads"/>
      <source dev="/dev/sdc"/>
      <target dev="sdc" bus="sata"/>
      <shareable/>
      <address type="drive" controller="0" bus="0" target="0" unit="2"/>
    </disk>
[...]
    <input type="mouse" bus="virtio">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x0e" function="0x0"/>
    </input>
    <input type="keyboard" bus="virtio">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x0f" function="0x0"/>
    </input>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
[...]
<qemu:commandline>
    <qemu:arg value="-object"/>
    <qemu:arg value="input-linux,id=mouse1,evdev=/dev/input/by-id/usb-A....._G3-event-mouse"/>
    <qemu:arg value="-object"/>
    <qemu:arg value="input-linux,id=kbd1,evdev=/dev/input/by-id/usb-CM_Storm_Keyboard_--_Trigger_Z_gaming-event-kbd,grab_all=on,repeat=on"/>
    <qemu:arg value="-audiodev"/>
    <qemu:arg value="id=pa,driver=pa,in.fixed-settings=on,in.frequency=48000,in.format=s32,in.channels=2,server=/run/user/1001/pulse/native"/>
  </qemu:commandline>
</domain>

It's also highly advisable to set the EA-modes of every virtual hard drive to "threads", otherwise QEMU will try to process all the I/O in the same thread, possibly causing input lag. I'm not sure if it's actually necessary if you pass your HDD through rawly (directly pointing at /dev/sdx), but it had no impact on its performance, so I did it anyway.

 

The passthrough also worked quite well, almost every single device on the X570 Aorus Pro got their own IOMMU group. AMD / Sapphire even put the HDMI audio device on their 5700 XT in an own group for whatever reason - never tinkered with that though. As far as I could see the board also got at least 2 different USB controllers, if not 3. Since there are three 3.1 (one being USB-C), three 3.0 and four 2.0 USB ports on the back panel with 4 front USB headers (2x 3.0) onboard, there surely is at least one controller I could passthrough. Didn't try though since I got everything I want in that Windows machine. Since the GTX 960 also appears as a device with their own IOMMU group (together with its audio device though) I can confirm that the first two PCIe 16x slots (running 2x 8-lanes) of this board are definitely working great for gaming VMs. the GTX 960 in the second slot showed up as PCIe 1.0 x8 in UEFI when I configured it, which was kinda weird - but didn't annoyed me too much since the 5700 XT gets 4.0 x8 being the high-performance card in this system.

The thing I definitely know I was lucky was the network working with their standard settings being set to macvtap (device e1000e, source-mode bridged). It just worked out of the box.
I only had to use pci-passthrough for the GPU. Everything else worked (after some tweaking) using vfio, libvirtd, qemu/kvm, virt-manager. My distro of choice was Arch Linux, using their kernel how it comes.
The only problem that is currently unfixable and somewhat annoying is the inability to shutdown and restart the virtual machine because of the "Unknown PCI header type 127" kernel/driver bug with AMD GPUs. Once the VM runs I usually don't shut it down, having Win10 configured to do as less as I can command it to do. My Ryzen is giving me more than enough power to do so until gnif got his patch working for my card.
Performancewise I can't really tell the difference to my native Win10 machine. Games like PUBG, Trackmania 2, NieR:Automata, GTA5 or FAF work flawlessly. I even get my ~120fps @ 3840x2400 in Rocket League using VSR (1920x1200 16:10 display). Didn't run benchmarks yet, but from everything I see I can't tell the difference until I see the 7 cores and 14 threads of a "3700X Eight-Core Processor" in Windows' Taskmanager.

So... yeah, works great for me. I actually started writing on a full Beginner's Guide for all of this since it worked so well for me, from picking your parts to having the Win10 Gaming VM running on a self-configured Arch Linux. Will take a while to complete that though, since I aim to explain every single step & command so people new to Linux actually know what they're copy-pasting.

Edited September 9, 2019 by Sereiya
Forgot to mention VM's performance