Passing through multi-channel USB card to multiple VMs

[AJMansfield]

I'm putting together a computer for flashing firmware via USB onto some custom hardware I'm manufacturing.

Unfortunately, the flashing software will only run on windows XP, and I can only flash one board at a time with it, but otherwise it takes only a tiny bit of memory and processing.

To solve this, I'm considering this build:

 

SuperMicro SYS-5019A-FTN4 : https://www.newegg.com/Product/Product.aspx?Item=9SIA5EM67J7117

1x Hynix 8GB DDR4-2400 ECC UDIMM : https://www.newegg.com/Product/Product.aspx?Item=9SIA5EM62K8176 (validated)

1x Samsung SM961 128GB NVMe SSD : https://www.newegg.com/Product/Product.aspx?Item=9SIA5EM5Z40944 (validated)

1x SuperMicro PCIe x8 riser : https://www.newegg.com/Product/Product.aspx?Item=9SIA5EM2JS8113

1x StarTech 4-port Quad Bus PCIe USB 3.0 Adapter : https://www.newegg.com/Product/Product.aspx?Item=9SIAB274849872

 

I'd run unRAID on it, and inside it run four Windows XP virtual machines.

Due to a bug with QEMU or libusb, I can't just use normal USB passthrough to connect the boards to the virtual machines though, which is what the 4-port quad-channel USB card is for - the hope is that it will be possible to pass through the USB controllers to separate virtual machines.

In theory this should be possible - the Atom C3758 processor and motherboard support VT-d - but in practice it depends on whether the CPU/USB card supports ACS for the separate channels, whether it's possible to map the functions on that card to separate IOMMU groups.

 

Does anyone have any any experience with Intel Atom C3758 or multi-channel USB cards who would know if this kind of virtualization setup is possible?

[brwainer]

I haven't done it yet, but I have plans to use this card: https://www.amazon.com/dp/B00XPUHO10/_encoding=UTF8?coliid=I1BRFWFKYQ628B&colid=1R2PD9HY1JUJ7

 

With some help/guidance from these pages:
https://www.redhat.com/archives/vfio-users/2016-June/msg00103.html

https://www.redhat.com/archives/vfio-users/2016-July/msg00013.html

https://www.redhat.com/archives/vfio-users/2016-July/msg00060.html

 

Note that there may be kernel, qemu, etc changes since June/Jluy 2016 that makes these not valid anymore. I did my research into this topic a few months ago, I didn't really find any other cases of people talking about splitting a multi-usb-chip card into multiple VMs. There is some more discussion if you look at the other messages in this same email thread.

 

I'll be looking forward to hearing what you find out and your experiences!

[AJMansfield]

Well, I've purchased the parts, hopefully it works for me.

Especially thanks to the links to those threads; gives me a much better idea how I need to set the system up.

[AJMansfield]

It works! At least with that card and that motherboard, the USB controllers end up in separate IOMMU groups and can be passed through to separate VMs.

[brwainer]

26 minutes ago, AJMansfield said:

It works! At least with that card and that motherboard, the USB controllers end up in separate IOMMU groups and can be passed through to separate VMs.

Great! You got the Sonnet card? Or the Startech? Supposedly both work well with ACS/IOMMU, but you didn’t say which you had used.

[AJMansfield]

The StarTech card. Although actually now I'm having more trouble with it, for some reason QEMU crashes whenever I try to install the driver in Windows XP.

[brwainer]

6 minutes ago, AJMansfield said:

The StarTech card. Although actually now I'm having more trouble with it, for some reason QEMU crashes whenever I try to install the driver in Windows XP.

Did you do the normal steps to make sure that the "driver" reported is vfio-pci when running "lspci -vvvv"? because if it is grabbed by the normal USB drivers, then you can't pass it through to any VMs, it is being used by the host as USB card

[AJMansfield]

On 11/21/2017 at 6:41 PM, brwainer said:

Did you do the normal steps to make sure that the "driver" reported is vfio-pci when running "lspci -vvvv"? because if it is grabbed by the normal USB drivers, then you can't pass it through to any VMs, it is being used by the host as USB card

Yes I did all that, it wasn't the passthrough or linux side that was the issue. It passed through, but the VM would crash (QEMU would go into a paused state with a vfio error) every time I tried to install the Windows XP drivers for it.

I also just tried the Sonnet Allegro Pro, but for some reason passing it through interferes with being able to boot from a VirtIO disk.

 

However, I was able to find a card that does work: the RocketU 1144D.

I had to enable PCIe ACS Override and add its PCI bridges to the override, but this boot configuration worked perfectly:

label unRAID OS
  menu default
  kernel /bzimage
  append pcie_acs_override=downstream,id:10b5:8609 vfio-pci.ids=1b21:1142 initrd=/bzroot

(As an added bonus, the drivers it comes with are in a format that's easy to slipstream into the windows install CD.)

[brwainer]

Well thanks for the report, I’ll have to try that card. I do wonder though if any of your issues would be different for me, I use Proxmox.

[AJMansfield]

Hope it works for you.