UnRAID on PCI GPU?

Reckless Maker · October 28, 2017

Hello All,

I'm currently building an UnRAID server into an old HTPC case I picked up recently. Unfortunately the form factor limits me to 2 PCIe slots. One is being used by the LSI RAID card and one by an R5 550. I'm trying to use the 550 in a windows VM but since Ryzen doesn't have onboard graphics and both PCIe lanes are taken, I had to resort to buying a PCI (non express) GPU for UnRAID. Unfortunately the system always boots PCIe before PCI and there's no way (that I can tell) to set PCI as first preference (just PCIe slot 1 or 2). My last resort is to buy a different Mobo with 3 PCIe slots but other than that I'm out of ideas. Does anyone familiar with UnRAID know of a way to make this setup work?

Paul

GDRRiley · October 28, 2017

remove the pcie gpu and tell it to make the pci its gpu, I don't remember exactly how to do it but LTT brought it up in 2 games 1 cpu.

Reckless Maker · October 28, 2017

Just now, GDRRiley said:

remove the pcie gpu and tell it to make the pci its gpu, I don't remember exactly how to do it but LTT brought it up in 2 games 1 cpu.

I already tried that and it still only let me choose between the two PCIe slots. I believe they used integrated graphics in 2 gamers.

Reckless Maker · October 28, 2017

So I tried passing through the PCI card just to see if it would work and it didn't. The error code got me looking into B350 IOMMU groups and in turn the seemingly baffling array of motherboard layouts and asterisks that come with each. After lots of research, I think I understand what's going on. Here's what I pieced together (sorry if this is old news):

On Ryzen, the CPU has 24 PCIe lanes; 16 for the graphics card (or 8+8 on X370), 4 for NVMe (or 2 SATA3 if no NVMe), and 4 for the chipset. The chipset then has a further 4-8 PCI lanes which are allocated based on board price. Typically the available features break down like this: PCIe x1 slot = 1 lane, PCIe x16 slot = 4 lanes (2 on A320), and PCI non-express slot = 2 lanes. Mix and match up to the 4, 6, or 8 available lanes on the chipset.

It's this chipset vs CPU PCIe lane breakdown matters in unRAID. In order to pass through something to a VM, it must have its own IOMMU group. Now on Ryzen, everything on the chipset PCIe lanes are in the same IOMMU group. Essentially, nothing going through the chipset can be passed through to a VM.

So what's the TL;DR if you're building a Ryzen storage + VM box? Well, it depends on the form factor you're looking for.

If you're looking for an ATX board or above, look for an X370 board with 3 PCIe x16 slots. One slot will be on the chipset (put LSI card here), and two will be on the CPU operating as either one x16 slot or two x8 slots. These two x8 slots will be in separate IOMMU groups, allowing one to be passed through to your VM (with no performance impact since x8 is still plenty fast) and the other to be captured by unRAID.

If you're looking for an mATX board, look for a B350 board with two PCIe x16 slots and at least one PCIe x1 slot. Speculation: Since the motherboard can be set to boot from the PCIe x1 graphics, this puts the card being used by unRAID in the same IOMMU group as the rest of the hardware being used by unRAID (mainly the LSI card). Thus the CPU-bound card running at full x16 speed is free to be passed through. (Can someone with one of these boards confirm this?) This will also work with a full ATX board with an x1 slot but x16 cards are much cheaper than x1 cards and are much more likely to work.

If you're looking for an ITX board, you're SOL.

And finally, if you're like me and have already bought a board that is none of the above, there may be hope. Unfortunately, booting from a PCI non-express graphics card doesn't seem to be possible at least on my board (and I'd imagine most others as well). However, if you're not using the M.2 slot, that means you have 4 open PCIe lanes in their own IOMMU group. I haven't tested this but there are m.2 to PCIe x4 and PCIe x4 to x16 risers. So theoretically it should be possible to pass through a card on the M.2 slot to a VM.

Where that leaves me, I'm not too sure. I'm not sure if my current case can fit the janky M.2 to PCIe x16 adapter but I'm also not sure if switching to a board with a PCIe x1 slot would work either. If anyone is able to test either of these scenarios that would be awesome. Thoughts?

TL;DR: Passing a GPU through to a VM on Ryzen is complicated.

NelizMastr · October 30, 2017

One thing you've failed to mention: which PCI card did you get? Putting some really ancient PCI 1.x or 2.x card in probably won't work anyway. I've tried an S3 Virge card and an ATI Rage II once in a P67 board and that refused to do anything at all, so you might need something a tad newer in the first place. Preferably PCI 3.x. And before you ask, yes, I mean PCI, not PCIe.

Reckless Maker · October 31, 2017

So I picked up a motherboard with a PCIe x1 slot from Fry's and did some more experimentation.

Even with a (powered) GTX 1080 in the x1 slot, the BIOS still only gives PCIe 1 and PCIe 2 as options. And even with PCIe 2 (on either board) selected, the only way to boot via the card in the PCIe x1 (or PCI) slot is if there is nothing else connected to the chipset x16 slot.

All of this testing was done on Gigabyte boards so perhaps other manufacturers have more options in their BIOS. Although this behavior could just be baked into the chipset.

However it turns out that all this is pointless because the primary GPU actually can be passed through to a VM. When windows didn't boot I figured it was because the primary card couldn't be passed through, but Linux booted just fine.

Now I just have to figure out why it keeps hard locking.