USB Expansion for 2vm unraid (IOMMU ?)

[JCBiggs]

The machine in my signature is now being repurposed for the guys in my shop. (im building a 3800x machine)     the 6850k has 40 cpu lanes, and I THINK the motherboard already has at least 2 controllers for usb ports, so i can put the peripherals into seperate IOMMU groups. BUT I am not 100% sure about that an I need this setup to work. The usb ports on this board have always been a little janky.

 

I want to just buy a PCIe USB expansion card, (hopefully usb3.2) that has at least 2 separate controllers.  That just makes it easy to  guarantee i can get the ports into separate groups.  Can anyone recommend a card, or let me know what I might need to look for in the specs?    if your interested,  Ill be installing 2 quadro p2000s, and running them in x8 mode.  Next year ill upgrade to the 6950x whenever I find one cheap. 

[Windows7ge]

What motherboard are you working with?

 

I don't know how well this would work for you. Have you tested if it's possible to have a single PCI_e USB AIC be split between two VM's even with two controllers? I fear the two controllers may appear in the same IOMMU group due to their association and I don't believe one can go to one VM and one to the other in that event.

[JCBiggs]

5 minutes ago, Windows7ge said:

What motherboard are you working with?

 

I don't know how well this would work for you. Have you tested if it's possible to have a single PCI_e USB AIC be split between two VM's even with two controllers? I fear the two controllers may appear in the same IOMMU group due to their association and I don't believe one can go to one VM and one to the other in that event.

x99 deluxe ii.. as I said the build is in my signature.

 

I havent tested anything, but from what I have read and watched on the internet "the way to do it"  is with a  card that has independent controllers.  beyond that I dont have any answers. 

 

[Windows7ge]

6 minutes ago, JCBiggs said:

x99 deluxe ii.. as I said the build is in my signature.

 

I havent tested anything, but from what I have read and watched on the internet "the way to do it"  is with a  card that has independent controllers.  beyond that I dont have any answers. 

 

I disabled signatures on the forum. I don't read them so I'd rather not see them (even my own). You would also be surprised as to how many people don't update their builds in their profile so my best option was to just ask.

 

If I had one on hand I'd test the theory. It sounds plausible but plausible isn't definitive. Really I know of a different forum who would have a lot more people who know about this type of setup but I think it's against the forum guidelines to direct people off the forum (to other forums).

 

Before spending money on something you may not need I would test how stable the ports on the motherboard are after passing them though. Maybe you'll only really need one extra controller. If you intend to use 2 GPU's you have enough slots for 2 GPU's and two USB controllers.

 

Looking over the motherboards spec page though:

Quote

Intel® X99 chipset : 
4 x USB 3.1 Gen 1 port(s) ()
Intel® X99 chipset : 
6 x USB 2.0/1.1 port(s) (4 at back panel, )
ASMedia® USB 3.1 controller : 
3 x USB 3.1 Gen 2 port(s) (3 at back panel, teal blue, Type-A)
ASMedia® USB 3.1 controller : 
1 x USB 3.1 Gen 2 port(s) (1 at back panel, , USB Type-CTM, Reversible)
ASMedia® USB 3.0 Hub : 
4 x USB 3.1 Gen 1 port(s) (4 at back panel, blue)

So you do have at least 2 onboard controllers. A couple that are a part of the X99 chipset itself. You'd have to test if they appear in their own IOMMU groups or not. Then you have a Hub which I can't say how that will appear as a device you can/can't pass-though.

 

You said they're unstable/fickle. How exactly? Has Windows been the primary OS up until this point? It could very well just be a software/driver issue that will be solved when switching to unRAID.

[JCBiggs]

3 minutes ago, Windows7ge said:

I disabled signatures on the forum. I don't read them so I'd rather not see them (even my own). You would also be surprised as to how many people don't update their builds in their profile so my best option was to just ask.

 

If I had one on hand I'd test the theory. It sounds plausible but plausible isn't definitive. Really I know of a different forum who would have a lot more people who know about this type of setup but I think it's against the forum guidelines to direct people off the forum (to other forums).

 

Before spending money on something you may not need I would test how stable the ports on the motherboard are after passing them though. Maybe you'll only really need one extra controller. If you intend to use 2 GPU's you have enough slots for 2 GPU's and two USB controllers.

 

Looking over the motherboards spec page though:

So you do have at least 2 onboard controllers. A couple that are a part of the X99 chipset itself. You'd have to test if they appear in their own IOMMU groups or not. Then you have a Hub which I can't say how that will appear as a device you can/can't pass-though.

 

You said they're unstable/fickle. How exactly? Has Windows been the primary OS up until this point? It could very well just be a software/driver issue that will be solved when switching to unRAID.

I would do that, but I have one chance to buy hardware and  I need to get it all in on this PO.  you know how owners are. dont want to guy back later. even if it is only a couple hundred bucks

 

[Windows7ge]

1 minute ago, JCBiggs said:

I would do that, but I have one chance to buy hardware and  I need to get it all in on this PO.  you know how owners are. dont want to guy back later. even if it is only a couple hundred bucks

 

Well, I can't recommend you a USB3.2 AIC w/ two controllers if I don't confidently know they can be passed to different VM's. I would say though if those Quadro P2000S's are single slot you have more than enough slots for independent USB AICs and even a 10Gig NIC if you like.

 

Do you have a specific reason for needing these USB controllers to be compressed down to 1 slot?

[JCBiggs]

1 minute ago, Windows7ge said:

Well, I can't recommend you a USB3.2 AIC w/ two controllers if I don't confidently know they can be passed to different VM's. I would say though if those Quadro P2000S's are single slot you have more than enough slots for independent USB AICs and even a 10Gig NIC if you like.

 

Do you have a specific reason for needing these USB controllers to be compressed down to 1 slot?

I considered using separate USB cards, However I have a thunderbolt card in the 4th slot that I intend to use for high speed storage outside of the machine.   That's another topic though as I dont know if the induvidual VMs would be able to access it since it would need to be shared across both workstations.  

[Windows7ge]

9 minutes ago, JCBiggs said:

I considered using separate USB cards, However I have a thunderbolt card in the 4th slot that I intend to use for high speed storage outside of the machine.   That's another topic though as I dont know if the induvidual VMs would be able to access it since it would need to be shared across both workstations.  

I don't have any experience with thunderbolt. When working with NIC's you can create a network bridge which enables multiple VM's to share a single NIC.

 

Reading a little bit into TB it says it's basically USB & Video. I've never heard of being able to share something like that between VM's so I think you may be SOL there.