Unraid/VM lstopo NUMA questions (EVGA SR-2 weirdness)

[bimmerman]

I (finally) was able to get a Windows VM set up on my EVGA SR-2 motherboard, with GPU passthrough on Unraid, despite the NF200 pcie multiplexer chips. For those that don't know anything about this board....it's a weird LGA1366 / Intel 5520 era part that is dual CPU but only single chipset, intended for overclockers and folks running quad-SLi......which means, all the PCIe is through one CPU supposedly. I got the VM running but with a bit disappointing performance, so am in the tweaking and tuning stage.

However, running the lstopo command, here's the diagram I get back (no ACS overrides or unsafe interrupts enabled). Both CPUs are in their own physical node, which makes sense, but all the IO and PCIe are just floating weirdly. The terminal output lists all that under "HostBridge P#0", but it's not visible on the diagram.

So here are my questions:

1. If I'm understanding this correctly, that means that all the IO/PCIe/etc are only connected to the P#0 Numa node?
2. If so, why isn't the IO/PCIe stuff inside the green box for Node P#0 as on other diagrams in this thread?
3. Would it be somewhat accurate to think of this topology as half of a Threadripper 2970X? I.e., there are two 'dies', one of which has PCIe access (though, both have memory access here), and tuning tricks may apply?
4. finally, for CPU allocation, my goal with the system was initially to set up a 4 gamer VM tower for group parties, if/when those happen again with 'Rona. Would it make sense to split CPUs between the two nodes i.e. allocate 1-2 from the PCIe connected CPU + 1-2 from the non-connected per VM, since I don't have a lot of cores to go around?
5. Or, is it going to be fine to set aside 2-3 vCPUs per physical CPU and not worry about the floating IO/PCIE?
   1. I have some physical evidence for this being an ok option-- the board will boot to a GPU with either CPU disabled via enable/disable jumpers present on the board. Found that out while troubleshooting other stuff on it, so my thinking is that the PCIe/IO may be addressable by either CPU?
6. Finally, am I better off going with a different motherboard given two numa nodes, only one of which is PCIe connected?

Thanks all!

 

not sure how much of this is useful, sooooo here's a bunch.

 

lstopo output:

 

lstopo text:

lstopo -s
depth 0:        1 Machine (type #1)
 depth 1:       2 NUMANode (type #2)
  depth 2:      2 Package (type #3)
   depth 3:     2 L3Cache (type #4)
    depth 4:    12 L2Cache (type #4)
     depth 5:   12 L1dCache (type #4)
      depth 6:  12 L1iCache (type #4)
       depth 7: 12 Core (type #5)
        depth 8:        24 PU (type #6)
Special depth -3:       13 Bridge (type #9)
Special depth -4:       11 PCI Device (type #10)
Special depth -5:       6 OS Device (type #11)

-------

lstopo
Machine (47GB total)
  NUMANode L#0 (P#0 24GB) + Package L#0 + L3 L#0 (12MB)
    L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0
      PU L#0 (P#0)
      PU L#1 (P#12)
    L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1
      PU L#2 (P#1)
      PU L#3 (P#13)
    L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2
      PU L#4 (P#2)
      PU L#5 (P#14)
    L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3
      PU L#6 (P#3)
      PU L#7 (P#15)
    L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4
      PU L#8 (P#4)
      PU L#9 (P#16)
    L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5
      PU L#10 (P#5)
      PU L#11 (P#17)
  NUMANode L#1 (P#1 24GB) + Package L#1 + L3 L#1 (12MB)
    L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6
      PU L#12 (P#6)
      PU L#13 (P#18)
    L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7
      PU L#14 (P#7)
      PU L#15 (P#19)
    L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8
      PU L#16 (P#8)
      PU L#17 (P#20)
    L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9
      PU L#18 (P#9)
      PU L#19 (P#21)
    L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10
      PU L#20 (P#10)
      PU L#21 (P#22)
    L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11
      PU L#22 (P#11)
      PU L#23 (P#23)
  HostBridge L#0
    PCIBridge
      PCI 1b4b:9123
        Block(Disk) L#0 "sdd"
        Block(Disk) L#1 "sde"
      PCI 1b4b:91a4
    PCIBridge
      PCIBridge
        PCIBridge
          PCI 10de:13c0
        PCIBridge
          PCI 1000:0072
        PCIBridge
          PCI 1002:67b1
    PCIBridge
      PCIBridge
        PCIBridge
          PCI 1002:67b1
    PCIBridge
      2 x { PCI 197b:2363 }
    PCIBridge
      PCI 11ab:4380
        Net L#2 "eth0"
    PCIBridge
      PCI 11ab:4380
        Net L#3 "eth1"
    PCI 8086:3a22
      Block(Disk) L#4 "sdc"
      Block(Disk) L#5 "sdb"
--------

numactl output

numactl -s
policy: default
preferred node: current
physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 
cpubind: 0 1 
nodebind: 0 1 
membind: 0 1 
------

numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 12 13 14 15 16 17
node 0 size: 24164 MB
node 0 free: 23133 MB
node 1 cpus: 6 7 8 9 10 11 18 19 20 21 22 23
node 1 size: 24189 MB
node 1 free: 23396 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 

numastat

numastat -c

Per-node numastat info (in MBs):
                Node 0 Node 1 Total
                ------ ------ -----
Numa_Hit          5840  10566 16406
Numa_Miss            0      0     0
Numa_Foreign         0      0     0
Interleave_Hit     576    579  1155
Local_Node        5838   9983 15821
Other_Node           3    583   586