NVENC bottleneck

[mushk1n]

Hi all,

 

Have a general question about a specific topic

 

Basically - I'm running multiple encoding jobs that use nvenc hardware encoding. I have a couple M4000 quadro cards, using only 1 GPU I see 100% enc utilization when using nvidia-smi to monitor the GPU. However, if I introduce a second GPU I see the utilization drop to 70% per card (both cards are installed in the system at all times during these tests).

 

Specs:

Z10PE

2699V4 (1 CPU)

4*16GB DDR4 ECC @2400MHz

NV M4000 * 2

4 * 275GB SSDs in RAID0

 

I'm currently running 2 ubuntu vm's under unraid (1 GPU per VM) and previously I ran a bare metal install of Ubuntu with both GPUs. However, both VM and bare metal have the exact the same bottleneck @ 70% per GPU for 2 cards and 100% utilization for a single GPU.

 

I've monitored resources of all the obvious culprits and I can't see anything getting close to their limit - so, my general question is, could it be something to do with the PCIE bandwidth? I figured both cards running at x16 PCIE gen3.0 was sufficient to assume no bottleneck there, but I'm out of ideas

 

Without wishing to confuse the matter, I  have also tested the system with a third concurrent VM using another quadro card (K2200). Using 2 VMs (K2200 with a single M4000) I see utilization on the K2200 of 100% and utilization on the single M4000 at 80%. Using 3 VMs (2*M4000 and K2200) I see the utilization even out, with the two M4000's at roughly the same ~45% and the K2200 with a higher encoder utilization ~80%. During these tests all 3 cards were installed in the system, therefore 1 M4000 was running at x16 and the other at x8 with the K2200 at x8.

 

Hopefully that makes some level of sense

 

Thanks in advance!