10 GbE network issues - help greatly appreciated

[wizzackr]

Hi guys, so here is the setup:

There are ten workstations here (lenovo p700 with older dual 14core xeons, 64gb ram), each with a Sun Sual Port 10GbE PCIe 2.0 Adapter, Base-T (which we got for cheap on ebay). All the NICs have their drivers installed.

All cabling is rated CAT7 (https://www.amazon.de/gp/product/B07W7TFN3C/ref=ppx_yo_dt_b_asin_title_o07_s00?ie=UTF8&psc=1), none of the cables is longer than 15m. The switch in use is the Netgear ProSafe XS716-T with 16 ports, all ports' LEDs show 10GbE connectivity with the exception of the 1GbE cable that comes from the router/DHCP.

 

Now all fine and dandy BUT I cannot get any file to transfer across the network (tested from a NAS with 10 Exos X10s in RAID 10 to a local SSD) with speeds exceeding 2Gbit/s. ...kinda underwhelming and I am at a loss here with regards to what to check.

 

I ran iPerf3 between four machines (2x two connections tested ) to check if the network or the discs were the culprit and it seems to be the network: All connections max out at around 2Gbit/s.

I triple checkd the NIC settings and in the switch there really is not much to get wrong... or is there? Man, I am going nuts here, so any help is greatly appreciated, guys!

[Skiiwee29]

Are you sure its reading Gigabit and not Gigabyte? Windows usually displays file transfers in Bytes, not bits. 

[Lurick]

10 minutes ago, Skiiwee29 said:

Are you sure its reading Gigabit and not Gigabyte? Windows usually displays file transfers in Bytes, not bits. 

2GB would be 16Gbit and would be well above what a 10Gbit adapter could do.

They can't be load balancing since it's single source to single destination as well (if I read correctly) so it will hash to a single port.

 

My suggestion is to check if jumbo frames are enabled, VMQ is disabled, and look for other settings to tweak.

 

 

 

Edit:

 

This is on a Solarflare adapter but I suspect you'll have similar settings.

If you see any of the bottom 3 settings "VMQ" or "Virtual Machine Queues" make sure they are disabled.

[mariushm]

Also check what pci-e slot you plugged those cards into, though I doubt that's an issue.

You have some ports that are pci-e x8 and some that are pci-e x4

See ThinkStation.indd - P700.pdf

Slot 1: PCIe 3.0 x16, Half length, full height, by CPU2

Slot 2: PCIe 3.0 x16, Full length, full height, by CPU1

Slot 3: PCIe 3.0 x8, Full length, full height, by CPU2

Slot 4: PCIe 3.0 x16, Full length, full height, by CPU1

Slot 5: PCI,Full length, full height, by PCH

Slot 6: PCIe 2.0 x4, Half length, full height, by PCH

 

I think it would be better to be in a slot belonging to CPU1 so slot 2 or 4

 

Maybe buy an intel x520-T2 from eBay and see if you get better results with it. You can find them from around 60$ and up. Here's one with more than 10 available : link

 

[wizzackr]

24 minutes ago, mariushm said:

Also check what pci-e slot you plugged those cards into, though I doubt that's an issue.

You have some ports that are pci-e x8 and some that are pci-e x4

See ThinkStation.indd - P700.pdf

Slot 1: PCIe 3.0 x16, Half length, full height, by CPU2

Slot 2: PCIe 3.0 x16, Full length, full height, by CPU1

Slot 3: PCIe 3.0 x8, Full length, full height, by CPU2

Slot 4: PCIe 3.0 x16, Full length, full height, by CPU1

Slot 5: PCI,Full length, full height, by PCH

Slot 6: PCIe 2.0 x4, Half length, full height, by PCH

 

I think it would be better to be in a slot belonging to CPU1 so slot 2 or 4

 

Maybe buy an intel x520-T2 from eBay and see if you get better results with it. You can find them from around 60$ and up. Here's one with more than 10 available : link

 

Thanks for the help, guys!!

Re slots: Will try swapping the slot - they are currently in Slot 3. Man, that would be one hell of a drop in throughput for a x8 slot on CPU2, though?

With regards to the "Virtual Machine Queues" I do not have any of those settings on the NICs.

...and man, I wish I had 2 GB/s going here

[wizzackr]

a yeah, would also be willing to try the Intel cards, but that additional cash for essentially debugging is annoying me...

[wizzackr]

Could it be the cables even though the switch reports the connection to be 10GbE?

[wizzackr]

I did run ntttcp tests and it shows pretty much the same result: I hit a cap at pretty much close to 2Gbit/s. I also checked Get-NetAdapterHardwareInfo to make sure I did not mess up the PCIe slot I put the NIC into, but the info there seems fine as well: Slot04 is where the NIC sits and it is reported as 5.0 GT/s with a PCIeLinkWidth of 8 (see attached)

 

What is odd though is that I copied a test file from the NAS to the local SSD and it again capped at 2Gbit/s. When I copied it BACK to the NAS, however, it reached transfer speeds of 540MB/s - about what I would expect when writing to a six disc RAID10 array, right? In that case I can also cancel my new cable order, as the cables seem to work fine...?

 

I also checked and re-installed the Intel network driver for the X540-T2 and disabled jumbo-frames on the NAS, switch and NICs to isolate the problem.

Any other pointers?

[wizzackr]

Ok, here are the latest findings. When running iperf3 on multiple threads and a fixed window size (iperf3 -c 192.168.0.139 -w 640k -l 640k -P 4) I get a combined throughput of 8.9 Gbits/s, which I deem ok, right?

On a single thread it never goes beyond 2Gbits/s - which seems kinda odd? Copying files from within windows explorer from the NAS it tops out at bad 2Gbit/s - when copying TO it I reach 550MB/s, which is clearly where the disc performance limits any more throughput.

If I am not mistaken the successful iperf test rules out the switch, the cables and the NIC/drivers, right?
So it leaves me with the Synology NAS and windows or am I missing something.

Any ideas??