10gbe LAN Networking issues - Slow, and asymmetric speeds

[osint]

I have a home server setup running proxmox and the server's mother board is a z790 pro art Wi-Fi creator with a built in 10Gbe NIC.

For my desktop, I am using a TP-Link TX-401 10Gbe NIC using the latest drivers from here (I have reinstalled and repaired them several times)

I have them connected via a 5 port 10Gbe Trendnet switch and the link indicators on the ports are green suggesting they are at full speed.

I have everything connected with Cat6 / Cat 7 cables with less than 15 meters of cable.

 

 

The issue I am running into is, either through iPerf3 or just real world data transfer of a single file that is 10 gigabytes, I'm getting nowhere near 10Gbe speeds. To make things more confusing, it seems the speeds are slower in one direction than the other.

 

I built the server and computer in the last 2 months, I did update the bios of the server when I set it up, but I did not for my desktop, although I did update the NIC's drivers as previously mentioned. When transferring the 10Gb file, it seems to peak around 120-140mbps.

 

I'm not really sure where to begin testing because I dont have any other 10gbe capable devices to see what's working or not working. Im a proxmox noob, do I need to update the drivers of the built in NIC somehow for the server?

 

Here are the iPerf logs

 

Server connecting to desktop

redacted@proxmox:/# iperf3 -c 192.168.0.54
Connecting to host 192.168.0.54, port 5201
[  5] local 192.168.0.250 port 37628 connected to 192.168.0.54 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   480 MBytes  4.03 Gbits/sec    0    301 KBytes       
[  5]   1.00-2.00   sec   514 MBytes  4.31 Gbits/sec    0    301 KBytes       
[  5]   2.00-3.00   sec   516 MBytes  4.33 Gbits/sec    0    301 KBytes       
[  5]   3.00-4.00   sec   510 MBytes  4.27 Gbits/sec    0    301 KBytes       
[  5]   4.00-5.00   sec   509 MBytes  4.27 Gbits/sec    0    301 KBytes       
[  5]   5.00-6.00   sec   512 MBytes  4.29 Gbits/sec    0    301 KBytes       
[  5]   6.00-7.00   sec   509 MBytes  4.27 Gbits/sec    0    301 KBytes       
[  5]   7.00-8.00   sec   517 MBytes  4.33 Gbits/sec    0    301 KBytes       
[  5]   8.00-9.00   sec   510 MBytes  4.28 Gbits/sec    0    301 KBytes       
[  5]   9.00-10.00  sec   517 MBytes  4.34 Gbits/sec    0    301 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  4.97 GBytes  4.27 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  4.97 GBytes  4.27 Gbits/sec                  receiver

iperf Done.

 

 

 

Desktop connecting to server

redacted@proxmox:/# iperf3 -s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 192.168.0.54, port 53956
[  5] local 192.168.0.250 port 5201 connected to 192.168.0.54 port 53957
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  14.1 MBytes   118 Mbits/sec                  
[  5]   1.00-2.00   sec  72.9 MBytes   611 Mbits/sec                  
[  5]   2.00-3.00   sec   284 MBytes  2.38 Gbits/sec                  
[  5]   3.00-4.00   sec   312 MBytes  2.62 Gbits/sec                  
[  5]   4.00-5.00   sec   284 MBytes  2.38 Gbits/sec                  
[  5]   5.00-6.00   sec   264 MBytes  2.21 Gbits/sec                  
[  5]   6.00-7.00   sec   280 MBytes  2.35 Gbits/sec                  
[  5]   7.00-8.00   sec   179 MBytes  1.50 Gbits/sec                  
[  5]   8.00-9.00   sec   228 MBytes  1.91 Gbits/sec                  
[  5]   9.00-10.00  sec   211 MBytes  1.77 Gbits/sec                  
[  5]  10.00-10.01  sec  1.13 MBytes  1.36 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  2.08 GBytes  1.79 Gbits/sec                  receiver

 

Desktop connecting to server again becasue wasnt sure if it was a fluke
-----------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------
Accepted connection from 192.168.0.54, port 53971
[  5] local 192.168.0.250 port 5201 connected to 192.168.0.54 port 53972
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  18.0 MBytes   151 Mbits/sec                  
[  5]   1.00-2.00   sec  21.0 MBytes   176 Mbits/sec                  
[  5]   2.00-3.00   sec   156 MBytes  1.31 Gbits/sec                  
[  5]   3.00-4.00   sec   241 MBytes  2.02 Gbits/sec                  
[  5]   4.00-5.00   sec   122 MBytes  1.02 Gbits/sec                  
[  5]   5.00-6.00   sec   212 MBytes  1.78 Gbits/sec                  
[  5]   6.00-7.00   sec   320 MBytes  2.68 Gbits/sec                  
[  5]   7.00-8.00   sec   354 MBytes  2.97 Gbits/sec                  
[  5]   8.00-9.00   sec   346 MBytes  2.91 Gbits/sec                  
[  5]   9.00-10.00  sec   414 MBytes  3.47 Gbits/sec                  
[  5]  10.00-10.00  sec  1.13 MBytes  1.96 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  2.15 GBytes  1.85 Gbits/sec                  receiver
 

 

As you can see, best case scenario is less than 1/2 of the expected bandwidth, and at worst its ~1/5th the expected bandwidth...

 

 

Point of note which *may* help narrow down the culprit - I have a 3gbe fiber network, so I can test internet speeds from the server and from the desktop since that also runs through the switch.

 

 

Heres those results - 

 

Speedtest from a TrueNAS VM on the server

redacted@truenas[~]# speedtest

   Speedtest by Ookla

      Server: redacted - redacted, redacted (id: redacted)
         ISP: redacted
Idle Latency:     4.36 ms   (jitter: 0.13ms, low: 4.22ms, high: 4.56ms)
    Download:  3133.23 Mbps (data used: 3.7 GB)
                 30.49 ms   (jitter: 1.24ms, low: 2.41ms, high: 37.87ms)
      Upload:  3209.73 Mbps (data used: 4.3 GB)

 

Speedtest from desktop

 

The first day I ran a speedtest after building the computer, I got a 3Gbps up/down so Im leaning towards the issues being with the NIC in my desktop, but Im really not sure becasue clearly the desktop is capable of around 4gbe when the server connects to it, and I have no idea why it would be different the other way around... 

 

 

 

 

 

Any and all help would be greatly appreciated

 

[osint]

bump

[jenson_dj]

Hi @osint. From the looks of it, the iperf test that you have run is in TCP mode. TCP transfer rate is hugely dependent on the device receiving the traffic and its TCP window size/buffer that it is offering.

You could do either of the 2 options to test:

1. Use UDF based iperf (-u on the client ) and set the target bandwidth (-b 10000M on the client). You might have to lower the bandwidth to about 90%(9000M) to account for the IP/ethernet headers which iperf does not consider in its calculations.

2. Run TCP based iperf with Parallel connections (-P n on client) where n is the number of connections. You can start with 2 and increase to see what kinds of total speed the link is able to handle. But again note that since this is TCP based session, the iperf server side window size could again be the limiting factor.

You can also use the various other options available in the iperf user docs: https://iperf.fr/iperf-doc.php#3doc 

[Bdavis]

What devices are you copying to and from? Usually your speeds with 10gb networking are limited by your storage devices rather than the network. Try setting up a ramdisk on each system and test speeds between them.

 

Are you running trueNAS core or scale? Scale is still relatively new and is known to have poorer speeds compared to core.

 

Tp- links nic is also suspect. Intel would be the only nic brand I would buy.