Network HELL! P2P Connection on 10gbe fast one way not the other.

[Bmoney]

Asus Rampage VI Extreme and Asus Zenith Exetreme Alpha
Ubuntu 20 and Windows 10
P2P Cat6A direct connection. Jumbo Packets and MTU 9000

Crappy Speeds Why

Ethtool read out

Settings for enp9s0:
        Supported ports: [ TP ]
        Supported link modes:   100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
                                2500baseT/Full
                                5000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
                                2500baseT/Full
                                5000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: Unknown
        Supports Wake-on: g
        Wake-on: d
        Link detected: yes

Iperf 3 result Server First then Client
Accepted connection from 10.2.2.3, port 54847
[  5] local 10.2.2.1 port 5201 connected to 10.2.2.3 port 54848
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   443 MBytes  3.72 Gbits/sec
[  5]   1.00-2.00   sec  80.6 MBytes   676 Mbits/sec
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.06  sec   524 MBytes   437 Mbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201

Client commad
 

Connecting to host 10.2.2.3, port 5201
[  5] local 10.2.2.1 port 39938 connected to 10.2.2.3 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   995 MBytes  8.34 Gbits/sec    0    228 KBytes
[  5]   1.00-2.00   sec  1.01 GBytes  8.66 Gbits/sec    0    228 KBytes
[  5]   2.00-3.00   sec  1.00 GBytes  8.60 Gbits/sec    0    228 KBytes
[  5]   3.00-4.00   sec  1017 MBytes  8.53 Gbits/sec    0    228 KBytes
[  5]   4.00-5.00   sec  1.01 GBytes  8.68 Gbits/sec    0    228 KBytes
[  5]   5.00-6.00   sec  1012 MBytes  8.48 Gbits/sec    0    228 KBytes
[  5]   6.00-7.00   sec   990 MBytes  8.31 Gbits/sec    0    228 KBytes
[  5]   7.00-8.00   sec  1013 MBytes  8.50 Gbits/sec    0    228 KBytes
[  5]   8.00-9.00   sec  1013 MBytes  8.50 Gbits/sec    0    228 KBytes
[  5]   9.00-10.00  sec  1004 MBytes  8.42 Gbits/sec    1    228 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  9.90 GBytes  8.50 Gbits/sec    1             sender
[  5]   0.00-10.00  sec  9.90 GBytes  8.50 Gbits/sec                  receiver

What the Hell is wrong. 

 

[SorryBella]

Are you sure the ISP plan is symetrical 10 gigabit? Many ISP plans only advertises its fastest speed and then have a different speed from one way or the other (usually faster speed on the download), so that could be your issue.

[Bmoney]

2 minutes ago, SorryClaire said:

Are you sure the ISP plan is symetrical 10 gigabit? Many ISP plans only advertises its fastest speed and then have a different speed from one way or the other (usually faster speed on the download), so that could be your issue.

Not ISP this is a P2P LAN connection directly attached. 

[Alex Atkin UK]

1 hour ago, Bmoney said:

Asus Rampage VI Extreme and Asus Zenith Exetreme Alpha
Ubuntu 20 and Windows 10
P2P Cat6A direct connection. Jumbo Packets and MTU 9000

Crappy Speeds Why

Ethtool read out

Settings for enp9s0:
        Supported ports: [ TP ]
        Supported link modes:   100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
                                2500baseT/Full
                                5000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
                                2500baseT/Full
                                5000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: Unknown
        Supports Wake-on: g
        Wake-on: d
        Link detected: yes

Iperf 3 result Server First then Client
Accepted connection from 10.2.2.3, port 54847
[  5] local 10.2.2.1 port 5201 connected to 10.2.2.3 port 54848
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   443 MBytes  3.72 Gbits/sec
[  5]   1.00-2.00   sec  80.6 MBytes   676 Mbits/sec
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.06  sec   524 MBytes   437 Mbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201

Client commad
 

Connecting to host 10.2.2.3, port 5201
[  5] local 10.2.2.1 port 39938 connected to 10.2.2.3 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   995 MBytes  8.34 Gbits/sec    0    228 KBytes
[  5]   1.00-2.00   sec  1.01 GBytes  8.66 Gbits/sec    0    228 KBytes
[  5]   2.00-3.00   sec  1.00 GBytes  8.60 Gbits/sec    0    228 KBytes
[  5]   3.00-4.00   sec  1017 MBytes  8.53 Gbits/sec    0    228 KBytes
[  5]   4.00-5.00   sec  1.01 GBytes  8.68 Gbits/sec    0    228 KBytes
[  5]   5.00-6.00   sec  1012 MBytes  8.48 Gbits/sec    0    228 KBytes
[  5]   6.00-7.00   sec   990 MBytes  8.31 Gbits/sec    0    228 KBytes
[  5]   7.00-8.00   sec  1013 MBytes  8.50 Gbits/sec    0    228 KBytes
[  5]   8.00-9.00   sec  1013 MBytes  8.50 Gbits/sec    0    228 KBytes
[  5]   9.00-10.00  sec  1004 MBytes  8.42 Gbits/sec    1    228 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  9.90 GBytes  8.50 Gbits/sec    1             sender
[  5]   0.00-10.00  sec  9.90 GBytes  8.50 Gbits/sec                  receiver

What the Hell is wrong. 

 

That's pretty much what I see tbh.  Both upgraded PCs and a refresh install of Windows 10 onto a brand new PC had similar results.

 

Connecting to host god.lan, port 5201
[  5] local 192.168.1.253 port 35186 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   982 MBytes  8.24 Gbits/sec    0    471 KBytes
[  5]   1.00-2.00   sec  1024 MBytes  8.59 Gbits/sec    0    448 KBytes
[  5]   2.00-3.00   sec  1.00 GBytes  8.62 Gbits/sec    0    425 KBytes
[  5]   3.00-4.00   sec  1020 MBytes  8.56 Gbits/sec    0    456 KBytes
[  5]   4.00-5.00   sec  1009 MBytes  8.46 Gbits/sec    0    422 KBytes
[  5]   5.00-6.00   sec  1000 MBytes  8.39 Gbits/sec    0    462 KBytes
[  5]   6.00-7.00   sec  1004 MBytes  8.42 Gbits/sec    0    439 KBytes
[  5]   7.00-8.00   sec  1020 MBytes  8.56 Gbits/sec    0    422 KBytes
[  5]   8.00-9.00   sec  1022 MBytes  8.57 Gbits/sec    0    422 KBytes
[  5]   9.00-10.00  sec  1009 MBytes  8.46 Gbits/sec    0    436 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  9.88 GBytes  8.49 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  9.88 GBytes  8.48 Gbits/sec                  receiver

iperf Done.
[root@server prosody]# iperf3 -Rc god.lan
Connecting to host god.lan, port 5201
Reverse mode, remote host god.lan is sending
[  5] local 192.168.1.253 port 35190 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   587 MBytes  4.92 Gbits/sec
[  5]   1.00-2.00   sec   287 MBytes  2.41 Gbits/sec
[  5]   2.00-3.00   sec   373 MBytes  3.13 Gbits/sec
[  5]   3.00-4.00   sec   468 MBytes  3.92 Gbits/sec
[  5]   4.00-5.00   sec   365 MBytes  3.06 Gbits/sec
[  5]   5.00-6.00   sec   392 MBytes  3.28 Gbits/sec
[  5]   6.00-7.00   sec   421 MBytes  3.53 Gbits/sec
[  5]   7.00-8.00   sec   311 MBytes  2.61 Gbits/sec
[  5]   8.00-9.00   sec   406 MBytes  3.41 Gbits/sec
[  5]   9.00-10.00  sec   304 MBytes  2.55 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  3.82 GBytes  3.28 Gbits/sec                  sender
[  5]   0.00-10.00  sec  3.82 GBytes  3.28 Gbits/sec                  receiver

 

I've tweaked everything I can think of, its some sort of bug in Windows 10 since an update around September 2020 I think as if you Google Windows upstream speed issues there are other people complaining.

 

I only managed to get it THAT fast by disabling interrupt moderation on the network cards and I found jumbo frames causes more harm than good in the long run as it caused delays when accessing the Internet. (but mine isn't point to point, its into a switch so you're probably fine there)

[Bmoney]

5 hours ago, Alex Atkin UK said:

That's pretty much what I see tbh.  Both upgraded PCs and a refresh install of Windows 10 onto a brand new PC had similar results.

 

Connecting to host god.lan, port 5201
[  5] local 192.168.1.253 port 35186 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   982 MBytes  8.24 Gbits/sec    0    471 KBytes
[  5]   1.00-2.00   sec  1024 MBytes  8.59 Gbits/sec    0    448 KBytes
[  5]   2.00-3.00   sec  1.00 GBytes  8.62 Gbits/sec    0    425 KBytes
[  5]   3.00-4.00   sec  1020 MBytes  8.56 Gbits/sec    0    456 KBytes
[  5]   4.00-5.00   sec  1009 MBytes  8.46 Gbits/sec    0    422 KBytes
[  5]   5.00-6.00   sec  1000 MBytes  8.39 Gbits/sec    0    462 KBytes
[  5]   6.00-7.00   sec  1004 MBytes  8.42 Gbits/sec    0    439 KBytes
[  5]   7.00-8.00   sec  1020 MBytes  8.56 Gbits/sec    0    422 KBytes
[  5]   8.00-9.00   sec  1022 MBytes  8.57 Gbits/sec    0    422 KBytes
[  5]   9.00-10.00  sec  1009 MBytes  8.46 Gbits/sec    0    436 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  9.88 GBytes  8.49 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  9.88 GBytes  8.48 Gbits/sec                  receiver

iperf Done.
[root@server prosody]# iperf3 -Rc god.lan
Connecting to host god.lan, port 5201
Reverse mode, remote host god.lan is sending
[  5] local 192.168.1.253 port 35190 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   587 MBytes  4.92 Gbits/sec
[  5]   1.00-2.00   sec   287 MBytes  2.41 Gbits/sec
[  5]   2.00-3.00   sec   373 MBytes  3.13 Gbits/sec
[  5]   3.00-4.00   sec   468 MBytes  3.92 Gbits/sec
[  5]   4.00-5.00   sec   365 MBytes  3.06 Gbits/sec
[  5]   5.00-6.00   sec   392 MBytes  3.28 Gbits/sec
[  5]   6.00-7.00   sec   421 MBytes  3.53 Gbits/sec
[  5]   7.00-8.00   sec   311 MBytes  2.61 Gbits/sec
[  5]   8.00-9.00   sec   406 MBytes  3.41 Gbits/sec
[  5]   9.00-10.00  sec   304 MBytes  2.55 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  3.82 GBytes  3.28 Gbits/sec                  sender
[  5]   0.00-10.00  sec  3.82 GBytes  3.28 Gbits/sec                  receiver

 

I've tweaked everything I can think of, its some sort of bug in Windows 10 since an update around September 2020 I think as if you Google Windows upstream speed issues there are other people complaining.

 

I only managed to get it THAT fast by disabling interrupt moderation on the network cards and I found jumbo frames causes more harm than good in the long run as it caused delays when accessing the Internet. (but mine isn't point to point, its into a switch so you're probably fine there)

Ok yeah I did iperf on my server and got 28GB/s back to itself
Then I did it with the windows machine only 6GB/s. 

I think I narrowed like you said to the Windows. Get lame speed on my 1GB port too with transfers at 80MB/s and bumpy. 

I disable all offloads and bupkis. 

My Laptop with a USB-3 NIC works great oddly. 

[Bmoney]

22 hours ago, Alex Atkin UK said:

That's pretty much what I see tbh.  Both upgraded PCs and a refresh install of Windows 10 onto a brand new PC had similar results.

 

Connecting to host god.lan, port 5201
[  5] local 192.168.1.253 port 35186 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   982 MBytes  8.24 Gbits/sec    0    471 KBytes
[  5]   1.00-2.00   sec  1024 MBytes  8.59 Gbits/sec    0    448 KBytes
[  5]   2.00-3.00   sec  1.00 GBytes  8.62 Gbits/sec    0    425 KBytes
[  5]   3.00-4.00   sec  1020 MBytes  8.56 Gbits/sec    0    456 KBytes
[  5]   4.00-5.00   sec  1009 MBytes  8.46 Gbits/sec    0    422 KBytes
[  5]   5.00-6.00   sec  1000 MBytes  8.39 Gbits/sec    0    462 KBytes
[  5]   6.00-7.00   sec  1004 MBytes  8.42 Gbits/sec    0    439 KBytes
[  5]   7.00-8.00   sec  1020 MBytes  8.56 Gbits/sec    0    422 KBytes
[  5]   8.00-9.00   sec  1022 MBytes  8.57 Gbits/sec    0    422 KBytes
[  5]   9.00-10.00  sec  1009 MBytes  8.46 Gbits/sec    0    436 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  9.88 GBytes  8.49 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  9.88 GBytes  8.48 Gbits/sec                  receiver

iperf Done.
[root@server prosody]# iperf3 -Rc god.lan
Connecting to host god.lan, port 5201
Reverse mode, remote host god.lan is sending
[  5] local 192.168.1.253 port 35190 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   587 MBytes  4.92 Gbits/sec
[  5]   1.00-2.00   sec   287 MBytes  2.41 Gbits/sec
[  5]   2.00-3.00   sec   373 MBytes  3.13 Gbits/sec
[  5]   3.00-4.00   sec   468 MBytes  3.92 Gbits/sec
[  5]   4.00-5.00   sec   365 MBytes  3.06 Gbits/sec
[  5]   5.00-6.00   sec   392 MBytes  3.28 Gbits/sec
[  5]   6.00-7.00   sec   421 MBytes  3.53 Gbits/sec
[  5]   7.00-8.00   sec   311 MBytes  2.61 Gbits/sec
[  5]   8.00-9.00   sec   406 MBytes  3.41 Gbits/sec
[  5]   9.00-10.00  sec   304 MBytes  2.55 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  3.82 GBytes  3.28 Gbits/sec                  sender
[  5]   0.00-10.00  sec  3.82 GBytes  3.28 Gbits/sec                  receiver

 

I've tweaked everything I can think of, its some sort of bug in Windows 10 since an update around September 2020 I think as if you Google Windows upstream speed issues there are other people complaining.

 

I only managed to get it THAT fast by disabling interrupt moderation on the network cards and I found jumbo frames causes more harm than good in the long run as it caused delays when accessing the Internet. (but mine isn't point to point, its into a switch so you're probably fine there)

Definitely a local windows problem. 
My Laptop and USB NIC is fine. Thinking about reinstall but don't know if it will work. 

[Alex Atkin UK]

4 hours ago, Bmoney said:

Definitely a local windows problem. 
My Laptop and USB NIC is fine. Thinking about reinstall but don't know if it will work. 

It didn't for me, I had it on a fresh install on a brand new Ryzen 5950X, its baffling.  Boot Linux on the same hardware, no problems at all.

I have two laptops, two desktops, they all have the same problem.  Even on WiFi the upstream seems to be limited compared to the downstream more than it used to be.  Same on Windows 11 even which I had kinda hoped it was a known issue they might have fixed.

[Bmoney]

On 7/24/2021 at 10:08 PM, Alex Atkin UK said:

It didn't for me, I had it on a fresh install on a brand new Ryzen 5950X, its baffling.  Boot Linux on the same hardware, no problems at all.

I have two laptops, two desktops, they all have the same problem.  Even on WiFi the upstream seems to be limited compared to the downstream more than it used to be.  Same on Windows 11 even which I had kinda hoped it was a known issue they might have fixed.

I wanna try Windows Pro for Workstation as it has more unlocked hardware features. 

[Alex Atkin UK]

Also worth noting, when I said I turned of interrupt moderation, I did so on the Linux box too.  Just turning it off on Windows wasn't enough, turning it off on my NAS is what allowed me to get decent downstream speeds to the Windows machine at the cost of higher CPU usage on my NAS.