Unable to get full full-duplex 2Gbps speeds

[sazrocks]

This is more of a proof of concept and fun thing than practical, but still I'm curious why this isn't working.

 

I have two 1Gbps Intel NICs (on the same card) teamed together using static link aggregation and have set up the ports they are connected to on my switch as a LAG group. I am using two instances of iperf3 client and two instances of iperf3 server in combination with two other devices on the network each with one instance of iperf3 server and one instance of iperf3 client. The client arguments I'm using across all of them are 

-w 1024KB -p PORT -t 60

Good:

I can get 2Gbps TX (1Gbps to each device)

I can get 1Gbps TX and 1Gbps RX  simultaneously for each client individually actually I can't

Bad:

I can only get 1Gbps RX with both clients trying together (random speeds on both clients but they always add up to ~1Gbps)

I can only get ~1.6Gbps TX peak and 800Mbps peak RX  simultaneously when testing all 4 instances of iperf3 at once.

 

Questions:

1. There seems to be some bottleneck around 1Gbps for RX. Is there a way to fix this?

2. There seems to be some bottleneck around  2.5-3.0 Gbps for combined RX and TX. Is there a way to fix this?

3. Is there a better way I could title this thread?

[beersykins]

What algorithm are you using on your switch for lacp?  By default you should get more server to multiple hosts bandwidth, but single host to single host usually hashes out to one interface.

[sazrocks]

17 minutes ago, beersykins said:

What algorithm are you using on your switch for lacp?  By default you should get more server to multiple hosts bandwidth, but single host to single host usually hashes out to one interface.

Unfortunately that isn't documented anywhere for my switch (TP-link TL-SG108E) and there doesn't seem to be any way of changing it. I'm guessing that means there's nothing I can do?

[brwainer]

The switch is probably doing a MAC-based algorithm distribution of traffic. Your host controls which port outbound traffic leaves on, but the switch controls with port inbound traffic comes in on. Try doing your tests with more than one other device sending traffic to the computer with the teamed interfaces - the different transmitting MACs will hopefully cause the switch to load balance the two connections to different ports in the team.

EDIT: This is exactly what @beersykins was referring to as well.

[brwainer]

7 minutes ago, sazrocks said:

Unfortunately that isn't documented anywhere for my switch (TP-link TL-SG108E) and there doesn't seem to be any way of changing it. I'm guessing that means there's nothing I can do?

I don't see any settings for the LAG in the user manual

[sazrocks]

5 minutes ago, brwainer said:

The switch is probably doing a MAC-based algorithm distribution of traffic. Your host controls which port outbound traffic leaves on, but the switch controls with port inbound traffic comes in on. Try doing your tests with more than one other device sending traffic to the computer with the teamed interfaces - the different transmitting MACs will hopefully cause the switch to load balance the two connections to different ports in the team.

Unfortunately when using two different other devices (which are each individually capable of 1Gbps) sending to my desktop with the 2Gbps teamed connection it caps out at 1Gbps, with a random distribution of that among the two other devices.

 

I'm guessing this would be the result of a crap lacp algorithm?

[brwainer]

3 hours ago, sazrocks said:

Unfortunately when using two different other devices (which are each individually capable of 1Gbps) sending to my desktop with the 2Gbps teamed connection it caps out at 1Gbps, with a random distribution of that among the two other devices.

 

I'm guessing this would be the result of a crap lacp algorithm?

More specifically it is a result of a crap LAG algorithm - this switch is too basic to actually support LACP. With a LAG you force the two ports together, and there is no verification that the two ports you forced together actually go to the same destination. With LACP you enable the ports for LACP and then the two ends verify that they are indeed connected to each other on both ports and can also inform each other of which traffic splitting algorithm they are using.