Getting ATT 5Gb working on 10Gb lan

Method · March 19, 2022

ATT has been rolling out 2.5 and 5gb internet for a little while now, and as I live in Dallas, I'm able to get their 5Gb internet. They provided me with a BGW320-500 modem that has a fiber line going into it, and 4 RJ45 lan ports. 3 of the lan ports are 1Gb, and the 4th is 5Gb.

Connecting that 5Gb link directly to my motherboards 2.5Gb port, I can get full 2.5Gb up and down. I figured rather than find some maybe crappy 5Gb rj45 card to put in my machine, I would instead upgrade my local network to 10gig. So I ordered the following:

2x SFP+ cables from "10Gtek"

2x Mellanox ConnectX-3 (from ebay)

1x Ipolex RJ45 transceiver (for the 5Gb connection from the modem)

1x CRS305-1G-4S+IN Mikrotik switch

Connected all of this up and ran a speed test. I can get the full 5Gb download speed, but the upload speed is _abysmal_ at less than 1Gb. I thought maybe this was the switch or maybe the cards/cables so I ran iperf between my nas and my desktop, and saw much better speeds:

Connecting to host 192.168.1.79, port 5201
[  4] local 192.168.1.216 port 56866 connected to 192.168.1.79 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  1.07 GBytes  9.19 Gbits/sec
[  4]   1.00-2.00   sec  1.07 GBytes  9.16 Gbits/sec
[  4]   2.00-3.00   sec  1.06 GBytes  9.12 Gbits/sec
[  4]   3.00-4.00   sec  1.06 GBytes  9.14 Gbits/sec
[  4]   4.00-5.00   sec  1.04 GBytes  8.91 Gbits/sec
[  4]   5.00-6.00   sec  1.01 GBytes  8.68 Gbits/sec
[  4]   6.00-7.00   sec  1.05 GBytes  9.00 Gbits/sec
[  4]   7.00-8.00   sec  1.07 GBytes  9.17 Gbits/sec
[  4]   8.00-9.00   sec  1.05 GBytes  8.99 Gbits/sec
[  4]   9.00-10.00  sec  1.05 GBytes  9.03 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  10.5 GBytes  9.04 Gbits/sec                  sender
[  4]   0.00-10.00  sec  10.5 GBytes  9.04 Gbits/sec                  receiver

iperf Done.

There is no ip filtering on the switch, it is _literally_ just a dumb switch at the moment, nothing is active on it. Looking at the ATT modem's interface, I can see that is says that 5Gb connection is connected at 5Gb. However, in the switch's software (winbox), it says it's connected at 10Gb. The thinking here then is that it isn't negotiating 5Gb properly. There is no way to "hard code" the interface to 5Gb. So, maybe a problem with the transceiver? I've just ordered a MikroTik S+RJ10 so we'll see how that fairs.

I suspect this kind of setup might become more common as people try to take advantage of 5Gb internet? Am I maybe missing something in my setup that might perform better? Any other thoughts people might have?

Lurick · March 19, 2022

You're definitely not alone. I had a similar issue with a 10G port and a mgig transceiver which hamstrung my uploads too. My current setup involves passing it through a switch that can do 1/2.5/5/10Gbps on some ports and putting it in a seperate VLAN to passthrough to my firewall. Current setup is:

BGW320 5Gbps port > Switch port at 5Gbps > Switch port at 10Gbps > 10Gbps Firewall

I would suggest getting a 5Gbps capable or nBase-T switch that can do 5Gbps as well and using that.

brwainer · March 19, 2022

6 hours ago, Method said:

However, in the switch's software (winbox), it says it's connected at 10Gb. The thinking here then is that it isn't negotiating 5Gb properly.

These RJ45 SFP+ modules are themself acting as a little two-port switch, because the SFP+ standard doesn’t technically support anything in between 1Gb and 10Gb. So the module links to its host at 10Gb, and the RJ45 side at whatever. There is also nothing in the standard for communicating this information, or for RJ45 modules to exist at all. If you look at the module type as reported by the switch, it is probably showing up as a DAC or even maybe a fiber module.

Mikrotik’s own S+RJ10 module uses one of the unused pin pairs to make an extra metadata connection to pass the RJ45 link speed, this is something proprietary to Mikrotik. It might be a waste of money now, but if you had asked before I would have recommended spending extra on the Mikrotik module purely for this reason. No other company has done this that I’m aware of.

https://mikrotik.com/product/s_rj10

Quote

The latest revision of S+RJ10 contains "/r2" by the end of serial number. It comes with following improvements:

- Jumbo frames up to 10218 Bytes are now supported for rates 2.5Gbps, 5Gbps and 10Gbps;

- Link speed reporting is fixed - actual link speed in the interface menu is listed. Previously 100M/1G/2.5G/5G link speeds were not reported correctly;

- DDM monitoring (Supply Voltage, Module temperature) is now supported.

They say “link speed reporting is fixed” in the 2nd revision, but truthfully it wasn’t possible to do when they were keeping within the SFP+ standards. They found a solution such that their module can still be used in other manufacturer’s equipment without issue.

So, all that out of the way, what are you using as your router?
Edit: I’m guessing it is the modem unit? Try putting the RJ45 module into one of your Connect X-3 cards and connecting that to the modem/router, just for more data points.