Using multiple WAN Connections

The_DR · October 7, 2023

Hey, I need some help. At home, I have three WAN Connections.

1) Cable with 100 Down and 20UP

2) 5G with ISP1 with about 400/400 MBit/s

3) 5G with ISP2 with about 400/200 MBit/s

Connection 1 slow, but more or less reliable

ISP1 has a Bandwidth Limit of 100 GB/day, then the connection is throttled to 64kbit/s.

On some days, there are multiple connection outages of ISP1 and ISP2 at the same time.

So now the question is, what can I do with these connections to get the most from it?

1) I tried it with my Omada Router. The problem was, that I'm not able to set some limits. So the connection switch was IMHO to slow and unreliable.

2) I tried speedify. As standalone and also on SmoothWAN, a Router OS, that is using speedify. This also wasn't reliable. First, many sites just blocked me, because I was using a more or less VPN. Then the Bandwith was slow, I didn't get even once close to 400MBit, I get on the 5G Line. And when I had outages from ISP1 and ISP2 at the same time, I got connectivity Issues.

3) Currently I'm using OPNSense. But here I also have issues. The first Issue is, when I reach the 100 GB/day limit. Because OPNSense is using ping to monitor the connectivity, the line isn't taken down. The ping comes back. Baybe because the ISP is QoS the ping higher. On the other hand, often lines, that are down were not taken up again. Or connections are sticking to the down line instead of using a line, that is up. And I have to restart the gateway service, to make everything work again.

That's why I want to ask here: What is the best way to use three WAN Connections for my case? Do you have any suggestions? Because the current condition really sucks!

NobleGamer · October 7, 2023

I have multi WAN with my Edgerouter X, and unfortunately there is not a native way to configure bandwidth limits on a WAN like ISP1 has. ERX uses ping too. If it's possible it would require custom scripting of some kind.

1 hour ago, The_DR said:

The ping comes back. Baybe because the ISP is QoS the ping higher.

QoS is largely irrelevant with ping that takes up <1 kb (based on my own bandwidth monitoring of ERX pings) relative to even the 64kbps after hitting ISP1's cap.

1 hour ago, The_DR said:

Or connections are sticking to the down line instead of using a line, that is up. And I have to restart the gateway service, to make everything work again.

ERX has several config flags around connection stickiness in a WAN failover, and a 7 second failover with some level of sticky works well for me even when doing live video conferencing. Try to find & play with those equivalent settings for your multi WAN router. If your device doesn't, then it will be up to your client devices to determine that each TCP IP connection is failing, close each one when it determines that, and more broadly for the client device to determine that the overall current internet connection is down.