Removing double or triple NAT on network with EdgeRouter & 2 ISPs

[NobleGamer]

I recently acquired a 5G mobile modem/router from my mobile phone provider, since my primary ISP has unreliable connectivity, wired both modems to an Ubiquiti Edgerouter-X 5, configured failover from primary to 5G, and wired that to my existing network.

 

It works fine right now, but I wanted to know is it feasible to pair down the amount of NATing on my network? I know that mutli-NATed network *may* be susceptible to some connectivity issues when there is more than one router at play.

 

So let me break down the different configured subnets:
Primary Modem: 98.XXX.XXX.XXX/22 (modem only)
5G Modem/Router: 192.168.XXX.XXX/24 (different from below subnets)
EdgeRouter-X: 192.168.100.1/24 (switch0 interface)
Gryphon Router/AP: 192.168.1.1 (/24?), with device DHCP range: 192.168.1.100-249

 

I think this results in routes to/from the Primary modem being double NAT: Primary Modem (No NAT) -> ER-X (NAT) -> Gryphon (NAT)

 

Is it advisable to remove the ER-X's Source NAT rule to Masquerade for the Primary Modem if my ER-X is only connected to the one Gryphon Router?  Would this interfere with when a failover occurs that changes routes from Primary Modem to 5G?

 

Then there's the routes to/from 5G modem/router that might be a triple NAT: 5G modem/router (NAT) -> ER-X (NAT) -> Gryphon (NAT)

 

I know that there is no way to remove NAT from the 5G modem/router, but what about removing it from the ER-X?  Would that work alright in a failover scenario?

 

As for the Gryphon Router: It seems NAT is built-in and cannot be disabled. All I can configure is LAN IP, static & DHCP Ranges for devices, and configure WAN for DHCP, Static IP, or PPPoE.

 

I know that the ideal solution would be to ditch Gryphon router+mesh alltogether, but I can't afford to replace it with an AP-only hardware, like a Ubiquiti AP.  I would also like to avoid setting any additional manual IP addresses if possible, aside from switch0 interface.

 

Lastly, is it possible for my ER-X to do traffic analysis of my Gryphon's subnet by changing ER-X or Gryphon config such as changing & broadening the ER-X's "switch0" interface's subnet, or does each router need to continue to have separate subnets?

 

I think my ER-X config is relatively simple, but here it is sanitized below, and also my network diagram below that if you are curious what's downstream from the routers:

Spoiler

firewall {
    all-ping enable
    broadcast-ping disable
    group {
        network-group PRIVATE_NETS {
            description "Excludes subnets of modems"
            network 192.168.0.0/21
            network 172.16.0.0/12
            network 10.0.0.0/8
            network 192.168.8.0/22
            network 192.168.13.0/24
            network 192.168.14.0/24
            network 192.168.15.0/24
            network 192.168.16.0/20
            network 192.168.32.0/19
            network 192.168.64.0/18
        }
    }
    ipv6-receive-redirects disable
    ipv6-src-route disable
    ip-src-route disable
    log-martians disable
    modify balance {
        rule 10 {
            action modify
            description "do NOT load balance lan to lan"
            destination {
                group {
                    network-group PRIVATE_NETS
                }
            }
            modify {
                table main
            }
        }
        rule 20 {
            action modify
            description "do NOT load balance destination public address"
            destination {
                group {
                    address-group ADDRv4_eth0
                }
            }
            modify {
                table main
            }
        }
        rule 30 {
            action modify
            description "do NOT load balance destination public address"
            destination {
                group {
                    address-group ADDRv4_eth1
                }
            }
            modify {
                table main
            }
        }
        rule 70 {
            action modify
            modify {
                lb-group G
            }
        }
    }
    name WAN_IN {
        default-action drop
        description "WAN to internal"
        rule 10 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
        rule 21 {
            action drop
            description "Drop 5G to NAS"
            destination {
                address 192.168.1.***
            }
            log disable
            protocol all
            source {
                group {
                    address-group ADDRv4_eth1
                }
            }
            state {
                established disable
                invalid disable
                new enable
                related disable
            }
        }
    }
    name WAN_LOCAL {
        default-action drop
        description "WAN to router"
        rule 10 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
    }
    receive-redirects disable
    send-redirects enable
    source-validation disable
    syn-cookies enable
}
interfaces {
    ethernet eth0 {
        address dhcp
        description Cable
        duplex auto
        firewall {
            in {
                name WAN_IN
            }
            local {
                name WAN_LOCAL
            }
        }
        speed auto
    }
    ethernet eth1 {
        address dhcp
        description 5G
        duplex auto
        firewall {
            in {
                name WAN_IN
            }
            local {
                name WAN_LOCAL
            }
        }
        speed auto
    }
    ethernet eth2 {
        description LAN
        duplex auto
        speed auto
    }
    ethernet eth3 {
        duplex auto
        speed auto
    }
    ethernet eth4 {
        duplex auto
        poe {
            output off
        }
        speed auto
    }
    loopback lo {
    }
    switch switch0 {
        address 192.168.100.1/24
        description Local
        firewall {
            in {
                modify balance
            }
        }
        mtu 1500
        switch-port {
            interface eth2 {
            }
            interface eth3 {
            }
            interface eth4 {
            }
            vlan-aware disable
        }
    }
}
load-balance {
    group G {
        exclude-local-dns enable
        flush-on-active enable
        gateway-update-interval 60
        interface eth0 {
            route-test {
                count {
                    failure 2
                    success 4
                }
                initial-delay 30
                interval 7
                type {
                    default
                }
            }
        }
        interface eth1 {
            failover-only
            route-test {
                count {
                    failure 2
                    success 6
                }
                initial-delay 60
                interval 15
                type {
                    default
                }
            }
        }
        lb-local disable
        lb-local-metric-change disable
        sticky {
            dest-addr enable
            dest-port enable
            source-addr enable
        }
    }
}
protocols {
    static {
        interface-route 192.168.1.***/32 {
            next-hop-interface eth0 {
                description "NAS to Cable Modem"
            }
        }
    }
}
service {
    dhcp-server {
        disabled false
        hostfile-update disable
        shared-network-name LAN {
            authoritative enable
            subnet 192.168.100.0/24 {
                default-router 192.168.100.1
                dns-server 192.168.100.1
                lease 86400
                start 192.168.100.38 {
                    stop 192.168.100.243
                }
                static-mapping Gryphonb*** {
                    ip-address 192.168.100.38
                    mac-address 1c:49:7b:d8:b0:40
                }
            }
        }
        static-arp disable
        use-dnsmasq disable
    }
    dns {
        forwarding {
            cache-size 300
            listen-on switch0
        }
    }
    gui {
        http-port 80
        https-port 443
        older-ciphers disable
    }
    nat {
        rule 5000 {
            description "masquerade for WAN"
            outbound-interface eth0
            type masquerade
        }
        rule 5002 {
            description "masquerade for WAN 2"
            outbound-interface eth1
            type masquerade
        }
    }
    ssh {
        port 22
        protocol-version v2
    }
    ubnt-discover {
        disable
    }
    unms {
        disable
    }
}
system {
    analytics-handler {
        send-analytics-report true
    }
    conntrack {
        expect-table-size 4096
        hash-size 4096
        table-size 32768
        tcp {
            half-open-connections 512
            loose enable
            max-retrans 3
        }
    }
    crash-handler {
        send-crash-report true
    }
    host-name EdgeRouter-X5
    login {
        user *** {
            authentication {
                encrypted-password ***
                plaintext-password ""
            }
            level admin
        }
    }
    ntp {
        server 0.ubnt.pool.ntp.org {
        }
        server 1.ubnt.pool.ntp.org {
        }
        server 2.ubnt.pool.ntp.org {
        }
        server 3.ubnt.pool.ntp.org {
        }
    }
    offload {
        hwnat enable
        ipsec enable
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone America/New_York
    traffic-analysis {
        dpi enable
        export enable
    }
}
traffic-control {
    smart-queue 5G {
        upload {
            ecn enable
            flows 1024
            fq-quantum 1514
            limit 10240
            rate 20mbit
        }
        wan-interface eth1
    }
    smart-queue Cable {
        upload {
            ecn enable
            flows 1024
            fq-quantum 1514
            limit 10240
            rate 30mbit
        }
        wan-interface eth0
    }
}

/* Warning: Do not remove the following line. */
/* === vyatta-config-version: "config-management@1:conntrack@1:cron@1:dhcp-relay@1:dhcp-server@4:firewall@5:ipsec@5:nat@3:qos@1:quagga@2:suspend@1:system@5:ubnt-l2tp@1:ubnt-pptp@1:ubnt-udapi-server@1:ubnt-unms@2:ubnt-util@1:vrrp@1:vyatta-netflow@1:webgui@1:webproxy@1:zone-policy@1" === */
/* Release version: v2.0.9-hotfix.4.5521907.220630.0657 */

Spoiler

Thanks for reading.

[Needfuldoer]

Can you set the Gryphon router up so it just acts as a switch and an access point?

[Ralphred]

17 minutes ago, NobleGamer said:

Is it advisable to remove the ER-X's Source NAT rule to Masquerade for the Primary Modem

You'll need SNAT but masquerade/overload is overkill, but the only issue you'll have is latency, but if it isn't being effected by the the connection tracking then it's moot really.

 

25 minutes ago, NobleGamer said:

As for the Gryphon Router: It seems NAT is built-in and cannot be disabled.

Well, if you don't plug into the WAN port it kind of is disabled, it's just a fancy switch with an AP and bunch of unused routing protocols, but then your masquerade/overload would need to be on the ubiquity router, and the gateway of last resort offered up with DHCP leases (assuming that the server  stays on the Gryphon) would have to point to the actual gateway (re: the LAN IP of ER-X)

27 minutes ago, NobleGamer said:

or does each router need to continue to have separate subnets?

Depends if you are asking them both to route or not, something will shit the bed if there are "different networks" with the same network address though.

 

If that 5G modem has to route, then it has to NAT, but it doesn't have to masquerade or owt a static SNAT rule will suffice, everything goes to the ER-X on a single IP, as does the (bridge mode?) modem. It's The gryphon that is superfluous in this set-up, and You can just side step it by not letting it route.

[NobleGamer]

10 minutes ago, Needfuldoer said:

Can you set the Gryphon router up so it just acts as a switch and an access point?

I would have done that if I could, but there is no config that turns off the router capabilities. Gryphon is targeted at non-tech people in that it has relatively limited configuration compared to say ASUS wifi routers.

 

4 minutes ago, Ralphred said:

You'll need SNAT but masquerade/overload is overkill, but the only issue you'll have is latency, but if it isn't being effected by the the connection tracking then it's moot really.

When I ran the ER-X config wizard for failover, it auto configured SNAT masquerade for each WAN.

When I look at relevant EdgeRouter docs, a functional difference (aside from config options) wasn't clear to me between a SNAT "masquerade" type and a SNAT of type "source", given that there's only one non-WAN physical port on ER-X used either way.

So does a SNAT of source with this pseudo config somehow result in less translations or less latency than masquerade?:

Outbound Interface = [ISP port]

Type = Source

Protocol = All

Outside address = [ISPs IP subnet]

Source address = [Gryphon Router subnet]

 

Quote

Well, if you don't plug into the WAN port it kind of is disabled, it's just a fancy switch with an AP and bunch of unused routing protocols, but then your masquerade/overload would need to be on the ubiquity router

I may consider trying that to remove the Gryphon's NAT, thanks.

 

Quote

Depends if you are asking them both to route or not, something will shit the bed if there are "different networks" with the same network address though.

I cannot ask the Gryphon to *not* route, but like you said I could plug ER-X into the Gryphon's non-WAN port and force Gryphon to not route.

 

[NobleGamer]

 

3 hours ago, Ralphred said:

You'll need SNAT but masquerade/overload is overkill, but the only issue you'll have is latency, but if it isn't being effected by the the connection tracking then it's moot really.

So ER-X SNAT gives me 2 translation types: "Use Masquerade" and "Specify address and/or port". TIL I cannot use the latter to specify the subnet of either ISP because I get this error: Translation subnet is not on a network boundary

 

Its pointless for me to specify a single  translation IP per ISP when either ISP could generate a new IP from their subnets at anytime. So that leaves me with SNAT translation type of Masquerade.

[Ralphred]

9 hours ago, NobleGamer said:

So that leaves me with SNAT translation type of Masquerade.

Like I said, it can be moot anyway. The only difference would be when you are masquerading/overloading, the router is "creating" DNAT/SNAT rules on the fly and remembering where each one "points to".

For example (in a standard single router set-up) your phone sends an http request to google.com from ip 10.0.0.2:[some port], this ends up on your routers LAN interface and it forwards the request via it's WAN interface, this is when the SNAT rule is applied, so google gets a request from an address it can actually reach (re: not a private class a|b|c) . When it comes back from google the DNAT rule starts playing a part, first it looks up who the request is for in it's "masquerade table" and DNAT is applied to the reply so it gets back to your phone. This needs to be "dynamic" as, if your pc was to make a request to google.com at the same time we don't want to be sending the packets back to the wrong device.

 

In your case the ER-X can't "send them back" to the wrong device, as it's always being sent back to the gryphon router which does the actual connection tracking/masquerading for the LAN network address(es). So if you consider that with a single DNAT/SNAT pair of rules you can create a situation where every possible dynamic DNAT/SNAT pair can be replicated without the overhead of [finding the right rule/looking up the right rule] you **might** experience lower latency << But this depends on how the UI manifests the rules you have asked to be created within the routing software (I would *expect* Ubiquity to do it "properly", especially in the case of the ER-X, but they are (IME) one of these borderline mfgrs that sometimes leave me thinking "G*d d*mn Ubi, I thought you'd do that properly!").

 

If the ER-X UI made it difficult for *me* to enter the SNAT rule in my mind i wanted to enter, and I wasn't having latency issues, I'd probably be "F*ck you then!" and leave it masquerading (or look into it's "DMZ" capabilities, 'cos I'm a stubborn b*stard).

 

Similar principles apply to the NAT rules on the 5G modem, as it's always sending packets on the LAN side to the same address, the ER-X.

 

 

Once your packets leave from your WAN address, all sorts of routing happens within your ISP's own network space before it reaches the big bad internet space (where even more goes on), but as none of this is [stateful/involves NAT contracking] the added latency is negligible: Ideally this is the kind of routing you would be doing within your own network, but so many consumer grade routers assume they are "the only router in the village" you can't turn off NAT, as you have experienced.

[Ralphred]

Just as a point of interest, when connecting two devices (as in your 5G router to the ER-x) where no other devices will ever be introduced, I use /30 network addresses to do so. In your example I'd assign 192.168.AAA.0/30 between these two so:

• 192.168.AAA.0 is the network address
• 192.168.AAA.1 is the Modem address
• 192.168.AAA.2 is the ER-X address
• 192.168.AAA.3 is the broadcast address 

If you wanted to be at the other end of the /24 address range:

• 192.168.AAA.252 is the network address
• 192.168.AAA.253 is the Modem address
• 192.168.AAA.254 is the ER-X address
• 192.168.AAA.255 is the broadcast address

[NobleGamer]

On 9/24/2022 at 8:46 AM, Ralphred said:

Just as a point of interest, when connecting two devices (as in your 5G router to the ER-x) where no other devices will ever be introduced, I use /30 network addresses to do so.

Maybe this is where I'm getting a bit over my head in terms of my networking knowledge: I thought the 5G router is the one dishing out IP addresses via DHCP to whatever interfaces connect to it, in this case the ER-X, and that results in ER-X auto creating a route of 192.168.XXX.XXX/24 for 5G's eth0 ethernet interface?

 

Are you saying that even though I don't have a static IP address from 5G, I can configure the ER-X ethernet interface to use a specific IP address, and as long as I pick one address within that defined subnet /24, then I can "keep" that IP forever even if 5G uses DHCP?  I thought static mappings had to be done on the DHCP server side (in this case, the 5G modem, which does not have DHCP/IP Config of any kind)?

 

I suppose that might resolve the non-masquerade config I tried earlier...

 

On 9/24/2022 at 8:31 AM, Ralphred said:

The only difference would be when you are masquerading/overloading, the router is "creating" DNAT/SNAT rules on the fly and remembering where each one "points to".

I tried SNAT+DNAT rules along with verifying that existing firewall policies supported them, and it didn't work.  Maybe its the fact that a specific singular translation address has to be defined for NAT rules to avoid masquerading.  The error I get is "Translation subnet is not on a network boundary".

Unfortunately, I don't have a static IP, and I'm not going to setup NAT rules for every single IP in the subnet range that I have access to, so I think I'll pass on non-masquerade.  Like you said, it may not even make a difference.

As for the earlier idea to connect ER-X to Gryphon's LAN (not WAN port), that made me realize that there was certain filtering capabilities that I wanted to retain from Gryphon.  Unfortunately, those filtering capabilities aren't just DNS-side, because if they were I'd just plug into Gryphon's LAN port, point ER-X DNS to Gryphon, and call it a day.  So I'm going to live with the multi-NAT involving the Gryphon.