VPLS tunnel between sites

[leadeater]

So I've setup a VPLS tunnel between two Mikrotik hEX's lab style, directly connected + private WAN IPs and it works perfectly. Can get a layer 2 tunnel between the two routers and pop traffic in and out of them at full 1Gbps speed.

 

Now the question/issue is, can this be migrated to an over the internet setup without any ISP assistance at all between two sites on different ISPs?

 

I changed all the tunnel endpoints to the actual WAN interface IPs, changed the LSRs etc, LDP negotiates and establishes and I can see the remote networks in the MPLS information but the VPLS tunnel just will not go active. The biggest difference between the lab setup and the real thing is one of the sites uses a PPPoE VLAN10 tagged interface for the WAN, messed with the default firewall rules but I think/thought I had that sorted out except for the VPLS tunnel not actually working.

 

Will this just not work across the internet like this? Is ISP MPLS awareness absolutely required? Do I need to create a GRE tunnel and do the MPLS/VPLS inside of that (kinda yuck)?

 

Basically the performance was just so outstandingly awesome compared to an IPSec tunnel (ERLite-3 pathetic 30-60Mbps) I really want to get it working using VPLS.

 

Anyone got any ideas?

@Lurick

@KuJoe

@LAwLz

 

[Lurick]

I haven't really done much with VPLS so unfortunately I'm not going to be the best resource but you could definitely try with a GRE tunnel and see if that at least gets the VPLS to come up that way.

 

@mynameisjuan

[brwainer]

I think you should give EoIP a try since you presently have Mikrotik at both ends:

https://wiki.mikrotik.com/wiki/Manual:Interface/EoIP you can run EoIP directly as is, or it will integrate directly with IPSec which on the hEX is hardware accelerated and capable of 470Mbps: https://www.manitonetworks.com/mikrotik/2016/3/9/eoip-tunnel - the only thing I’m not clear about is where the simple/direct IPSec setup built into the EoIP is compatible with the hardware acceleration of the hEX, you might have to build an IPSec tunnel using the hardware accelertation profile, then EoIP on top of that - at that point I guess you could use VPLS on top instead.

[leadeater]

1 minute ago, brwainer said:

I think you should give EoIP a try since you presently have Mikrotik at both ends:

https://wiki.mikrotik.com/wiki/Manual:Interface/EoIP you can run EoIP directly as is, or it will integrate directly with IPSec which on the hEX is hardware accelerated and capable of 470Mbps: https://www.manitonetworks.com/mikrotik/2016/3/9/eoip-tunnel - the only thing I’m not clear about is where the simple/direct IPSec setup built into the EoIP is compatible with the hardware acceleration of the hEX, you might have to build an IPSec tunnel using the hardware accelertation profile, then EoIP on top of that - at that point I guess you could use VPLS on top instead.

EoIP is my fallback plan but VPLS is a decent amount faster, especially in the smaller packet sizes.

[mynameisjuan]

8 hours ago, Lurick said:

 

 

@mynameisjuan

-mikrotik

-vpls

-mpls

 

Well I thought my heart murmurs were gone....

 

We deploy mikrotik and I do have a ton of experience with it but I never tried vpls. I can try it in my lab quick

[brwainer]

2 hours ago, mynameisjuan said:

-mikrotik

-vpls

-mpls

 

Well I thought my heart murmurs were gone....

 

We deploy mikrotik and I do have a ton of experience with it but I never tried vpls. I can try it in my lab quick

The problem @leadeaterhas is that it works in the lab but not real life for him.... but I also look forward to anything youfigude out.

[leadeater]

3 hours ago, mynameisjuan said:

Well I thought my heart murmurs were gone....

Don't like Mikrotik much? So far better than the ERLite but the hEX model without a proper VLAN capable switch chip and the whole bridge VLAN software stuff is just weird. Getting a port to be a trunk port for VLANs and also getting that port properly working with a different PVID is far as I can tell not quite working correctly...

 

At the stage of 2 steps forward and 1 step backward.

 

Very tempting to just throw more money at this and get something a lot better but at that point the number of options sky rockets and so does the possibilities of different tunneling options, like using VXLAN.

 

Might just end up tucking away the second hEX and continue to use my Fortigate 60D at my end of the tunnel.

[mynameisjuan]

10 minutes ago, leadeater said:

Don't like Mikrotik much?

No I love Mikrotik, its the backasswards configuration it needs at times that ruffles my jimmies.

 

12 minutes ago, leadeater said:

Getting a port to be a trunk port for VLANs and also getting that port properly working with a different PVID is far as I can tell not quite working correctly...

Vlan trunking took me weeks to understand mikrotiks logic and to reliable setup and troubleshoot. I have a config with native vlan working on the trunk port. I just have to find it lol

 

 

[leadeater]

Just now, mynameisjuan said:

No I love Mikrotik, its the backasswards configuration it needs at times that ruffles my jimmies.

Coming from using other equipment omg yes I agree with that so much lol

[LAwLz]

Sorry but I have not used VPLS before.

I don't see why it wouldn't work if the tunnels are pointed at the WAN addresses and both routers can reach each other over non-tunneled traffic.

[mynameisjuan]

On 12/22/2018 at 7:28 PM, leadeater said:

Coming from using other equipment omg yes I agree with that so much lol

Are you able to post or PM me your config (export command) so I can compare it with mine? Please remove any private info (IP, uname, passwd, just replace them with XXX)

[leadeater]

@mynameisjuan

@Lurick

 

Got it working, did need to setup a GRE tunnel between the routers. After that gave the GRE interfaces IP addresses, added static routes for the loopback interfaces (private IPs), setup MPLS/LSR to use those loopback interfaces and to listen to LDP on the GRE interface. Setup VPLS with remote peers pointing to the loopback IPs, created a new bridge and added the VPLS interface and enthernet interface to the bridge.

 

Plugged laptop at Site 1 in to tunneled ethernet interface and it got DHCP address from Site 2 DHCP server (the hEX router) and could ping local devices on the same ethernet segment transparently as far as the client device can tell.

 

Speed test from the laptop at Site 1 tunneled through to Site 2 and out it's internet connection.

 

Both sites are 100/100 connections. Upload seems a little off, might need to do some tuning, possibly MTUs? I'm betting those are totally messed up because tunnel in a tunnel all hidden from the devices actually using it. Might also need to set firewall mss clamp.

 

[leadeater]

Also in case anyone does for some weird random reason find this and tries to copy it please note the tunnels are not encrypted only encapsulated, use IPSec on the GRE tunnel. I'm not using it because I want to do performance comparisons with and without it.

[Falconevo]

Do the Microtiks not support VXLAN?  I would imagine that would of been a better fit unless im missing something with the desired end result.

 

Also good job getting it sorted

[leadeater]

16 hours ago, Falconevo said:

Do the Microtiks not support VXLAN?

I wish, tried looking for something cheap that supported VXLAN and basically nope. Open vSwitch as far as I know is the only realistic choice unless there is some cheap used equipment on ebay that supports it I'm not aware of.

 

After running a lot of my networking stuff in the past as VMs, like Open vSwitch, it gets too annoying/disruptive when doing configuration changes means no internet at all. If I were to use Open vSwitch I'd have to build something dedicated for it or use something like this https://northboundnetworks.com/products/zodiac-gx.

 

Open to any VXLAN options you know of.

[Lurick]

2 hours ago, leadeater said:

I wish, tried looking for something cheap that supported VXLAN and basically nope. Open vSwitch as far as I know is the only realistic choice unless there is some cheap used equipment on ebay that supports it I'm not aware of.

 

After running a lot of my networking stuff in the past as VMs, like Open vSwitch, it gets too annoying/disruptive when doing configuration changes means no internet at all. If I were to use Open vSwitch I'd have to build something dedicated for it or use something like this https://northboundnetworks.com/products/zodiac-gx.

 

Open to any VXLAN options you know of.

Couple ASR1002-HX and maybe an N7702 or two

[leadeater]

1 minute ago, Lurick said:

Couple ASR1002-HX and maybe an N7702 or two

Sure, if you'll sell them to me for under $1000 lol.