LAN setup from hell

[SpaceGhostC2C]

Dear forum, I'm back in search of wisdom. I'm encountering multiple problems in my current LAN setup, and while I have a couple of suspects, everything tells me I should be getting better results or worse results, but not the exact results I get. It makes no sense to me at this point.

 

Let me preface with two things:

1) I know there are many ways to connect two computers, so having "a" connection could be achieved by trying a different configuration and see how it goes. However, the goal here is to get this layout to work, or alternatively, understanding why it can't (as in, shouldn't) work, and then move to something else. If I just quit I'll learn nothing.

2) This is a graph of my current setup. Each line represents a 1Gig Ethernet connection. Ellipses represent bonds of ethernet connections working as one (LACP a.k.a. 802.3ad). Wireless connections go directly to the ISP modem, and are in the same space as the modem and the PFSense WAN (say, 192.168.xx.zz). PFsense LAN and all the PCs are in 192.168.yy.zz, xx=/=yy. Dashed lines delimit rooms, connections between rooms go through the walls.

 

Spoiler

 

The problems

 

1) The old: It all started after installing the PFSense router (although I remember some previous oddities preparing this post, more on that later). I made a thread about it here:

Spoiler

 

In a nutshell, the connection between PFSense and Switch breaks down, triggering a "possible flapping" message on the PFSense logs. It goes back up with no user intervention. The current state of the problem is reproducible: it happens every time I turn on or turn off either of PC1, PC2 or, since Installing that, PC3 (I've checked ans it's not exactly at power up, but when the OS starts loading). If you are using any of the PC1 or PC2, you're suddenly limited to the Switch, then after waiting a bit (varies between seconds and minutes) you can go all the way to the internet again.

 

2) The new: I installed Ubuntu Server on another PC last week, configuring a bond as main network interface, but without connecting to the internet during the installation. Once done, I connected the cables and try connect to the outside world to no avail. Now here's the puzzle: I can reach all the way from PC3 to the PFSense box and SSH into it. That is, the path between PC3 and PFSense seems no different than the path between PC1 and PC2, and PFSense. But PC3 gets no internet. The obvious candidate was DNS, but I checked with various built-in tools (route, netplan --debug, looking at resolv.conf and whatever the netplan config files are, etc), and in all cases it is correctly set to 1.1.1.1, 8.8.8.8 (it's the same in PC1 and PC2, PFSense is set to DNS Relay). But no internet.

 

I did some ping and web GUI / SSH tests, as follows:

 

PC1: can ping Switch, can access Switch GUI, can ping PFSense, can access PFSense GUI, can ping PC3, can ping 1.1.1. has internet

PC3: cannot ping Switch, can ping PFSense, can SSH into PFSense, can ping PC1, cannot ping 1.1.1., has no internet

 

I checked the PFSense logs and there's no trace of PC3 trying to reach the internet, or anywhere, and being blocked. The only signs of any blocking are stuff coming from my laptop (connected through WiFI to ISP modem) towards PFSense LAN and inwards. No clue what that is, but glad to have PFSense Anyway, laptop on or off does not affect my results. So that's my puzzle: I would understand having internet on PC3, and I would understand if PC3 couldn't reach PFSense due to some misconfiguration somewhere. I would understand less if PFSense was blocking PC3 while not the others, but at least it would be a lead to follow. But right now it's as if PC3 was simply not sending the requests to, say, 1.1.1.1 correctly, or at all

 

3) The unresolved oddities from the past

 

Before even having the PFSense box, everything seem to work fine (PC1 and PC2 to Switch same as now, Switch to ISP), but I remember now two issues that I overlooked:

i) PC1 dual-boots Windows and Linux Mint. Windows configured the team (a.k.a. bond) in two clicks, and worked since then. In Linux Mint I never got any Ethernet connectivity whatsoever as far as I remember, and 100% sure the team does not work at all in Linux to this day. I'm limited to WiFi on PC1 when on Mint.

ii) I have two available modes for the "Etherchannels" (how Cisco calls the teams/bonds) in my Switch: "Static" and "LACP". THe description of "Static" was not very informative to me, but my understanding is that the correct option for my setup is "LACP". That's how all the Linux / PFSense bonds are set, and they will not work if I change that to "Static" in the switch (while keeping the settings unchanged in the computers). This is the expected behavior, I think. However, the link between Switch and PC1 is set as "Static" in the Switch. Looking back, when first setting this up, there were instances in which it would work in Static but not LACP, it would work in LACP but not Static, neither, both, etc, until I finally got it all set. My understanding is that it shouldn't work because the team is set as LACP in PC1, but OK, I guess what does work is not my main concern  

 

So, at this point: what could be preventing PC3 to reach the internet through PFSense like the other computers? What's causing the link between PFSense and the Switch yo break down when a computer connected to the Switch initializes its NICs / powers them down? I'd blame the Switch, but then, why does a direct link between ISP and Switch not break in the same situation?

 

I do hope it's the Switch: While it would suck not being able to use it this way, I think I could setup a similar layout with PC3 acting as both server and switch. Still, I'd want to understand first what's wrong with the damned CE500G, and salvage it if possible.

 

Thanks in advance to anyone with 2 hours to read this post and able to provide any hint

 

[SpaceGhostC2C]

21 hours ago, SpaceGhostC2C said:

I do hope it's the Switch: While it would suck not being able to use it this way, I think I could setup a similar layout with PC3 acting as both server and switch. Still, I'd want to understand first what's wrong with the damned CE500G, and salvage it if possible.

 

Update: bad news for me. Bypassing the Switch and connecting PC3 directly to the PFSense gives worse results: no access to the PFSense whatsoever from PC3, no ping, no SSH, nothing (those were working through the Switch, only the last step -internet- was missing).

However, I'm getting no "possible flapping" message in the PFSense log with this layout. So it seems the "flapping" thing may be a PFSense-Switch thing, while the no internet is a PFSense-PC3 thing... maybe?

Still no clue what to try next, although I have to restore the switch to its place so PC1 and PC2 can access the internet.

[Alex Atkin UK]

Any particular reason why you are doing LACP to the pfSense router?  If this just so you have more bandwidth from the switch back to room 2?

 

Its really not a great configuration to have double-NAT and especially so with WiFi clients effectively being stuck on the WAN side of the pfSense box.  Would be far better to get a dedicated Access Point into the switch and put the modem into bridge mode.  You're kinda killing most of the benefits of using pfSense by allowing WiFi clients to bypass it and not having it do the WAN routing.

[SpaceGhostC2C]

1 hour ago, Alex Atkin UK said:

Any particular reason why you are doing LACP to the pfSense router?  If this just so you have more bandwidth from the switch back to room 2?

Yes. That would allow for PC3 to serve multiple devices in room 2 in the future (and/or bond the two links going to room 2).

I could always try a simple link from PFSense (I plugged a laptop to a port in room 2 and that seems unaffected by all this mess) as a test, but if that works it would narrow the problem to the PFSense LAGG interface. At least for Problem 1, Problem 2 seems specific to PC3 (going to try a different OS, given the dual-boot experience in PC1).

 

1 hour ago, Alex Atkin UK said:

Its really not a great configuration to have double-NAT and especially so with WiFi clients effectively being stuck on the WAN side of the pfSense box.  Would be far better to get a dedicated Access Point into the switch and put the modem into bridge mode.  You're kinda killing most of the benefits of using pfSense by allowing WiFi clients to bypass it and not having it do the WAN routing.

Yep, that's in the roadmap but first I have to finish having a working PFSense setup, since right now those direct WiFi connections are what's saving me during the ethernet experimentation, otherwise I'd have no way to google for answers while I try things  Plus I may keep it separate anyway, it's not clear that I want WiFi devices having access to the computers connected by cable. I could do that from inside PFSense, though.

Since I'm not exploiting PFSense as a Firewall right now (beyond whatever default rules), I could also check if the Switch would physically fit in the cabinet, then remove PFSense altogether from the equation and still have LAGGs to both rooms. Then I would have to make PC3 act also as a Switch for room 1. However, PC3 isn't very promising: after a few tests, not I got consistent failure: no more SSH into PFSense no ability to  ping anyone, despite the same configuration as yesterday  I really don't want to put that one on Windows...

[jagdtigger]

Are you using static arp entries by any cahance for thoe PC's? There was an issue with one my prebuilt NAS that t randomly switched to a different MAC(it jumped between the two interface MAC) then pfsense blocked it because the IP was tied to a different MAC address.

[Alex Atkin UK]

10 hours ago, SpaceGhostC2C said:

Yep, that's in the roadmap but first I have to finish having a working PFSense setup, since right now those direct WiFi connections are what's saving me during the ethernet experimentation, otherwise I'd have no way to google for answers while I try things  Plus I may keep it separate anyway, it's not clear that I want WiFi devices having access to the computers connected by cable. I could do that from inside PFSense, though.

Since I'm not exploiting PFSense as a Firewall right now (beyond whatever default rules), I could also check if the Switch would physically fit in the cabinet, then remove PFSense altogether from the equation and still have LAGGs to both rooms. Then I would have to make PC3 act also as a Switch for room 1. However, PC3 isn't very promising: after a few tests, not I got consistent failure: no more SSH into PFSense no ability to  ping anyone, despite the same configuration as yesterday  I really don't want to put that one on Windows...

I'd argue its not exactly a "working" configuration whilever its a double-NAT.   You're not going to really see any benefit from having pfSense sat there in this configuration, as the biggest benefit is usually that pfSense is quicker at routing, firewalling, etc than consumer routers.

 

As for keeping WiFi separate,  the easiest way to do that would be to have a second LAN on a different port on pfSense and plug a WiFi AP into that port.  Dedicated WiFi AP tend to perform much better than all-in-one routers, plus you can more easily place them for optimal signal range without struggling with wiring the broadband side up too.

[SpaceGhostC2C]

Thanks for your replies, @Alex Atkin UK.

 

12 hours ago, Alex Atkin UK said:

I'd argue its not exactly a "working" configuration whilever its a double-NAT.   You're not going to really see any benefit from having pfSense sat there in this configuration, as the biggest benefit is usually that pfSense is quicker at routing, firewalling, etc than consumer routers.

I don't know enough about NAT other than it exists and having to manually set it for a cluster some 10 years ago, but I think I can't do much about your last statement: my ISP provides one of these "modem-router" devices which takes the coaxial on one end and provides a bunch of ethernet + WiFi. So, ultimately, PFSense has to go through the ISP device to get to the outside world, and the latter isn't very knowledgeable about whether it's a proper router or a smartphone that it's sending the requests, it just does all of its default routing/"firewalling" anyways. Hence, whatever performance penalty it imposes on my WAN side, it's pretty much fixed. Maybe there is some setting to turn off part of what it does in its rather barren GUI, but I won't mess with that until everything goes through PFSense first.

 

 

Also, last night's update:

- replaced Ubuntu Server with CentOS, seemed to do a better/easier job at setting up the bond, it reported success everywhere, all slaves and the bond itself look like up and running. Except no ping to anywhere, no internet. May as well try installing PFSense on PC3 - it should work as well as the other  

- Tried disconnecting the PC3 bond and setting up a single link connection to the switch on different ethernet ports (both ends). Different IP in the appropriate range, everything else (gateway, DNS, mask) same as before (and same as PC1 and PC2). No ping, no internet. Tried setting the Switch as gateway (I don't know if it can be told to do that, but in its current config it shouldn't work), not like PC1 and PC2. Didn't work, as expected

- Tried connecting a Linux Mint laptop to the Switch. It's the same laptop I've been using to connect directly to PFSense from room 2. It failed to connect with DHCP, but it sill triggered the "possible flapping" message in PFSense (It looks more and more like the Switch gets some hiccup when detecting new links, and PFSense is particularly sensitive to it...). Since I set DHCP on the links to room 2, as everything in room 2 is static, I wasn't surprised. I Set a manual IP (same way that PC1 or PC2 in Windows) for the laptop, it did not work either. I mean, network manager reported a successful connection but no ping, no internet.

- Restarted the laptop in Windows. No connection on automatic as no DHCP to provide an IP address. I set it manually, same settings as in Linux Mint, and immediately got ping everywhere and internet access.

 

 

It looks like I may have one specific issue with LAGG in PFSense or in the particlar PFSense+Switch combination (problem 1), and I may have a more general problem getting the Switch to interact with Linux  Well, the switch would be the common denominator, at least  Except for the time I connected Ubuntu server directly to  PFSense with no luck... I should try bypassing PFSense and see if Linux->Switch->ISP does work, or somehow Linux can't get to any gateway through this Switch nor talk to PFsense without further tweaking compared to Windows. But I don't think "my Linux box can't talk to PFSense" is something that would go undetected, so... 

[Alex Atkin UK]

6 minutes ago, SpaceGhostC2C said:

Thanks for your replies, @Alex Atkin UK.

 

I don't know enough about NAT other than it exists and having to manually set it for a cluster some 10 years ago, but I think I can't do much about your last statement: my ISP provides one of these "modem-router" devices which takes the coaxial on one end and provides a bunch of ethernet + WiFi. So, ultimately, PFSense has to go through the ISP device to get to the outside world, and the latter isn't very knowledgeable about whether it's a proper router or a smartphone that it's sending the requests, it just does all of its default routing/"firewalling" anyways. Hence, whatever performance penalty it imposes on my WAN side, it's pretty much fixed. Maybe there is some setting to turn off part of what it does in its rather barren GUI, but I won't mess with that until everything goes through PFSense first.

There's two possibilities:

 

1) Many cable companies allow the option to switch their modem/routers into modem-only mode, but they may need to do that for you remotely if its nowhere in the routers UI.

2) Some cable companies let you buy your own modem.

 

Honestly I don't see the point in using pfSense if its not actually doing the routing.  Double-NAT means you are routing twice, but all the CPU intensive stuff is happening on the first router, negating the point of the second one.

[SpaceGhostC2C]

16 minutes ago, Alex Atkin UK said:

Honestly I don't see the point in using pfSense if its not actually doing the routing.  Double-NAT means you are routing twice, but all the CPU intensive stuff is happening on the first router, negating the point of the second one.

Well, I would still get the ability to act as a LACP router/switch (or so I wished ), and more flexibility in setting firewall rules, house-level blocking of sites, possible pi-hole replacement?, connecting to a VPN at one single point instead of every PC by itself... To be honest, I haven't even heard of performance benefits before, I arrived through PFSense from a different path: versatility.

 

Having said that, if I can turn the ISP's into modem only, all the better, I guess, I won't complain about performance advantages either. But I'll need the wireless AP set first. Long to-do list for one person in his spare time  One of the items being the ongoing troubleshooting in this thread 

[SpaceGhostC2C]

2 hours ago, SpaceGhostC2C said:

- Tried connecting a Linux Mint laptop to the Switch. It's the same laptop I've been using to connect directly to PFSense from room 2. (...) I Set a manual IP (same way that PC1 or PC2 in Windows) for the laptop, it did not work either. I mean, network manager reported a successful connection but no ping, no internet.

- Restarted the laptop in Windows. No connection on automatic as no DHCP to provide an IP address. I set it manually, same settings as in Linux Mint, and immediately got ping everywhere and internet access.

(...)

and I may have a more general problem getting the Switch to interact with Linux  

(...) 

or somehow Linux can't get to any gateway through this Switch nor talk to PFsense without further tweaking compared to Windows.

OK, one thing sorted it out: Linux requires a different configuration of the Switch than Windows. Got the laptop to connect on Linux by doing the following:

- Set the relevant "smartport role" to "Desktop" in the Switch

- Manually set the link speed to "1000 full duplex" in both Linux and the Switch (as opposed to the auto-negotiate default).

 

The second  point speaks for itself, but what about the first one? Well Cisco has (or had in the ancient times of the CE500G) a role-specific "optimized" configuration for the ports:

Spoiler

"Desktop" or "Server" seems the natural choice for the role. I have, however, all ports set as "Switch". This doesn't seem to be a problem for Windows (whether 7, Server, or 10), but whatever the "Switch" behavior is, it doesn't play nice with Linux. Now, why would I have every port as "switch" instead of the appropriate role, you may ask? Well, it kind of is the appropriate role for my intentions given that is mandatory, and the only option, for "Etherchannels":

Spoiler

Obviously Cisco didn't have in mind bonding for workstations or server, or really anything other than deep trunks of the network (I'm sure the technical term is "layer something" or thereabouts ).

Hence, if I'm ever to have Linux bonds going through this switch, I'll need to find out what Windows does differently in order to not care about the "switch" role and replicate it in Linux. Otherwise, it's either an all-Windows network or putting PC3 in place of the switch (if I get PFSense to PC3 to work via LAGG, that is) and then have a direct bond to PC1, making everyone else on room 1 be single-link through the switch. It would suck, though; the original plan was a 4-way bond from PC3, a 2-way bond in each of PC1 and PC2, and either to two links or a bond  to room 2 

[SpaceGhostC2C]

So... I think I kind of sorted this out. It's been more of a blog than a thread, but in case anyone ever googles something similar, let's give it closure.

 

So, it kinda was the Switch in a way. It seems that proper 802.3ad LACP is feasible, but hard for it. And the implementation probably ever expected to be used between Cisco switches and some corners were cut or something. I'll try to break it down in parts:

 

1. "Smartport" roles: only Windows (and possibly other Cisco switches, can't test) seem to interact successfully with "switch" ports. PFSense can to some extent, but not ideal. No Linux flavor could do it, although Ubuntu Server got a bit further (ping locally, no route to WAN), but I don't know how and could not reproduce it. "Desktop" (to a laptop), "Router" (to PFSense) and "Server" (to PC3) all worked well with the corresponding OSes. That takes care of most of Problem 2.
2. Link aggregation: my switch can't do LACP with any port role other than "switch", but I still got teams / bonds / laggs to work. First, I noticed I made a mistake in previous post: the Intel NIC in PC1 (Windows) was set to "SLA", not LACP, consistent with the Switch's Etherchannel being "static", making me think that probably LACP wasn't doing that well with Windows either (static Etherchannel does work, though). Intel's SLA is equivalent to LACP in my case, since I'm really not doing anything dynamically, everything is fixed both in hardware and logical terms, and both modes balance outgoing traffic across interfaces and receive incoming on any. I looked for similar static options in PFSense and CentOS: for PFSense, "LOADBALANCE" looked equivalent to me, with the benefit of specifying "no switch negotiation required" (which makes me think a simple unmanaged switch would suffice? Not sure). Similarly, I went with "balance-alb" (mode 6) in CentOS, with similar description and no switch support required. I removed all Etherchannels from the switch except the Windows one, and set the port roles individually as mentioned above. I now have a working bond in CentOS, a working lagg in PFSense, a working team in Windows, and all (reported) link speeds are 2.0Gbps Oh, and tried rebooting PC3 while monitoring from PC1, and no experienced no downtime and no "possible flapping" message!

Now it's time to settle on a server OS for PC3, extend its bond to x4, complete the x2 team in PC2, and wait for the switch to catch fire so I can justify getting one from this century