VLAN Headaches....

[OhioYJ]

So perhaps my planning is wrong, or I have a lack of understanding. Hopefully someone more knowledgeable will see something I'm not. So I'm using a NUC running pfsense, and have a Netgear XS512EM ("plus" switch, supports VLANs)

 

These devices all have dedicated LAN ports:

Spoiler

Bed Room 1
Kids PC

 

Bed Room 2
HTPC
Steam Link

 

Bed Room 3
Gaming Desktop
Server

 

Living Room
HTPC
Xbox
Steam Link

 

Laundry Room
AP1 (obviously these access points are not located in the laundry room just the ports end up by the switch in there)
AP2
Alarm
Firewall

Then I setup these VLANs in PFSense:

 

Spoiler

2 - Kids

3 - Wifi

4 - IOT

5 - PCs

 

All of these are active interfaces, in pfsense, and each has dhcp enabled. They each just go 192.168.x.100 (x) being the vlan tag ID

This is where I start to run into problems. If I go into the switch I think I've setup the VLANs correctly. I could have messed up the untagged / tagged (trunk) selection though:

 

Spoiler

So I added all the devices to their associated VLAN IDs

 

 

For reference this is how they are plugged into the switch:

 

1. Firewall

2. AP1

3. AP2

4. Bedroom 2 HTPC

5. Gaming Desktop

6. Server

7. Kids PC

8. Living Room HTPC

9.  Xbox

10. Steam Link

11. Steam Link

12. Alarm

 

When I try and "activate" (change the Port PVID) the Wifi VLAN my first access point gets assigned the the new correct ip address as I would expect but the second goes offline? I don't have any rules listed in pfsense yet for this VLAN, but being on different ports on the switch they at least get an IP?

 

This is why I think perhaps my layout is wrong. I can't assign the firewall or APs multiple PVIDs, but in my head, I was thinkng WIFI should have it's own VLAN?

 

This would be my end goal:

Spoiler

- I would like to isolate my IOT devices, which some are wifi, some are connected directly to the alarm.

- I would like to still be able to keep my son's devices in a separate rule set that I can schedule.

- I still need all the desktop's, laptops (wifi), and HTPCs to be able to talk to the "server" (TrueNAS)

My apologies for all the spoiler tags, I was trying to prevent this from being a wall of text.

[leadeater]

35 minutes ago, OhioYJ said:

When I try and "activate" (change the Port PVID) the Wifi VLAN my first access point gets assigned the the new correct ip address as I would expect but the second goes offline? I don't have any rules listed in pfsense yet for this VLAN, but being on different ports on the switch they at least get an IP?

Did you set port 2 and port 3 both to PVID 3 at the same time?

[OhioYJ]

2 minutes ago, leadeater said:

Did you set port 2 and port 3 both to PVID 3 at the same time?

Yes (changed both ports at the same time). I left port 1 (the firewall) as PVID 1. I also just tried adding a rule for allow all on VLAN Wifi. The First AP gets the correct IP, the second goes offline. No devices see either AP though, I lose all Wifi all together. (to be fair I didn't open my scanner app to see if it was transmitting, they just might not have internet.)

[leadeater]

36 minutes ago, OhioYJ said:

This is why I think perhaps my layout is wrong. I can't assign the firewall or APs multiple PVIDs, but in my head, I was thinkng WIFI should have it's own VLAN?

No you cannot, PVID is the same thing as untagged or the default VLAN. A port can only have a single PVID and usually a client/endpoint device would use that such as a PC where you wouldn't have VLAN tagging being done.

 

Firewalls, switches, servers, enterprise APs etc would all be devices that would use a trunk port with multiple VLAN IDs assigned.

 

When going between switches, firewall, APs and other networking devices each hop in the chain must have the VLAN ID on the trunk port or the frame will not be allowed to pass. Once you get to the last hop you must then decide if you want the VLAN tag removed from the frame aka untagged which would typically be if it were a PC.

 

Based on your requirements I will list what I think the PVID for each port should be:

Port 1: PVID 1

Port 2: PVID 3

Port 3: PVID 3

Port 4: PVID 5

Port 5: PVID 5

Port 6: PVID 1

Port 7: PVID 2

Port 8: PVID 5

Port 9: PVID 5

Port 10: PVID 5

Port 11: PVID 5

Port 12: PVID 4

 

 

VLAN 1 does not need to be tagged on any other ports at all

 

Port 1 is the only one that needs to be a trunk with multiple VLANs assigned

 

[leadeater]

Also forgot to mention a consumer AP will not have VLAN tagging support so whatever device that connects to the AP will get an IP address from that same VLAN as the AP as everything will be using the PVID of the port the AP is connected to.

 

If you want wireless devices on different VLANs then you need a VLAN capable AP and either setup SSID to VLAN rules on the AP or a RADIUS server to assign the correct VLAN based on device type, MAC Address or username/password etc.

[OhioYJ]

16 minutes ago, leadeater said:

Also forgot to mention a consumer AP will not have VLAN tagging support so whatever device that connects to the AP will get an IP address from that same VLAN as the AP as everything will be using the PVID of the port the AP is connected to.

 

If you want wireless devices on different VLANs then you need a VLAN capable AP and either setup SSID to VLAN rules on the AP or a RADIUS server to assign the correct VLAN based on device type, MAC Address or username/password etc.

 

They are Ubiquiti Unifi Access points, according to the datasheet they should support VLANs? If I'm reading it correctly. However that maybe just forwarding of tags. I'll have to look into that more.

 

I'm saying "server", it is a TrueNAS PC. It still counts as a Trunk device? I was originally assuming it would belong in the PC group. Or is that so other PCs can connect to it? Sorry every time I think I fully understand this trunk / tagged concept, something throws it off.  My understanding was it was for connecting other devices, switchers, routers, APs, essentially?

 

If my understanding is correct, after all the PVIDs are set, then I would just setup rules in pfsense to allow the devices to talk between these networks if need be? Like obviously everything will need to be able to reach the firewall again at a minimum to see the internet.

[leadeater]

8 minutes ago, OhioYJ said:

They are Ubiquiti Unifi Access points, according to the datasheet they should support VLANs?

Yes they do so that's good

 

For your situation I would just create multiple SSIDs and connect your IOT devices to the IOT SSID which should be setup to use the IOT VLAN within the AP configuration. That means the ports to the APs should be trunk ports with multiple VLANs with the PVID left as VLAN 1.

 

I haven't touched a Unfi AP in a long time though so I can't give you specific how to on this but their docs aren't too bad.

 

12 minutes ago, OhioYJ said:

I'm saying "server", it is a TrueNAS PC. It still counts as a Trunk device? I was originally assuming it would belong in the PC group.

The server in this situation doesn't need to be a trunk device. Your pfsense firewall will take care of adding and removing the VLAN tags for you so devices on other VLANs will still be able to access the server. You will however need firewall rules to allow it. You can put it on the same VLAN as the other PCs and would probably be easier and better performance if you did so.

 

Where a server would typically use a trunk port is a VM host that is running multiple VMs and those VMs need to be on different VLANs.

 

15 minutes ago, OhioYJ said:

If my understanding is correct, after all the PVIDs are set, then I would just setup rules in pfsense to allow the devices to talk between these networks if need be? Like obviously everything will need to be able to reach the firewall again at a minimum to see the internet.

Yes you understand correctly

[leadeater]

Also just as a note for what might be getting you confused is that you have VLANs and routing involved in your equation and that is likely part of what might be tripping you up in your understanding.

 

Each switch vendor does it slightly differently and web interfaces can make implied changes based on what you do and you may not be aware of it.

 

First it's important that a switch port should be an access port and never a trunk port unless the port actually needs to be a trunk and carry multiple VLANs. This will simplify the configuration of the port and also make it more simple to understand how it works.

 

When a port is in trunk mode it needs to have a PVID which is the default VLAN for the port. When an Ethernet frame is going to leave the port, outbound, if it has the matching VLAN ID in the frame header that VLAN information will be removed and the receiving device will get the frame without a VLAN tag. When an Ethernet frame enters the port, inbound, without any VLAN headers the switch will add the VLAN ID that matches the PVID.

 

This is also how an access port works but it will always be doing this since only a single VLAN is allowed. That can be any VLAN ID and may be shown as the untagged VLAN or the PVID, I've seen it both ways in switch web interfaces. I've even seen it represented both ways in different pages on the web configuration. That's why I usually prefer to configure switches using CLI, it's actually less confusing in my opinion once you have enough understanding of the CLI commands.

 

All other Ethernet frames with different VLAN IDs than the PVID will be left alone, additionally they will only be allowed to pass through the port if that VLAN is assigned to the port.

 

When you need to go between VLANs that is when you need Layer 3 networking and routing and each VLANs needs a router, this would be your pfsense firewall.

 

Traffic from one of your PCs on VLAN 5 can come in to the pfsense firewall with a destination IP address of a device on another VLAN, if there is a matching firewall rule to allow it, then pfsense will strip the VLAN header and then add the destination VLAN ID to the frame and send it back out. Pfsense is doing both Layer 2 and Layer 3 functions.

 

P.S. I'm explaining it rather badly right now so I can explain it better later when I'm in a better thinking mood if you need   My main tip here is to separate out VLANs and routing and be careful in understanding which is being used and when/why.

[OhioYJ]

Ok Thank you so much @leadeater, I'm almost there. Today I got a little closer. I got one of the VLans up an working today. It appears I had some rules backwards. I was trying to just allow what I wanted, and it appears it's easier to do the opposite (allow everything, and block what I don't want).

 

So this brings me with the thing that stopped me dead today. Once I did this, I could no longer manage the switch. Since I could get to the firewall, and internet, obviously it was passing through it.

[leadeater]

35 minutes ago, OhioYJ said:

So this brings me with the thing that stopped me dead today. Once I did this, I could no longer manage the switch. Since I could get to the firewall, and internet, obviously it was passing through it.

What is the switch IP? The switch management interface will need a default gateway set otherwise it won't know how to get to devices in other subnets. I would leave another port on the switch default PVID 1 so if you need an emergency way of accessing it then you can plugin to that port. Once you get the required routing and firewall rules sort out to access the switch mgmt IP you won't need this switch port anymore.

[Jarsky]

Just to help clarify, there are 2 different types of Ports. Depending on vendor theyre referred to differently but they are:

• Tagged (aka Trunk port)
• Untagged (aka Access port)

 

With Tagged ports, they carry multiple VLAN ID's, and are primarily used for connecting network devices. So you would use tagged/trunk ports between your router & switches, and your switches & AP's

 

Untagged ports, or access ports, typically are a single VLAN configuration. They're for connecting your end devices like PC's, Consoles, TV's, etc.... 

 

Pretty much everyone uses VLAN1 as its default or "native" VLAN. That is, any packet that is sent not tagged with an ID, will go over VLAN1. 

VLAN1 is generally reserved for the network itself for various communication between routers and switches etc....(It can be changed to any VLAN, but by default is just normally 1)

 

With Tagged ports, they essentially pass the tag all the way to the end device. To connect a device like a PC etc...the device must understand VLAN's.

By default a Windows computer wont be configured with a VLAN for its network interface, hence the computer will just have no network. 

 

Untagged ports end at the port where it is turned back into "native" traffic for the device connected to it. So if you have VLAN20 with 192.168.20.0/24 assigned to that port and plug in a PC, if your DHCP is setup it should instantly get a 192.168.20.0 IP address. 

 

As for the AP's, as leadeater said, they support VLAN's. 

You can leave them on the default ALL, or if you have a UniFi Controller you can define specific VLAN's allowed to the AP

 

Then setup different SSID's for every network you'll be connecting wifi devices to, and assign the VLAN ("Network") to that SSID

 

 

[OhioYJ]

7 hours ago, leadeater said:

What is the switch IP? The switch management interface will need a default gateway set otherwise it won't know how to get to devices in other subnets. I would leave another port on the switch default PVID 1 so if you need an emergency way of accessing it then you can plugin to that port. Once you get the required routing and firewall rules sort out to access the switch mgmt IP you won't need this switch port anymore.

The firewall is 192.168.1.100 and I set a static IP in for the switch (in pfsense) of 192.168.1.101. When I log into the switch, it sees the gateway as 192.168.1.100 (pfsense). When this happened last night, I had only changed the PVID, on the PCs group. So my thought was I would go to my son's PC, and change it back. Didn't work, he had no access either. I then briefly tried from my phone, and which did have access through the wifi for a moment, and changed my PC back to PVID 1, but it still didn't gain me access. I ended up just resetting the swtich.

 

5 hours ago, Jarsky said:

As for the AP's, as leadeater said, they support VLAN's. 

You can leave them on the default ALL, or if you have a UniFi Controller you can define specific VLAN's allowed to the AP
 

Then setup different SSID's for every network you'll be connecting wifi devices to, and assign the VLAN ("Network") to that SSID

My Unifi APs connect into my switch directly. I setup the VLANs in the Unifi software, attaching SSIDs to the individuals VLANs (so each VLAN has it's own SSID).  So I have:

 

10 - Secure SSID (For PCs that need to talk to the NAS)

20 - Everyday SSID (For our devices, phones tablets etc)

30 - IOT SSID (Cameras etc)

40 - Guest SSID

 

I'm thinking I'm still messing up the switch configuration. Should I change VLAN 1s mememship perhaps? By default my switch shows VLAN1 as all Untagged. However I'm thinking since these ports, 1 - Firewall, 2 - AP1, 3 - AP2 (my first three) should be trunks / tagged?  Any group that needs to talk to these, they belong in correct?

 

 

[OhioYJ]

54 minutes ago, OhioYJ said:

I'm thinking I'm still messing up the switch configuration. Should I change VLAN 1s mememship perhaps? By default my switch shows VLAN1 as all Untagged. However I'm thinking since these ports, 1 - Firewall, 2 - AP1, 3 - AP2 (my first three) should be trunks / tagged?

Marking APs as Trunked / Tagged takes my wifi down. At least in PVID 1 with the current setup. Marking the firewall as "T" takes the internet down.

 

So this is what I have right now:

 

 

10 -  Secure Wifi

20 - Every Day Wifi

30 - IOT Wifi

40 - Guest Wifi

50 - PCs

60 - Server (NAS)

70 - Kids

80 - IOT (Wired)

 

So I believe my PVIDs need to be pretty much using @leadeater's advice above, just using the new ID Tags (thanks again)

 

1. 1  (Firewall stays in 1 since it has multiple groups right?)

2. 1  (APs stays in 1 since it has multiple groups right? They should send a tag back with a VLAN ID tag attached and devices should get a correct IP address assigned right?) **

3. 1  (Another AP)

4. 50

5. 50

6. 60

7. 70

8. 50

9. 50

10. 50

11. 50

12. 80

 

** Leaving those set to 1, and attaching to the different SSIDs currently does not get me assigned to the IP addresses I would expect.

[LIGISTX]

5 hours ago, OhioYJ said:

Marking APs as Trunked / Tagged takes my wifi down. At least in PVID 1 with the current setup. Marking the firewall as "T" takes the internet down.

 

So this is what I have right now:

 

 

10 -  Secure Wifi

20 - Every Day Wifi

30 - IOT Wifi

40 - Guest Wifi

50 - PCs

60 - Server (NAS)

70 - Kids

80 - IOT (Wired)

 

So I believe my PVIDs need to be pretty much using @leadeater's advice above, just using the new ID Tags (thanks again)

 

1. 1  (Firewall stays in 1 since it has multiple groups right?)

2. 1  (APs stays in 1 since it has multiple groups right? They should send a tag back with a VLAN ID tag attached and devices should get a correct IP address assigned right?) **

3. 1  (Another AP)

4. 50

5. 50

6. 60

7. 70

8. 50

9. 50

10. 50

11. 50

12. 80

 

** Leaving those set to 1, and attaching to the different SSIDs currently does not get me assigned to the IP addresses I would expect.

Not to increase e-waste, but it may be worthwhile to pick up some (really inexpensive) UniFi managed switches, and sell the netgear. That way your entire switch infrastructure is a single vendor, and can be managed from a single controller, with the same logic behind the policies. I can attest, the 30 dollar 5 port switches from UniFi rock. 

[OhioYJ]

21 minutes ago, LIGISTX said:

Not to increase e-waste, but it may be worthwhile to pick up some (really inexpensive) UniFi managed switches, and sell the netgear. That way your entire switch infrastructure is a single vendor, and can be managed from a single controller, with the same logic behind the policies. I can attest, the 30 dollar 5 port switches from UniFi rock. 

The only reason I didn't do that in the first place was the Netgear is 2.5 GbE, which keep my entire Network 2.5 GbE (with the exception of a couple devices, and the WiFi). When I looked at the Unfi switches they were all 1 GbE from what I saw?

[LIGISTX]

45 minutes ago, OhioYJ said:

The only reason I didn't do that in the first place was the Netgear is 2.5 GbE, which keep my entire Network 2.5 GbE (with the exception of a couple devices, and the WiFi). When I looked at the Unfi switches they were all 1 GbE from what I saw?

Ah, yes. They don’t have anything between 1 and 10gbe. 
 

Personally, I am just sitting on 1gbe for my physical infrastructure, waiting for hopefully 10 to get more affordable and more ubiquitous. I do plan on a direct fiber run using 10gbe transceivers and PCIe cards from NAS to main PC at some point…. I am just lazy lol. I already have the cards, just need a few more parts. One of these days…

[OhioYJ]

Ok, so after so much trouble, on something that seemed like it would be easy..... I'm up and running. I'm still not entirely sure why. While searching about this topic, I came across a few posts on Lawerence System's forum from a user talking about how Netgear switches did weird things if you tried to set them up connected to the firewall. This user had to setup the firewall, then the switch, then connect them together. I also found I had two networks on my APs without VLAN tags (could be why the Wifi always dropped out).  After setting them up individually though, it does indeed work. Now I just need to start in on rules to actually make sure the VLANs that shouldn't be able to talk to each other actually can't.

 

The only thing I've found that I actually can't do anymore seems to be WOL. It appears in my searching there isn't a way to pass WOL packets between VLANs. The only way to do WOL is use the built in service in pfsense, which isn't as convenient.

[leadeater]

6 hours ago, OhioYJ said:

The only thing I've found that I actually can't do anymore seems to be WOL. It appears in my searching there isn't a way to pass WOL packets between VLANs. The only way to do WOL is use the built in service in pfsense, which isn't as convenient.

You can send a WOL to the broadcast address of the other subnet and it should work, however this type of traffic is usually blocked by default and you'll have to allow it. It's not really a good idea. The other way is a WOL relay/proxy or IP Helper (vendor specific naming).

 

In the past I've used a PC proxy and psexec to do it but this is on a larger network with thousands of computers so odds are a PC is on in the required subnet and I instruct that PC to do the WOL.

 

Anyway it's possible but you'll have to decide if it's worth the effort to setup.

[Jarsky]

On 1/26/2023 at 3:04 PM, OhioYJ said:

While searching about this topic, I came across a few posts on Lawerence System's forum from a user talking about how Netgear switches did weird things if you tried to set them up connected to the firewall.

 

I was just helping someone set up VLAN last with pfSense and a TPLink last week that has a similar looking configuration UI and we had major headaches as well. It didnt want to "just work" with what looks like it should be a valid config to someone whos done a lot on Cisco & Ubiquiti.  Good to see you got it working

[OhioYJ]

I started in on the rules last night, and while I it did seem to work, I'm confused by something I had happening. So if I have two VLANs

 

10 - Guest Wifi

50 - PCs

 

I make a rule that says on 10 - Guest WiFi that says Block Any traffic from Guest WiFi Net to PC Net.  

 

Then when I ping from PCs to GuestWifi device I still get a response. I get my PC can send out the request, but shouldn't the firewall stop my phone from sending the response? 

 

I get the expected result when I add a second rule on PC blocking all traffic to Guest net as well. Then there is no response. 

[leadeater]

8 hours ago, OhioYJ said:

Then when I ping from PCs to GuestWifi device I still get a response. I get my PC can send out the request, but shouldn't the firewall stop my phone from sending the response? 

This is actually desired behavior. The name for this is Stateful Firewall which means the traffic session information is tracked and associated traffic to the original rule that allowed it is also allowed. It greatly simplifies firewall rules because you don't have to define rules for both directions. Also return traffic is on random RPC port range so having to explicitly allow the return traffic is actually very problematic. You want a Stateful Firewall not a Stateless Firewall.

 

Your phone will only be able to reply back to a ping made to it from the PC subnet/vlan, it will not be able to initiate the ping from itself to a PC.

 

So when thinking about firewall rules think about it from the perspective of the originating device, any return traffic is implicitly allowed. This holds true most of the time, way more complex environment with multiple firewalls means it gets rather bit more difficult but you don't have to worry about that. 

[OhioYJ]

Ok, how about this one... is there a way to block access to the switches web interface? For the firewall, I can change the port that it operates on, and block that port for the VLANs that shouldn't be able to reach the firewall GUI. The switch however it does not appear I can change the port (it operates on port 80). So if I block the IP address of the switch or port 80, I'll kill the internet and pretty much all internal traffic as well as I don't think it will communicate with the switch either?

[leadeater]

14 minutes ago, OhioYJ said:

So if I block the IP address of the switch or port 80, I'll kill the internet and pretty much all internal traffic as well as I don't think it will communicate with the switch either?

You can block that fine without affecting your internet. Scope the rule correctly to traffic destined for the switch IP and that's all the rule will apply to.

[OhioYJ]

13 minutes ago, leadeater said:

You can block that fine without affecting your internet. Scope the rule correctly to traffic destined for the switch IP and that's all the rule will apply to.

Oh, that makes sense now that I think about it, because I can block, just that IP address:80, and nothing else. After getting locked out and starting from scratch, I got a little gun shy of just making rules, if I wasn't sure. This time though, I've started making backups after I'm sure something works...

 

Thanks again!