Pfsense Subnet Routing Issues

[LIGISTX]

Hi,

I have been runing pfsense for a few years now, and it only now occures to me that my IoT turned Guest & IoT network has some sort of routing or DNS issue that I really do not understand. I believe it is DNS related since if I cange the DNS address handed out by DHCP for this subnet (which also has pfblockerng running) to say someting like 9.9.9.9 either in the DNS server settings in pfsesne or manually on a device, things work normally.

 

This tells me I likely have my rules for the subnet to agressive and its blocking the ability for clients to talk to pfsense’s DNS server. What I do not understand is… I have explicate pass rules for DNS. Does anyone see any glaring issues with my config?

Nat (the .69 network is my IoT network…)

 

Furthermore as I am on vacation right now trying to troubleshoot since I am bored and have some downtime and I am noticing the same exact seemingly DNS issue on my VPN split tunnel setup. I have wireguard running on pfsense and have 2 VPN’s set up for my laptop, one for split tunnel and 1 for full tunnel.

I am starting to thing it is not DNS…

The only difference between my wireguard configs on the client side are the “Interface Address” (same subnet, only off by 2 numbers in the last octet, and the “AllowedIPs” for the full tunnel being:

AllowedIPs = 0.0.0.0/0, ::/0

And the split tunnel being a set of my prive IP’s that I use:

AllowedIPs = 10.1.15.0/24, 10.90.5.0/24, 10.80.5.0/24, 10.81.5.0/24, 10.70.5.0/24, 192.168.69.0/24

All other settings (minus keys) are the same. Both VPN’s work fine except for some websites I just can’t get to on the split tunnel much like the issue I have with certain subnets within my LAN. While on the split tunnel, I can ping a website like CNN via terminal, and I get responses, but the website just does not load.

In wireguard, for both tunnels, DNS is set to 10.1.15.1 which is the “wireguard” subnet I have set up in pfsnese. I really don’t know the full tunnel works fine, and the split tunnel reacts exactly like I am on one of the internal subnets that doesn’t seem to be getting correct routing. If I edit the split tunnel AllowedIP’s to be 0.0.0.0/0, ::/0, that connection suddenly works fine.

I am entirely at a lost, but I alst only know enough to know enough… I don’t even know where to start with this issue. I know at some previous point in time my split tunnel worked perfectly, but I have not used it in a while and I am not sure when it started to work incorrectly. I can’t imagine what I would have changed that caused this for the split tunnel or the other subnets - I assume its a single issue affecting both scenarios.

Rules for this subnet are extremely simple:

Wireguard settings for split tunnel:

[Interface]
PrivateKey = xxxx
Address = 10.1.15.4/24
DNS = 10.1.15.1

[Peer]
PublicKey = xxxx
PresharedKey = xxxx
AllowedIPs = 10.1.15.0/24, 10.90.5.0/24, 10.80.5.0/24, 10.81.5.0/24, 10.70.5.0/24, 192.168.69.0/24
Endpoint = xxxx

I can't get the split tunnel interface to work correctly even just removing 1 subnet at a time from the AllowedIPs list. Only once I set it to 0.0.0.0/0, ::/0 does it work correctly. 

[Alex Atkin UK]

28 minutes ago, LIGISTX said:

Hi,

I have been runing pfsense for a few years now, and it only now occures to me that my IoT turned Guest & IoT network has some sort of routing or DNS issue that I really do not understand. I believe it is DNS related since if I cange the DNS address handed out by DHCP for this subnet (which also has pfblockerng running) to say someting like 9.9.9.9 either in the DNS server settings in pfsesne or manually on a device, things work normally.

 

This tells me I likely have my rules for the subnet to agressive and its blocking the ability for clients to talk to pfsense’s DNS server. What I do not understand is… I have explicate pass rules for DNS. Does anyone see any glaring issues with my config?

Nat (the .69 network is my IoT network…)

 

Furthermore as I am on vacation right now trying to troubleshoot since I am bored and have some downtime and I am noticing the same exact seemingly DNS issue on my VPN split tunnel setup. I have wireguard running on pfsense and have 2 VPN’s set up for my laptop, one for split tunnel and 1 for full tunnel.

I am starting to thing it is not DNS…

The only difference between my wireguard configs on the client side are the “Interface Address” (same subnet, only off by 2 numbers in the last octet, and the “AllowedIPs” for the full tunnel being:

AllowedIPs = 0.0.0.0/0, ::/0

And the split tunnel being a set of my prive IP’s that I use:

AllowedIPs = 10.1.15.0/24, 10.90.5.0/24, 10.80.5.0/24, 10.81.5.0/24, 10.70.5.0/24, 192.168.69.0/24

All other settings (minus keys) are the same. Both VPN’s work fine except for some websites I just can’t get to on the split tunnel much like the issue I have with certain subnets within my LAN. While on the split tunnel, I can ping a website like CNN via terminal, and I get responses, but the website just does not load.

In wireguard, for both tunnels, DNS is set to 10.1.15.1 which is the “wireguard” subnet I have set up in pfsnese. I really don’t know the full tunnel works fine, and the split tunnel reacts exactly like I am on one of the internal subnets that doesn’t seem to be getting correct routing. If I edit the split tunnel AllowedIP’s to be 0.0.0.0/0, ::/0, that connection suddenly works fine.

I am entirely at a lost, but I alst only know enough to know enough… I don’t even know where to start with this issue. I know at some previous point in time my split tunnel worked perfectly, but I have not used it in a while and I am not sure when it started to work incorrectly. I can’t imagine what I would have changed that caused this for the split tunnel or the other subnets - I assume its a single issue affecting both scenarios.

Rules for this subnet are extremely simple:

If you are using Unbound then DNS should not be an issue as by default I believe its set to respond on all network interfaces.  You can always doubled check this in Services > DNS Resolver.  I'd be tempted to point the finger at pfBlockerNG as it can break websites.

 

My NAT rule is setup as:

 

Firewall rule for IoT subnet:


I do have a Port Forward rule to force any attempt to connect to a DNS or Time server to go via the router though:


Wireguard I have had plenty of problems but not with specific sites, it just sometimes wont load anything or goes incredibly slow, both remoting in and when connecting from pfSense to a VPN.  I stick to OpenVPN for this reason.

[LIGISTX]

10 minutes ago, Alex Atkin UK said:

If you are using Unbound then DNS should not be an issue as by default I believe its set to respond on all network interfaces.  You can always doubled check this in Services > DNS Resolver.  I'd be tempted to point the finger at pfBlockerNG as it can break websites.

 

My NAT rule is setup as:

 

Firewall rule for IoT subnet:


I do have a Port Forward rule to force any attempt to connect to a DNS or Time server to go via the router though:


Wireguard I have had plenty of problems but not with specific sites, it just sometimes wont load anything or goes incredibly slow, both remoting in and when connecting from pfSense to a VPN.  I stick to OpenVPN for this reason.

I am using unbound. 
 

I have tried to disable pfblocker testing to see if this changes anything, it doesn't appear to… and pfblocker seemingly works fine on my private LAN…….. and things work fine on the WireGuard interfaces when setting allowed IP’s to all, so I just don’t think it’s pfblocker either. It seems like there is some IP Conflict somewhere or things are getting sink holed. I just don’t understand what is causing this. 
 

I have disabled pfblocker a few times attempting to try this, but maybe I should try again and confirm I reset states. It still doesn’t really explain why some subnets work fine, and why changing the allowedIP’s in my WireGuard client make it work tho, at least not in my mind. 

[Alex Atkin UK]

2 hours ago, LIGISTX said:

I am using unbound. 
 

I have tried to disable pfblocker testing to see if this changes anything, it doesn't appear to… and pfblocker seemingly works fine on my private LAN…….. and things work fine on the WireGuard interfaces when setting allowed IP’s to all, so I just don’t think it’s pfblocker either. It seems like there is some IP Conflict somewhere or things are getting sink holed. I just don’t understand what is causing this. 
 

I have disabled pfblocker a few times attempting to try this, but maybe I should try again and confirm I reset states. It still doesn’t really explain why some subnets work fine, and why changing the allowedIP’s in my WireGuard client make it work tho, at least not in my mind. 

Nothing in the firewall logs for the host having issues?

[LIGISTX]

1 hour ago, Alex Atkin UK said:

Nothing in the firewall logs for the host having issues?

What should I check? This is where I lose my knowledge of pfsense. What should I look for, and where should I look?

[Alex Atkin UK]

2 hours ago, LIGISTX said:

What should I check? This is where I lose my knowledge of pfsense. What should I look for, and where should I look?

I'd be looking in Status > System Logs > Firewall to see if anything is being blocked that shouldn't be.  Its not the most helpful way of displaying the information, but I'm not sure where else to look.

 

Another thing would be to check Firewall > Rules and on your blocking rules observe to see if the number is going up (I think you have to manually reload the page).  Annoyingly if you click on this it wont tell you anything, as it links to Active States, not blocked connections or states that have expired.  Bit of an oversight IMO to not have it filter the firewall log automatically to show what it has blocked.  This is definitely something pfSense could improve.

[LIGISTX]

47 minutes ago, Alex Atkin UK said:

I'd be looking in Status > System Logs > Firewall to see if anything is being blocked that shouldn't be.  Its not the most helpful way of displaying the information, but I'm not sure where else to look.

I dno't seem to see anything... I refresh a page like cnn.com and I dont seem to see anything strange in the firewall logs.

 

48 minutes ago, Alex Atkin UK said:

Another thing would be to check Firewall > Rules and on your blocking rules observe to see if the number is going up (I think you have to manually reload the page). 

On my wireguard subnet, I don't even have any block rules, the subnet is entirely wide open. Should I be looking in WAN or something?

 

The fact wireguard (when not a full tunnel) gives me this issue makes this so much stranger since that subnet is extremely simple. This is the sort of thing that makes you just want to give up and try unifi or something... I have a feeling I will never be able to solve this short of starting over from scratch which I cant even imagine having to do :/. Probably some setting somewhere that no one would even think of checking.

[Alex Atkin UK]

4 minutes ago, LIGISTX said:

The fact wireguard (when not a full tunnel) gives me this issue makes this so much stranger since that subnet is extremely simple. This is the sort of thing that makes you just want to give up and try unifi or something... I have a feeling I will never be able to solve this short of starting over from scratch which I cant even imagine having to do :/. Probably some setting somewhere that no one would even think of checking.

Are you in the same country as your pfSense router?  I wonder if split-tunneling is pulling a geographically close IP address locally and then its being rejected when the page tries to load over the tunnel?

[LIGISTX]

6 hours ago, Alex Atkin UK said:

I'd be tempted to point the finger at pfBlockerNG as it can break websites.

Hmmm, after doing a bunch more testing, it does look like its pfblocker. Why and how, I do not understand. But after some testing it looks like its the DNSBL being enabled that causes it. When I disable that and restart unbound and clear states, thigns seem to work normal.... Once I reenable and force a update for pfblocker and reset states, thigns stop working. 

[LIGISTX]

2 minutes ago, Alex Atkin UK said:

Are you in the same country as your pfSense router?  I wonder if split-tunneling is pulling a geographically close IP address locally and then its being rejected when the page tries to load over the tunnel?

Yup, I am even on the same ISP.... and physcially only a few hundread miles away. Of note, this does seem to also happen on local subnets as well. I first noticed it on my IoT subnet when I was logged in on my phone after setting up a IoT device, and was like "why can't I get to websites". I can't check that at the moment (unless I spin up a VM and stick it on that network... which I guess I can do to test........), but it isn't just the wireguard subnet that has this issue. My main private LAN doesn't have the issue thus I never noticed it.

[LIGISTX]

FWIW, after lots more testing, it does look to be pfblockerng just being.... fucky. I don't know what its even doing wrong, or how to explain why its doing what its doing, but I switched to pihole and everything is fixed. So I spun up 2 pihole VM's for a pseudo HA solution with DHCP handing out each of their IP's in case one goes down. Things are happier for sure, and I am using unbound on pfsense as their upstream DNS.

[Alex Atkin UK]

9 hours ago, LIGISTX said:

FWIW, after lots more testing, it does look to be pfblockerng just being.... fucky. I don't know what its even doing wrong, or how to explain why its doing what its doing, but I switched to pihole and everything is fixed. So I spun up 2 pihole VM's for a pseudo HA solution with DHCP handing out each of their IP's in case one goes down. Things are happier for sure, and I am using unbound on pfsense as their upstream DNS.

I got the impression pfBlockerNG is pretty much designed on the basis of having it apply to everything.  Its why I only use it to manage IP blocklists, not the DNS functionality.  As I'd only want it to DNS block a specific subnet, not the whole LAN, as blocking Google Ads especially can be problematic.

This pretty much sounds like the ideal scenario for pihole, seeing as its completely independent.  I may end up trying it myself as I'd like to adblock when over the VPN, where I'm more likely to be somewhere bandwidth constrained.