Pfsense: WTF is going on?!?!?!?!

[jooroth18]

So i have Pfsense as my router of choice for a good 6 months now. Recently, the server has been doing something weird, where the built in DNS resolver that I set up would stop replying to requests, and the only way to get it to work is if i leave a tab open on a computer with the Pfsense Dashboard. Its extremely strange, and if i close the tab, after about a minute it stops responding again. i tried switching back to the normal DNS forwarder, but no use.

- Im running Pfsense 2.4.2 (amd64).

- i did swap out the old HDD for a new cheap SSD from china, but i highly doubt that caused it

-i have a broken UniFi pfsense installation. (even if i kill the service, it still persists) (by broken i mean it fails to load properly and i have given up on it)

-internet works fine if i use 1.1.1.1 on my PC. i woudl just rather use the built in one on pfsense to get better latency

 

[Razor Blade]

I've experienced a similar issue... I run PFsense in a VM and occasionally after an upgrade it will exhibit the same symptoms (except for working with PFsense dashboard open). Even restarting the VM it would work okay for about an hour then start acting up again...Restoring the VM to previous snapshot yielded the same result. The only thing I was able to do to seemingly correct it when the problems happened was going to the interfaces tab then released the lease on my modem (as in the WAN interface), waited about 60 seconds then renewed it. After that it works fine. I haven't been able to track down a root cause yet as it happens so very rarely... Also I wasn't sure if it had to do with running PFsense in ESXI or not.

 

Hopefully you are able to find a resolution soon.

[Falconevo]

Sounds like an ARP cache or possibly pfBlocker (DNSBL) *assuming you use this package* issue to me, what switching (outside of virtual switching) are you using and do you have any VLANs configured on internal (LAN side) interfaces?

 

Also just to check have you verified that there are no other devices on your network are attempting to use your or the pfSense boxes IP address...? This should show in the logs.

[jooroth18]

 

9 hours ago, Falconevo said:

Sounds like an ARP cache or possibly pfBlocker (DNSBL) *assuming you use this package* issue to me, what switching (outside of virtual switching) are you using and do you have any VLANs configured on internal (LAN side) interfaces?

 

Also just to check have you verified that there are no other devices on your network are attempting to use your or the pfSense boxes IP address...? This should show in the logs.

i do not have any VLANS set up, however i do have pfblocker installed. it is however disabled at the moment. i checked the ip list and aswell looked at each device on the network, and there are none overlapping. To be clear, i can access the internet, each computer reports the correct arp table. Its just the DNS server doesnt respond to any request. it only responds if i have the dashboard tab open on any computer.

 

10 hours ago, Razor Blade said:

I've experienced a similar issue... I run PFsense in a VM and occasionally after an upgrade it will exhibit the same symptoms (except for working with PFsense dashboard open). Even restarting the VM it would work okay for about an hour then start acting up again...Restoring the VM to previous snapshot yielded the same result. The only thing I was able to do to seemingly correct it when the problems happened was going to the interfaces tab then released the lease on my modem (as in the WAN interface), waited about 60 seconds then renewed it. After that it works fine. I haven't been able to track down a root cause yet as it happens so very rarely... Also I wasn't sure if it had to do with running PFsense in ESXI or not.

 

Hopefully you are able to find a resolution soon.

I have tried this, and to no avail.

 

Thank you for your replies, they do help narrow down the situation.

[jooroth18]

Here is my services config, if it helps. I have disabled the resolver and attempted to use the forwarder, and even that doesn't solve the problem.

[jooroth18]

I updated the title of this thread so it doesn't go dead, as I really need this issue solved. I'm planning to reach out to the negate forums as I really need this fixed.

[Falconevo]

On 3/12/2019 at 2:22 AM, jooroth18 said:

Here is my services config, if it helps. I have disabled the resolver and attempted to use the forwarder, and even that doesn't solve the problem.

First thing, backup your config.

 

Remove pfBlocker to clear the DNSBL component and reboot the pfSense box to make sure it is not part of the problem.

 

Provide the following config information;

• System > General Setup
  Please copy or screenshot the DNS Server config here
• Services > DNS Resolver
  Please copy or screenshot the DNS Resolver config here
  Please also provide the DNS Resolver log from Status > System Logs > DNS Resolver
  If you are not using the DNS Resolver and use the DNS Forward instead, please provide the following
• Services > DNS Forwarder
  Please copy or screenshot the DNS Forwarder config here

[jooroth18]

14 hours ago, Falconevo said:

Remove pfBlocker to clear the DNSBL component and reboot the pfSense box to make sure it is not part of the problem.

 

I just now attempted to remove it, but Pfsense says i have no packages installed, which I do.

[jooroth18]

14 hours ago, Falconevo said:

Provide the following config information;

• System > General Setup
  Please copy or screenshot the DNS Server config here
• Services > DNS Resolver
  Please copy or screenshot the DNS Resolver config here
  Please also provide the DNS Resolver log from Status > System Logs > DNS Resolver
  If you are not using the DNS Resolver and use the DNS Forward instead, please provide the following
• Services > DNS Forwarder
  Please copy or screenshot the DNS Forwarder config here

Here it is. i don't have any system logs as i have disabled the resolver for a while as a troubleshooting step, but i have just now enabled it. Probably about tomorrow i should have some useful logs.

 

Also i have changed the log level to 5 to help narrow down the problem.

[Falconevo]

Perform the following changes (backup before hand so you can revert back easily)

 

Set Outgoing interfaces to ALL 
Disable DNSSEC in the resolver general options

In Advanced options set the following;

Message Cache size - 4MB
Outgoing TCP buffers - 10
Incoming TCP buffers - 10
Unwanted Reply Threshold - Disabled

 

Click on the Access List tab in the Resolver

Click Add

Enter a name for the DNS list like 'InternalSubnets'

Action Allow

Enter a description for your reference

Add the networks you are using in the internal networks you have for example; 192.168.0.0/24 etc etc
Add all of the networks you have configured on the internal pfSense interfaces (including IPv6 if you have this configured for DNS)

 

Save changes
Restart the DNS Resolver service

 

Check each of the LAN networks you have configured and make sure UDP 53 is allowed in each of the outbound rule sets.  If you don't have this already, configure the following;

Source 'Internal Subnet Network'
Source Port 'ANY'

Destination 'ANY'
Destination Port UDP53

[jooroth18]

9 minutes ago, Falconevo said:

Perform the following changes (backup before hand so you can revert back easily)

Disable DNSSEC in the resolver general options

In Advanced options set the following;

Message Cache size - 4MB
Outgoing TCP buffers - 10
Incoming TCP buffers - 10
Unwanted Reply Threshold - Disabled

 

Click on the Access List tab in the Resolver

Click Add

Enter a name for the DNS list like 'InternalSubnets'

Action Allow

Enter a description for your reference

Add the networks you are using in the internal networks you have for example; 192.168.0.0/24 etc etc
Add all of the networks you have configured on the internal pfSense interfaces (including IPv6 if you have this configured for DNS)

 

Save changes
Restart the DNS Resolver service

Alright, I have applied the changes. The issue is sporadic at times, so I will report tomorrow night if it still persists. Thanks for the response!

 

edit: It still persists, so I went ahead and rebooted the entire box. Hopefully that fixes it.

[Falconevo]

1 minute ago, jooroth18 said:

Alright, I have applied the changes. The issue is sporadic at times, so I will report tomorrow night if it still persists. Thanks for the response!

I edited the post as I missed a bit off, can you just recheck you have everything  report back if you have anymore drama.

Also another thing to note, I came across an odd DNS issue on pfSense when using RealTek network interfaces, It was something to do with their driver UDP checksum offload not working correctly, I just binned the interfaces and went with Intel.  I haven't seen that issue in about 6 years though.

[jooroth18]

After a bit of testing, it seems to work. Im going to switch all my clients off their VPN's that i set up as a temporary fix, and see what happens after a full 24 hours. Hopefully i come back with good news.

[jooroth18]

8 minutes ago, Falconevo said:

I edited the post as I missed a bit off, can you just recheck you have everything  report back if you have anymore drama.

Also another thing to note, I came across an odd DNS issue on pfSense when using RealTek network interfaces, It was something to do with their driver UDP checksum offload not working correctly, I just binned the interfaces and went with Intel.  I haven't seen that issue in about 6 years though.

That's a good thing that i chose Intel for my NIC's. I have two of their gigabit ones. Ill check over my settings and see if I missed anything. Hopefully this is the solution as i want my network to work the way it used to.

[jooroth18]

The ARP hint was extremely useful to me for another issue, as i had another problem where i couldn't find my UniFi access points, as they wouldn't show up in the DHCP leases, or on the Ubiquiti discovery tool. I saw them listed in the ARP tables and i was able to enter the correct IP's into the controller. I can finally control my access points again!

[Falconevo]

49 minutes ago, jooroth18 said:

The ARP hint was extremely useful to me for another issue, as i had another problem where i couldn't find my UniFi access points, as they wouldn't show up in the DHCP leases, or on the Ubiquiti discovery tool. I saw them listed in the ARP tables and i was able to enter the correct IP's into the controller. I can finally control my access points again!

Are you sure you don't have ARP issues?  Most of the time ARP issues aren't caused by pfSense but the attached switching, either with incorrect VLAN tagging or incorrectly configured trunk ports (assuming a single LAN interface).

What switches and VLAN configurations do you use on the internal side?  I've seen unmanaged switches that are piggybacking managed switches cause a load of grief if the port they are attached to is a trunk with a native untagged vlan also configured (usually vlan1).