Full memory in router??

[MRVodkaaa]

hello last month i get two asus RT-AX55 routers there are conected in AIMesh and like 2 weeks was everething working without any problems and one day my lan on pc start disconnecting. also i wasnt able to go to webpage of my router 192.168.1.1 so i restart it and everething start working again but in log i seen this messages

Mar  2 11:53:41 dnsmasq-dhcp[2631]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Mar  2 11:54:41 dnsmasq-dhcp[2631]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Mar  2 11:55:41 dnsmasq-dhcp[2631]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Mar  2 11:56:41 dnsmasq-dhcp[2631]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Mar  2 11:57:41 dnsmasq-dhcp[2631]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Mar  2 11:58:41 dnsmasq-dhcp[2631]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Mar  2 11:59:41 dnsmasq-dhcp[2631]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Mar  2 12:00:41 dnsmasq-dhcp[2631]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Mar  2 12:01:41 dnsmasq-dhcp[2631]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Mar  2 12:02:41 dnsmasq-dhcp[2631]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Mar  2 12:03:41 dnsmasq-dhcp[2631]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Mar  2 12:04:41 dnsmasq-dhcp[2631]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Mar  2 12:05:41 dnsmasq-dhcp[2631]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Mar  2 12:06:41 dnsmasq-dhcp[2631]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)

anyone know what this is? what memory? what is caising this issue?

[Falcon1986]

10 minutes ago, MRVodkaaa said:

anyone know what this is? what memory? what is caising this issue?

Perform on both units:

1. Reset to factory defaults through the router OS.
2. Update the firmware to the latest as found on the Asus website. Only flash firmware if you’re directly wired into the unit.
3. After the flash, allow to boot up and reset to factor defaults through the router OS again.
4. Reconfigure both units as you’d like.

[Eigenvektor]

dnsmasq is a simple DNS and DHCP server. It's most likely trying to write a DHCP lease into the file "/var/lib/misc/dnsmasq.leases" and that fails, because either the router's memory or disk is full. Since it is solved by a reboot, I'm going to assume that particular file is actually in memory.

 

Do you have a large number of clients on your network?

 

Most likely this is a bug in the router's software that slowly causes it's RAM to fill up and once that's the case it can no longer do stuff like issue DHCP leases or give access to its web management. Try to see if there's a firmware update available,

[MRVodkaaa]

19 hours ago, Eigenvektor said:

dnsmasq is a simple DNS and DHCP server. It's most likely trying to write a DHCP lease into the file "/var/lib/misc/dnsmasq.leases" and that fails, because either the router's memory or disk is full. Since it is solved by a reboot, I'm going to assume that particular file is actually in memory.

 

Do you have a large number of clients on your network?

 

Most likely this is a bug in the router's software that slowly causes it's RAM to fill up and once that's the case it can no longer do stuff like issue DHCP leases or give access to its web management. Try to see if there's a firmware update available,

hi before this issue both roters was allready updated to latest firmware and they was reset after.

clients on wifi 6

clients on cable 5

[Falcon1986]

44 minutes ago, MRVodkaaa said:

hi before this issue both roters was allready updated to latest firmware and they was reset after.

clients on wifi 6

clients on cable 5

OK.

 

Turn off AiProtect and anything else that you don’t really need, e.g. IPv6 (if your ISP doesn’t use it), WPS (mainly for security). For wireless security/encryption, use WPA2/AES. Don’t use TKIP and WPA1 if possible; older devices might require WPA1.

 

When these units run in their conventional wireless router mode, do you have the same problem? Do you have a stats page in the router OS that shows memory use, CPU load, etc?

[MRVodkaaa]

4 hours ago, Falcon1986 said:

OK.

 

Turn off AiProtect and anything else that you don’t really need, e.g. IPv6 (if your ISP doesn’t use it), WPS (mainly for security). For wireless security/encryption, use WPA2/AES. Don’t use TKIP and WPA1 if possible; older devices might require WPA1.

 

When these units run in their conventional wireless router mode, do you have the same problem? Do you have a stats page in the router OS that shows memory use, CPU load, etc?

well see cpu load and memory is no chance no stats at all. AIprotect is off but traffic analyser is on no chance to turn off dont know why they remove all those settings. looks like i dont have most stuff under control at all.

also AES/wpa2 is in use

[MRVodkaaa]

4 hours ago, Falcon1986 said:

OK.

 

Turn off AiProtect and anything else that you don’t really need, e.g. IPv6 (if your ISP doesn’t use it), WPS (mainly for security). For wireless security/encryption, use WPA2/AES. Don’t use TKIP and WPA1 if possible; older devices might require WPA1.

 

When these units run in their conventional wireless router mode, do you have the same problem? Do you have a stats page in the router OS that shows memory use, CPU load, etc?

aslo i am getting weird stuff in log like 

Mar  7 17:03:10 miniupnpd[1978]: upnp_event_process_notify: connect(192.168.1.199:2869): Connection timed out
Mar  7 17:03:15 miniupnpd[1978]: upnp_event_process_notify: connect(192.168.1.199:2869): Connection timed out
Mar  7 17:03:15 miniupnpd[1978]: upnpevents_processfds: 0x403200, remove subscriber uuid:3ddcd1d3-2380-45f5-b069-a85e45fb81d4 after an ERROR cb: http://192.168.1.199:2869/upnp/eventing/qncaqtzhfu
Mar  7 17:03:34 miniupnpd[1978]: upnp_event_process_notify: connect(192.168.1.199:2869): Connection timed out
Mar  7 17:03:34 miniupnpd[1978]: upnp_event_process_notify: connect(192.168.1.199:2869): Connection timed out

 

 before i had asus 

ASUS RT-AC87U and that router was running smooth without any problems at all i could set anything on and off

i replece it because i want to use wifi6 in mesh

[Falcon1986]

@MRVodkaaa

 

This seems to be an issue unique to Asus wireless routers. A bunch of errors get logged repeatedly and the NVRAM fills up fast, thereby crippling the overall performance of the router.

 

Other things to try:

1. Disable AiCloud.
2. Disable any firewall logging or filters from Advanced > Firewall.
3. Disable any ping/DNS query from Administration > System > Network Monitoring.

You still haven’t reported on how the units perform in regular (non-mesh) mode.

[MRVodkaaa]

57 minutes ago, Falcon1986 said:

@MRVodkaaa

 

This seems to be an issue unique to Asus wireless routers. A bunch of errors get logged repeatedly and the NVRAM fills up fast, thereby crippling the overall performance of the router.

 

Other things to try:

1. Disable AiCloud.
2. Disable any firewall logging or filters from Advanced > Firewall.
3. Disable any ping/DNS query from Administration > System > Network Monitoring.

You still haven’t reported on how the units perform in regular (non-mesh) mode.

they never been running without aimesh i tryed all things you told me so i will  wait and see how it will work.

AIcloud is not there at all

logging firewall is off

ping/DNS was off aswell

 

[Falcon1986]

8 hours ago, MRVodkaaa said:

they never been running without aimesh i tryed all things you told me so i will  wait and see how it will work.

AIcloud is not there at all

logging firewall is off

ping/DNS was off aswell

Which firmware version do you have running on these units?

[MRVodkaaa]

8 hours ago, Falcon1986 said:

Which firmware version do you have running on these units?

 3.0.0.4.386_41700-gb567bc9

router tell me after check that is on latest firmware

[Falcon1986]

5 hours ago, MRVodkaaa said:

 3.0.0.4.386_41700-gb567bc9

router tell me after check that is on latest firmware

There is a later beta version that you could try flashing, but this would be using a manual flash (download file, upload file to router through interface, wait for flash). Do it in this order before setting it up as you want...

 

FULL RESET —> FIRMWARE FLASH —> FULL RESET

 

Also, try running 1 of the devices in regular router mode (not AiMesh) to see how it behaves.

 

The next thing to do would be to SSH into the router and extract the log files to see what is causing the NVRAM to fill up so quickly. Googling those exact errors don’t seem to return any working solution, unfortunately. Hopefully, someone else has input.

 

If the problem continues to occur after a regular number of days, you could put the units on a power-cycle schedule just before that happens to clear the NVRAM.

[Chewy2]

Hey MRVodkaaa. Just wondering if you ever resolved this? I have the exact same setup. I'm having intermittent issues connecting to webpages and seeing those messages in the logs. Thanks.

[Lebowski_DG]

I have this same router and this morning it would not stay connected to the Internet. The system log printed this line each time the issue surfaced:

WAN Connection: ISP's DHCP did not function properly.

It was not a problem with my modem, as a direct connection works properly.

After numerous router reboots, the issue was only resolved temporarily. Within 5-10 minutes the issue would come back.

I looked further at the system log and noticed this line:

dnsmasq-dhcp[1439]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)

I enabled SSH to the router and checked available disk space with "df -h" command. Sure enough /var was near 100% usage. As I did not know what to clear out from that path, I opted for a router hard reset. 

1. Power off router.
2. Hold WPS button.
3. Power on router still holding the WPS button, wait for power light to turn on then off. Once off, release the WPS button.
4. Power on router.
5. Go through the guided setup, and install the latest firmware while you're at it.
6. Check disk space and see /var now is 0% usage.

Not sure what the root cause is, but thought I might chime in to explain what resolved this annoying issue for me.

[Cavefishes]

50 minutes ago, Lebowski_DG said:

I have this same router and this morning it would not stay connected to the Internet. The system log printed this line each time the issue surfaced:

WAN Connection: ISP's DHCP did not function properly.

It was not a problem with my modem, as a direct connection works properly.

After numerous router reboots, the issue was only resolved temporarily. Within 5-10 minutes the issue would come back.

I looked further at the system log and noticed this line:

dnsmasq-dhcp[1439]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)

I enabled SSH to the router and checked available disk space with "df -h" command. Sure enough /var was near 100% usage. As I did not know what to clear out from that path, I opted for a router hard reset. 

1. Power off router.
2. Hold WPS button.
3. Power on router still holding the WPS button, wait for power light to turn on then off. Once off, release the WPS button.
4. Power on router.
5. Go through the guided setup, and install the latest firmware while you're at it.
6. Check disk space and see /var now is 0% usage.

Not sure what the root cause is, but thought I might chime in to explain what resolved this annoying issue for me.

Just wanted to chime in here because I had an identical experience waking up today with my RT-AX55 router (updated to latest FW 3.0.0.4.386_51598). Connected but no internet, but bypassing the router and plugging directly into my FiOS Ethernet cable worked totally fine. 

Reboot fixes the problem for around 8-10 minutes, at which point the web interface slows to a crawl and the log starts returning the same "failed to write dnsmasq.leases: No space left on device" error you've been seeing, along with a whole bunch of "out of memory [1837]" and "firewall: apply rules error(****). This is around when the unit starts giving me DNS errors on any connected devices before losing internet connection entirely. 

I hadn't done any setting customization other than setting a SSID and password when I first got it a year ago, and it had been working perfectly fine for a full year. I don't think I ever had more than 10 devices connected. 
 

Either something weird got pushed on the ASUS side that caused a bunch of errors overnight and filled up the storage with logs, or they're doing some stupid stuff with what they store on the device. Something is definitely filling up that limited storage and essentially ruining the unit until you do a factory reset. I wonder if I'll have the same problem in another year?

 

I didn't bother with setting up SSH into the router since it seemed pretty obvious the storage was full! I did a factory reset as described and lo and behold, the thing runs fine for more than 10 minutes. Also noticed a significant drop in post-boot CPU and RAM usage. Before the reset, it would boot to 79% ram and climb to 82-85%%, and a few CPU cores were close to maxing out regularly. After the reset, it booted to 65-66% RAM without climbing, and the CPU cores chilling at 0-2%, with small spikes to 10%. 

Anyways I dunno what secret sauce they're putting in these things cause it's not good haha

[Ben846]

I work for a small ISP and we use Asus routers. We just started having customers call in with these same symptoms.

Have the same lines in the log files on the routers.

I'm about to SSH in to one of the routers to find out if the /var folder is full

[asussuss]

I just started having this same problem today (17 May 2023) after about a year with no issues. Mostly factory default settings. Nothing additional enabled (no AIMesh, no AIProtect etc.) just the default firewall enabled with no firewall packet log saved.

 

Prior to reboot, the router can't connect to the internet even though a direct connection to the fibre line works fine. Logging into Asus UI router.asus.com shows one CPU pegged at 100% and errors in the logs:

 

May 17 09:18:05 dnsmasq-dhcp[1127]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)

 

Rebooting the ASUS RT-AC86U fixes the problem for a short time:

 

# df -h /var
Filesystem                Size      Used Available Use% Mounted on
tmpfs                   215.1M    184.0K    214.9M   0% /var

 

Presumably a bug is filling tmpfs (in memory mount not a real disk?). 

 

I'm on firmware 3.0.0.4.386_48260-gd4c241c

 

It's unusual that this started happening to several of us today.

 

I'd rather not wipe and reload. If someone else finds a solution, please post. Thank you!

 

Edit:

Updated to firmware 3.0.0.4.386_51529-g70acb9d and anxiously awaiting the problem's return.

 

# while true; do df -h; sleep 10; done
Filesystem                Size      Used Available Use% Mounted on
ubi:rootfs_ubifs         77.2M     63.4M     13.7M  82% /
devtmpfs                207.9M         0    207.9M   0% /dev
tmpfs                   208.1M    192.0K    207.9M   0% /var
tmpfs                   208.1M    912.0K    207.2M   0% /tmp/mnt
mtd:bootfs                4.4M      3.3M      1.1M  75% /bootfs
tmpfs                   208.1M    912.0K    207.2M   0% /tmp/mnt
mtd:data                  8.0M    584.0K      7.4M   7% /data
tmpfs                   208.1M    912.0K    207.2M   0% /tmp
/dev/mtdblock9           47.0M      2.9M     44.1M   6% /jffs

 

Edit: I have two of these ASUS RT-AC86U. The one I updated to firmware 3.0.0.4.386_51529-g70acb9d is no longer having the issue. The one still on the old firmware 3.0.0.4.386_48260-gd4c241c repeatedly has the issue.

 

Edit: stable for more than an hour now on the latest firmware. Previously the issue would recur within 10-15 minutes.

Edited May 18, 2023 by asussuss
Added firmware upgrade info

[GM-NY]

How funny, have been on a tech support call with Asus on the exact same issue. This also just started today for me. I gave them the log and they will get back to me within two days.

[lyndwyrm]

I have an RT-AX92U and this just started this morning for me. My firmware version is 3.0.0.4.386_46061, and there is a version 3.0.0.4.388.22525 on the ASUS website, but automatic firmware updates and manual firmware updates have been unsuccessful for me so far today.

Same issue where 

May 17 21:24:31 dnsmasq-dhcp[1298]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)

starts repeating. There are some other message that start popping up after this one but I haven't noticed them as consistently. Things like

May 17 21:17:39 watchdog: restart_firewall due DST time changed(1->0)
May 17 21:17:39 rc_service: watchdog 1327:notify_rc restart_firewall
May 17 21:17:39 rc_service: watchdog 1327:notify_rc restart_wan
May 17 21:17:39 rc_service: waitting "restart_firewall" via watchdog ...
May 17 21:17:39 firewall: apply rules error(2857)
May 17 21:17:39 firewall: apply rules error(2892)
May 17 21:17:39 services: apply rules error(17168)
May 17 21:17:39 firewall: apply rules error(4499)
May 17 21:17:40 dnsmasq[1298]: script process exited with status 127
May 17 21:17:40 dnsmasq[1298]: script process exited with status 127
May 17 21:17:40 dnsmasq[1298]: script process exited with status 127
May 17 21:18:10 watchdog: restart_firewall due DST time changed(0->1)

It's weird that it looks like the Daylight Savings Time setting is repeatedly switching. Which makes me suspicious of the NTP server. Under Administration > System > Basic Config, the NTP server is listed as "pool.ntp.org" which I have to assume is the default since it's not something I've ever touched.

 

Hmmm, 6 unique posters within a few hours on a topic that went for over a year with nothing.

[spilledice]

Hey all, I am running 2 RT-AX92U in mesh mode at my house. I am also having the same issues. I am running on 3.0.0.4.386_46061 firmware. I haven't been able to get into the router when it crashes to view Ram / CPU status. 

 

Hope they get back to you soon! I will open a case as well. 

[Ben846]

7 minutes ago, lyndwyrm said:

Hmmm, 6 unique posters within a few hours on a topic that went for over a year with nothing.

Through my company it's about 30 routers that have called in with this issue.....we have 3500 of these ASUS routers deployed and I'm kinda worried

[Lochinver]

I was running into this as well.  I have an Asus RT-AC66U B1. I was getting the 

 

dnsmasq-dhcp[473]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)

 

also a 

 

May 17 10:06:18 kernel: firmware_check_/19761: potentially unexpected fatal signal 11.
May 17 10:06:18 kernel: Pid: 19761, comm:      firmware_check_
May 17 10:06:18 kernel: CPU: 1    Tainted: P             (2.6.36.4brcmarm #1)

 

I was only able to keep connected to the internet for 5-15 min, I would also loose connectivity to the router.  Then I had to do a hard reset

 

I updated to firmware to

 

3.0.0.4.386_51665-g8072e52

 

and have been stable for 40 min now.

 

HOpe this helps.

[asussuss]

Updating the ASUS RT-AC86U to firmware 3.0.0.4.386_51529-g70acb9d seems to have fixed the issue for me as well. No factory reset was necessary. Best of luck everyone.

 

 

 

[Cavefishes]

Definitely concerning, but also good to know it's probably something on the ASUS end that's affecting a whole lot of units and not a hardware failure. I'd usually be more willing to open a support case with them, share logs, etc but I need solid internet for work and a device reset was the quickest solution short term. 
 

Hopefully they get enough identical reports that they get to work on a fix soon! Glad to have some forum troubleshooting solidarity

[Mirrin]

Had the issue mentioned above router would die after 20 mins of uptime with high CPU and RAM usage. Factory reset no issues past 2hrs

 

DSL-AC68U - 3.0.0.4.386_50117

 

This started as of today 18-5-23 no issues prior to this point in time