Full memory in router??

[PlainMn]

I just googled this same problem for my RT-AC66U B1 that has been running great until yesterday.  Firmware update says 5/15 version 3.0.0.4.386_46065-ge51f2dc and it is showing the similar errors of out of space:

 

May 17 22:14:29 dnsmasq-dhcp[220]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
May 17 22:15:21 dnsmasq[220]: failed to allocate 1028 bytes
May 17 22:15:23 dnsmasq[220]: failed to allocate 1028 bytes
May 17 22:15:23 dnsmasq[220]: failed to allocate 1028 bytes
May 17 22:15:29 dnsmasq-dhcp[220]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
May 17 22:16:02 dnsmasq[220]: failed to allocate 1028 bytes
May 17 22:16:29 dnsmasq-dhcp[220]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
May 17 22:16:31 dnsmasq[220]: failed to allocate 1028 bytes
May 17 22:17:01 dnsmasq[220]: failed to allocate 1028 bytes
May 17 22:17:29 dnsmasq-dhcp[220]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
May 17 22:17:59 kernel: dcd/1618: potentially unexpected fatal signal 11.

I enabled ssh and logged in, but could not "see" anything because it said no home directory was available and bash was not found.  Router set to manually update firmware, but it updated anyway apparently.  I did not initiate the update.

 

EDIT:  updated to firmware RT-AC66U_B1_3.0.0.4_386_51665-g8072e52 from asus website and seems to be working.  Log no longer shows failure to write or allocate

[Ben846]

The original routers we rebooted 2 hours ago started getting the dnsmasq errormessage again after 20-30 minutes.

Routers rebooted less than an hour ago have so far been stable with no dnsmasq error messages.

 

I don't think firmware updates (ours are all on the latest) solved this. I think whatever external force was causing this has stopped for the night.....

[8bitRunner]

This started happening to me today too!

[ChaseMeredith]

+1 for Firmware Update solving the issue. 

 

Because I didn't want to get my laptop out, I rebooted the router, connected to the web UI with my phone as soon as it came up, checked for updates and installed the latest version.  Super simple once the problem was identified, but man was that annoying at first haha. 

[GnatGoSplat]

This started happening to me this morning and still going on.  Seems to be not enough DHCP space for clients.

I've got an RT-AC3100 on 3.0.0.4.386_48260, which is unfortunately the newest.

 

[angl]

Started having the same problem this morning. Restarting would fix for 10-20 minutes then fail again.

I was already on the latest firmware: 3.0.0.4.386.49380.

Factory reset the router and now hoping it doesn't drop again in 15 minutes. This is on a RT-AX56U.

 

Update:

Possibly useful for others: First comment I found here https://www.downtowndougbrown.com/2023/05/what-happened-with-asus-routers-this-morning/trackback/

 

They say they deleted  the file /jffs/asd/chknvram20230516 then rebooted and that seems to have fixed the issue. That might be simpler that a factory reset.

 

Second update:

Router has been working for about 40 minutes after the factory reset. It hasn't lasted that long all day, so I'm feeling optimistic.

 

[Jolly Raunchy]

Same issue started happening to me yesterday and has continued today. The number of people seeing this, seemingly "out of the blue", seems to be too much of a coincidence. Hopefully Asus is diligent in troubleshooting and resolving this, at this point they don't need (yet) another black eye.

[PiotrG]

Whoah what a wonderful nightmare this is  My 56U also goes down every 20 minutes. Started today, morning.

ASUS RT-AX56U Firmware version 3.0.0.4.386.49380 - got the latest so no upgrade for me.

 

telnet

o "router ip"

give cedentials

rm /jffs/asd/chknvram20230516

 

works for close to an hour now and cpu/ram load is half of what it was early today

 

[UnexpectedSegue]

Had the same problem when I woke up this morning.

Updated RT-AX55 FW to 3.0.0.4.386_51598 and the router have been stable since for a few hours. CPU cores runs at 1-2 % with spikes in 10% and RAM with stable 81% usage.

[lyndwyrm]

4 hours ago, angl said:

Started having the same problem this morning. Restarting would fix for 10-20 minutes then fail again.

I was already on the latest firmware: 3.0.0.4.386.49380.

Factory reset the router and now hoping it doesn't drop again in 15 minutes. This is on a RT-AX56U.

 

Update:

Possibly useful for others: First comment I found here https://www.downtowndougbrown.com/2023/05/what-happened-with-asus-routers-this-morning/trackback/

 

They say they deleted  the file /jffs/asd/chknvram20230516 then rebooted and that seems to have fixed the issue. That might be simpler that a factory reset.

 

Second update:

Router has been working for about 40 minutes after the factory reset. It hasn't lasted that long all day, so I'm feeling optimistic.

 

I tried this on my RT-AX92U, but I ssh'd in and actually just moved chknvram20230516 to chknvram20230516.bak. After rebooting and ssh'ing in again, the /jffs/asd/ folder was empty. But after that reboot, the RAM usage stopped climbing and is currently at a stable 76% (was climbing to above 90% before). I didn't think the CPU usage was odd before, hopping around 20-50%, but the CPU usage now is pretty consistently down near 1-2%.

 

I'll feel more confident if I still have internet in 12 hours, but it appears that getting chknvram20230516 to go away has resolved the problem.

 

Edit: I no longer anticipate it had any effect, but I did also change the NTP server to time.nist.gov rather than pool.ntp.org

[InAUGral]

  Hi all, FYI RT-AC86U user here. I started having issues some time during the night and when I woke up this morning the internet was dead, I restarted the router then didnt have issues again for around 10 hours. From that point it happened every 5 minutes so had to find a fix for it. After reading the advice I updated my firmware and it seems to be ok so far.  Imagine my shock when I came across this thread with many others having the same issue at the same time, very strange. Unluckily for me SSH did not work when the errors were occuring I assume because the disk was full so not sure of the exact cause other than firmware bug 

[GnatGoSplat]

Deleting chknvram20230516 seemed to have fixed mine.  CPU utilization also went WAY down after reboot.

 

[MagnificentBen]

Started having this issue yesterday. I was happy to find this forum and upgraded firmware (fingers crossed). What's surprising to me is that I disabled automatic firmware upgrades as I didn't want the latest and greatest version to be automatically pushed and break something. How the hell did this started happening overnight if supposedly nothing is pushed from asus side?? Are they pushing changes to machines that I can't turn off?

[GnatGoSplat]

6 minutes ago, MagnificentBen said:

Started having this issue yesterday. I was happy to find this forum and upgraded firmware (fingers crossed). What's surprising to me is that I disabled automatic firmware upgrades as I didn't want the latest and greatest version to be automatically pushed and break something. How the hell did this started happening overnight if supposedly nothing is pushed from asus side?? Are they pushing changes to machines that I can't turn off?

I suspect it's a firmware bug that's been there for some time.  It seems it hit everyone on the same date regardless of firmware version.  There is a repeating error in asd.log:

[chknvram_action] Invalid string

That's clearly a date in the chknvram20230516 filename of the corrupted file.  So my hypothesis is there's just something about May 16, 2023 that caused a corrupt chknvram file to be generated that the Asus Security Daemon (asd) trips on.

 

[Zewtastic]

RT-AC68U 

 

My issue may have started on the 16th.

I simply updated my firmware to the latest AFTER a fresh router restart. 

 

Currently at Firmware 3.0.0.4.386_51665-g8072e52.

 

We'll see how it goes but no more of these messages in the log.

 

May 18 07:50:21 dnsmasq-dhcp[725]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
May 18 07:50:21 dnsmasq-dhcp[725]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
May 18 07:50:21 dnsmasq-dhcp[725]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
May 18 07:50:21 dnsmasq-dhcp[725]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
May 18 07:50:21 dnsmasq-dhcp[725]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)

May 18 07:50:22 dnsmasq-dhcp[725]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
May 18 07:50:22 dnsmasq-dhcp[725]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
May 18 07:50:23 dnsmasq-dhcp[725]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
May 18 07:50:23 dnsmasq-dhcp[725]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)

 

So do a manual restart of your AC68U (so you do not get an outage during firmware DL).

DL latest firmware, if it is not at latest.

Cross your fingers.

 

UPDATE: General log has been empty since updating to latest firmware, so some sort of coded bug that was date triggered seems to have been the issue. 

[GnatGoSplat]

2 minutes ago, Zewtastic said:

Well my issue started on the 17th, so that invalidates the 16th theory.

Not necessarily.  Mine started on the 17th as well, but that doesn't necessarily mean the asd process starts looking at the file right away, or that it was even generated at 0:00.  I didn't check the timestamp before wiping mine away, but it could have been generated at 23:59.  Or the date could even be zero-based and actually mean the 17th.  I've worked with some languages in the past where there's inconsistency between year, month, and day being 0 or 1 based.

[Zewtastic]

2 minutes ago, GnatGoSplat said:

Not necessarily.  Mine started on the 17th as well, but that doesn't necessarily mean the asd process starts looking at the file right away, or that it was even generated at 0:00.  I didn't check the timestamp before wiping mine away, but it could have been generated at 23:59.  Or the date could even be zero-based and actually mean the 17th.  I've worked with some languages in the past where there's inconsistency between year, month, and day being 0 or 1 based.

I agree, might have been triggered early than I identified.

[Ticonivus]

Asus has officially released this:

Interruption in Router Product Connectivity and Urgent Mitigation Measures. During routine security maintenance, our technical team discovered an error in the configuration of our server settings file, which could potentially cause an interruption in network connectivity on part of the routers.   Our technical team has urgently addressed the server issue, and impacted routers should be returning to normal operation. If your device is still affected, we recommend the following: 1. Manually reboot your router 2. If rebooting does not resolve the issue, please save the settings file, perform a hard reset (factory default), and then re-upload the settings file (follow the directions in the https://www.asus.com/support/FAQ/1001376/) If there are any further developments around this issue, we will immediately update our users. We deeply apologize for any inconvenience this incident may have caused and are committed to preventing such an incident from happening again. Thank you for your understanding and support.

[GnatGoSplat]

So it sounds like the asd process phones home and downloads settings files from Asus on a regular (daily?) basis.  Kind of tempted to disable it.

 

[Ticonivus]

I have also found that reducing the LAN DHCP lease time to 3600 seconds solves the issue and it doesn't come back. This is actually the only thing that has fixed it so far on our end.

[ChewySoftware]

I just created an account so I could add my story to the pile. I don't see any previous posts about the AX6600 which I'm using, so I wanted the community to know the issue is affecting this router model too. As other posters have noted, the issues started out of the blue on 5/17 with unexplained internet failures. Looking in the router log I saw many of the same mysterious errors including:

 

May 17 09:40:29 dnsmasq-dhcp[1501]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)

...

May 17 10:23:51 WAN Connection: ISP's DHCP did not function properly.

 

I also saw some bizarre behavior in which the router was switching back and forth between the correct time and an incorrect time on May 5. Typical log lines include the following. Note that continuous text blocks denote consecutive log lines showing the transition points where the router switches from the correct time late on May 17 to the incorrect time, and then back again to the correct time (after an apparently successful NTP check) about 5 minutes later:

 

May 17 19:20:50 watchdog: restart_firewall due DST time changed(0->1)
May 17 19:20:50 rc_service: watchdog 1552:notify_rc restart_firewall
May 17 19:20:50 rc_service: watchdog 1552:notify_rc restart_wan
May 17 19:20:50 rc_service: waitting "restart_firewall" via watchdog ...
May 17 19:20:51 miniupnpd[18183]: shutting down MiniUPnPd
May  5 00:05:18 crashlog: LOG
May  5 00:05:18 crashlog: pgd = cfe68000
May  5 00:05:18 crashlog: [00000004] *pgd=139c7831, *pte=00000000, *ppte=00000000
May  5 00:05:18 crashlog: Internal error: Oops: 17 [#1] PREEMPT SMP ARM

...

May  5 00:05:34 ntp: start NTP update
May 17 19:24:48 rc_service: ntp 2390:notify_rc restart_diskmon

 

My router is configured for manual firmware update and I haven't updated it in over 6 months. The firmware version these problems occurred under is 3.0.0.4.386_43170. I have since rebooted my router and updated the firmware. My router has been running for 1.5+ hours now with no error messages appearing in the log, so I'm fairly confident the issue has been resolved. Interestingly, the latest firmware version for the AX6600 seems to be several revs ahead of the 386_50117 version which is fixing problems for other Asus router models. My AX6600s (2 of them in my home mesh network) are now running firmware version 3.0.0.4.388_23285. I will update my post if I see additional problems but for now all looks good.

 

Thank you to everyone here for sharing the problems seen and the solutions that worked! This made it massively easier to track down what was going wrong in my own network. Hopefully my notes above will help others with the AX6600 do the same.

 

UPDATE 5/18/12 @ 9:00pm: My router has now been running for 10+ hours with no error messages appearing in the log. Internet has been up and running all day with no noticeable problems. Looking good for a complete fix after the firmware upgrade.

[shaddai]

5 minutes ago, ChewySoftware said:

I just created an account so I could add my story to the pile. <snip>

Same. This has been bugging me for a few days now, but diving into it today has been interesting. I've seen the same dnsmasq-dhcp failed to write errors, as well as the NTP time discrepancies.

 

What I just did that seems to also line up is I noticed the asd process was flooring the CPU (and apparently memory too). I killed it and the dnsmasq errors stopped, and free memory greatly increased.

 

Interestingly... I also saw these messages in my syslog... 

 

May 18 09:17:28 ZenWiFi_XD4-81D0-4E08CE2-C dnsmasq-dhcp[2614]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
May 18 09:18:19 ZenWiFi_XD4-81D0-4E08CE2-C kernel: potentially unexpected fatal signal 11.
May 18 09:18:19 ZenWiFi_XD4-81D0-4E08CE2-C kernel: CPU: 2 PID: 12159 Comm: wl Tainted: P                4.1.52 #4
May 18 09:18:19 ZenWiFi_XD4-81D0-4E08CE2-C kernel: Hardware name: Generic DT based system
May 18 09:18:19 ZenWiFi_XD4-81D0-4E08CE2-C kernel: task: d410c800 ti: d3e8e000 task.ti: d3e8e000
May 18 09:18:19 ZenWiFi_XD4-81D0-4E08CE2-C kernel: PC is at 0xb6d6b5dc
May 18 09:18:19 ZenWiFi_XD4-81D0-4E08CE2-C kernel: LR is at 0x2fccc
May 18 09:18:19 ZenWiFi_XD4-81D0-4E08CE2-C kernel: pc : [<b6d6b5dc>]    lr : [<0002fccc>]    psr: 60070010
May 18 09:18:19 ZenWiFi_XD4-81D0-4E08CE2-C kernel: sp : bede63c4  ip : b6d6b5d0  fp : 003134ec
May 18 09:18:19 ZenWiFi_XD4-81D0-4E08CE2-C kernel: r10: 003134ec  r9 : 000739df  r8 : 0007714e
May 18 09:18:19 ZenWiFi_XD4-81D0-4E08CE2-C kernel: r7 : 0000000b  r6 : 000739df  r5 : 003134bc  r4 : 003134a4
May 18 09:18:19 ZenWiFi_XD4-81D0-4E08CE2-C kernel: r3 : 00000000  r2 : 003134bc  r1 : 003134a4  r0 : fffffff4
May 18 09:18:19 ZenWiFi_XD4-81D0-4E08CE2-C kernel: Flags: nZCv  IRQs on  FIQs on  Mode USER_32  ISA ARM  Segment user
May 18 09:18:19 ZenWiFi_XD4-81D0-4E08CE2-C kernel: Control: 10c5387d  Table: 13f5804a  DAC: 00000015
May 18 09:18:19 ZenWiFi_XD4-81D0-4E08CE2-C kernel: CPU: 2 PID: 12159 Comm: wl Tainted: P                4.1.52 #4
May 18 09:18:19 ZenWiFi_XD4-81D0-4E08CE2-C kernel: Hardware name: Generic DT based system
May 18 09:18:19 ZenWiFi_XD4-81D0-4E08CE2-C kernel: [<c0026fe0>] (unwind_backtrace) from [<c0022c38>] (show_stack+0x10/0x14)
May 18 09:18:19 ZenWiFi_XD4-81D0-4E08CE2-C kernel: [<c0022c38>] (show_stack) from [<c04ff078>] (dump_stack+0x8c/0xa0)
May 18 09:18:19 ZenWiFi_XD4-81D0-4E08CE2-C kernel: [<c04ff078>] (dump_stack) from [<c003ac6c>] (get_signal+0x490/0x558)
May 18 09:18:19 ZenWiFi_XD4-81D0-4E08CE2-C kernel: [<c003ac6c>] (get_signal) from [<c00221d0>] (do_signal+0xc8/0x3ac)
May 18 09:18:19 ZenWiFi_XD4-81D0-4E08CE2-C kernel: [<c00221d0>] (do_signal) from [<c0022658>] (do_work_pending+0x94/0xa4)
May 18 09:18:19 ZenWiFi_XD4-81D0-4E08CE2-C kernel: [<c0022658>] (do_work_pending) from [<c001f4cc>] (work_pending+0xc/0x20)

[ChewySoftware]

Out of curiosity, how are people able to see CPU load and which processes are running and using the most resources? Is there an external tool to do this or is this info buried somewhere in the router's admin interface? FYI, I'm using just the included web based admin UI for my ZenWiFi AX6600 router. Thanks, in advance, for any tips on how to do this!

[shaddai]

6 minutes ago, ChewySoftware said:

Out of curiosity, how are people able to see CPU load and which processes are running and using the most resources? Is there an external tool to do this or is this info buried somewhere in the router's admin interface? FYI, I'm using just the included web based admin UI for my ZenWiFi AX6600 router. Thanks, in advance, for any tips on how to do this!

You can see some of these details by going to the web interface, Network Map, and then choose the Status tab off to the right.  (edit - admittedly it only shows you usage percentages/amounts)

[VerityLambert]

Created an account for this etc, chiming in to add my 2cents to this thread, started having the same issue, RT68Uv2 (?) initially thought it was related to timeout on PADO packets (called my ISP to resolve.) But this kept happening, the factory reset has seems to have kept the memory goblins at bay.