Kernel Panic - Not syncing | Ubuntu 20.04

[Koen Peters]

Hello everyone!

 

Thank you for taking the time to look at this as i am learning to manage my own server to run Multiple versions of one application with different configurations.

 

What this needs is each of the applications needs its own IPv4 so this server has multiple ipv4 connected and configured in the netplan file. 

 

Basically what i do when i setup the server from scratch is this:
- Allow SSH root login

-configure netplan

- edit limits.conf in etc/security to allow more files to be open

- apt update & upgrade

- install the following: "sudo apt-get -y install -qq --force-yes jq iptables-persistent fail2ban htop git apache2"

- setup swap file

- edit ip tables for the ip addresses and each application has its own internal port with

"iptables -t nat -A PREROUTING -i eno1 -p tcp -d $IP_ADDRESS.$i --dport 7112 -j DNAT --to $IP_ADDRESS.$i:$val"
"iptables -t nat -A OUTPUT -o lo -p tcp -m tcp -d $IP_ADDRESS.$i --dport 7112 -j DNAT --to $IP_ADDRESS.$i:$val"

 

I do not know what is causing this kernel panic as the servers have been running for about 9 months some a bit shorter some longer. But after restart they all have this issue that after a bit of running (1 day?) they crash into this kernel panic.

 

I hope someone here can help me figure this out! 

 

Thank you for your time to read this and if you need any more information to help please let me know

 

 

[Ralphred]

1 hour ago, Koen Peters said:

running for about 9 months some a bit shorter some longer. But after restart

Check your package managers log for a kernel update, if you find one - roll it back. Regressions are not unheard of, and you may have stumbled across one. Whether it's a vanilla regression or something to do with your distros patchset is another question, but get your servers working first, then start playing around on a test system.

[Koen Peters]

2 hours ago, Ralphred said:

Check your package managers log for a kernel update, if you find one - roll it back. Regressions are not unheard of, and you may have stumbled across one. Whether it's a vanilla regression or something to do with your distros patchset is another question, but get your servers working first, then start playing around on a test system.

I found that it went to kernel version 5.4.0-122 on 12th of july on one of the servers via unattended upgrades.

 

When i look at possible boot kernel version i see these:
`:~# sudo grep 'menuentry \|submenu ' /boot/grub/grub.cfg | cut -f2 -d "'"
Ubuntu
Advanced options for Ubuntu
Ubuntu, with Linux 5.4.0-122-generic
Ubuntu, with Linux 5.4.0-122-generic (recovery mode)
Ubuntu, with Linux 5.4.0-107-generic
Ubuntu, with Linux 5.4.0-107-generic (recovery mode)
UEFI Firmware Settings`

I have edited the Grub file for the older kernel that is still installed, lets hope that this fixes it!

 

Thank you for your input!

[Chunchunmaru_]

5 hours ago, Koen Peters said:

Hello everyone!

 

Thank you for taking the time to look at this as i am learning to manage my own server to run Multiple versions of one application with different configurations.

 

What this needs is each of the applications needs its own IPv4 so this server has multiple ipv4 connected and configured in the netplan file. 

 

Basically what i do when i setup the server from scratch is this:
- Allow SSH root login

-configure netplan

- edit limits.conf in etc/security to allow more files to be open

- apt update & upgrade

- install the following: "sudo apt-get -y install -qq --force-yes jq iptables-persistent fail2ban htop git apache2"

- setup swap file

- edit ip tables for the ip addresses and each application has its own internal port with

"iptables -t nat -A PREROUTING -i eno1 -p tcp -d $IP_ADDRESS.$i --dport 7112 -j DNAT --to $IP_ADDRESS.$i:$val"
"iptables -t nat -A OUTPUT -o lo -p tcp -m tcp -d $IP_ADDRESS.$i --dport 7112 -j DNAT --to $IP_ADDRESS.$i:$val"

 

I do not know what is causing this kernel panic as the servers have been running for about 9 months some a bit shorter some longer. But after restart they all have this issue that after a bit of running (1 day?) they crash into this kernel panic.

 

I hope someone here can help me figure this out! 

 

Thank you for your time to read this and if you need any more information to help please let me know

 

 

I can't understand the environment 

Same OS version on those servers? 

Multiple physical servers?

Virtualized? 

[Ralphred]

1 hour ago, Koen Peters said:

Ubuntu, with Linux 5.4.0-122-generic
Ubuntu, with Linux 5.4.0-107-generic

This is quite a small version bump, if it does fix it, it should be fairly trivial to trace.

[Koen Peters]

26 minutes ago, Chunchunmaru_ said:

I can't understand the environment 

Same OS version on those servers? 

Multiple physical servers?

Virtualized? 

Multiple physical servers, same OS ubuntu 20.04 all configured the same way.

 

There are some hardware differences to what was available at the time.

 

Most of them run: 
AMD Epyc 7313 - 16c/32t - 3 GHz/3.7 GHz

128 GB ECC 3200 MHz

3×1.92 TB SSD NVMe

 

 

[Koen Peters]

22 hours ago, Ralphred said:

Check your package managers log for a kernel update, if you find one - roll it back. Regressions are not unheard of, and you may have stumbled across one. Whether it's a vanilla regression or something to do with your distros patchset is another question, but get your servers working first, then start playing around on a test system.

It worked, all servers are still running! For now the issue is fixed, will need to make time to see what changed in the versions between to see what broke something

 

 

Thank you for your input!