Ubuntu 18.04.3 LTS Crashing on Startup

[Windows7ge]

All I can really say to describe the issue is that the system was fine earlier today but turning it off for a while and coming back to turn it on I'm greeted by this (I would shrink the image but the text is small):

I don't have a complete understanding of what's going on in the text or if it in any way says why it's hung-up.

 

I've done some of the most basic troubleshooting:

1. Reset it
2. Turn it off and on
3. Cut power, turn back on
4. Unplug GPU
5. Unplug unnecessary peripherals

That's where I'm at right now but I have a strange feeling the issue isn't hardware. Ubuntu didn't tell me an update ran (usually does) but I still think something happened in the background and it bricked my install. Does anybody know where to find and how to read the crash log? I've been told there usually is one when there's a kernel panic.

 

Any other suggestions on how to get this back up & running are appreciated.

 

Hardware:

AMD TR-1950X

G.Skill Ripjaws 4x8GB 2400MHz

ASUS PRIME X399-A 

2x Sapphire R9-290X

Samsung 960 PRO 1TB

Corsair AX1200i

Broadcom 2x 10Gbit NIC.

[Electronics Wizardy]

try entering single user mode.

 

Is it really stuck on running vms, that seems odd, try disabling those.

[Windows7ge]

26 minutes ago, Electronics Wizardy said:

try entering single user mode.

So I looked up how to do that and after entering it here I am back at my desktop...I'm going to give it a minute to see if we crash or not. I see "single user mode" is a one shot deal. Does this help up narrow down a more underlying issue?

 

29 minutes ago, Electronics Wizardy said:

Is it really stuck on running vms, that seems odd, try disabling those.

I did not know how to do that from CLI so since I no longer needed the VM I just deleted the ovmf & virt-manager packages. The start-up text changed but it still crashed. I'm going to reboot and see if exiting single user mode cause another hang-up..

[Windows7ge]

It did.

At least single user mode works. A good temp solution at least.

[Electronics Wizardy]

10 minutes ago, Windows7ge said:

So I looked up how to do that and after entering it here I am back at my desktop...I'm going to give it a minute to see if we crash or not. I see "single user mode" is a one shot deal. Does this help up narrow down a more underlying issue?

 

I did not know how to do that from CLI so since I no longer needed the VM I just deleted the ovmf & virt-manager packages. The start-up text changed but it still crashed. I'm going to reboot and see if exiting single user mode cause another hang-up..

do you get a shell in single user mode?

 

Does it respond to things like numlock or caps lock?

 

I think there is a verbose grub settings as well.

[Windows7ge]

5 minutes ago, Electronics Wizardy said:

do you get a shell in single user mode?

The whole GUI/Desktop mode loads as it should in single user mode. I can try it again to make sure it wasn't a fluke.

 

8 minutes ago, Electronics Wizardy said:

Does it respond to things like numlock or caps lock?

On the stuck [ OK ] loading screen? It responds to Num but not Caps. Weird.

 

10 minutes ago, Electronics Wizardy said:

I think there is a verbose grub settings as well.

Probably. I can try looking up enabling that so we get more detail on exactly where it's stuck.

 

Since much of the issue does look to be VM related I do have some settings I can try disabling from CLI to see if it solves the problem. Things like hugepages, disabling IOMMU groups. I can remove the VM config files. Etc. It doesn't really make any sense to me though. None of this was an issue before. I saw no software changes it just up and quit.

[Windows7ge]

16 hours ago, Electronics Wizardy said:

I think there is a verbose grub settings as well.

I looked up how to enable verbose mode and it yields a lot more text. I can't say it really states what it's hung on though.

I'm thinking I'll just do a clean install. All my data stays on my network server or in the Chrome browser and the repositories for Ubuntu make re-installing programs a short script away.

 

Before I do though I will mention by mistake I removed "quiet splash" from the advanced startup menu and it enabled the system to boot properly. Whatever broke it really got in there and did something funky.

[Windows7ge]

@Electronics Wizardy Good news and bad news.

 

Good news: Ubuntu works again.

 

Bad news: OpenCL is broken. I've lost the ability to compute on my GPUs...looks like I'm going to have to take another look at rocm because installing the amdgpu-pro driver bricks the whole OS install.

 

None of this makes any sense. Either AMD or Ubuntu must have pushed an update and it's broken compatibility.

[Windows7ge]

@Electronics Wizardy Fixed it. Everything's working again (for the time being).

 

TLDR:

It was a kernel/driver incompatibility.

 

Long Explanation:

I had downloaded the .ISO from the Ubuntu website. It came with v5.3.0-26-generic. I decided to check my File Server for a copy of Ubuntu and found one for 18.04.3 LTS. I installed it and it had kernel 4.15.0-74-generic. So I tried installing the amdgpu-pro driver on that and it works.

 

So two things have happened here:

1. Ubuntu has updated their current 18.04.3_LTS.iso file with kernel version 5.3.0-26-generic.
2. AMD has NOT updated their current amdgpu-pro driver to be compatible with Ubuntus kernel update.

None of this explains why my rig committed seppuku in the first place but it means we're back up and running fully.