VM's going into Read Only

LIGISTX · March 8, 2020

I am really not sure where to turn here, so I hope some folks here can lend me a hand…

I have a homelab that has been stable for a few years now but I am now running into the strangest of issues. I run ESXi with ~4 Ubuntu LTSC 18.04’s and freenas with baremetal HDD access, along with a Win 10 LTSC for Veeam backup of the Ubuntu VM’s.

A few weeks ago, I would SSH into an Ubuntu VM and ZSH would prompt me of the file system being Read Only. I shrugged it off as a weird one time thing and restored from Veeam, but that it started happening more, and to more VM’s. Last night, sometime between when I went to sleep and woke up, all 4 VM’s had it happen…

Initially I thought maybe it was a Veeam backup durring an Ubuntu auto-security update (I do have that enabled), but Veeam did not run last night. I am restoring one of the smaller VM’s right now to an earlier state just to try and collect some info, but I am not even sure what to collect.

When I google the issue, it seems like the main answer is a mounted file system has errors, or FSTAB is the issue. I am not sure how this would be as one of the VM’s is not mounting any of my FreeNAS storage at all… It has no network links at all. I just got that VM restored (it just runs pihole… pretty simple VM, literally nothing to it at all except pihole), this is the fstab config:

UUID=2d945db3-1ff7-4c22-8022-0479207bb427 / ext4 defaults 0 0
/swap.img none swap sw 0 0

I run the VM’s (and ESXo) on a consumer SSD, and I upgraded to it maybe 5 months ago so its not exactly old, and its not like my VM’s hit it hard at all, its a very low use homelab, its on 24/7, but realistically its doing almost nothing 100% of the time. Could the VM’s be doing some sort of SMART check, seeing an issue with the SSD, and going to read-only as some sort of protection? How would I determine this? What else could be happening?

LIGISTX · March 8, 2020

Trying some things out, I tried this. So, looks like Ubuntu put itself in read-only as it found some corruption? So is this indicative of a failing boot SSD?

LIGISTX · March 8, 2020

With this information, and the fact the VM seems to be “ok” now that it rebooted, I am inclined to think these are the possible issues I am facing. Bad/loose SATA cable, dying SSD, bad RAM.

I am inclined to rule out RAM as FreeNAS and ZFS would likely be throwing all sorts of errors if it was seeing checksum issues… FreeNAS has bare metal access to my HBA, and I run ECC RAM. I would like to think somewhere along this chain FreeNAS would have been the first thing to throw issues at me if it was in fact a RAM issue. This leads me to believe its a bad SSD/SATA sable to the SSD.

Some info on the homelab if it will help:

Homelab/ Media Server: ESXi 6.5 - - 250 GB SSD for VM’s/ESXi boot - - FreeNAS 11.2-U5 - -HPE Proliant ML10 Gen 9 backbone - - i3 6100 - - 28 GB ECC - - 10x4 TB WD Red RAID Z2