PVE stops responding, hardware is stable, nothing gets logged. IO related issues?

[Levent]

I am having issues with my home server where, I made sure RAM and CPU are as stable as they get (even tested them for 72 hours non stop) however i am still getting hardlocks and "CPU stuck" errors if I was logged to server via SSH.

 

Nothing ever show up on journalctl, there is nothing abnormal logged in VMs. Everything simply stops. I cant ssh into PVE or any of the VMs afterwards. Only common thing is, around these hardlocks happen, server is scheduled do IO heavy tasks.

 

I previously posted about this briefly as a status update.

 

I changed my disk options in all VMs to follow virtio-scsi-single, all vm disk have thread=1 enabled.

 

Any thoughts? Only hardware change happened to the server in the last month is that I added another NVME using M2 to PCIE1x adapter.

[leadeater]

Is all your NVMe SSDs direct to CPU or through chipset?

[Levent]

Just now, leadeater said:

Is all your NVMe SSDs direct to CPU or through chipset?

I have one plugged to the x4 slot that is wired to the CPU. Other, I have no clue but considering its rated for PCIE 2 X1, I would assume its through the chipset. I actually havent thought of that before.

[leadeater]

4 minutes ago, Levent said:

I have one plugged to the x4 slot that is wired to the CPU. Other, I have no clue but considering its rated for PCIE 2 X1, I would assume its through the chipset. I actually havent thought of that before.

CPU and chipset?

 

Either one could be bugging out for some reason and you might have to switch to CPU only or chipset only. Also try setting PCIe version down to like 2.0 in the BIOS. Personally I think you are getting errors on the PCIe bus and it's locking up.

[Levent]

5 minutes ago, leadeater said:

CPU and chipset?

 

Either one could be bugging out for some reason and you might have to switch to CPU only or chipset only. Also try setting PCIe version down to like 2.0 in the BIOS. Personally I think you are getting errors on the PCIe bus and it's locking up.

B450M S2H V2, R5 3600, Samsung PM981A (plugged to M2 socket), SKHynix HFM512GD3HX015N (connected via PCIE to M2 adapter) I also got a GTX970 in there as well. It does makes sense, ill remove the graphics card and plug the PCIE1x card there and test again.

 

I should also mention that I got 4x HDDs in this system and I assume these crashes happen when all HDDs and both NVMEs are running full tilt.

 

Thanks!

[LIGISTX]

38 minutes ago, leadeater said:

CPU and chipset?

 

Either one could be bugging out for some reason and you might have to switch to CPU only or chipset only. Also try setting PCIe version down to like 2.0 in the BIOS. Personally I think you are getting errors on the PCIe bus and it's locking up.

I like this idea. @Levent see if this has any impact. 

What PCIe to NVMe adapter card are you using? 

[Levent]

6 minutes ago, LIGISTX said:

I like this idea. @Levent see if this has any impact. 

What PCIe to NVMe adapter card are you using? 

I honestly didnt even see any text on the thing.

(it looks like this)

Spoiler

 

[LIGISTX]

15 minutes ago, Levent said:

I honestly didnt even see any text on the thing.

(it looks like this)

  Hide contents

 

It’s probably™, but this is why I use Supermicro PCIe to nvme cards in my server… less chance of issues with server gear. But I also run an older Supermicro mobo, so it was sort of required.

 

Id try moving the drive to the top slot as suggested, if that doesn’t work, maybe remove it all together for a bit and see if anything changes? I assume it holds some of your VM’s on it tho? If so, do you have a SATA SSD laying around you could use in the meantime for testing? 

[Levent]

26 minutes ago, LIGISTX said:

It’s probably™, but this is why I use Supermicro PCIe to nvme cards in my server… less chance of issues with server gear. But I also run an older Supermicro mobo, so it was sort of required.

 

Id try moving the drive to the top slot as suggested, if that doesn’t work, maybe remove it all together for a bit and see if anything changes? I assume it holds some of your VM’s on it tho? If so, do you have a SATA SSD laying around you could use in the meantime for testing? 

I do have a SATA SSD but I got no SATA ports free. My weekly pipelines are almost finished, once they are finished ill remove the GPU, move the pcie m2 to x16 slot and trigger disk scrub in truenas vm and my database caching worker along with couple of benchmarks on the hypervisor itself. Ill report back.

[Levent]

Quick update, I moved the PCIE NVME adapter to top slot. It wouldnt work in X16 no matter what I did. lspci would show the controller but not the disk? I plugged the GPU back in and moved the adapter to the top pcie1x slot. I ran my cache worker non-stop overnight, I also did couple of terribly long disk benchmarks while scrubbing HDDs. It didnt crash at all.

 

I dont want to say anything just yet and jinx myself.

[leadeater]

3 hours ago, Levent said:

moved the adapter to the top pcie1x slot. I ran my cache worker non-stop overnight, I also did couple of terribly long disk benchmarks while scrubbing HDDs. It didnt crash at all.

Top one 100% is off the CPU btw and the bottom one is off the chipset. Sounds like it wasn't happy running off the chipset. Personally quite hopeful that it has fixed the issue.