[Update on the diagnostic]
So I eventually could replace the GPU with a new one, which did not solve the issue. Now I have to check the entirety of the PC to find the failing component(s).
I removed the CPU from its socket, the socket smelled like heated dust, but definitely not like it burnt. The CPU smells fine, its pins are straight and clean. What's surprising to me is that the motherboard passes the CPU check successfully, so it might be the main culprit here.
What do you suggest I do? Before replacing either the motherboard or the CPU that is.