New Components, No POST

[Swankey]

Hi all,
Recently I have upgraded a crappy pre-built PC by replacing every single part except for the RAM and the GPU.
However, over the past couple of weeks, my PC has not been booting, and my motherboard makes 5 short beeps.
I'm pretty sure this means CPU error, but the thing is, I have already gotten my CPU replaced, and it is working perfectly when I boot the PC without the GPU.
What is so strange, is that I've ruled out the idea that the CPU is the problem and therefore can confidently say that the GPU is the problem. But when I plug the GPU back in, why does the beep code say that the CPU is faulty?

My specs are:
GPU - Galax GTX 1060 6GB
CPU - i3 9100
Mobo - Gigabyte B365M (HD3)
PSU - Be Quiet! Pure Power 11 600W Gold Rated
RAM - Not really sure what this is, but the PC works with it so...?
Storage - MX500

I'm pretty much convinced that the GPU is the issue, so I'm planning to grab a RX 5500 xt, but I would love to hear some thoughts on what the issue is first.
Any help will be greatly appreciated!

[Master Disaster]

The GPU is directly connected to the CPU by PCIe so its possible the CPU is reporting an error in POST because it cannot talk to the GPU correctly. It could also be because of an issue on the board as well though, broken trace somewhere or a bad PCIe slot.

[Swankey]

Ok thanks for the reply, I guess I'm grabbing that RX 5500 xt then