Ok I solved it, this is fucking weird. Turns out the card only crashes on the HDMI port. My monitor is connected over DisplayPort so I never saw anything in my machine, however just now booting up VR it crashes the headset and nothing else. Looks like we just got another faulty card :/