At first this did not work, so I tried clearing the CMOS aswell, this helped it and made me boot into the USB, finally. Don't know how I managed to forget that part though.
Also for other people, what fixed my issue is going into the repair of the windows recovery on my USB drive, and then select solve start-up issues. The troubleshooter said that it could not find any issues with my drive, and told me it did nothing. When I tried to boot again, it suddenly booted up again, so it did do something, I just have no clue what.