It appears the switch is on the 12V line and all other connections are normal.
So you need to remove the 5V line (red).
Swap the switched 12V line (yellow) to the 5V pins. Best done by rehousing the pins if you can.
You will then need either 5V fan or add a relay that can switch the 12V off the 5V signal.
I suppose you could also just put the relay on the 12V to switch the 5V
Or you could purchase a DPST switch and switch both sides at the same time.