I will just post some infos to compare...
i7 8700k@5GHz and GTX 1080Ti@2GHz both watercooled with a HWLabs GTS and 3xNF-A12x25 and the radiator was NOT in the case (on top of the case, like a open test bench, for testing).
I ran a test with The Witcher 3 and Prim95 on 8 Threads (to give enough CPU resources to The Witcher.
fans @ 100% ~dT 10K
fans @ 80% ~dT 11,5K
fans @ 60% ~dT 13,1K
I would guess (since the overall energy your system creates is lower than mine) that your heat transfer from water to air is not working properly. Is anything restricting the airflow? How is your rad/fans mounted?