You want to have a core unused for every two gpus for maximum gpu potential, hopefully with the arm processor on nvidias 900 series this ratio will become a relic but for now most GPUs still have to run a lot of code with CPU. If you can enable any more cores beyond that ratio, great do it, just watch for a performance drop off.