Limit GPU for app/Prioritize app

[Afthrast]

I was looking around for a solution for my issue, but never found a direct one:
I'd like to use my GPU to encode recordings with StreamlabsOBS with the new nvenc encoder, but most of the time my PC is limited by my GPU, not by my CPU.

Nvenc needs based on experience ~10% GPU overhead (90% usage) to make sure that the recording/stream won't get any stutter. I don't want to buy a recording card.

Does anyone know a reliable way of doing this? Basically I'd either need Streamlabs to always have 10% GPU available or the game to use max of 90%.

I'm open to creating VMs for it as well if that could be a possibility.

Some extra info: Streamlabs priority is set to high already - can't use CPU encoding because even those games that leave some headroom, several of them are heavily single-threaded and for just the hell of it, usually use different cores, so core affinity would have to be set up per game which is an annoyance, plus sometimes even these games become GPU bottlenecked from time-to-time, so it's difficult to separate games for CPU or GPU encoding as well.

I don't want to limit FPS even though twitch has a 60fps limit because: I have a 144hz monitor and 60fps isn't pleasant on it and even if I set it, there can always be dips below the set framerate at which point the GPU is again 100% used and the overhead for Streamlabs/OBS is gone.

Anyone bumped into the same issue?

[agent_x007]

Drop game graphics settings for GPU to have more headroom/free resources.

[Afthrast]

If I'm not limiting my FPS/enabling vysnc I'd just have to lower graphics until the graphics heavy, multi-thread optimized game becomes fully CPU bottlenecked or else it will just push out crazy amounts of FPS while still running on max GPU usage.

 

But for an example: let's say a GPU bottlenecked game runs with 100FPS while recording/streaming and will be choppy due to 100% gpu usage.

Fine, I'll turn it down to 80FPS which is still fine on 144hz monitor (it's easier to limit FPS if there's more than needed, than to lower graphics so much, that it becomes CPU bottlenecked without fps limit/vsync) - so there's headroom, but if at any point during the game there's a huge clusterf**k of graphics heavy stuff, it will eat up that overhead and the stream/recording is stuttering again, despite giving it more headroom - by either lowering graphics to the point it's CPU bottlenecked or limiting FPS.

I'm not trusting a game to always remain CPU bottlenecked even with lowest graphics, take for example World of Warcraft, it's heavily CPU single-thread bottlenecked so it wouldn't even be a problem to stream it with nvenc decoder, but to my surprise, there are some new locations in the game, one just added in the latest patch, where my recording went haywire, because somehow they put in enough stuff that it went from being 60% GPU usage to 100% while remaining at around 80FPS.

 

TL:DR: I'm asking if anyone knows about a way how to GPU limit a game, or assign x% GPU to an application, I've read all the basic "if you just started streaming" tips.