I say GPU. But you should wait for the prices to drop again
if you want to play native gsync 165hz you r gonna have to lower settings by a considerable amount
but just save up and buy them both at the same time(or just wait for a sale on either one)
buying that monitor and using a 780 is a waste of the monitors potential because you will have to compromise a lot to get the results you want from a monitor like that.
Upgrading the GPU is a waste of the GPU's potential as the 1080 TI will literally wreck any game of today no problem
but the 1080 TI should be able to handle that monitor very well though to keep it at a constant 165hz at 1440p with gsync will be very hard so most likely you won't be able to max out every game and get the most out of your monitor on triple a games