Was the file, you tested with, an 8bit or 10bit file?
On a side note, I used my i7 770k pc as plex server. I was unable to stream 4K content to my TV. Also buffering the whole time. So, did some research and ended up getting a HTPC with a GT 1030 in, because I needed 4K HVEC 10bit decoding. My setup is now perfect, using a Dell 3040 with I3 6100, 8GB ram and a GT 1030. I have tested numorous media players and they all failed at 4K.
So, first of all, I would say the issue is with your TV and Xbox, but IIRC the Xbox should be able to do HVEC decoding.