Transcoding (in our application) is the process of taking a video file and transforming that file to different resolutions and quality levels (colors etc) to make it more playable on certain devices. A common use case is to transcode a 1080p 20mbps file down to 1080p 8mbps to make it play better on a poor connection external to your network. Other things that are related are audio transcoding. An example would be going from 7.1 surround sound capable file to 2.1 for a common soundbar + subwoofer setup. Subtitle burn-in is also related, as I believe it re-encodes the video with the subtitles burned into the image for a cleaner look during playback.
For optimal playback with Plex, direct play everything you can. If you want to watch 4k movies that are HDR, make sure your playback device and TV support 4k HDR. 1080p transcoding isn't as difficult and most computers can do it fairly easily. Intel processors with the quicksync feature perform very well when transcoding plex content.