Single Status Update
I compiled a custom version of ffmpeg with Netflix's VMAF enabled and I've spent all day today trying to figure out the best settings to use when transcoding my H.264-files to HEVC using NVENC. VMAF is a tool for assessing quality of transcoded video when compared to the original file. It's quite neat. It's also horribly, horribly slow.
I am only interested in constant quality, ie. constqp in ffmpeg-parlance, and I have no interest in playing around with CBR or VBR. With that in mind, I've come to the following conclusions:
- preset: No difference between slow and hq, doesn't matter which one I use.
- nonref_p: No difference whatsoever, didn't change VMAF-score in any way.
- weighted_pred: Enabling does improve quality and seemingly also reduced output's filesize slightly.
- rc-lookahead: Enabling does improve quality, but a value of 4 produced the exact same quality as 32, so it doesn't look like there's any point in going past 4.
- spatial_aq: Enabling does improve quality, with a caveat!
- aq_strength: The higher the value, the larger the output-file and quality, but the filesize grows disproportionally when compared to the improvement in VMAF-score. In fact, in my testing it was better to use aq_strength of 1 and go for lower qp as I can get a similarly-sized file with much higher PSNR, SSIM and VMAF that way!
- profile: Couldn't test, output is somehow broken. Causes VLC to crash if using DXVA to decode and messed-up image using D3D-decoding. No idea if the issue is with ffmpeg or NVIDIA's drivers, but the result is the same whether I am encoding under Windows or Linux and whether I am using self-compiled ffmpeg or someone else's build.
- tier: No effect.
With the above in mind, my ffmpeg-command looks as follows at the moment:
ffmpeg -hwaccel nvdec -c:v h264_cuvid -hwaccel_output_format cuda -i input.file -c:v hevc_nvenc -preset:v slow -rc:v constqp -qp:v 26 -profile:v main -tier:v main -spatial_aq:v 1 -aq-strength:v 1 -rc-lookahead:v 4 -weighted_pred 1 -c:a copy output.file
Disclaimer: this is on a Pascal GPU. The results might look different on e.g. Turing. Ain't got no Turing, though, so can't test. Also, I ain't no ffmpeg- or video-encoding guru, so these may not be the actual bestest of best settings; these are only the best I've been able to come up with so far.
EDIT: Oof. Apparently my test-clip wasn't long enough as when I tested with a full-length video, weighted_pred caused my GPU to crash! Seems to only work on short clips, which sucks!