I have spent the last 2 weeks trying to build a system for a friend with a recording studio. Audio was just fine, but since he has a fairly high-end older NVIDIA card he wanted to have his videos encoded with HEVC. HEVC is rather CPU intensive unless you go with the lower resolutions, so I decided to get NVENC working. I followed all the recipes I could find, and if NVENC worked in one utility, it failed in another. I followed the rules for building FFMPEG for Ubuntu from the FFMPEG site, and at first we got errors because the instructions only build static libraries. I altered and configurations to build shared libraries, and again, one or two applications might be able to use them while another would not.
Then I found this guide from NVIDIA's website: developer.download.nvidia.com/compute/redist/ffmpeg/1511-patch/FFMPEG-with-NVIDIA-Acceleration-on-Ubuntu_UG_v01.pdf
Follow the instructions, but, when configuring FFMPEG, add
to the ./configure command and compile. Make and install as instructed by the guide and it should work.
Caveat: AVIDEMUX, a quick linear editor, gives an error "bitrate too low" when using NVENC (x265), but it works fine in others. I wish I had found these instructions 2 weeks ago. Nothing else worked.
This is only tested on an NVIDIA card; you would have to disregard the CUDA installation if you want to try to compile FFMPEG from these instructions.
Good luck!