I have a TS-873 and recently had a ticket open with QNAP about transcoding on the fly. I even had an Nvidia GPU installed and was having problems getting transcoding on the fly working.
In this process it was pointed out to me that my load averages were way too high (using the Linux command “top” in an ssh shell). They were in the 30s when they should be under 10. Turns out I just had too much running on my 873. Removing QVR Pro drastically reduced my usage and now hardware transcoding works great.
I’ve since removed my GPU as I wanted to use my 10Gbit fiber NIC to communicate with another NAS I recently purchased.
But I would check what load you have on your NAS. The Ryzen CPU in the 873 is a lot less powerful than a Core i5 or even a core i3. Transcoding is an intensive process.
Also Plex works great for free if you use a VPN. I use it that way all the time.