Nvidia drivers broken after updating QuTS hero h6.0.3.3670

English Container & Virtualization
1

After updating to QuTS hero h6.0.3.3670, I can’t run Plex using Container station. I’m getting this error

Failed to start application "plex". Error message: operateApp action [--project-name plex start] failed: exit status 1:  Container plex Starting Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Using requested mode 'legacy'nvidia-container-cli: detection error: open failed: /share/ZFS530_DATA/.qpkg/NVIDIA_GPU_DRV/usr/nvidia/libnvidia-tls.so.575.64.05: no such file or directory: unknown

I’ve tried to remove and reinstall NVIDIA kernel drivers (matching the OS version) and the GPU driver one without any success. Additionally, I can’t downgrade to the previous OS version as it’s blocked by the UI.

When would we expect a fix?

2

Btw, the version inside the NVIDIA_GPU_DRV is 580.159.04 which doesn’t mach with the one requested by docker (575.64.05)

3

Why do you want to run Plex in Container Station? There is a native app…

4

And why not? I really want to use always the latest app versions. Your comment doesn’t provide any helpful information or fix ….. the fact is that Nvidia Drivers have issues on the latest firmware version / app, and it should be fixed

5

There may be a problem, but you are adding complexity to something and making it more difficult. Plex always has the latest QPKGs on their website. I get notified in the app and download them.

There may be an Nvidia driver problem. But there’s a solution for you that is simpler and has no issues.

Your call though. Run something that is more difficult and complex or something easy…

6

I’m not asking for alternatives, I have my reasons to use that, this issue is also affecting other containers that depend on the gpu. Please, let’s focus on the real issue.

7

Same problem here after updating to h6.0.3.3670. I found the cause and a fix.

What happened: The update moved the NVIDIA driver to 580.159.04. The kernel module and the GPU Driver app both match, so the driver itself is fine. But Container Station keeps its own NVIDIA library cache, and that cache still listed the old 575.64.05 files, which no longer exist. That’s what causes the libnvidia-tls.so.575.64.05: no such file or directory error in the first post. Rebooting, reinstalling the drivers and running NVIDIA_GPU_DRV.sh start all left that cache unchanged, so it has to be rebuilt by hand.

Fix: Run these over SSH as an administrator. I’m just a user, not QNAP, so use at your own risk.

1. Set the paths (this works whatever your pool is called):

NV=$(getcfg NVIDIA_GPU_DRV Install_Path -f /etc/config/qpkg.conf)
CS=$(getcfg container-station Install_Path -f /etc/config/qpkg.conf)
C="$CS/opt/nvidia/etc/ld.so.cache"

2. Check it’s the same problem:

cat /proc/driver/nvidia/version     # should show 580.159.04
sudo grep -E '(root|ldcache) *=' "$CS/etc/nvidia-container-runtime/config.toml"
grep -c 575.64.05 "$C"              # anything above 0 = stale cache

The config line should show root = "/" and an ldcache line pointing at that same ld.so.cache. If the kernel module isn’t 580.159.04, you have a different problem and this won’t help.

3. Back up and rebuild the cache:

sudo cp "$C" "$C.bak"
sudo /sbin/ldconfig -X -C "$C" "$NV/usr/nvidia"
grep -c 575.64.05 "$C"; grep -c 580.159.04 "$C"   # want 0, then a number above 0

-C makes ldconfig write only to Container Station’s cache file, and -X stops it changing any symlinks, so nothing else on the system is touched.

4. Start your GPU containers from Container Station. You don’t need to restart Container Station itself. Mine came straight back up.

To undo, run sudo cp "$C.bak" "$C". A future driver or Container Station update may leave the cache stale again. If so, repeat step 3.

Credit to mjellybaby’s PaddlePaddle guide, which mentions this cache going stale after GPU driver updates. A reboot didn’t fix it for me, though: Getting GPU-Accelerated PaddlePaddle Working in Docker on QNAP NAS

For QNAP: The root cause is that Container Station’s NVIDIA ld.so.cache isn’t rebuilt when the GPU driver package is updated. Please make driver or Container Station updates regenerate it automatically.

8

Awesome! Thank your for the detailed explanation, will look into it! :heart::heart::heart: