That server monitor is impressive. You should bottle that and sell it! 
Or at least make it available…
Na9d,
Thanks for the compliment, but it’s tough to offer as a standalone, for a couple of reasons.
First, a lot of it exists purely to work around QNAP structural issues. The busybox that ships with the QNAP OS is ancient and can’t be modified or updated, so I had to spin up a container holding my own modern toolset and write everything against that instead of the stock tools.
But because of how QNAP obsessively protects the default configuration nothing you set up in a running shell survives a reboot or firmware update. Launched daemons, all of it is per-session. So an autorun.sh script has to rebuild the whole environment at startup; relaunch the monitoring daemons, wire up the toolbox, and reinstall the cron entries, all because QNAP also overwrites the live crontab on reboots, including crash reboots, so persistent entries have to live in the right config file and get put back by the startup script. And after every firmware update the autorun hook itself has to be re-verified, since updates can reset it.
So a standalone version would mean asking people to modify autorun and cron on their own boxes Both of these offer opportunities for potentially destructive changes if fat-fingered. There’s a lot I love about QNAP, but the limits they impose to protect users from catastrophic config changes mean the workarounds are inherently invasive.
Could I package it? Probably. But then I’m on the hook the first time someone breaks their NAS or loses data and wants someone to blame.
Just thought I’d mention I installed Rel 6 on both my TS-h1290FX and TS-j1277AFX and both installed fine with no problems. However, there was a long wait for the 1277 to re-boot after the install, but all is well. I delayed the upgrade for over a week
I don’t think it’s beta, as there is no beta tags, nor even a RC (release candidate) tag. It looks like a full production release of QuTS hero 6 to me.
Ha! Good catch. I’m so used to the h6.x line being in beta I didn’t even notice 3550 was a real release.
Though some might argue it actually IS still beta and the tag was just missed on the release. 
In all reality you have to start somewhere and go live, or it forever lives in beta/RC.
Hello,
During this weekend, I completely reset my NAS, using the latest firmware: h6.0.3564 Build 20260723.
During the installation before the reset, the new nvkernerdriver drivers installed themselves.
Following the reset, it was impossible to restart a model on Ollama; in the end, it worked after a reboot (following the installation of the Nvidia drivers). Despite everything, this morning, it was again impossible to launch a model on the GPU.
Like this weekend, I had the presence of these errors in “dmesg”:
[59082.857111] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[59082.871729] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[59083.013898] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[59083.028519] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[59083.568050] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[59083.582684] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[59083.638931] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[59083.653570] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[59084.839297] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[59084.853921] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
A restart of the Ollama container was not enough to find a functional card. I had to restart the NAS.
Since the reset, I have finally been able to properly install a VM, without any error in the virtual switch.
But this undeniably leads to a waste of time, and backups must be managed properly.
During my research this weekend to get my graphics card to work, I came across this post; I have a limited desire to follow its recommendations: Getting GPU-Accelerated PaddlePaddle Working in Docker on QNAP NAS
Unless QNAP improves the sitation
@achimede333 — that dmesg output identifies your problem. It is not the drivers, not Ollama, and not your VRAM. Stop chasing those.
Before you read further
I am a fellow QNAP user, not a QNAP employee, and I have no connection to QNAP support. Everything below is offered in good faith from my own experience on similar hardware, with no warranty of any kind.
You are responsible for your own NAS. Please read and understand every command before you run it, and make sure you have a current, verified backup of anything you care about first.
To be clear about what carries risk:
- The diagnostic commands (
df,free,du,dmesg,uptime) are read-only and do not change anything.- The remount and the autorun edit are system-level changes. A mistyped
mountcommand or a broken startup script can leave a NAS in a state that needs recovery.I accept no responsibility or liability for any damage to your hardware, data, or configuration arising from following this advice. If you are not comfortable with that, open a ticket with QNAP support instead — and given what the logs show, that may be the better call anyway.
What the error actually says
faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer
nvCheckOkFailedNoLog: Out of memory [NV_ERR_NO_MEMORY]
The NVIDIA kernel driver is trying to allocate the client shadow fault buffer for MMU non-replayable faults. That buffer lives in system memory, not in VRAM. Registering it is part of setting up a CUDA/UVM context. When that allocation fails, no CUDA context can be created — so when Ollama starts and probes for a GPU, it finds nothing usable and falls back to the CPU.
This explains everything you have observed:
- Your 17GB quantized models on a 24GB RTX PRO 4000 were never the issue. This is host RAM, not VRAM.
- Rolling Ollama back to 0.30.8 could not have helped. The failure is below Ollama entirely.
- Reinstalling the drivers could not have helped. The drivers are loading correctly.
nvidia-smikept working throughout, because it uses the management library and does not need a CUDA context. This is the most misleading symptom on QNAP.
The timestamp is the key evidence
Your dmesg lines are at 59082 seconds — 16 hours and 24 minutes of uptime. Combine that with your own observation that restarting the Ollama container was not enough and you had to reboot the NAS, and the pattern is unmistakable: it works from a fresh boot, it dies after hours of uptime, and only a reboot brings it back.
That is memory being exhausted over time. No configuration error behaves that way, which is why your complete system reset did not fix it and was never going to.
Confirm it in 30 seconds
These are read-only. Run them right now, while it is broken:
df -h /
free -h
Then after your next reboot, run the same two commands and compare. If root is much larger in the broken state than it is after a reboot, you have your answer and you can stop guessing.
The fix is one line from that guide, not the whole guide
I understand your reluctance about the PaddlePaddle post — most of it is a container build that has nothing to do with you. You do not need the Dockerfile, the pip pins, or any of it. You need this one thing:
mount -o remount,size=800M /
QNAP’s root filesystem is a ramdisk with a small default cap. That single change gives the GPU driver the headroom it needs to allocate its fault buffer.
Test it in the broken state — if a model then loads onto the GPU without a reboot, you have confirmed both the diagnosis and the fix in one step. If it does not help, you have learned something useful and lost nothing, since the setting reverts on the next reboot anyway.
Making it survive a reboot — and here I would not follow the guide
The remount is not persistent. mjellybaby puts it in the NvKernelDriver QPKG start script, which works, but that script is replaced every time you update the driver QPKG — and you have already updated drivers several times through this.
I would put it in autorun.sh instead. That survives QPKG updates. It still needs re-verifying after a firmware update, but that is one check instead of a silent regression every time you touch the drivers.
Be aware that enabling and editing autorun is itself a change worth understanding before you make it — a broken autorun script runs on every boot.
What is actually filling your ramdisk
Worth finding out, because expanding the cap treats the symptom rather than the cause.
Note that the build you are running, h6.0.1.3564, contains this in its release notes: “Fixed an issue where improper API handling would cause abnormal log writing and lead to ramdisk errors.” So QNAP already knows ramdisk exhaustion is a real defect on this firmware line. Either that fix is incomplete, or something else on your system is consuming it.
To find the consumer, in the broken state (read-only):
du -xk -d1 / 2>/dev/null | sort -n | tail -20
The -x keeps it on the root filesystem so it does not wander into your storage pools.
A reference point
One request: whatever the outcome, post your df -h / from before and after. If a ramdisk that fills over uptime is causing GPU failures on h6.0.1, that is worth QNAP seeing in this thread.
Thank you for your great response.
Granted, I may not be looking in the right place to find the solution.
I just tested the orders:
- “free -h”, but the option -h does not exist in base
- “from -xk -dl”, but the “-dl” options don’t exist.
Do you use Busybox or an installation of tools, such as those via Entware?
Do you have the same configuration as mine, aren’t you experiencing the same problem?
I wasn’t reluctant to follow your advice and the ones the person posted on Paddle’s thread;).
Actually, I was saying that this thread of discussion was interesting and highlighting a structure specific to QNAP that is not “obvious.”
Best regards
achimede333,
Sorry mate that one’s on me. I load an entire toolbox into a container on every boot cycle and put it on the path, so I forget that not everyone has the same tools available. The commands I gave you assume a modern busybox. Yours is the stock QNAP one, which is a good deal older.
Stock-busybox equivalents
freewith no flags works on every busybox (output is in KB). If even that gives you trouble,head -3 /proc/meminfois bulletproof — it reads the kernel directly and has no options to get wrong.du -xks /*gets you one level deep without needing-dat all.
Those should give you the same information from the stock QNAP busybox “may its ancient, crippled arse burn in hell.” This is one of my long-standing gripes with QNAP and their stranglehold on the OS environment. The shipped busybox can’t be updated or replaced, so you end up either working around it or, as I did, carrying your own tools in a container.
What I’d like to see
dmesg | grep -i NVRM
df -k /
uptime
head -3 /proc/meminfo
Note I’ve used df -k rather than df -h — same reason free -h failed on your box. -k is the default on busybox anyway, so it will work regardless of version. Sizes come back in KB.
Why I’m asking about RAM
I’m used to my own system, which has 192GB installed. If yours has significantly less, I want to rule out the root ramdisk being sized as a proportion of available memory rather than as a fixed cap.
If that turns out to be the case, it would explain a great deal: the same firmware and the same GPU family could behave completely differently on two machines purely because one has more headroom before the NVIDIA driver runs out of room to allocate its fault buffer. That would make this a memory-capacity-dependent problem rather than a configuration problem — which is exactly the sort of thing that stays invisible until someone with a smaller machine hits it.
The df -k / and /proc/meminfo output together will tell us which it is.
Thank you for your help!
The defect was present before the model loading. Here is the result before and after the model loading, in the… CPU 
Before the load :
]$ dmesg | grep -i NVRM
[ 284.005891] NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64 575.64.05 Release Build (dvs-builder@U22-A23-13-1) Fri Jul 18 16:00:10 UTC 2025
[27877.210663] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[27877.225294] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[27877.601371] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[27877.616030] NVRM: sysmemConstruct_IMPL: *** Cannot allocate sysmem through fb heap
[27877.623639] NVRM: nvAssertOkFailedNoLog: Assertion failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from pRmApi->Alloc(pRmApi, device->session->handle, isSystemMemory ? device->handle : device->subhandle, &physHandle, isSystemMemory ? NV01_MEMORY_SYSTEM : NV01_MEMORY_LOCAL_USER, &memAllocParams, sizeof(memAllocParams)) @ nv_gpu_ops.c:4647
[27878.531209] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[27878.545826] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[27878.601580] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[27878.616207] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[27880.102423] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[27880.117054] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[27900.022239] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[27900.036867] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[28156.315466] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[28156.330119] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[28156.551884] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[28156.566518] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[28157.186096] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[28157.200726] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[28157.258996] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[28157.273628] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[28158.257236] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[28158.271863] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[28182.958643] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[28182.973267] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
]$ df -k /
Filesystem 1K-blocks Used Available Use% Mounted on
none 473088 390860 82228 83% /
]$ uptime
23:37:41 up 8:18, load average: 21.30, 23.85, 26.89
]$ head -3 /proc/meminfo
MemTotal: 32328040 kB
MemFree: 562976 kB
MemAvailable: 357008 kB
After :
]$ dmesg | grep -i NVRM
[ 284.005891] NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64 575.64.05 Release Build (dvs-builder@U22-A23-13-1) Fri Jul 18 16:00:10 UTC 2025
[27877.210663] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[27877.225294] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[27877.601371] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[27877.616030] NVRM: sysmemConstruct_IMPL: *** Cannot allocate sysmem through fb heap
[27877.623639] NVRM: nvAssertOkFailedNoLog: Assertion failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from pRmApi->Alloc(pRmApi, device->session->handle, isSystemMemory ? device->handle : device->subhandle, &physHandle, isSystemMemory ? NV01_MEMORY_SYSTEM : NV01_MEMORY_LOCAL_USER, &memAllocParams, sizeof(memAllocParams)) @ nv_gpu_ops.c:4647
[27878.531209] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[27878.545826] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[27878.601580] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[27878.616207] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[27880.102423] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[27880.117054] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[27900.022239] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[27900.036867] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[28156.315466] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[28156.330119] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[28156.551884] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[28156.566518] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[28157.186096] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[28157.200726] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[28157.258996] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[28157.273628] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[28158.257236] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[28158.271863] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[28182.958643] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[28182.973267] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
]$ df -k /
Filesystem 1K-blocks Used Available Use% Mounted on
none 473088 390860 82228 83% /
]$ uptime
23:50:18 up 8:31, load average: 33.62, 35.05, 32.14
]$ head -3 /proc/meminfo
MemTotal: 32328040 kB
MemFree: 6922760 kB
MemAvailable: 6712488 kB
I unloaded model and execute :
mount -o remount,size=800M /
I reloaded the model again in Ollama, without any change.
]$ dmesg | grep -i NVRM
[ 284.005891] NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64 575.64.05 Release Build (dvs-builder@U22-A23-13-1) Fri Jul 18 16:00:10 UTC 2025
[27877.210663] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[27877.225294] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[27877.601371] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[27877.616030] NVRM: sysmemConstruct_IMPL: *** Cannot allocate sysmem through fb heap
[27877.623639] NVRM: nvAssertOkFailedNoLog: Assertion failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from pRmApi->Alloc(pRmApi, device->session->handle, isSystemMemory ? device->handle : device->subhandle, &physHandle, isSystemMemory ? NV01_MEMORY_SYSTEM : NV01_MEMORY_LOCAL_USER, &memAllocParams, sizeof(memAllocParams)) @ nv_gpu_ops.c:4647
[27878.531209] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[27878.545826] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[27878.601580] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[27878.616207] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[27880.102423] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[27880.117054] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[27900.022239] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[27900.036867] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[28156.315466] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[28156.330119] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[28156.551884] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[28156.566518] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[28157.186096] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[28157.200726] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[28157.258996] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[28157.273628] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[28158.257236] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[28158.271863] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[28182.958643] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[28182.973267] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[31392.811153] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[31392.825783] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[31393.035352] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[31393.049973] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[31395.347200] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[31395.361828] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[31395.438280] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[31395.452912] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
[31403.016203] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353
[31403.030848] NVRM: faultbufCtrlCmdMmuFaultBufferRegisterNonReplayBuf_IMPL: Error allocating client shadow fault buffer for non-replayable faults
]$ df -k /
Filesystem 1K-blocks Used Available Use% Mounted on
none 819200 390860 428340 48% /
]$ uptime
00:21:08 up 9:02, load average: 28.42, 28.40, 31.70
]$ head -3 /proc/meminfo
MemTotal: 32328040 kB
MemFree: 6029652 kB
MemAvailable: 5823852 kB
Well, it seems my guess was wrong.
The remount did exactly what it was supposed to: cap 462 MiB → 800 MiB, 83% → 48%, 418 MiB free. And the NVRM errors kept accumulating as seen in the burst at uptime 8:43 (t=31392–31403), after the 8:31 sample.
What is obvious from these traces is that your system RAM situation is dire. MemTotal 30.8 GiB, MemAvailable 349 MiB at the first sample. That’s 1.1% available on a 48 GB machine after ZFS eats into your memory. The 381 MiB ramdisk is a rounding error next to that. I had a bunch of early problems on my TVS-AIH1688ATX due to low memory and finally gave in and bought more. QNAP support, to their credit, never said a word about the additional memory and continued to provide spectacular support. I can’t say it will resolve your problems; I’m just sharing my personal experience.
I think it’s time to open a helpdesk ticket and get tech support assistance on this.
Good luck and keep us posted!
The memory information that @achimede333 is reporting seems pretty normal. My TVS-672XT has 32 GB of memory and running head 03 /proc/meminfo gives me roughly the same amount of memory.
Now what is interesting in Resource Monitor, it tells a different story:
Shows I have 11 GB free not 6.
I’m only using about 4 GB of swap space.
I am guessing the moral of the story here is that if you want to run large language models on your NAS you need a LOT of memory.
QuTShero have a problem with the ZFS ARC.
Hello to both of you,
In my previous message, the NVIDIA drivers were already incorrect before using a model on Ollama.
Since I reset, I have been restoring files and reconfiguring, which is a lot of work for me.
This can have an influence and is where monitoring with supervision could facilitate this.
I actually have 32 GB of RAM since the purchase, and given the price right now, I can’t buy any more:P
Before the rest, I had already received messages during the night telling me that the NAS was out of memory, without knowing the reason (without using Ollama, for example, and I know it comes from somewhere else, but I wanted to clarify).
Afterwards, we find some error (“[NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1353…”) traces on the internet because of the NVIDIA drivers themselves, but indeed you didn’t seem to encounter this problem with your system that has more RAM.
To be continued, but as for the supervisions, for now, I don’t think I’ll have time to discuss this topic (one day certainly).
you can lower the maximum ARC RAM usage. With 50% i have no problems
Why is RAM usage so high in QuTS hero? | QNAP
Thank you, I can test that.
However, I discovered these messages after dark. It was certainly a maintenance/nocturnal operation that led to this.
Edit :
After checking, I might find some bars that aren’t “very” (that’s relative :)) expensive.
@kzboray What did you put on your side?
I found 3x"KSM56E46BD8KM‐48HM" (48Go, 5600, ECC), but I need to check the seller.
Or 3x"KSM48E40BD8KM-32HM", similar of my current “TS4GLA72V8E” (DDR5, 4800, ECC).
The “KSM56E46BD8KM‐48HM” it’s better, but the seller is foreign (what about the warranty) and is moderately well rated on Amazon.
I ordered these from Newegg.com. They came with a lifetime warranty which I liked. And I didn’t want to mix vendors and memory timings, so I removed the stock UDIMM and installed two new sticks.
I first ordered one set and verified they worked and would pass POST on the NAS. They did, so I ordered a second set for the full 192GB.The first time booting after I installed the first two sticks, the NAS took about 5 minutes to go through POST performing memory validation.
It looked interesting, especially before the price rise…
This brand is really for the American brand.
Of the ECC on-die, I imagine.
In Europe, there seems to be PHS-Memory, but I only know the big brands (Kingston, Corsair, Crucial, HPE, Samsung, etc.).
The memory I received from NEMIX identified as Micron memory, and it is ECC on-die. I can’t say that they only ship Micron, but both sets of two sticks that I purchased a month apart were identical Micron UDIMMS.