Disk performance extremely slow after raid 5 disk failure rebuild

Have a QNAP TS-h886 with Quts Hero h5.2.9.3410

Originally - disk 1 to 5 were in a raid 5 set with WD 16 TB Red Pro NAS drives

Disk 1 recently failed, and my hot spare in disk 6 tookover - rebuild took about 10 days, everything shows green.

Didn’t notice anything until I started to shutdown my VMs in VMware and they were extremely slow in powering down (used to be instant). I went ahead and shutdown all my VMs, and rebooted the NAS. Everything comes up fine after reboot, ESXi hosts come back and see the disks. Everything is connected via 10GB and multi-honed into the NAS.

Ran a quick CrystalDiskMark test and I’m only seeing 369 MB/s reads (used to be over 1500 MB/s). All disk activity just seems slow. Already opened up a ticket but wanted to check here to see if anyone has experienced this and if they’ve addressed it. Thanks!

Networking on the NAS is all 10GbE talking to each ESXi host via iSCSI multhipathing

iSCSI luns are connected through both 10GbE interfaces using multhi-pathd active/active

A rebuild of 10 days seems very slow, please do a sequential read benchmark of all drives

There is a cache drive that was not mentioned, please disable cache and test again

While QuTS cache is supposedly not busted like QTS cache, it’s worth to test it anyways

2 Likes

I’ve never had an issue with the cache enabled all these years … don’t want to make too many changes at the moment either, only culprit so far was the data being rebuilt to the spare on slot 6

Well if you don’t want to troubleshoot, then I cannot help you

Holy moly, why the heck is cache slowing this down???

Would cache affect a raid rebuild as well?

I just did a google search if ZIL/L2ARC is slowing down re-silvering, but it was sort of inconclusive.

Maybe open a ticket, if this is indeed an issue, QNAP could change the OS routines to switch off ZIL during a rebuild and only rebuild ZIL/L2ARC, it once it’s completed.

1 Like

Yep got a ticket open too, will definitely keep cache turned off from now on - thanks a bunch!

Here’s my raid rebuild… it was less than 9 days, but was still surprised it took so long

I always had issues with cache on my NAS with quts. I finally gave up on using it. I also was using WD RED m.2 drives so wasnt like it was consumer hardware.

1 Like

IMO it seems to have gone downhill once they released h5.2.x … with h5.1.x I was seeing over 2000 MB/s on reads easily

yeah that is about when i gave up on it