TS-1232PXU-RP low sequential read speed

English Storage
1

Hi,
This is my new hardware:

TS-1232PXU-RP QTS 5.0.1.2425 Build 20230609 12 x HDD used: WD102KRYZ-01A5AB0 (10 TB each) All disks in RAID type 5 Single static Volume Single 10 GbE port used No data encryption used

My typical use scenario is mounting NAS resources on the other Linux (Ubuntu 20.04) machine:

sudo mount -t nfs 192.168.200.28:/dirname ~/qnap/dirname

and sequentially read a series of files, 1.0 GB each. That PC has Mellanox ConnectX-3 MCX311A-XCAT (10GbE SFP+) network card and is connected with NAS by CRS309-1G-8S+IN MicroTik switch. Everything worked great with the old NAS server (QNAP TVS-1282, 8 x HDD used: WD102KRYZ-01A5AB0, RAID type 5). I reached a sequential read speed about 900 MB/s witch is close enough to a network speed bottleneck.

My problem is the new server described above. I get 150-400 MB/s sequential read speed, whitch is way below my expectations (900 MB/s). I’m admin and a sole user of this NAS, so it’s surely not busy with other tasks.
I tried hints form this article:
https://www.qnap.com/en/how-to/faq/article/what-should-i-do-if-im-experiencing-slow-file-transfer-issue

  1. Check Disk raw Read Speed → I reached 250-260 MB/s on each single HDD.
  2. Examine the RAID performance → Here’s my results:
[administrator@NAS8016FE ~]$ qcli_storage -t force=1
fio test command for LV layer: /sbin/fio --filename=test_device --direct=0 --rw=read --bs=1M --runtime=15 --name=test-read --ioengine=libaio --iodepth=32 &>/tmp/qcli_storage.log
fio test command for File system: /sbin/fio --filename=test_device/qcli_storage --direct=0 --rw=read --bs=1M --runtime=15 --name=test-read --ioengine=libaio --iodepth=32 --size=128m &>/tmp/qcli_storage.log
Start testing!
Performance test is finished 100.000%...
VolID   VolName             Pool     Mapping_Name            Throughput      Mount_Path                    FS_Throughput
1       DataVol1            288      /dev/mapper/cachedev1   1.89 GB/s       /share/CACHEDEV1_DATA         [color=#FF0000]407.64 MB/s[/color]
[administrator@NAS8016FE ~]$ cat /tmp/qcli_storage.log
test-read: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=32
fio-2.2.10
Starting 1 process
test-read: Laying out IO file(s) (1 file(s) / 128MB)

test-read: (groupid=0, jobs=1): err= 0: pid=13005: Wed Jan  8 15:25:15 2025
  read : io=131072KB, bw=417427KB/s, iops=407, runt=   314msec
    slat (usec): min=249, max=35288, avg=2443.60, stdev=5885.57
    clat (usec): min=3, max=103719, avg=55024.96, stdev=30099.57
     lat (usec): min=262, max=135355, avg=57469.55, stdev=31296.17
    clat percentiles (usec):
     |  1.00th=[  274],  5.00th=[17024], 10.00th=[18560], 20.00th=[20096],
     | 30.00th=[24192], 40.00th=[51456], 50.00th=[59136], 60.00th=[69120],
     | 70.00th=[77312], 80.00th=[86528], 90.00th=[90624], 95.00th=[100864],
     | 99.00th=[102912], 99.50th=[103936], 99.90th=[103936], 99.95th=[103936],
     | 99.99th=[103936]
    lat (usec) : 4=0.78%, 500=0.78%
    lat (msec) : 20=13.28%, 50=24.22%, 100=54.69%, 250=6.25%
  cpu          : usr=0.00%, sys=12.78%, ctx=45, majf=0, minf=525
  IO depths    : 1=0.8%, 2=1.6%, 4=3.1%, 8=6.2%, 16=12.5%, 32=75.8%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.0%, 8=0.0%, 16=0.0%, 32=1.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=128/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=131072KB, aggrb=417426KB/s, minb=417426KB/s, maxb=417426KB/s, mint=314msec, maxt=314msec

Disk stats (read/write):
    dm-0: ios=92/0, merge=0/0, ticks=1060/0, in_queue=1240, util=57.85%, aggrios=256/0, aggrmerge=0/0, aggrticks=3410/0, aggrin_queue=3410, aggrutil=64.78%
    dm-1: ios=256/0, merge=0/0, ticks=3410/0, in_queue=3410, util=64.78%, aggrios=256/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    md1: ios=256/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=21/1, aggrmerge=0/0, aggrticks=284/12, aggrin_queue=296, aggrutil=46.75%
  sda: ios=20/1, merge=0/0, ticks=200/0, in_queue=200, util=30.43%
  sdb: ios=20/1, merge=0/0, ticks=200/0, in_queue=200, util=32.45%
  sdc: ios=22/1, merge=0/0, ticks=330/10, in_queue=340, util=40.57%
  sdd: ios=20/1, merge=0/0, ticks=230/0, in_queue=230, util=32.45%
  sde: ios=22/1, merge=0/0, ticks=400/10, in_queue=410, util=44.62%
  sdf: ios=22/1, merge=0/0, ticks=240/10, in_queue=250, util=36.51%
  sdg: ios=22/1, merge=0/0, ticks=220/10, in_queue=230, util=38.54%
  sdh: ios=22/1, merge=0/0, ticks=380/10, in_queue=390, util=42.60%
  sdi: ios=22/1, merge=0/0, ticks=260/20, in_queue=280, util=40.57%
  sdj: ios=20/1, merge=0/0, ticks=230/0, in_queue=230, util=36.51%
  sdk: ios=22/1, merge=0/0, ticks=420/80, in_queue=500, util=46.75%
  sdl: ios=22/1, merge=0/0, ticks=300/0, in_queue=300, util=38.62%

Measured FS_Throughput is disappointing, but consistent with measurements done by my custom program that reads NFS-mapped files. What should I do to improve this?

2

Firmware is a 1 1/2 years out of date, have you tried an update ?

3

Thank you for your reply.
Just updated it to version 5.2.3.3006 build 20250108.
Unfortunately results are nearly the same: qcli_storage tool indicates FS_Throughput 450 MB/s. My custom application indicates sequential read speed varying from 200 MB/s up to 470 MB/s.