I’m trying to plan a 2PB+ NAS set up using QNAP rackmount hardware. I’m considering the TS-h1677AXU-RP-R7-32G, upgraded to 128GB of RAM and 2x QXP-820S-B3408 cards, along with 4x TL-R1620SEP-RP with a total of 80x Seagate 32TB SATA 6.0Gb/s 3.5" HDDs. I have two questions: 1) would this set up work and be capable of supporting the mentioned storage, and 2) would a 25Gbe adapter make any sense with this set up or is 10Gbe already enough? I’m interested in hearing any alternative recommendations that would be comparable to my requirements of 2PB+. I’m also potentially interested in the NAS having an SSD cache.
See the small print here (But I guess you are aiming for QuTS with ZFS)
https://www.qnap.com/en-ca/compatibility-expansion?expansion=tl-r1620sep-rp
The maximum size supported for a RAID group is 308 TB.
The maximum size supported for a storage pool is 308 TB. (Starting from QTS version 4.2, the minimum size of a storage pool is 144 GB.)
The maximum capacity for volumes and iSCSI LUNs depends on both the capacity of the drives and the system specifications.
- f the NAS has less than 4GB RAM, then the maximum capacity is 144 TB.
- If the NAS has at least 4GB RAM, then the maximum capacity is 250 TB.
Also with storage that large, there is no cache option on QNAP that could even come close to providing any benefit (2,3 or 4 drives of NVMe cache)
@Bob, would OP’s plan work ? (never used expansion chassis)
1 Like
I based my current thinking off of: Petabyte storage solution | QNAP
TS-h2287XU-RP has 16x 3.5” bays and 6x 2.5” bays, which I had assumed was for SSD cache. Using 15TB enterprise SSDs would be ~90TB, so I’d think cache would be possible. I’ve never worked with any of this hardware before and want to get it planned out as well as possible before ordering anything.
ZFS QuTS NAS with more than 8GB of memory use ARC cache already.
https://www.qnap.com/en/how-to/faq/article/why-is-ram-usage-so-high-in-quts-hero
Also cache is not magic, unless you have applications that read the same blocks over and over and over again, there would not be any benefits. (And with that many disks, your sequential reads would be pretty fast already)
2 Likes
Thanks for your help. Well the QNAP Petabyte Storage page says “At least 128 GB RAM is required for the NAS.“ so I’m not sure how much RAM would be left over. It’s for a big data project that would often read the same files over and over, but I don’t strictly require a cache. My main requirement is 2PB+ of storage. It can dip a little below after redundancy, but ~2PB+ gives us some room to grow.
you are thinking incorrectly. You get a QNAP TS-h2477AXU-RP. You run QuTS only. You install two M.2 NVMe drives in the TS-h2477AXU-RP as Storage Pool 1 > RAID 1 configuration for the System. Then you install Twenty Four Seagate Ironwolf Pro SATA drives. With 28 TB drives in a RAID 60 configuration (where 4 drives can fail - QNAP won’t let you do RAID 5 or RAID 6 with 24 drives) - this is 560 TB of usable storage. You can also get Seagate Ironwolf Pro or Seagate EXOS 32 TB drives now, so after RAID 60, that would be 640 TB of usable storage. OK - now you need MORE storage - so you buy the QNAP TL-R2400PES 24 bay expander. This uses the QNAP QXP-3X8PES expander. You can have EIGHT of these 24 bay expanders on the TS-h2477AXU-RP, and now with the main chassis and 3 expanders, you now have your 2 Pedabytes.
And if this is still not enough storage for you - QNAP is about to release the new 60 bay TL-R6020Sep expander, and you can have multiples of those.
For reference, you would not use a TL-R1620SEP-RP with a new TS-h1677AXU-RP - you could use the TL-R2400PES if you wanted a 24 bay expander on this, but for a 16 bay expander you would use the TL-R1600PES-RP, and the same expander card I just mentioned. -
Got more questions ? Ask away. I do giant systems like this all the time.
Bob Zelin
edit - you have not stated your application. So should you have a 10G user interface to your switch or 25G ? What are you doing ? Video editing ? Insurance company? Hospital ? For Video editing, I would put in the QNAP 25G CX6 card, and plug into the new wonderful QNAP QSW-M7230 switch that has 24 10G copper ethenet ports, four 25G SFP28 ports, and 2 100G QSFP ports.
2 Likes
I guess you mean this:
ARC has a max usage of 70% (set in global storage settings) and I think ZFS/QuTS snapshots have no influence in terms of RAM usage, at least according to this page.
So you might have 35ish GB left over (above page mentions 265GB min for 3+ PB though.)
1 Like
Thanks @Bob. Our use case is a big data one, where petabytes of files, typically ~1MB to ~250MB each get worked with on an ongoing basis. We already have a 100Gbe network for some other hardware, so I’ll probably go with 25Gbe (I’m assuming 40Gbe or 100Gbe would have no benefit here since they’re HDDs?).
So just to check my understanding, using the QNAP TS-h2477AXU-RP with one QNAP QXP-3X8PES and three QNAP TL-R2400PES would be what you’re suggesting, with one port on the QNAP QXP-3X8PES each going to a single QNAP TL-R2400PES and then the third QNAP TL-R2400PES being daisy chained to one of the first two QNAP TL-R2400PES?
The QNAP QXP-3X8PES would take up PCI slot 2 (with slot 1 empty so slot 2 gets the x8 bandwidth) and a 25Gbe card could go in the slot 3 x4 PCI?
Also, could the QNAP TS-h2477AXU-RP really support 8 24-bay expansions with only 192GB of RAM? I’m now considering doing 4x of the QNAP TL-R2400PES, but might that need 256GB of RAM in the NAS?