Storage Pool / RAID / Volume / Data Recovery after QNAP Hardware Failure - A Viable Solution using NON-QNAP Hardware.

English Storage
1

Backstory

  • My QNAP TS-451A had hardware failure on bay 4. The drive in bay 4 became stale, but all drives (including 4) were operational.
  • Unwilling to get a new QNAP
  • While QNAP RAID originated with Linux, the LVM headers have been modified to the extent that stock Linux can not access them.
  • Encrypted volume recovery is all but impossible.

Foreword

  • I make no claims to be an expert in this area. Please feel free to suggest improvements.
  • The below method will likely work for a number of other circumstances with minimal adjustments.
  • The forum limited me to 2 links. Please find the rest via Google.

What is Needed - for this specific method.

  • Remaining or all members of the QNAP RAID array
  • Enough storage, on other drives, as target for RAID recovery
  • Proxmox (ideally bare metal), with required number of drive bays
  • Qnap QuTScloud subscription - start at $4.99 USD / Month
  • Patience

General Procedure

Move Physical Drives

  • Initial processing of drives removed from QNAP
    • Get the model number and size of the drives from the physical sticker on the drive
      • Taking a photo of each label can be helpful
    • Install the QNAP drives into the Proxmox machine

From Proxmox: Get Drive Info

  • install packages in Proxmox
    • apt install mdadm lvm2 thin-provisioning-tools
    • You may need more / less, this will get you started
  • Find the QNAP drives
    • Proxmox UI > Node drives are installed in (first column) > Disks (second column)
      • Look for the model number of the drives you recorded earlier
      • QNAP drives will generally have 5 partitions and partition usage will be ‘linux_raid_member’:
        • Partition 1 & 2: Small partitions for QNAP OS/Config (RAID 1).
        • Partition 3: This is usually your actual data (RAID 5).
        • Partition 4 & 5: Swap and reserved space.
        • These may be changed with other QNAP versions, the largest partition will almost certainly be the data partition.
  • Find the Target Drives (where you will export the data to)
  • Get additional information on drives. – I found these helpful
    • ls -al /dev/disk/by-id
    • lsblk
    • lsblk -af -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT,UUID,WWN,HCTL,TRAN,SUBSYSTEMS
    • lsscsi -g
    • fdisk -l

From Proxmox: Setup QuTScloud

  • Determine which QuTScloud license key is appropriate
    • QuTScloud
      • Current pricing starts at: 1 core: 4.99 USD/mo
    • You don’t have to get the license yet, but you need to know the core count to setup your Virtual Machine (VM)
  • Create a QuTScloud VM in Proxmox
    • Download QNAP QuTScloud
    • Import the image (customize the command as required)
      • qm importdisk 101 /root/QuTScloud_c5.2.4.3041.qcow2 local-data-00
      • When creating the VM shell in Proxmox
        • I had to use SeaBIOS (most docs say use UEFI)
        • Set the machine to ‘q35’
        • Set SCSI controller to VirtIO SCSI Single
        • include a Network Device (virtio) as this will be needed to validate your license.
  • From within the Proxmox UI, initialize QuTScloud:
    • Select the QuTScloud VM
    • click ‘Start’
    • click ‘console’
      • This will display the console, which after booting, will display the LAN IP, and login details.
    • Using a browser, login to QuTScloud using the IP address and provided Username and Pswd (default is ‘admin’)
  • Get a QuTScloud license key & enter into local QuTScloud.

From Proxmox Shell: Connect drives to QuTScloud

  • In my case, one of my drives was degraded (TS-451A Bay Failure). If that is your case as well, only connect the non-degraded drives - this keeps QuTScloud from having to figure it out.
  • From within Proxmox, use something like this to pass the drives to QuTScloud (for 3 drives or 4 drive RAID 5):
    • qm set 101 --scsi1 /dev/sde,backup=0
    • qm set 101 --scsi2 /dev/sdf,backup=0
    • qm set 101 --scsi3 /dev/sdg,backup=0
    • Be sure to match VM number (101 here) and device nodes (drive designators) with your actual system
    • It is better to use persistent designators here (e.g. by-id)
    • You can try to add the drives as read-only (ro=1).
      • Setting to Read-Onlly was not successful when I tried due to QuTScloud requiring write access to update drive metadata
    • IMPORTANT: Send the individual drives to QuTScloud, do not assemble the drives in Proxmox and send the RAID.

From QuTScloud: Determine QuTScloud RAID Setup path (GUI or Shell)

  • Either the GUI or Shell may work for this, I used the shell as the GUI did not find the drives.

From QuTScloud GUI: Setup SSH access to QuTScloud

  • Use the GUI to enable SSH access.
    • Control Panel > Network & File Services

From QuTScloud Shell: Get Drive Info

  • Check the drives – the following are helpful
    • sudo cat /proc/mdstat
    • sudo qcli_storage -d
    • sudo fdisk -l /dev/sdc /dev/sdd /dev/sde
    • sudo lvs
    • QuTScloud shell specific
      • qcli_storage -d

From QuTScloud Shell: Assemble RAID

  • Once you have all the device nodes, assemble the RAID with something like this:
    • sudo mdadm --assemble --force /dev/md1 /dev/sdc3 /dev/sdd3 /dev/sde3

From QuTScloud Shell: Create Mounting Point & Mount

  • sudo mkdir /mnt/recovery
  • sudo vgchange -ay vg1 (may be required here).
  • sudo mount -t ext4 /dev/mapper/vgl-lvl /mnt/recovery
    • If this does not work, try the GUI: Storage & Snapshots > Recover > Scan and Import Storage Pool

From QuTScloud GUI: Check ‘Storage & Snapshots’

  • Hopefully your volume will now be visible in the QNAP GUI > Storage Pools & Volumese

Notes

  • QuTScloud will constantly complain about the size of the OS disk.
    • This is because of the way it was created from the qcow2 image
    • This was never an issue for me
  • I was able to recover both unencrypted and encrypted volumes
    • You do need a valid encryption key for encrypted volumes
  • Do NOT attempt to rebuild the array before recovering the data. – Always: Recover > rebuild

Keywords

  • storage pool recovery
  • volume recovery
  • encrypted volume recovery
  • QNAP hardware recovery
1 Like
2

Nice job! This will come-in handy, I think. :nerd_face:

3

Thank you very much for sharing your experience!

In addition, if you require any repair services, please feel free to open a support ticket via the link below.

We may be able to provide further assistance once we review your specific case. Thank you!

Support Portal: https://service.qnap.com/