453D and ZFS / QuTS - seeking data degradation strategy

So QuMagie takes your photos and sorts them in the interface by date.

Folders are set up/configured as folders containing image data in Multimedia Console:

Albums is maybe what you would be looking to use. There seems to be a good bit of flexibility with them.

1 Like

Thanks for the extra effort to assist. I wasn’t quite sure how QuMagie prefers photo image folders because the dates of the photos can be unclear. For example, sometimes photos have EXIF data and at others they don’t. Same goes for correct create dates, which you have to hope is embedded into the file name if all the others are absent. I recall that QuMagie can also use date based hierarchies, e.g.

photos/2010/01/John Doe Birthday
photos/2010/01/Richard Roe Anniversary Party
photos/2010/02/Grateful Dead Concert

so that photos in these buckets are identified as being within January 2010 and February 2010 if all else fails. This may be too much of an effort as so many sets span several days and can go across months, etc. I may just stick with year and events and only split up those that cross the year mark.

I wrote QNAP about this and got a ridiculously great email back with setup recommendations and references to the website so it looks like this shouldn’t be too great of a problem, including confirmation of using the single pool and RAID 5 setup which seems most popular and logical if using RAID. The other aspect I didn’t consider was the storage of keywords and AI assisted search, which I’m told is stored in the NVME cache as that would make the most sense for quick retreival and search to identify images. Glad I maxed out at the time when all of this was affordable.

Thanks again and very much appreciated.

Backups won’t help on bit rots. Without file checksums, you can’t tell which copy is valid unless one of them just doesn’t load. ZFS has data checksum and refuses to load the blocks when checksums mismatch, which at least reminds you to check your backup when bit rot happens. Many multmedia files still can be loaded after bit rot, unless the rot is on the metadata part.

I once got a brand new QNAP with memory issue. Using QTS, it just runs and crashes randomly. After switching to QuTS Hero, the checksum mimsmatch blocked me opening a file on my RAID 1, indicating that a same error occured on both disks and reminding me to check its memory.

QTS does have scrubbing but that’s on block device level rather than filesystem level (and the behavior when a mismatch is found is unknown). Also QTS doesn’t support scrubbing on RAID 1.

2 Likes

I’ve seen issues with QuMagie getting old pics assigned with new dates. I’m not sure what it was about these handful of photos (bad EXIF data so it used the date when the file was copied over to the NAS?). But everything else I have going back over 20 years is great.

If you back up a file that has bit rot you are correct. But the idea in my mind was that an archived file that was backed up before the bit rot on another piece of media is likely still good.

But I get your point. It is a good one…

Thank you for your post and agreed. The assumption is that I will have assembled a set that is complete from all my files and validated.

I’m considering creating a crc32 or md5 hash for each file or kept as mentioned with a catalogued checksum that will identify when a file has been corrupted. I’ve done this for other media, such as a video library, and keep 2 additional backups in addition to the live set, which rarely changes and usually only has new files added.

Bit rot rarely happens but bad sectors do happen as do bad writes on occasion (such as when there are unexpected power outages) and that is where it gets problematic. For backups, many have suggested using WinRAR for datasets and having them as both with built in hashing to validate and also for rebuilding due to corruption. The alternative is to use ZIP with PAR files. While it is a laborious process, if I’ve got 500 folders of photos and videos from different years, it might be a feasible project to do over several months time.

RAID 1 is never a proper backup solution IMHO. The moment one copy fails so will the other. What you describe on QTS is another very good point and hence I’m left with the above types of validation and recovery approaches.