When using HBS to back up NAS data to cloud (eg. OneDrive), are there upper limits to be aware of around the number of files/directories to be backed up?
Also, does individually selecting multiple sub-directories (as opposed to selecting the parent directory) force HBS to process each selected sub-directory sequentially?
I am trying to backup a NAS directory containing thousands of files in sub-directories to OneDrive, with no version management - simply a clone of the data in case of NAS data loss. Each time I run the backup, it compares the source and destination to then replicate changes (adds, deletes, etc.) in the source to the backup copy. The comparison takes hours, and I think has resulted in the NAS memory being consumed to the point that the NAS stops processing altogether, and needs to be rebooted.
There should be no limits. If you don’t have version management selected, you are going to basically be doing a full backup of every file on the NAS each time you run the backup. HBS does not do incremental backups unless version management is selected. The replication you are saying is happening is not happening without version management turned on. Instead, it is taking so long because it is backing up every single file over and over and over each time.
Actually, I think HBS does compare files at both locations and then only transfers the needed changes. If you have thousands of files, this will still take a long time, especially over a slow connection or with limited cpu or memory.
Thank you for the replies. I can confirm that subsequent backups are not full backups all over again. In the Advanced setting section of Policies under Rules there are options to “Only back up updated files” and “Remove deleted data from the destination”
I tested this by creating smaller backup job and rerunning it immediately after the first backup finished and it took only a few minutes to run the compare and copied no files as nothing had changed.
It is the sheer number of files that I have in the source that is taking so long to compare. I was curious if this would cause excessive memory usage, to the point where the NAS pretty much stops responding.
Thanks for your question! We’ll look into this issue. Could you also let us know how many files you have specifically, the total size, and whether they’re mostly small files or large files? We’d like to get a better sense of the scale involved.
Could you also provide us with the HBS debug log? Thanks!
I have rebuilt the backup and have run it a few times after the initial full copy. The subsequent backups have run without issue - albeit with no files to copy/delete. I will monitor over the next few days and if the problem(s) return I’ll send through the debug logs.
Regarding the files themselves, they are mostly photos with some videos, and are spread across around 15,000 folders. The number of files is around 70,000 and total size around 185Gb They vary in size from typically a few hundred KB to several hundred MB.
I have encountered another more serious issue. All volumes unmounted overnight and several restarts and attempted file system checks have not had any impact. The file system check tries to start and fails immediately. The event log simply states “Failed to check file system for volume “Apps”. Storage pool: 1”. The disks themselves are all showing as healthy. Is there any other options I have to try and recover the volumes or is this terminal?
PS: The volumes are showing they still have data on them as per example attached.
