REF: Optimize directory discovery functions
Opened this issue · 1 comments
Since we started gathering more details (from stat on the directories) I've noticed that the application is all of the sudden very slow even on small differential backups since it needs to process so many directories for metadata....when we eventually may have to also stat files, this will quickly get out of control. I need to find a way to batch process the jobs or something in order to reduce the amount of time it takes to perform the preliminary directory/data discovery phase.
_getDirectoryData - duplicates lstat on _generateBackupTreeFromRoot possibly causing delays?
Will have to look at changes introduced to the discovery phase in #43
Due to the cause of the slowdown being directory and file stat, it would seem that the problem is much less severe when dealing with small (non-complex) directory structures.
For backups with large files and simple directory structures the problem is almost non-existent and the time to create a backup is therefore within reason.