cache pmar folder's that have been downloaded
Closed this issue · 4 comments
Similar to how the basestation has a processed files cache, create a similar text doc for all the pmar folders that have been downloaded because that is a slow step in downloadBasestationFiles
@DMellinger Here is an example of an 'issue' I created for myself...
the downloading process through all the various PMAR folders is slow so I thought caching a list of what has already been done and scanning that (similar to what the glider basestation does with the processed_files.cache) might be faster. I've been working on it slowly on my 'dev' branch
Modified downloadBasestationFiles to now create, or if it already exists, read in a text file called 'downlaoded_files_cache.txt'. The download function will now compare the basestation files to the cache list; if files are NOT in the cache list, they will be downloaded, and once something is downloaded, it is appended to the end of the cache list.
The cache list file can be deleted to redownload all files, or individual lines can be deleted to redownload just those specific files.
This is used for all .log, .nc, .eng, .dat, .asc files + pm and ws files but does NOT currently work for cmdfiles or pdoscmds.bat files - those are unique because there may be multiples for a single dive.
Working for SG639/PMAR glider, still needs testing on WISPR glider.
TIMING TEST (at SG639 Dive 256, timing to download NO new files except combined .nc file)
- new approach = 49 sec
- old approach = 123 sec (yay!)
Found a bug - wasn't saving pmar files within the nexted folders but fixed with commit 705f84b
WISPR test
- new approach - 29 secs
- old approach - 27.5 secs
No significant improvement as expected because the nested pmar folders were really the slow problem