Keeping updated
Opened this issue · 3 comments
Do you plan to keep this repo up to date with the sourceforge source? If so, will that be automated?
@jcuenod that is my goal, but right now I am doing so manually. The repo doesn't appear to be updated frequently.
I've also contemplated recursively expanding the zip files so the raw data are accessible from this repo.
I mainly just wanted to mirror the data should the repo somehow become defunct. Aside from writing a local script that runs on a recurring basis to check last update time and pull down the data when it changes, I'm not really sure how to go about this. I don't have a server I can run this on freely. I'm open to any ideas you have!
Ja, maybe using a github actions cron job you could schedule a polling system to do it. I thought about this a while ago but just didn't feel like the effort was worth it :)
@jcuenod it looks possible, but I don't know how to use GitHub Actions yet, and the script I'm currently using to download the data would have to be significantly improved to only check for updated files and selectively download them. For now I'll likely just manually pull them. I'm hoping to back some additional data up but have been having trouble scraping it (the sites no longer exist so I'm using web.archive.org but the pages seem to error/freeze or no longer have the data in the snapshots I've reviewed so far.
With that said, I'm going to leave this issue open for being addressed in the future.