Replace weekly files with yearly file once available (check in early 2021)
andybega opened this issue · 1 comments
This is a pre-emptive issue, it may turn out to not be an actual problem down the road.
Writing this on 2020-05-22. Currently there are, for 2020:
- an annual file,
events.2020.20200506093336.tab
, with events for the first part of 2020 - weekly update files, named like
20200505-icews-events.tab
It seems possible that in early 2021, the events.2020...
file will be updated to contain all 2020 events, and thus the weekly 2020 files will become superseded. What should properly happen on update is:
- Regarding the files, the weekly files should be removed; the 2020 annual file updated
- In the DB: events from the weekly files should be removed, and events from the 2020 annual file ingested. Those should in theory be the same set of events, so technically it is not strictly necessary to update, but for the sake of simplicity do it anyways.
If the weekly files are removed from dataverse, then the update might work correctly.
This came up during the pause and subsequent resumption of ICEWS between 2019 and 2020. Aside from the daily to weekly update file change, there were a bunch of non-yearly files for 2018 and 2019 that after the updating resumed again were replaced by annual 2018 and 2019 files. This caused problems in updating, and also those non-annual files would locally not have been replaced by the annual files.
This more or less seems to work. Didn't even notice when the 2020 data were replaced by the annual file. Of course a little hiccup because of the unusual monthly file for January 2021 this time around, but otherwise the functionality for handling this seems to work. Closing.