RMI-PACTA/workflow.data.preparation

save pre-flight files to a temp directory

cjyetman opened this issue · 2 comments

cjyetman commented

Currently, pre-flight files are saved to the inputs directory, before ultimately the properly formatted RDS versions of them are saved to the outputs directory. I think it might be safer to initially save the pre-flight files to a temp directory so that the inputs directory could potentially be read-only. @jdhoffa and @AlexAxthelm do you have any concerns with that?

jdhoffa commented

I think the pre-flights were (if I'm not mistaken) most useful because it was nice to have a cache for the FactSet and Scenario data.

Since it seems like we are moving towards those datasets being pulled/ stored/ versioned separately anyway (https://github.com/RMI-PACTA/workflow.factset and https://github.com/RMI-PACTA/workflow.scenario.preparation)

Since pacta.scenario.preparation offers a stable and versioned view of the scenario input datasets (and since FactSet is long gone), i think you're safe to do this now.

  • 👍1
cjyetman commented

Given the discussion beginning here, I'm closing this in favor of a strategy to eventually (optionally) copy all input files (including any "pre-flight" files) to a specific directory for archival (notably not a temp directory, and notably not a/the (possibly read-only) input directory).

  • 👍1