Log changes to each dataset
Closed this issue · 4 comments
In the dataset, there should be a "story" of what happened: identification of allocation method because of reason X; this product was flipped because Y, application of this allocation factor, scaled, etc. This would make.
Other option, a bit more ambitious: keep in the dataset copies of itself after each transformation. The datasets might become a bit large, but there are not that many transformations. Then the dataset would become its own documentation of what has been done, and it is not necessary to store multiple verseions of the database after different steps.
Adapt also the dataset_to_excel function to output the changes, with the option to show or not the previous steps.
This will require some substantial work, but should be done sooner rather than later.
Basically, we need a separate log at a custom log level, and to capture this log output with a separate handler. But this means that we need to add this separate logging to each function... and there are a lot of functions.