Implement "equilibrium state" test for SA part of calibration process.
Opened this issue · 9 comments
Need to expand upon the concept that is drafted in "mads_calibration/equlibrium_check.py".
Thinking that this might get built as part of drivers/Sensitivity.py such that the "collate results" function will run the various equilibrium state checks (stability, slope, etc) and output a file equilibrium_state.csv in each sample directory. Each column of the csv file may have a boolean value for whether each of the respective tests pass. Then this csv file can be read at a later date to assess whether the sample run reached equlibrium via various metrics.
So far checks we have thought of are:
- slope of a linear regression over N years
- "stability" which is the coefficient of variation over N years
- long term stability where you look at a long time series and see if there are periods of equilibrium separated by distinct steps from one equilibrium state to another
Propose that the "equlibrium check" logic is added to the util/metrics.py module.
In general, it would be nice to build the equilibrium check into the C++ code of the dvm-dos-them. This will eliminate the extra steps that need to be taken to make sure that the model is equilibrated. In addition, this could help with the overall model -eq case when the model will check and decide when the equilibrium run needs to be terminated without the user specifying the number of years because some cases might take longer or shorter to equilibrate. When we provide a hard number, usually 2000 years, if the model is not at equilibrium, then it will continue to lose carbon during the -tr and -sc stages, which would make it harder to match observations.
@Elchin yeah, we would like to put this in the C++ code but decided that we should implement in Python first as a "post hoc" check because it will be faster for development and will allow us to experiment with different methods of testing for equilibrium state.
@tobeycarman not sure if this structurally makes sense, but in thinking about the workflow post SA, right now the equilibrium check is for all samples run from SA, but am curious if there's another way to have it work from the SA_post_hoc_analysis.n_top_runs(), which selects the samples which meet these criteria based on r2lim and have an equilibrium check work from these top samples as a way to complement or supplement the SA results? Just a thought! Have been trying to work through it but its messy
@valeriabriones, @Benjamin-Maglio, @jsclein-uaf, just pushed changes (commit: ffd82c0) implementing a draft roughly as discussed above. Details in the commit message.🤞 that it might run for you. Note that I did not push anything that actually "checks" the equilibrium - the code I pushed makes various "measurements" and summarizes the data. The "check" still needs to be written...i.e. "only use samples with slope < .1" or "only use samples with cv<.5".