Automatic single-parameter sensitivity analysis
EwoutH opened this issue · 2 comments
Automatic, one-line single-parameter sensitivity analysis (a.k.a. univariate or "one-factor-at-a-time" (OFAT) sensitivity analysis) would be a great addition for exploratory model testing and model validation. The idea is that we add a function to the EMAworkbench which can fully test sensitivities in a model with a single line of code. The function would work like this:
- Request all variables and default values from model
- Request all reporters (KPIs) from model
- For each variables, start runs with -X% and +X% in that variable for the number of replications
- Log the results
- Normalize if specified
- Return graphs and DataFrame as specified
The user can specify:
- The number of replications
- The number of time steps (= the measurement time)
- The amount to vary (for example 5%, 10% or 20%)
- The default scenario (values for each variable)
- The variables to test for sensitivity (default=all)
- The KPIs (reporters) to test against (default=all)
- Normalization (none, normalize to base, normalize to base and calculate ratio)
- The desired output data (DataFrame, graph, or both)
The output graph could like this:
No normalization (absolute)
Normalization (relative to base case)
I think the function could be called univariate_sensitivity() and a function call could look like this:
dataframes, graphs = univariate_sensitivity(model=Model, time_steps=60, replications=6, amount_to_vary=0.20)
I have a lot of code already from a recent agent-based modelling course, mainly run_experiments.py and process_experiments.ipynb
Open to feedback on how to improve this!
Edit: Maybe we can make it a Class, which has functions built-in to normalize and graph. That way you only have to run the runs once, and then can use the Class functions to get data and graphs from it. That would be a two-line solution but is more scalable and robust.
Saltelli et al. (https://www.sciencedirect.com/science/article/pii/S1364815218302822) have strong opinions about one-factor-at-a-time sensitivity analysis, and claim it is only adequate if the model is provably linear. Just some food for thought.
Personally, I would love to see a smoother integration of the "bubble plot" introduced in https://pynetlogo.readthedocs.io/en/latest/_docs/SALib_ipyparallel.html with EMA and SALib, but that's a personal wish :)