Look at normalising error by active agents
ksuchak1990 opened this issue · 8 comments
Is there a case to be made for an error metric that normalises by the number of active agents instead of the number of agents in the population? Try implementing this.
- When we say the number of active agents, do we mean the number of active agents in the ground truth model?
- The number of active agents in one of the ensemble member models?
- The average number of active agents in all of the ensemble models? (average: mean? median? mode?)
It might be worth working through #115 as that may help to clarify some of these issues
Now that different approaches to normalising the average agent error have been
implemented, we need to add a method for switching this normalisation option
on and off (probably via some parameter passed at initialisation.
Now time to create a notebook which runs instances of the EnKF for the same parameter values but calculate errors using different normalisation approaches and plot the error profiles against each other.
Initial set of experients run, analysis in the morning
At present, the modified error calculation methods are being applied to all
forms of error:
- forecast,
- analysis,
- baseline model and
- observations.
We don't want to normalise the observation error by the number of active agents,
but instead by the number of observations. We should, therefore, make some
changes to the methods that are used to calculate the observation errors.
Having re-run some experiments, we can see from the figures at the end of this notebook that the use of different types of normalisation definitely has an impact on the errors that we observe. Unsurprisingly this doesn't change the fact that the use of the filter with the model outperforms the model on its own.
This set of runs is for the stationsim toy model, so next up we can run the same process wtih stationsim gcs.
Experiments have been run for both toy model and gcs model - discuss the results at the next meeting.