This project collects examples of analysis scripts for each application supported by AtmoRep. Please refer to the model page for instructions on how to run the model.
The output of the evaluation step is a set of .zarr
files. Example:
model_idc96xrbip.json
= model settings used in the evaluation phaseresults_idc96xrbip_epoch00000_pred.zarr
= file storing all predicted tokensresults_idc96xrbip_epoch00000_target.zarr
= file storing the masked target tokens (aka ground truth) in the same format as predictionsresults_idc96xrbip_epoch00000_source.zarr
= file storing all the loaded tokens (masked tokens are stored as zeros)results_idc96xrbip_epoch00000_ens.zarr
= file storing the ensemble predictions- (optional)
results_idc96xrbip_epoch00000_attention.zarr
= file storing the attention scores. Written only ifattention = True
at evaluation stage.
The read_atmorep_data.py
package contains all the functionalities to read the AtmoRep data. The output of read_data is a set of xarrays.
from utils.read_atmorep_data import HandleAtmoRepData
ar_data = HandleAtmoRepData(model_id, input_dir)
da_target = ar_data.read_data(field, "target", ml = levels)
da_pred = ar_data.read_data(field, "pred" , ml = levels)
the output is interfaced with the metrics.py
package, which contains a set of common metrics to evaluate the model performance.
from utils.metrics import Scores, calc_scores_item
metrics = ["rmse", "acc", "l1", "bias"]
atmorep_scores = calc_scores_item(da_pred, da_target, metrics, avg = ["lat", "lon"])
in this case the scores will be averaged over latitude and longitude. Please use avg = None
to avoid averaging or avg = all
if you want to average over all dimensions.
Here below you can find a brief description of the examples available for each application. This is a WIP and new use cases will be added shortly.
This package, contained within the trainings
folder contains a plotting routine to inspect your trainings. It supports the computation of all the metrics defined within metrics
:
- 1D plots over local token variables (latitude, longitude, time, local position within the masked area etc..)
- 2D maps averaged over time and per time-step
- plot single token predictions for the 10 examples with the highest and lower absolute and relative errors as well as 50 random examples.
- analysis of the ensemble distributions (standard deviation, spread vs error etc..).
The forecasting analysis is in the forecasting
folder. This is intended to be an example of how to compare AtmoRep data -- computed with the global_forecasting
option in evaluate.py
-- against e.g. PanguWeather data obtained with the ai-models interface provided by ECMWF.
The folder attention
contains the code to inspect the attention scores. The values can be obtained by setting attention = True
at evaluation stage in evaluate.py
.
The code 'plot_attention.py' plots the attention maps from the *_attention.zarr
file.