A collection of tools and configurations that allow you to interpolate/regrid DWD's NWP ICON model data from icosahedral(triangular) ICON grids onto regular latitude-longitude (geographical) grids using MPIMET CDO and ECMWF ecCodes. DWD's NWP data is freely available for download from DWD's Open Data File Server at https://opendata.dwd.de/weather/nwp/ .
- Available images
- Usage
Table of contents generated with markdown-toc
There are a variety of different docker images available for different regridding tasks.
Docker images that are tagged with a model name such as icon-eu-eps or all provide all necessary files to regrid data from these models. Currently the following models are supported:
- ICON Global:
deutscherwetterdienst/regrid:icon - ICON Global Ensemble:
deutscherwetterdienst/regrid:icon-eps - ICON EU Nest Ensemble:
deutscherwetterdienst/regrid:icon-eu-eps - ICON-D2 (pre-operational):
deutscherwetterdienst/regrid:icon-d2 - ICON-D2 Ensemble (pre-operational):
deutscherwetterdienst/regrid:icon-d2-eps
The general nomenclature for tags is as follows:
deutscherwetterdienst/regrid:[<model>|all][-samples|-grids][<version>|-<version>]
These images contain precomputed interpolation weights located in /data/weights/ for remapping model data to a regular-lat-lon/geographical grid at full resolution, e.g.:
- ICON EU Nest Ensemble:
deutscherwetterdienst/regrid:icon-eu-eps - all supported models:
deutscherwetterdienst/regrid:all
These images contain GRIB2 samples files for the given models e.g.
- image with sample data for all models:
deutscherwetterdienst/regrid:all-samples - image with ICON global samples:
deutscherwetterdienst/regrid:icon-samplesSamples are located in/data/samples/.
These images contain NetCDF grid definition files. These files are necessary to create customized interpolations for the given model and are located in /data/grids/. Sample files are also included in /data/samples/ to simplify tests. Examples:
- image with definition and sample files for all models:
deutscherwetterdienst/regrid:all-grids - image with definition and sample files for icon model:
deutscherwetterdienst/regrid:icon-grids
Images tagged with a version suffix correspond to a specific version tag in the VCS and are considered immutable. Images without a version tag signify the latest available versions.
We provide a slim/minimal image that contain all precompiled tools such as ecCodes and CDO, which are required to regrid/interpolate GRIB2 and NetCDF files:
deutscherwetterdienst/regrid
These images are intended for advanced users with existing knowledge of CDO. They do not provide any model specific configurations or data and hence can not be used to regrid data without providing additional data and configurations.
You can use the provided images interpolate data onto predefined grids or fully customize all details of the interpolation.
You can output versions the CDO version and the versions of all libraries by running:
docker run --rm deutscherwetterdienst/regrid cdo --version
The result should look like this:
Climate Data Operators version 1.9.8 (https://mpimet.mpg.de/cdo)
System: x86_64-pc-linux-gnu
CXX Compiler: g++ -g -O2 -fopenmp
CXX version : g++ (Debian 8.3.0-6) 8.3.0
C Compiler: gcc -fPIC -fopenmp
C version : gcc (Debian 8.3.0-6) 8.3.0
F77 Compiler: gfortran -g -O2
F77 version : GNU Fortran (Debian 8.3.0-6) 8.3.0
Features: 1GB 4threads C++14 Fortran DATA PTHREADS OpenMP HDF5 NC4/HDF5/threadsafe OPeNDAP SSE2
Libraries: HDF5/1.12.0
Filetypes: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5
CDI library version : 1.9.8
cgribex library version : 1.9.4
ecCodes library version : 2.17.1
NetCDF library version : 4.7.4 of Jun 23 2020 18:51:27 $
hdf5 library version : 1.12.0 threadsafe
exse library version : 1.4.1
FILE library version : 1.8.3
We provide tailor made images for our models that allow you to directly interpolate all available data from icosahedral/triangular ICON grids to regular-latitude-longitude/geographical grids with maximum resolution.
You can use images tagged with -samples to test interpolation on sample files. Run:
docker run --rm deutscherwetterdienst/regrid:icon-eu-eps-samples
This will convert the sample file, save the result in /data/samples/icon/icon_output.grib2 and exit. The output should look like this:
cdo remap: Processed 40 variables over 1 timestep [0.32s 79MB].
You can customize which data should be interpolated and where the result should be written by overriding the following environment variables in the docker image using the --env <VARIABLE>=<VALUE> option of docker.
INPUT_FILE: Absolute file path to the input file in GRIB2 format that needs to be transformed, e.g./data/samples/icon/icon_sample.grib2OUTPUT_FILE: Absulute path where the result should be written, e.g.:/data/samples/icon/icon_output.grib2
You need to mount a folder from you local hard drive and specify which file to regrid and where to save the output by setting the environment variables accordingly:
docker run --rm \
--volume ~/mydata:/mydata \
--env INPUT_FILE=/mydata/my_icon-eps_icosahedral_file.grib2 \
--env OUTPUT_FILE=/mydata/regridded_regular_lat_lon_output.grib2 \
deutscherwetterdienst/regrid:icon-eps
The output should look something like this:
cdo remap: Processed 40 variables over 1 timestep [0.32s 79MB].
Attention: The model of the docker image and the grid of the selected file have match!
To simplify the batch interpolation of files, the script convert.sh is included in every image. You can use environment variables or command line options to control the script's behaviour:
Regrids grib2 files and writes the output to the output directory.
Usage: convert.sh [-i <INPUT_FILE>] [-o <OUTPUT_FILE>] [-w <WEIGHTS_FILE>] [-d <DESCRIPTION_FILE>]
Alternatively the following environment variables can be used to control this script:
- INPUT_FILE : grib2 file or directory containing grib file(s)
- OUTPUT_FILE : grib2 file or directory to output regridded file(s)
- WEIGHTS_FILE : netCDF file that contains interpolation weights
- DESCRIPTION_FILE : cdo target grid description file for the given weights file
Example (with command line options):
convert.sh \
-i /data/samples/icon/ \
-o /output/ \
-w /data/weights/icon/icon_weights.nc \
-d /data/descriptions/icon/icon_description.txt
Example (with environment variables):
export INPUT_FILE=/data/samples/icon/
export OUTPUT_FILE=/output/
export WEIGHTS_FILE=/data/weights/icon/icon_weights.nc
export DESCRIPTION_FILE=/data/descriptions/icon/icon_description.txt
convert.sh
- Place all GRIB2 files that need to converted into a single folder, e.g
~/mydata. It's important that all these files are from the same model, e.g. the modelicon-d2-eps. Now the contents of you folder should look something like this:
eduard@Eduards-MacBook-Air mydata % ls -la
total 262080
drwxr-xr-x 8 eduard staff 256 Jul 29 19:44 .
drwxr-xr-x@ 127 eduard staff 4064 Jul 29 19:38 ..
-rw-r--r-- 1 eduard staff 22361460 Jul 29 19:44 icon-d2-eps_germany_icosahedral_single-level_2020072900_000_2d_t_2m.grib2
-rw-r--r-- 1 eduard staff 22361460 Jul 29 19:44 icon-d2-eps_germany_icosahedral_single-level_2020072900_001_2d_t_2m.grib2
-rw-r--r-- 1 eduard staff 22361460 Jul 29 19:44 icon-d2-eps_germany_icosahedral_single-level_2020072900_002_2d_t_2m.grib2
...
...
- Create a folder that will hold regridded data, e.g.
~/output.
mkdir ~/output
- Mount the folders inside your container (make sure you use absolute paths) and run the
convert.shscript located in file system root. Make sure you selected an image that matches the data you want to regrid, e.g.deutscherwetterdienst/regrid:icon-d2-epsforicon-d2-epsdata:
docker run --rm \
--volume ~/mydata:/mydata \
--volume ~/output:/output \
--env INPUT_FILE=/mydata \
--env OUTPUT_FILE=/output \
deutscherwetterdienst/regrid:icon-d2-eps \
/convert.sh
The output should look something like this:
Regridding all files in directory '/mydata' ...
Regridding '/mydata/icon-d2-eps_germany_icosahedral_single-level_2020072900_000_2d_t_2m.grib2' > '/output/regridded_icon-d2-eps_germany_icosahedral_single-level_2020072900_000_2d_t_2m.grib2'...
cdo remap: Processed 20 variables over 1 timestep [1.01s 170MB].
Regridding '/mydata/icon-d2-eps_germany_icosahedral_single-level_2020072900_001_2d_t_2m.grib2' > '/output/regridded_icon-d2-eps_germany_icosahedral_single-level_2020072900_001_2d_t_2m.grib2'...
cdo remap: Processed 20 variables over 1 timestep [0.93s 170MB].
Regridding '/mydata/icon-d2-eps_germany_icosahedral_single-level_2020072900_002_2d_t_2m.grib2' > '/output/regridded_icon-d2-eps_germany_icosahedral_single-level_2020072900_002_2d_t_2m.grib2'...
cdo remap: Processed 20 variables over 1 timestep [1.04s 170MB].
...
Done.
Your folder ~/output should now contain the regridded files:
eduard@Eduards-MacBook-Air output % ls -la
total 621800
drwxr-xr-x 20 eduard staff 640 Jul 29 19:45 .
drwxr-xr-x@ 127 eduard staff 4064 Jul 29 19:38 ..
-rw-r--r-- 1 eduard staff 32220340 Jul 29 19:44 regridded_icon-d2-eps_germany_icosahedral_single-level_2020072900_000_2d_t_2m.grib2
-rw-r--r-- 1 eduard staff 32220340 Jul 29 19:44 regridded_icon-d2-eps_germany_icosahedral_single-level_2020072900_001_2d_t_2m.grib2
-rw-r--r-- 1 eduard staff 32220340 Jul 29 19:44 regridded_icon-d2-eps_germany_icosahedral_single-level_2020072900_002_2d_t_2m.grib2
...
You can use all images by running command direcrly inside the container using an interactive shell.
Run:
docker run --rm -it \
deutscherwetterdienst/regrid:icon-samples \
bash
Analyse the input file by running:
grib_ls /data/samples/icon/icon_sample.grib2
The output should look something like this:
/data/samples/icon/icon_sample.grib2
edition centre date dataType gridType stepRange typeOfLevel level shortName packingType
2 edzw 20200618 fc unstructured_grid 0 heightAboveGround 2 2t grid_simple
1 of 1 messages in /data/samples/icon/icon_sample.grib2
1 of 1 total messages in 1 files
Run:
cdo -f grb2 remap,/data/descriptions/icon/icon_description.txt,/data/weights/icon/icon_weights.nc /data/samples/icon/icon_sample.grib2 /data/samples/icon/icon_regridded_output.grib2
To analyse the output file run:
grib_ls /data/samples/icon/icon_regridded_output.grib2
The result should look something like this:
/data/samples/icon/icon_regridded_output.grib2
edition centre date dataType gridType stepRange typeOfLevel level shortName packingType
2 edzw 20200618 fc regular_ll 0 heightAboveGround 2 2t grid_simple
1 of 1 messages in /data/samples/icon/icon_regridded_output.grib2
1 of 1 total messages in 1 files
The gridType changed from unstructured_grid (ICON triangular grid) to regular_ll (geographical grid).
You can fully customize which and how data will be interpolated by modifying environment variables and providing custom grid description files and weights files.
The following environment variables are supported:
INPUT_FILE: Absolute file path to the input file in GRIB2 format that needs to be transformed, e.g./data/samples/icon/icon_sample.grib2OUTPUT_FILE: Absulute path where the result should be written, e.g./data/samples/icon/icon_output.grib2DESCRIPTION_FILE: The CDO grid description file, e.g./data/descriptions/icon/icon_description.txtWEIGHTS_FILE: pre-computed interpolation weights - see DWD's CDO How-To-Guide, e.g./data/weights/icon/icon_weights.ncGRID_FILE: NetCDF grid definition file, e.g./data/grids/icon/icon_grid.nc
You can fully customize how to regrid DWD grib data by first creating a CDO grid description file and then genrating the interpolation weights. It is possible to extract an arbitrary subset of the available data and/or interpolate the data onto a custom grid.
First you need to create a custom (CDO grid description file)[https://code.mpimet.mpg.de/projects/cdo/embedded/index.html#x1-150001.3.2]. This file defines the grid type, the area and the resolution of the desired grib file after interpolation. The file below is an example of a cutout over Europe in a regular latitude-longitude (geographical) grid with a resolution of 0.25° x 0.25° degrees:
# File: output_grid.txt
# Climate Data Operator (CDO) grid description file
# Input: ICON (Global)
# Area: Europe
# Grid: regular latitude longitude/geographical grid
# Resolution: 0.25 x 0.25 degrees
gridtype = lonlat
xsize = 601
ysize = 301
xfirst = -75.0
xinc = 0.25
yfirst = 5.0
yinc = 0.25
Save this file to a local folder, e.g. to ~/mydata/custom_grid.txt.
In order to generate weights, you need the grid definition files (in NetCDF) that are necessary to generate custom interpolation weights for the icon model you want to interpolate. To do this we will use the corresponding grids images. In this example we will use the ICON Global image tagged icon-grids, which we will run interactively:
docker run --rm -it \
--volume ~/mydata:/mydata \
--workdir /mydata \
deutscherwetterdienst/regrid:icon-grids \
bash
Now, to precompute the weights for the custom grid definition run the following command inside the container:
cdo gennn,/mydata/custom_grid.txt \
/data/grids/icon/icon_grid.nc \
/mydata/custom_weights.nc
Attention: For ICON-D2 (and ICON-D2-EPS) you need to use the -setgrid option and select grid no. 2 in the netCDF grid definition file to generate weights:
cdo gennn,/mydata/custom_grid.txt \
-setgrid,/data/grids/icon-d2/icon-d2_grid.nc:2 \
/data/samples/icon-d2/icon-d2_sample.grib2 \
/mydata/custom_weights.nc
The output should look something like this:
cdo gennn: Nearest neighbor weights from unstructured (2949120) to lonlat (601x301) grid
cdo gennn: Processed 6 variables over 1 timestep [22.84s 932MB].
Now you can use the newly generated custom weights file custom_weights.nc to regrid the icon sample file:
cdo -f grb2 remap,/mydata/custom_grid.txt,/mydata/custom_weights.nc /data/samples/icon/icon_sample.grib2 /mydata/icon_custom.grib2