PyFlexPlot is a Python-based tool to visualize FLEXPART dispersion simulation results stored in NetCDF format.
Pyflexplot allows to visualize data on a map plot and save the output in either PDF or PNG format. To utilize this feature, simply adjust the outfile variable with the appropriate file extension.
Furthermore, Pyflexplot provides the functionality to export data into shape files (.shp) to utilize them in GIS programs such as QGIS 3. The output is a ZIP archive containing the essential components of a shapefile: .shp, .dbf, .shx, .prj, and .shp.xml. Key aspects of this feature include:
- Filtering Zero Values: The tool initially removes zero values from fields (e.g., concentration) before processing.
- Logarithmic Transformation: Field values undergo a log_10 transformation to optimize the visualization of data ranges.
- Precision Handling: The transformed field values are recorded with 15 decimal places, accommodating the precision limitations of some GIS software.
- Metadata Storage: Information, such as details about released materials, are stored within a .shp.xml file as metadata.
Another feature is to manipulate the field values by scaling with an arbitrary factor. This factor can be set in the preset with the variable multiplier
.
You can install pyflexplot from MCH pypi repository using pip:
pip install pyflexplot -i https://service.meteoswiss.ch/nexus/repository/python-all/simple
The primary command for pyflexplot follows this structure:
pyflexplot [OPTIONS] CONFIG_FILE_DIRECTORY
To see the available options, run:
pyflexplot --help
If you want to run the following examples interactively, you may want to allocate parallel resources with the help of SLURM (if available), e.g. 10 cores:
salloc -c 10
To use all allocated cpus, add the --num-procs
option to the pyflexplot command
(note that for a complete pyflexplot command, the definition of a preset or input and output need to be added, see below):
pyflexplot --num-procs=$SLURM_CPUS_PER_TASK
Important: Free resources when done!
exit
Example using default input file. This example assumes you are in the pyflexplot directory.
Default input files are searched for in ./data
.
If you want to use the files as defined in the presets for your tests,
link thm into the root of the repository. At CSCS on Alps, use:
ln -s /store_new/mch/msopr/pyflexplot_testdata data
There are several default config files available under src/pyflexplot/data/presets/opr
.
To produce graphics for a specific FLEXPART output, select the
corresponding preset from the table below and define the preset
variable accordingly:
Model | Type | Define Preset Variable |
---|---|---|
FLEXPART-IFS | global | preset=opr/ifs-hres/all_pdf |
FLEXPART-IFS | Europe | preset=opr/ifs-hres-eu/all_pdf |
FLEXPART-COSMO-1E-CTRL | deterministic | preset=opr/cosmo-1e-ctrl/all_pdf |
FLEXPART-COSMO-2E-CTRL | deterministic | preset=opr/cosmo-2e-ctrl/all_pdf |
FLEXPART-COSMO-1E | ensemble | preset=opr/cosmo-1e/all_pdf |
FLEXPART-COSMO-2E | ensemble | preset=opr/cosmo-2e/all_pdf |
FLEXPART-ICON-CH1-CTRL | deterministic | preset=opr/icon-ch1-ctrl/all_pdf |
FLEXPART-ICON-CH2-EPS | ensemble | preset=opr/icon-ch2-eps/all_pdf |
You may use the *
wildcard to operate pyflexplot
with several presets at once. For example, to run pyflexplot with all presets
that produce the graphics in PDF format for a specific
NWP model, define the preset
variable as one of:
preset='opr/cosmo*/all_pdf'
preset='opr/icon*/all_pdf'
preset='opr/ifs*/all_pdf'
Note however that the preset in this form requires the respective default input files to be accessible through the ./data
directory link.
Define an output directory and create it, if it does not exist, e.g.
nwp=$(echo ${preset} | cut -d/ -f2 | sed 's/*//g') # Extract NWP model name
dest=plot_$nwp
mkdir $dest
After selecting a preset, you may run pyflexplot interactively for the default test data:
pyflexplot --preset "$preset" --merge-pdfs --dest=$dest
On the production server at the CSCS, it is however highly recommended to create and run a batch job using the batchPP
utility:
batchPP -t 2 -T 10 -n pfp_$nwp -- \
$CONDA_PREFIX/bin/pyflexplot --preset "$preset" \
--merge-pdfs --dest=$dest --num-procs=\$SLURM_CPUS_PER_TASK
To use your own or an operational FLEXPART output file in NetCDF format as input for pyflexplot,
modify the settings of the preset with the help of the --setup
option as follows:
pyflexplot --preset "$preset" --merge-pdfs --dest=$dest --setup infile <netcdf-file>
To use a FLEXPART ensemble as input, the placeholder {ens_member:03}
may be used within the path of <netcdf-file>.
Instead of 03
(for %03d
), another C-style field width for the ensemble member field can be used.
Example using operational Flexpart ensemble output based on ICON-CH2-EPS:
preset=opr/icon-ch2-eps/all_pdf # Preset for ICON-CH2-EPS
nwp=$(echo ${preset} | cut -d/ -f2 | sed 's/*//g') # Extract NWP model name
basetime=$(date --utc --date="today 00" +%Y%m%d%H) # Recent base time
# Get name of first input file
infile000=$(echo /store_new/mch/msopr/osm/ICON-CH2-EPS/FCST${basetime:2:2}/${basetime:2:8}_???/flexpart_c/000/grid_conc_*_BEZ.nc)
infile=${infile000/\/000\//\/\{ens_member:03\}\/} # Input file definition
dest=plot_${basetime:2:8} # Output directory with base time of NWP model
mkdir $dest # Create output directory
# Submit job with the help of the batchPP utility
batchPP -t 1 -T 10 -n pfp-$nwp -- \
$CONDA_PREFIX/bin/pyflexplot --preset $preset \
--merge-pdfs --setup infile $infile --setup base_time $basetime --dest=$dest \
--num-procs=\$SLURM_CPUS_PER_TASK
The following expamles use FLEXPART output generated with the test-fp
script
in the test_meteoswiss
subdirectory of the flexpart repository of MeteoSwiss. Define FP_JOBS
as path to the FLEXPART output files that are to be used as input for pyflexplot, e.g.
FP_JOBS=$SCRATCH/flexpart/job
Write output to a location where you have write access, e.g.
FP_OUT=$SCRATCH/flexpart/job
After additionally defining the preset
as above and nwp
as the
job name (directory name below FP_JOBS), create the output directory with
infile=$(echo $FP_JOBS/$nwp/output/*.nc)
basetime=$(cat $FP_JOBS/$nwp/output/plot_info)
dest=$FP_OUT/$nwp/plots
mkdir -p $dest
and submit the job with the batchPP command as above.
For ensembles, the infile
needs to be a pattern rather than a single file:
infile000=$(echo $FP_JOBS/$nwp/output/000/*.nc)
infile=${infile000/\/000\//\/\{ens_member:03\}\/}
After job completion, list and visualize results e.g. with evince
:
ls $dest/*pdf
evince $dest/*pdf
In order to download input NETCDF data from S3, and S3 URI can be specified as the setup parameter infile
as below:
pyflexplot --preset "$preset" --merge-pdfs --dest=$dest --setup infile s3://<s3-bucket-name>/flexpart_cosmo-2e_2021030503_{ens_member:03d}_MUE.nc
In order to output the resulting plots to an S3 bucket, specify the S3 bucket name as the --dest
. The plots will still be created locally at the dest dir path defined in the config/settings.yaml
pyflexplot --preset "$preset" --merge-pdfs --dest=s3://<s3-bucket-name>
Prerequisites: Git, Miniconda (for installation of Conda) or Poetry
Clone the repo and enter the project folder:
git clone git@github.com:MeteoSwiss-APN/pyflexplot.git && cd pyflexplot
Create a Conda (or mamba/micromamba) environment with only the desired Python version and activate:
conda create --yes --prefix ./.conda-env python=3.10
conda activate ./.conda-env
Install Poetry into this environment and configure Poetry to not create a new virtual environment. If it detects an already enabled virtual (eg Conda) environment it will install dependencies into it:
conda install --yes poetry
poetry config --local virtualenvs.create false
Install packages:
poetry install
Run tests:
poetry run pytest
Run pylint to check code style of Python files (if any):
poetry run pylint src
Run mypy to check typing:
poetry run mypy
Pyflexplot includes a set of functionality tests that compare generated output against predefined reference data.
These reference files, which contain summary dicts, begin with ref_
and have
the nomal Python file ending .py
, and are stored in the directory
tests/slow/pyflexplot/test_plots
.
To update these reference files, uncomment the following line near the end
of the file
shared.py
in the same directory:
_TestBase = _TestCreateReference
Then re-run the (slow) tests to generate the new reference files. After generating the new reference files, comment out the above line again or simply revert the file with git.
- pyshp - Python module to generate Shapefiles
This project is licensed under the terms of the MIT License. The full license text can be found in the LICENSE file. In essence, you are free to use, modify, and distribute the software, provided the associated copyright notice and disclaimers are included.