PyGLDA: a fine-scale Python-based Global Land Data Assimilation system for integrating satellite gravity data into hydrological models
Data assimilation (DA) has been widely used to merge the hydrological 'model' and the remote sensing 'observations', to complement each other towards a better temporal, horizontal and vertical disaggregation of the TWS estimates. In this study, we present a free, open-source, parallel and Python based PyGDA system to address the challenges of the High-Resolution Global DA (HRGDA) for GRACE and W3RA hydrologocal model, where the framework has been completely re-designed relative to previous studies to be able to run HRGDA (up to 0.1 degree and daily) efficiently and stably.
Fan Yang (fany@plan.aau.dk) , Ehsan Forootan (efo@plan.aau.dk), Maike Schumacher (maikes@plan.aau.dk)
Geodesy Group (see https://aaugeodesy.com/), Department of Sustainability and Planning, Aalborg University, Aalborg 9000, Denmark
This work is supported by the Danmarks Frie Forskningsfond [10.46540/2035-00247B] through the DANSk-LSM project. Additional supports come from and national Natural Science Foundation of China (Grant No. 42274112 and No. 41804016).
- A novel unified framework to enable it to address both the basin-scale and grid-scale DA, where the area to be computed is defined as the 'basin' and the grid is treated as the 'sub-basin'. Any shape file that has well defined the basin and sub-basins is accepted to launch PyGDA;
- A novel framework to seamlessly transit from regional DA to global DA by introducing domain localization and weighting algorithms;
- Computation of the spatial covariance of the observations, i.e., GRACE based TWS, is available between arbitrary sub-basins/grids for the first time. Accounting for the spatial covariance into the DA is also allowed in PyGDA;
- Flexible options in the grid resolution: a choice of 0.1 degree and 0.5 degree is available for the employed global W3RA model; a flexible choice of the grid/sub-basin to be assimilated, e.g., from 1 degree to 5 degree with an increment of 0.5 degree;
- Flexible choice of perturbation, e.g., which forcing data and which model parameters to be perturbed in which noise distribution (Gaussian or Triangle; additive or multiplicative);
- In addition to the general Ensemble Kalman filter, a novel Kalman smoother that achieves optimal temporal disaggregation from monthly increment to daily increment is facilitated.
- A flexible modular structure to de-couple PyGDA into three individual modules: (1) hydrological model, (2) GRACE processing and (3) mathematical DA integration. This makes possible to easily develop/modify/replace any individual module;
- A high-level programming language (object-oriented Python) for easy comprehension of the code and to facilitate extensibility. The Python translation of W3RA model is distributed;
- Easy/fast installation for cross-platform to the users’ needs and capacity (e.g., Windows, Linux and parallel computation at high performance clusters);
- Optimization by using high-performance Numpy package as the basic data container to reach comparable numerical efficiency as C++, Fortran and Matlab;
- User-friendly interaction by (1) controlling/configuring PyGDA with JSON setting file, where a wide options are available for a different purpose; and (2) state-of-the-art data structure H5DF for reading and writing spatiotemporal data to allow for efficient management of data storage.
Please follow below the step-by-step instruction to do the installation, given that the Conda environment has been established already. A simpler installation via .yml to copy the environment will also be released soon. While PyGLDA does not specify the version of Python, it is strongly recommended to use python > 3.9.
- pip install netcdf4
- conda install metview-batch -c conda-forge
- conda install metview-python -c conda-forge
- pip install cdsapi
- conda install -c conda-forge mpi4py mpich
- pip install h5py
- pip install scipy
- pip install geopandas
To test if it is properly installed, please type below command and see if it passes.
- mpiexec -n 5 python -m mpi4py.bench helloworld
- metview -slog
- python3 -m metview selfcheck
Potential installation troubles:
- metview-related issues please refer to https://metview.readthedocs.io/en/latest/index.html
- mpi4py-related issues please refer to https://mpi4py.readthedocs.io/en/latest/install.html
Three demo (demo1, demo2, demo3) are present under /py-w3ra/demo to showcase the use of PyGLDA. Each demo has its detailed comments in its script. To run the demo, a bunch of sample data are necessary to be installed.
The sample data is distributed together with the code at the given data repository, named after External Data.
To begin with the demo, we suggest to place the sample data at its default place, which is under /PyGLDA/External Dara/.
Nevertheless, as an advanced user, one can place data at any desired place as long as the setting files are well configured.
Below we give a brief introduction of three demo.
- demo_1.py. In this demo, we show how to update the meteorological forcing field (ERA5-land) online
- demo_2.py. In this demo, we show how to configure and implement the W3RA water balance model.
- demo_3.py. In this demo, we show how to implement basin-scale regional data assimilation, grid-scale data assimilation and the grid-scale global data assimilation. Please feel free to contact us for possible more examples.