Add support for NetCDF files following the UGRID convention

Question

Add support for NetCDF files following the UGRID convention

Closed this issue 8 months ago · 6 comments

Answer 1 · 2022-08-16T23:37:24.000Z

Some MPAS grid file examples and information are here: https://mpas-dev.github.io/atmosphere/atmosphere_meshes.html

Answer 2 · 2023-08-31T20:54:12.000Z

UGridConfig_default is introduced. The dimension names and variable names should be added to support more unstructured grid files.

I used "face" instead of "cell" from UGRID Conventions (v1.0).
This is for MPAS (first name) and LFric (second name). node and edges are for the future implemenations:

ugrid_metadata_map = [
   { key = "dim_face";  val = "nCells,nMesh2d_face"; },
   { key = "dim_node";  val = "nVertices,nMesh2d_node"; },
   { key = "dim_edge";  val = "nEdges,nMesh2d_edge"; },
   { key = "dim_time";  val = "Time,time_counter"; },
   { key = "dim_vert";  val = "nVertLevels"; },
   { key = "cell_id";   val = "indexToCellID,"; },
   { key = "lat_face";  val = "latCell,Mesh2d_face_y"; },
   { key = "lon_face";  val = "lonCell,Mesh2d_face_x"; },
   { key = "vert_face"; val = "zCell"; },
   { key = "lat_edge";  val = "latEdge,Mesh2d_edge_y"; },
   { key = "lon_edge";  val = "lonEdge,Mesh2d_edge_x"; },
   { key = "lat_node";  val = "latVertex,Mesh2d_node_y"; },
   { key = "lon_node";  val = "lonVertex,Mesh2d_node_x"; },
   { key = "time";      val = "xtime,time_centered"; }
];

Answer 3 · 2023-09-12T23:27:04.000Z

The default ugrid_metadata_map from UGridConfig_default will be overridden by the "UGridConfig_user" at the current working directory. This will be overridden by the configuration file, "ugrid_map_config".

ugrid_user_map_config = "/d1/personal/hsoh/data/MET-2231/ugrid_metadata_map.config";

Answer 4 · 2023-09-21T20:04:23.000Z

How to handle vertical levels for MPAS?
MET does not allow a negative lower bound. Without handling vertical levels, FCST_LEV columns are empty (NA).

        double zCell(nCells) ;
                zCell:units = "m" ;
                zCell:long_name = "Cartesian z-coordinate of cells" ;

 zCell = 2849300.22215193, 2849300.23592602, 6371229, -2849300.22215219,
    -2849300.23592647, -2849300.18911285, 2849300.22944048,
    -2849300.22944073, -6371229, 2849300.2026535, -2849300.20265379,
    2849300.18911245, 3349553.28907785, -0.00621810533595908,
    0.0103280223263834, 3349553.3173988, 5419691.09266147,
    0.0245392207624739, 0.0289636509723273, 3349553.32185776,

Answer 5 · 2023-09-21T20:28:22.000Z

As discussed on 9/21/23, the ugrid_user_map_config setting may get confusing if the metadata for all supported models is included in the same file. Consider providing separate config files for each model type, e.g. MPAS, LFriC, Netpune, and so on.

Answer 6 · 2023-11-15T17:22:01.000Z

Working on the feature_2231_unstructured_grid branch to resolve linker issues we discovered when compiling inside the container. I'll update this comment with details about this process.

First, launch a dtcenter/met-base-unit-test:v3.1 container with the feature branch source code volume-mounted in. Run a script to compile the code within the container.

docker run -it --volume `pwd`/MET-feature_2231_unstructured_grid:/met/MET-feature_2231_unstructured_grid dtcenter/met-base-unit-test:v3.1 /bin/bash
export MET_GIT_NAME=feature_2231_unstructured_grid
internal/scripts/docker/build_met_docker.sh

I note that the updated version of autoconf in the container produces changes to the autoconf output (Makefile.in, configure, and so on). It's a whopping total of 151 modified files. I compared those to the contents of Julie's feature_2611_oneapi branch.

All of the Makefile.in files, aclocal.m4, compile, decomp, install-sh, missing, and ylwrap exactly match.
There are small diffs in config.guess which look to be bugs/typos fixed in this 2.71 autoconf instance. My version has a newer date (2022-01-09) than the Julie's version (2021-06-03).
- Committed the files modified by upgrading from using autoconf version 2.69 to 2.71.
The output files generated by flex and bison are also updated since the Bison version is updated from 3.3.2 to 3.8.2.
The compilation fails for pb2nc:

calcape.o: file not recognized: file format not recognized

This is not actually an issue. It was due to the fact that calcape.o already existed and was compiled by the Fortran compiler on my Mac rather than the one in the container. Those versions are incompatible. Running make clean install deletes the .o files, forcing them to be recreated, which fixes the problem.

Update build_met_docker.sh by adding --enable-ugrid when running the configure script, and rerun the build_met_docker.sh script. That errors out as expected.
Modified configure.ac to for GRIB2, Python, and UGRID to differentiate between the internal MET libraries from the associated external dependencies.
- Change from GRIB2_LIBS to GRIB2_MET_LIBS and GRIB2_DEP_LIBS.
- Change from PYTHON_LIBS to PYTHON_MET_LIBS and PYTHON_DEP_LIBS.
- Change from UGRID_LIBS to UGRID_MET_LIBS and UGRID_DEP_LIBS.
Run the following to replaces all instances of the former with references to the latter:

for file in `find ./ -name "Makefile.am"`; do
  cat $file | \
    sed 's/$(PYTHON_LIBS)/$(PYTHON_MET_LIBS) $(PYTHON_DEP_LIBS)/g' | \
    sed 's/$(GRIB2_LIBS)/$(GRIB2_MET_LIBS) $(GRIB2_DEP_LIBS)/g' | \
    sed 's/$(UGRID_LIBS)/$(UGRID_MET_LIBS) $(UGRID_DEP_LIBS)/g' > new
  mv new $file
done

Re-run build_met_docker.sh to get updated autoconf output files and commit those changes.
Manually modify 45 Makefile.am files that reference PYTHON_MET_LIBS, GRIB2_MET_LIBS, or UGRID_MET_LIBS to link the *_MET_LIBS entries with the other "data2d" libs after -lvx_data2d_factory and the *_DEP_LIBS entries at the end, just after -lvx_log.
Re-run build_met_docker.sh to see if these changes solve the compilation errors.
It all finally compiled (YEAH), and I committed/pushed the Makefile changes that work.

Add support for NetCDF files following the UGRID convention

Describe the New Feature

Acceptance Testing

Time Estimate

Sub-Issues

Relevant Deadlines

Funding Source

Define the Metadata

Assignee

Labels

Projects and Milestone

Define Related Issue(s)

New Feature Checklist