Enhancement: Add support for a Point2Grid config wrapper
Opened this issue · 11 comments
Describe the Enhancement
In this discussion, Need Point2Grid metplus wrapper for EVSv2.0 #2539, Ho-ChunHuang-NOAA mentioned the need to use Point2Grid app to map GOES ABI AOD to model grid for EVS2 AOD verification for AQM, RAP, HRRR, and GEFS-Chem. A Point2GridConfig_wrapped file does not currently exist at https://github.com/dtcenter/METplus/tree/develop/parm/met_config. It seems that METplus does not currently include logic for populating and passing a config file to the MET point2grid tool.
Please note that currently, the METplus documentation states with regard to the MET Configuration, "None. Point2Grid does not use configuration files.", but the MET Point2Grid usage statement indicates that it supports an optional configuration file. The METplus documentation is incorrect and should be updated.
Time Estimate
(For @georgemccabe to fill in.)
Sub-Issues
Consider breaking the enhancement down into sub-issues.
- Add a checkbox for each sub-issue here.
Relevant Deadlines
6.0.0-beta5
Funding Source
TBD
Define the Metadata
Assignee
- Select engineer(s) or no engineer required
- Select scientist(s) or no scientist required
Labels
- Select component(s)
- Select priority
- Select requestor(s)
Projects and Milestone
- Select Repository and/or Organization level Project(s) or add alert: NEED CYCLE ASSIGNMENT label
- Select Milestone as the next official version or Future Versions
Define Related Issue(s)
Consider the impact to the other METplus components.
Enhancement Checklist
See the METplus Workflow for details.
- Complete the issue definition above, including the Time Estimate and Funding Source.
- Fork this repository or create a branch of develop.
Branch name:feature_<Issue Number>_<Description> - Complete the development and test your changes.
- Add/update log messages for easier debugging.
- Add/update unit tests.
- Add/update documentation.
- Add any new Python packages to the METplus Components Python Requirements table.
- Push local changes to GitHub.
- Submit a pull request to merge into develop.
Pull request:feature <Issue Number> <Description> - Define the pull request metadata, as permissions allow.
Select: Reviewer(s) and Development issues
Select: Repository level development cycle Project for the next official release
Select: Milestone as the next official version - Iterate until the reviewer(s) accept and merge your changes.
- Delete your fork or branch.
- Close this issue.
Simply a reminder to check discussion of 2539 regarding the problem of using "POINT2GRID_QC_FLAGS=" in METplus config file to select the quality of mapped AOD.
@jprestop @georgemccabe I need some answer of whether there is a bug on the usage of "POINT2GRID_QC_FLAGS =" in Point2Grid. Please see detail of comment on April 4 2024 in issue 2539.
I set the POINT2GRID_QC_FLAGS = 0, but the result is like POINT2GRID_QC_FLAGS ="0,1,2".
Thank you.
Hi @Ho-ChunHuang-NOAA, there was a bug in the Point2Grid wrapper that did not properly apply the -qc 0 argument if the value set was 0. The code incorrectly interpreted a zero value as an empty value. This was fixed for METplus bugfix release v5.0.2 (PR: #2101) and v5.1.0 (PR: #2140).
Please let me know if you need any additional information.
Hi, George:
Simply to confirm that the bug fix version will accepts
POINT2GRID_QC_FLAGS =
0
1
2
0,1
0,1,2 (I think it is the default)
Yes, POINT2GRID_QC_FLAGS accepts a comma-separated list of integers, so all of those examples you provided are valid inputs. I am not sure what the default behavior is if no QC flags are specified, but my best guess is that it does not filter by the QC field if no value is specified. @JohnHalleyGotway, can you provide any additional insight?
@georgemccabe and @Ho-ChunHuang-NOAA, I see a potential source of confusion here.
In general, Point2Grid reads point data and processes it onto a regular grid. It can accept multiple types of input point data, including both GOES satellite data and the NetCDF output created by other MET point pre-processing tools, like PB2NC, ASCII2NC, and MADIS2NC.
Let me clarify two details about quality control:
- The
-qccommand line option applies specifically to GOES data inputs. It's described in the Point2Grid Usage as follows:
The -qc flags option specifies a comma-separated list of quality control (QC) flags, for example “0,1”. This should only be applied if grid_mapping is set to “goes_imager_projection” and the QC variable exists.
This GOES filtering logic is only applied when the -qc option is specified on the command line. You can see that logic on this line of point2grid.cc. To that end, the default behavior is to NOT APPLY any quality control filtering to GOES data unless -qc is specified on the command line.
- I see a Point2Grid config file entry named quality_mark_thresh. Checking in the actual code for Point2Grid, I see that that entry is being parsed, it is NOT actually be used at all.
I strongly suspect that this was just copied from the PB2NC tool when Point2Grid was originally written. It does not actually belong here. Instead, we should remove quality_mark_thresh and replace it with the obs_quality_inc and obs_quality_exc options seen here in the PointStatConfig file.
These would apply when processing the NetCDF output from PB2NC, ASCII2NC, and MADIS2NC and give the user to option to specifically include or exclude obs based on their quality control values.
I'd recommend that we write up an issue to change this for MET version 12.0.0.
I'm laying out these details here to prevent any confusion between the -qc command line option and the spurious quality_mark_thresh config file option that doesn't currently actually do anything!
@georgemccabe @JohnHalleyGotway Hi, George: Thanks for the information. I was using the METPlus v5.0.2 when I encountered the POINT2GRID_QC_FLAG = 0 issue in #2539 . I rerun my scripts in METPlus v6.0.0-beta3, and the log output indicates the METPlus did interpret POINT2GRID_QC_FLAG = 0 as "-qc 0". The comparison of AOD graphics also confirms that the the mapped point2grid nc files by METPlus wrapper is high quality AOD as expected.
@Ho-ChunHuang-NOAA Is this enhancement required for EVS v2.0, and should therefore be included in the METplus-6.0.0 release?
@AliciaBentley-NOAA YES. This is for AQM AOD and GEFS AOD verification and they are scheduled to be in EVSv2.0.
Thanks, @Ho-ChunHuang-NOAA. This particular METplus issue was marked as "stalled" ~3 weeks ago when I originally asked. I'll tag the DTC folks and see what can be done. If you need something in METplus for EVS v2.0, please continue to advocate for it. Not all of the open METplus issues will make it into the official release.
Tagging @JohnHalleyGotway @j-opatz @georgemccabe @jprestop @DanielAdriaansen
Hi @AliciaBentley-NOAA and @Ho-ChunHuang-NOAA,
This work is actually completed in a branch (draft pull request #2580) but is waiting for dtcenter/MET#2880 to be completed since there are some corresponding changes needed to the Point2Grid MET tool. I just marked that issue as required for the release to make sure that work gets completed.