Bugfix: Fix inconsistent handling of point observation valid times processed through Python embedding

Question

Bugfix: Fix inconsistent handling of point observation valid times processed through Python embedding

JohnHalleyGotway opened this issue 7 months ago · 0 comments

Describe the Problem

The METplus Point-Stat TCI Use Case recently produced unexpected differences when run via GitHub Actions. Careful investigation by @DanielAdriaansen (documented here) showed the following:

Re-create the dtcenter/METbaseimage v3.2 release triggered these differences.
The updated image lead to the Python glob serving up files to a MET Python embedding script in a different order.
The different order of input files lead to a different order of point observations being served up in memory to Point-Stat.
A bug in how valid time strings are stored for Python embedding of point observations produced different results from Point-Stat. Specifically, the observation are filtered by their valid time in an inconsistent way.

Expected Behavior

The order of point observations served up via Python embedding to Point-Stat should have no impact on filtering by valid time.

To Reproduce

Describe the steps to reproduce the behavior:

Download this sample data file:
issue_2897_data.tar.gz
Unpack the file and update the MET_BIN setting in run_ps.sh to point to a local MET binary directory.
Execute run_ps.sh.
The following command:

for OBS_SID in `cat out_test2/point_stat_000000L_19790101_000000V_mpr.txt | awk '{print $27}'`; do
   grep $OBS_SID test1.csv;
   grep $OBS_SID test2.csv;
   echo "$OBS_SID test1: `grep $OBS_SID out_test1/point_stat_000000L_19790101_000000V_mpr.txt | awk '{print $8, $28, $29}'`";
   echo "$OBS_SID test2: `grep $OBS_SID out_test2/point_stat_000000L_19790101_000000V_mpr.txt | awk '{print $8, $28, $29}'`";
   echo;
done

Note the following timestamp mismatch:

ADPSFC CH-Oe1 20020128_000000 47.2858 7.7319 10 TCI 10 0 NA 1.1331664358420377
CH-Oe1 test1: 20020128_000000 47.2858 7.7319
CH-Oe1 test2: 20051201_000000 47.2858 7.7319

In test2, the timestamp for CH-Oe1 is WRONG.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

Select engineer(s) or no engineer required
Select scientist(s) or no scientist required

Labels

Review default alert labels
Select component(s)
Select priority
Select requestor(s)

Milestone and Projects

Select Milestone as the next bugfix version
Select Coordinated METplus-X.Y Support project for support of the current coordinated release
Select MET-X.Y.Z Development project for development toward the next official release

Define Related Issue(s)

Consider the impact to the other METplus components.

METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy

Bugfix Checklist

See the METplus Workflow for details.