metadata

[akubaryk]

GSMWAM-IPE branch swio_metadata has been created to provide the output_metadata tag in the SWIO resource file with four static parameters:

1. initialization date (init_date)
2. model start date (start_date)
3. model end date (end_date)
4. model CDUMP (run_type, e.g. wfs for the long forecast, wfr for realtime run, compset_run for scripts/compsets)

Each of the first three will be YYYYMMDDHHMM strings to be interpreted by Python's datetime.datetime.strptime in post-processing/viz tasks. We also need another field (valid_date? fcst_date?) that represents the datetime at which the exported fields are valid.

[rmontuoro]

Commit 2d8603d adds the ability to write user-provided metadata to SWIO output files as global attributes.

Metadata is provided as a list of space-separated <key> and <value> pairs, listed (one pair per line) under the new output_metadata:: table included in the SWIO resource file.

For example, adding the following section to swio.ipe.rc:

output_metadata::
    init_date     2013031600
    start_date    2013031600
    end_date      2013031601
    run_type      "This run is wfs"
    compset_run
::

will add the global attributes below to each of the ipe.*.nc output files:

netcdf ipe.20130316_000600 {
...
// global attributes:
		:init_date = "2013031600" ;
		:start_date = "2013031600" ;
		:end_date = "2013031601" ;
		:run_type = "This run is wfs" ;
		:compset_run = "N/A" ;
}

Note that if a <value> is missing, it will be replaced with 'N/A'

[akubaryk]

Thanks for the update. I have merged swio_metadata into that pull -- is fcst_date/valid_date a possibility, or should we go ahead without that for now and post-process that in?

[rmontuoro]

You may use arbitrary strings for both <key> and <value>. SWIO won’t interpret such fields.

[akubaryk]

Right, I was wondering if we can grab the date when the fields are exported to SWIO, so each output file has a dynamic timestamp for the actual datetime it's outputting for -- I thought I had heard that on Monday but I may have misheard or misunderstood?

[rmontuoro]

Ok. This feature needs to be implemented if still needed.

[gmillward]

Raffaele - yes we need a forecast date in the metadata - fcst_date - this is the date-time of the actual model result - what we often have in the filename

[rmontuoro]

@gmillward and @akubaryk, commit 7132717 implements a special metadata <value>, __swio_field_timestamp__ that is replaced by the current field timestamp by SWIO at runtime. You may use this special value as shown in the following example:

output_metadata::
  init_date     $SWIO_IDATE
  start_date    $SWIO_SDATE
  end_date      $SWIO_EDATE
  fcast_date    __swio_field_timestamp__
  run_type      $CDUMP
::

[akubaryk]

Addressed in NOAA-SWPC/GSMWAM-IPE#31. Thanks @rmontuoro.

EDIT: for reference, timestamp is now YYYYMMDD_HHMMSS