Wind speed and wind direction issues R.06

[sjnoone]

We have found that some new average wind observations over specific time frames have been introduce into the r.06 merge.

For example CAW00064757.qff has 60minute average wind speed obseravtions enocded in the measurement code column as “R-60min-av”. In the C3S release we are currently only using "N" normal and "C" calm hourly wind speed observations.

For example I checked the CAW00064757.qff from source (NCEI/CRN) for occurrences of “R-60min-av” in the wind_speed_Measurement_Code columns and the file has 148967 rows of wind observations at “R-60min-av” out of the total rows of 1743012 over the period (2005 -2021). There are the same amount of wind_direction rows with -999.0. There are no other wind observations in this files beside the “R-60min-av”.

The corresponding wind direction for the R-60min-av wind speed observation is set to -999.0 because it is difficult to calculate hourly average wind direction.

Apparantly there are other potential new averages of wind obseravation in the r.06 merge (see list below). We will also need to check if other variables have averages introduced.

Potential solutions

1. For the C3S r06 release we could remove all wind speed observations that have any of the following codes (see highlighted queries) present in the wind_speed_measurement_code column and remove the corresponding wind_direction observations.

Remove codes for C3S
'A':'A-Abr-Beauf',
'B':'B-Beaufort',
'H':'H-5min-avg-spd',
'R':'R-60min-avg-spd',
'Q':'Q-Squall',
'T':'T-180min-avg-spd',
'V':'V-Variable',
'9':'9-Missing'

Keep codes for C3S:
'C':'C-Calm',
N':'N-Normal',

2. Or we could for example encode “R-60min-av” wind speeds in cdm format observed_value [106] wind speed, duration [9] 1 hour and significance value [2] mean. Replace corresponding wind direction -999.0 with blank. We could do this when converting to cdm format files from qff.

Team decision

The team agreed that it is a lot to get our heads around for r.06 to proceed with option 1 at this time. So we will only bring forward "N" normal and "C" calm wind observations in the C3S release but leave in them in the qff for NCEI to use in GHCNh. Lets look at the balance of wind observations over the next year and make an informed decision on whether/what we include them in r.07.

Other wind issue

It has also come to our attention that we will have to change the current encoding for wind speed observations as they are 10-min averges and not instatenous measurements.

At the moment all hourly wind speed observations in cdm format files are set to observed_value [106] = wind speed, duration [0] = instantaneous and significance value [12] = instantaneous.

The QFF to cdm_lite and cdm_obs CDM conversion code should be changed to:
At the moment all hourly wind speed observations in cdm format files are set to observed_value [106] = wind speed, duration [8] = 10 minutes and significance value [2] = mean.

[sjnoone]

Another issue with the hourly_qff_to_cdm_obs_v1.py wind speed code section is not picking up the observed wind speed value or the original wind speed value from the df (code section ws starts line 490-530) . I do not think there is any conversion in wind speed values. Needs the piece of code to add the values see other variables.
Also as far as I can ascertain the secondary_ID is needed to produce the primary_station_ID2 so information form the record_iD.csv can be merged with the file.

[rjhd2]

OK - I think this now works. There is quite some confusion through the scripts (Obs and Lite checked so far) as to how NaN, null and Null values are being created, masked and dropped. This meant that some code lines aren't doing anything in the remove_missing_data_rows() as some NaNs were converted to string "nan", and others replaced by "Null" in an earlier routine.

Currently this routine now does drop rows without observation values, and the fix for the masked wind speed and direction measurement codes also works. Issue #80 raised to address other items once Rel 6 completed.

[sjnoone]

Thanks for doing these edits.
I've checked the updated cdm conversion code and the wind direction is also set to 10-minute [8] and mean[2] is this correct?
Everything else looks good!

[rjhd2]

TBH, I'm not sure. I presumed that it would be the same instrument so handled in the same way. But I don't actually know.