GEUS-Glaciology-and-Climate/pypromice

Mixing temporal resolutions when merging files with combine_first

Opened this issue · 1 comments

If an hourly STM table covering 2022-2024 is retrieved from the field and is added after a raw 10 min file covering 2022-2023, then the merging of L1 datasets into the L1A dataset will have a strange behavior:

self.L1A = reduce(xr.Dataset.combine_first, reversed(self.L1))

The latest hourly data ds_latest_hourly = AWS.L1[-1] will be merged with the previous raw dataset ds_previous_10min = AWS.L1[-2] with :
ds_merged = ds_latest_hourly.combine_first(ds_previous_10min )

In that statement, the round-hours lines from the hourly file will be unchanged, but 10 min data will be inserted between these values.

Due to this this, winter daily tx have been injected at KAN_U's hourly data:
billede

Again, it all comes from the fact that combine_first only looks at the order at which the files come and is not setting different priorities to the different data types (10 min > STM > tx).