pysat/pysatNASA

BUG: `test_load_multiple_days` failures

Opened this issue · 5 comments

Description

pysat 3.2.0 has a new universal test: test_load_multiple_days. Several instruments fail this test

To Reproduce this bug:

run pytest with pysat develop

Test configuration

  • OS: Ventura
  • Version: Python 3.10.9
  • pysat develop

Additional context

  • ace_epam_l2, inst_id='12sec', tag='base'
  • ace_mag_l2, inst_id='16sec', tag='base'
  • ace_mag_l2, inst_id='16sec', tag='key'
  • ace_sis_l2, inst_id='256sec', tag='base'
  • ace_swepam_l2, inst_id='164sec', tag='base'
  • cnofs_vefi, tag='dc_b'
  • de2_fpi
  • de2_lang
  • de2_nacs
  • de2_rpa
  • de2_vefi, tag=''
  • icon_euv
  • icon_fuv, tag='day'
  • icon_fuv, tag='night'
  • icon_mighti, inst_id='a', tag='los_wind_green'
  • icon_mighti, inst_id='a', tag='los_wind_red'
  • icon_mighti, inst_id='a', tag='temperature'
  • icon_mighti, inst_id='b', tag='los_wind_green'
  • icon_mighti, inst_id='b', tag='los_wind_red'
  • icon_mighti, inst_id='b', tag='temperature'
  • maven_insitu_kp
  • ses14_gold, tag='tdisk'
  • ses14_gold, tag='o2den'
  • timed_guvi, inst_id='high_res', tag='sdr_spectrograph'
  • timed_guvi, inst_id='low_res', tag='sdr_spectrograph'

Some of these are cause by an issue with CDAS download routines, where the localize function adds a timestamp of '00:00:00' to a date, meaning only the first day will be returned rather than two days. BUGFIX downstream of #198

if isinstance(start, pds._libs.tslibs.timestamps.Timestamp):
start = start.tz_localize('utc')
stop = stop.tz_localize('utc')

Using pysat.utils.time.filter_datetime_inputinstead fixes this problem for some of the data products. Fixes 15 of the above errors

There's also an issue if data files do not begin exactly at zero seconds. CDAS WS interprets the end date as a beginning of day, so if data starts at 00:00:01 it will ignore the file. Adding 86399 seconds to the end date fixes this for another 5 instruments.

DE2 VEFI fixed by #217. New tests run when those changes are pulled in

TODO:

  • fix icon tests
  • version bump for other insts

The remaining issues (icon_euv, maven_insitu_kp) are basically the same => the datasets have covariance matrices with duplicate dimension names, ie dims = (time, compno_3, compno_3). Will write up a new issue.