BUG: `test_load_multiple_days` failures
Opened this issue · 5 comments
Description
pysat 3.2.0 has a new universal test: test_load_multiple_days
. Several instruments fail this test
To Reproduce this bug:
run pytest with pysat develop
Test configuration
- OS: Ventura
- Version: Python 3.10.9
- pysat develop
Additional context
-
ace_epam_l2
, inst_id='12sec', tag='base' -
ace_mag_l2
, inst_id='16sec', tag='base' -
ace_mag_l2
, inst_id='16sec', tag='key' -
ace_sis_l2
, inst_id='256sec', tag='base' -
ace_swepam_l2
, inst_id='164sec', tag='base' -
cnofs_vefi
, tag='dc_b' -
de2_fpi
-
de2_lang
-
de2_nacs
-
de2_rpa
-
de2_vefi
, tag='' -
icon_euv
-
icon_fuv
, tag='day' -
icon_fuv
, tag='night' -
icon_mighti
, inst_id='a', tag='los_wind_green' -
icon_mighti
, inst_id='a', tag='los_wind_red' -
icon_mighti
, inst_id='a', tag='temperature' -
icon_mighti
, inst_id='b', tag='los_wind_green' -
icon_mighti
, inst_id='b', tag='los_wind_red' -
icon_mighti
, inst_id='b', tag='temperature' -
maven_insitu_kp
-
ses14_gold
, tag='tdisk' -
ses14_gold
, tag='o2den' -
timed_guvi
, inst_id='high_res', tag='sdr_spectrograph' -
timed_guvi
, inst_id='low_res', tag='sdr_spectrograph'
Some of these are cause by an issue with CDAS download routines, where the localize function adds a timestamp of '00:00:00' to a date, meaning only the first day will be returned rather than two days. BUGFIX downstream of #198
pysatNASA/pysatNASA/instruments/methods/cdaweb.py
Lines 965 to 967 in 4f0df46
Using pysat.utils.time.filter_datetime_input
instead fixes this problem for some of the data products. Fixes 15 of the above errors
There's also an issue if data files do not begin exactly at zero seconds. CDAS WS interprets the end date as a beginning of day, so if data starts at 00:00:01 it will ignore the file. Adding 86399 seconds to the end date fixes this for another 5 instruments.
TODO:
- fix icon tests
- version bump for other insts
The remaining issues (icon_euv, maven_insitu_kp) are basically the same => the datasets have covariance matrices with duplicate dimension names, ie dims = (time, compno_3, compno_3)
. Will write up a new issue.