BUG: bad datetimes in DE2 RPA data result in non-monotonic series
Opened this issue · 0 comments
jklenzing commented
Description
Some of the data files for DE2 RPA improperly store the time epoch. This results in a non-monotonic index
To Reproduce this bug:
Steps to reproduce the behavior:
import pysat
rpa = pysat.Instrument('de2', 'rpa', strict_time_flag=False)
rpa.load(1983, 29)
rpa.data
yields
dataQuality x y z ionTemperature ... glat glon ilat mlt alt
1983-01-29 23:59:55.969 0 411.0 105.610001 310.600006 1463.0 ... 68.669998 30.740000 65.360001 3.23 350.589996
1983-01-29 23:59:59.969 60 126.0 83.610001 343.600006 1385.0 ... 68.940002 30.730000 65.610001 3.29 350.850006
1983-01-29 00:00:03.967 60 226.0 92.610001 337.600006 1357.0 ... 69.199997 30.709999 65.860001 3.32 351.109985
1983-01-29 00:00:07.967 60 279.0 103.610001 402.600006 1343.0 ... 69.459999 30.700001 66.110001 3.34 351.369995
1983-01-29 00:00:11.967 0 276.0 91.610001 440.600006 1505.0 ... 69.720001 30.680000 66.360001 3.36 351.619995
... ... ... ... ... ... ... ... ... ... ... ...
1983-01-29 19:10:17.039 0 1.0 12.610000 -95.599998 1131.0 ... -33.669998 102.120003 46.459999 1.55 252.960007
1983-01-29 19:10:20.039 0 -43.0 34.610001 -103.599998 1041.0 ... -33.470001 102.110001 46.230000 1.55 253.009995
1983-01-29 19:10:21.039 0 11.0 45.610001 -93.599998 1136.0 ... -33.400002 102.099998 46.160000 1.55 253.020004
1983-01-29 19:10:24.039 0 0.0 23.610001 4544.600098 993.0 ... -33.200001 102.089996 45.930000 1.55 253.070007
1983-01-29 19:10:25.039 0 0.0 23.610001 4587.600098 1112.0 ... -33.130001 102.089996 45.849998 1.55 253.089996
Manual inspection of the data here and on the preceeding day confirm that the first two data points should be for 1983-01-28. Loading the file with cdflib independently of pysat and plotting the raw epoch value confirms this is an issue with the file, not the datetime conversion in either package.
Test configuration
- OS: Ventura 13.6.1
- Python 3.10.9
Additional context
Since this is outside of our control, an optional routine to scrub non-monotonic files. Over the 1.5 months of 1983 data inspected, this affects four data points (including the two shown here), so dropping these points is a potential fix.