chime-experiment/ch_pipeline

Fix timing correction to work with most recent data

Closed this issue · 2 comments

jrs65 commented

At the moment the ApplyTimingCorrection task fails when running on more recent data that doesn't need it at all. We need to fix it such that it simply doesn't apply anything on this data.

Seth suggested that the best way to do this would be to use a dataflag to indicate the range of time over which a timing correction should in theory be applied, although it might be easier to just hard code the switch over data in the code.

Creating a dataflag using cdf:

https://bao.chimenet.ca/wiki/index.php/Cdf

Sample code that excludes days based on a dataflag:

base = Path("/path/to/files/")

available_lsds = np.array(sorted([int(path.stem.split("_")[-1]) for path in base.glob("*/sstream_lsd_*.h5")]))

dates = ctime.unix_to_datetime(ephemeris.csd_to_unix(np.array(available_lsds)))

core.connect()

bad_flags = [
    'needs_timing_correction',
]


flags = df.DataFlag.select().join(df.DataFlagType).where(df.DataFlagType.name << bad_flags)

flag_time_spans = [(f.type.name, f.start_time, f.finish_time) for f in flags]

lsd_time_range = ephemeris.csd_to_unix(available_lsds[:, np.newaxis] + np.array([[0, 1]]))
id: None
type: needs_timing_correction
start: 2018-10-11 14:00:00+00:00
finish: 2020-03-04 01:13:01+00:00
metadata: 
    {instrument: chime, user: Anja Boskovic}