operands could not be broadcast together in get_mag_from_img

[javierggt]

This week there were a few failures in several OBSIDs. The underlying errors were like this:

ValueError: operands could not be broadcast together with shapes (20946,) (20947,) 

The one thing that changed right before this error was that commands-v2 was set to be the default, and I started running with the KADI_COMMANDS_VERSION environment variable set to 1. The error can be reproduced using this script:

from agasc.supplement.magnitudes import mag_estimate, star_obs_catalogs as cat
cat.load()
obs = cat.STARS_OBS[(cat.STARS_OBS['agasc_id'] == 109584104) & (cat.STARS_OBS['obsid'] == 5606)][0]
mag_estimate.get_telemetry(obs)

[javierggt]

This issue can be traced back to the following.

This script:

import numpy as np
from cheta import fetch


start, stop, slot = (221360666.38525, 221360676.76287502, 6)

msids = fetch.Msidset(
    [
        f"AOACIIR{slot}",
        f"AOACMAG{slot}",
    ],
    start,
    stop,
)

AOACIIR = msids[f"AOACIIR{slot}"]
AOACMAG = msids[f"AOACMAG{slot}"]

tmin = np.min([np.min(msids[m].times) for m in msids])

t1 = np.round((AOACMAG.times - tmin) / 0.25625) * 0.25625
t2 = np.round((AOACIIR.times - tmin) / 0.25625) * 0.25625
print(f"AOACMAG{slot} times", t1)
print(f"AOACIIR{slot} times", t2)

gives the following output:

AOACMAG6 times [ 0.     1.025  2.05   4.1    5.125  6.15   7.175  8.2    9.225 10.25 ]
AOACIIR6 times [ 0.     1.025  2.05   3.075  5.125  6.15   7.175  8.2    9.225 10.25 ]

and you can see that AOACMAG6 and AOACIIR6 usually match in time, except in the 4th sample.

If one uses slot=0 or slot=1 or instead, the times align. The others don't.

AOACMAG for slots 0 an 1 are supposed to arrive in the same minor frame as all AOACIIR<N>, and for other slots it arrives in minor frame later:

In [4]: [msids[f'AOACIIR{slot}'].Tloc['START_MINOR_FRAME'][0] for slot in range(8)]
Out[4]: [2, 2, 2, 2, 2, 2, 2, 2]

In [5]: [msids[f'AOACMAG{slot}'].Tloc['START_MINOR_FRAME'][0] for slot in range(8)]
Out[5]: [2, 2, 3, 3, 3, 3, 3, 3]

I would have expected that in MAUDE, but I thought things were aligned in CXC telemetry (and indeed they are in most cases).

I still do not know why this issue did not appear before and it appears now.

[javierggt]

That last comment is a good question for @taldcroft

[javierggt]

one thing to note is that in both cases there is a missing time, which presumably was marked bad, but it is different in different MSIDs so the times do not match.

This has to do with the latest change in cheta, where it now does not try to be smart when fetching multiple MSIDs with bad samples (sot/cheta/pull/235).