check arc and LED standards for drifts
sybenzvi opened this issue · 6 comments
Starting on 2023-03-28, the SOs began reporting a significant increase in warnings for the arc and LED flux checks in Nightwatch. The cause is estimated flux levels across most of the B and R cameras dropping 5% to 10% below the nominal values.
The warnings persisted for two days, then disappeared, then reappeared on 2023-04-01 and 04-02, then largely went away on 2023-04-03 and 2023-04-04.
In the survey ops call on 2023-04-03, @julienguy suggested the setup on the white spot during afternoon calibrations could be moving around. This was reported to Steve Lane and Bob Marshall in the #on-mountain-status slack channel, who confirmed the setup on the white spot is actually very consistent over time.
We need plots of the estimated fluxes vs. time to see if there are obvious drifts that might be correlated with environmental conditions. For a quick and dirty version, a script to plot the data out in the QA files is good enough. Ultimately, the capability to plot QA time series needs to be integrated directly into Nightwatch.
This plot represents a time series of a single calibration and petal, and seems like the drifts in late March and early April were unusual during the last couple of months.
We found a strong hint of negative correlation between the temperature registered in TAIRTEMP and the flux for the same calibration. Will investigate whether the trend holds for the rest of the arc and LED calibrations.
Interesting... if this is correct, we could add a temperature dependence to the calibration check. Let's see if the other calculated lines and LED fluxes behave similarly.
@jose-bermejo, now that we are in shutdown we should build the temperature database for the flux levels and make a new nightwatch release. I have an update to your conditions plotting script that I'll commit to a branch tomorrow.
@jose-bermejo, now that we are in shutdown we should build the temperature database for the flux levels and make a new nightwatch release. I have an update to your conditions plotting script that I'll commit to a branch tomorrow.
There is now a script that fits the line fluxes vs temperature (see #346) but let's wait for the restart before updating the calibration standards checks. We may end up having to regenerate the standards from scratch if the instrument throughput has changed by enough.