pandas.errors.InvalidIndexError
Closed this issue · 6 comments
Describe the bug
I have a calibrated dataset flagged around 63% from MeerKAT, the output is also on the markdown file attached. I need to correct for the phase on the target field as well as the phase slope.
quartical_error_proxima_cen_target.md
To Reproduce
I have attached both commands I ran to derive and apply the gains on the markdown file above
Expected behavior
I expect for the derived gain solutions to be applied to the relevant data columns on the measurement set. Such that when I image the corrected data I get improved images.
Version
quartical==0.2.0
Hmm, that looks very suspicious. I would definitely recommend updating to the latest version of QuartiCal.
My instinct is that your gain solutions may not have been written correctly. Are you running QuartiCal locally or somewhere I have access to (Rhodes/CHPC boxes)? If so, I can take a quick look.
Hmm, that looks very suspicious. I would definitely recommend updating to the latest version of QuartiCal.
My instinct is that your gain solutions may not have been written correctly. Are you running QuartiCal locally or somewhere I have access to (Rhodes/CHPC boxes)? If so, I can take a quick look.
I might agree the calibration step ended with a killed - I am running this on young.ru... on this path /home/bngcebetsha/cal_quartical
Kill your run please - your config is completely swamping memory.
The reason this happens is because you haven't set input_ms.time_chunks or input_ms.freq_chunks appropriately, and by default QC is trying to use all the cores on the node (which is something I should change). Setting dask.threads will let you control that. The error you originally reported is simply caused by the fact that the gains were never written (because the job died due to an OOM condition I would guess).
Edit: QC may have been fine but other work on the machine may have led to am OOM death.
Kill your run please - your config is completely swamping memory.
we have DynSpecMS to thank for this. Ok thanks @JSKenyon . I will try out your suggestions and update/close the issue when done.
Kill your run please - your config is completely swamping memory.
we have
DynSpecMSto thank for this. Ok thanks @JSKenyon . I will try out your suggestions and update/close the issue when done.
Great! Sorry - I was struggling to communicate with young so I was just guessing that it was QC to blame. But yeah, swap was full and that usually means game over. Hitting swap is bad enough.