Load-and-apply not working (TypeError: 'NoneType' object does not support item assignment)
Closed this issue · 4 comments
IanHeywood commented
Could be a bug, could be user error. I duplicated the parset used for the solving and:
- Set
--out-mode ar - Cleared
save-toand specifiedload-fromin the Jones terms - Set
solvable = 0in the Jones terms - Cleared
save-toin the[bbc]section
This is using the stimela 1.5.4 container, which contains cubical 1.5.3.
Singularity cubical-1.5.4.simg:/ndidi/ianh/CDFS_L/CDFS_2_4/peeling-apply.cc-out> gocubical --version
gocubical version 1.5.3
INFO 13:07:11 - cc [0.1 0.1 0.0Gb] set global console verbosity level 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] started /usr/local/bin/gocubical parsets/cubical/apply_peelsols.parset --data-ms=1561266559_sdp_l0_1024ch_CDFS_2_
4.ms
INFO 13:07:11 - main [0.1 0.1 0.0Gb] output basename explicitly set to peeling-apply.cc-out/peeling-apply, --out-dir setting ignored
INFO 13:07:11 - main [0.1 0.1 0.0Gb] creating new output directory peeling-apply.cc-out
INFO 13:07:11 - main [0.1 0.1 0.0Gb] Selected Options:
INFO 13:07:11 - main [0.1 0.1 0.0Gb] [data] Visibility data options
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - ms ................................................ = 1561266559_sdp_l0_1024ch_CDFS_2_4.ms
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - column ............................................ = DATA
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - time-chunk ........................................ = 72
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - freq-chunk ........................................ = 256
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - rebin-time ........................................ = 1
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - rebin-freq ........................................ = 1
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - chunk-by .......................................... = SCAN_NUMBER
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - chunk-by-jump ..................................... = 1
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - single-chunk ...................................... =
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - single-tile ....................................... = -1
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - normalize ......................................... = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] [sel] Data selection options
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - field ............................................. = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - ddid .............................................. = None
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - taql .............................................. =
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - chan .............................................. =
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - diag .............................................. = False
INFO 13:07:11 - main [0.1 0.1 0.0Gb] [out] Options for output products
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - dir ............................................... = peeling-apply
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - name .............................................. = peeling-apply.cc-out/peeling-apply
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - overwrite ......................................... = False
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - backup ............................................ = 1
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - mode .............................................. = ar
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - apply-solver-flags ................................ = True
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - column ............................................ = CORRECTED_DATA
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - derotate .......................................... = None
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - model-column ...................................... =
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - weight-column ..................................... =
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - reinit-column ..................................... = False
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - subtract-model .................................... = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - subtract-dirs ..................................... = 1
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - correct-dir ....................................... = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - plots ............................................. = 1
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - casa-gaintables ................................... = True
INFO 13:07:11 - main [0.1 0.1 0.0Gb] [model] Calibration model options
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - list .............................................. = MODEL_DATA+-DIR1_DATA:DIR1_DATA
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - ddes .............................................. = auto
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - beam-pattern ...................................... = None
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - beam-l-axis ....................................... = None
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - beam-m-axis ....................................... = None
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - feed-rotate ....................................... = auto
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - pa-rotate ......................................... = False
INFO 13:07:11 - main [0.1 0.1 0.0Gb] [montblanc] Montblanc simulation options
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - device-type ....................................... = CPU
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - dtype ............................................. = double
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - mem-budget ........................................ = 1024
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - verbosity ......................................... = WARNING
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - threads ........................................... = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - pa-rotate ......................................... = None
INFO 13:07:11 - main [0.1 0.1 0.0Gb] [weight] Weighting options
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - column ............................................ = WEIGHT_SPECTRUM
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - fill-offdiag ...................................... = False
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - legacy-v1-2 ....................................... = False
INFO 13:07:11 - main [0.1 0.1 0.0Gb] [flags] General flagging options
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - apply ............................................. = -cubical
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - auto-init ......................................... = legacy
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - save .............................................. = cubical
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - save-legacy ....................................... = auto
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - reinit-bitflags ................................... = False
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - warn-thr .......................................... = 0.3
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - see-no-evil ....................................... = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] [degridding] Options for the degridder. Only in use when predicting from DicoModels using DDFacet
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - OverS ............................................. = 11
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - Support ........................................... = 7
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - Nw ................................................ = 100
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - wmax .............................................. = 0.0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - Padding ........................................... = 1.7
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - NDegridBand ....................................... = 16
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - MaxFacetSize ...................................... = 0.25
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - MinNFacetPerAxis .................................. = 1
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - NProcess .......................................... = 8
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - BeamModel ......................................... = None
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - NBand ............................................. = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - FITSFile .......................................... = beam_$(corr)_$(reim).fits
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - FITSFeed .......................................... = None
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - FITSFeedSwap ...................................... = False
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - DtBeamMin ......................................... = 5.0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - FITSParAngleIncDeg ................................ = 5.0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - FITSLAxis ......................................... = -X
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - FITSMAxis ......................................... = Y
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - FITSVerbosity ..................................... = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - FeedAngle ......................................... = 0.0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - FlipVisibilityHands ............................... = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] [postmortem] Options for "postmortem" flagging based on solution statistics
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - enable ............................................ = False
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - tf-chisq-median ................................... = 1.2
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - tf-np-median ...................................... = 0.5
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - time-density ...................................... = 0.5
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - chan-density ...................................... = 0.5
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - ddid-density ...................................... = 0.5
INFO 13:07:11 - main [0.1 0.1 0.0Gb] [madmax] Options for the "Mad Max" flagger
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - enable ............................................ = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - residuals ......................................... = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - estimate .......................................... = corr
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - diag .............................................. = True
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - offdiag ........................................... = True
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - threshold ......................................... = 10
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - global-threshold .................................. = 12
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - plot .............................................. = 1
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - plot-frac-above ................................... = 0.01
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - plot-bl ........................................... =
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - flag-ant .......................................... = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - flag-ant-thr ...................................... = 5
INFO 13:07:11 - main [0.1 0.1 0.0Gb] [sol] Solution options which apply at the solver level
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - jones ............................................. = ['G', 'DE']
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - precision ......................................... = 32
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - delta-g ........................................... = 1e-06
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - delta-chi ......................................... = 1e-06
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - chi-int ........................................... = 5
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - last-rites ........................................ = True
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - stall-quorum ...................................... = 0.99
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - term-iters ........................................ = [50, 50]
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - flag-divergence ................................... = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - min-bl ............................................ = 150.0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - max-bl ............................................ = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - subset ............................................ =
INFO 13:07:11 - main [0.1 0.1 0.0Gb] [bbc] Options for baseline-based corrections (a.k.a. BBCs, a.k.a. interferometer gains).
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - load-from ......................................... =
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - compute-2x2 ....................................... = False
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - apply-2x2 ......................................... = False
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - save-to ........................................... =
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - per-chan .......................................... = True
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - plot .............................................. = True
INFO 13:07:11 - main [0.1 0.1 0.0Gb] [dist] Parallelization and distribution options
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - ncpu .............................................. = 8
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - nworker ........................................... = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - nthread ........................................... = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - max-chunks ........................................ = 1
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - min-chunks ........................................ = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - pin ............................................... = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - pin-io ............................................ = False
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - pin-main .......................................... = io
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - safe .............................................. = True
INFO 13:07:11 - main [0.1 0.1 0.0Gb] [log] Options related to logging
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - memory ............................................ = True
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - stats ............................................. = chi2:.3f
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - stats-warn ........................................ = chi2:10
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - boring ............................................ = True
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - append ............................................ = False
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - verbose ........................................... = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - file-verbose ...................................... = None
INFO 13:07:11 - main [0.1 0.1 0.0Gb] [debug] Debugging options for the discerning masochist
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - pdb ............................................... = False
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - panic-amplitude ................................... = 0.0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - stop-before-solver ................................ = False
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - escalate-warnings ................................. = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] [misc] Miscellaneous options
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - random-seed ....................................... = None
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - parset-version .................................... = 0.1
INFO 13:07:11 - main [0.1 0.1 0.0Gb] [g] Options for G-Jones term
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - solvable .......................................... = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - type .............................................. = complex-2x2
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - delay-estimate-pad-factor ......................... = 8
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - load-from ......................................... = peeling.cc-out/peeling-G-field_0-ddid_None.parmdb
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - xfer-from ......................................... =
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - save-to ........................................... =
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - dd-term ........................................... = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - fix-dirs .......................................... =
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - update-type ....................................... = full
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - estimate-pzd ...................................... = False
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - time-int .......................................... = 16
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - freq-int .......................................... = 256
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - max-prior-error ................................... = 0.3
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - max-post-error .................................... = 0.3
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - low-snr-warn ...................................... = 75
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - high-gain-var-warn ................................ = 30
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - clip-low .......................................... = 0.1
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - clip-high ......................................... = 10
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - clip-after ........................................ = 5
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - max-iter .......................................... = 50
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - pin-slope-iters ................................... = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - epsilon ........................................... = 1e-06
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - delta-chi ......................................... = 1e-06
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - conv-quorum ....................................... = 0.99
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - ref-ant ........................................... = None
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - prop-flags ........................................ = default
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - diag-only ......................................... = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - offdiag-only ...................................... = False
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - robust-cov ........................................ = compute
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - robust-scale ...................................... = 1
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - robust-npol ....................................... = 2
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - robust-int ........................................ = 1
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - robust-save-weights ............................... = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - estimate-delays ................................... = False
INFO 13:07:11 - main [0.1 0.1 0.0Gb] [de] Options for G-Jones term
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - solvable .......................................... = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - type .............................................. = complex-2x2
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - delay-estimate-pad-factor ......................... = 8
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - load-from ......................................... = peeling.cc-out/peeling-dE-field_0-ddid_None.parmdb
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - xfer-from ......................................... =
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - save-to ........................................... =
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - dd-term ........................................... = 1
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - fix-dirs .......................................... = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - update-type ....................................... = full
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - estimate-pzd ...................................... = False
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - time-int .......................................... = 72
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - freq-int .......................................... = 64
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - max-prior-error ................................... = 5.0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - max-post-error .................................... = 5.0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - low-snr-warn ...................................... = 75
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - high-gain-var-warn ................................ = 30
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - clip-low .......................................... = 0.0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - clip-high ......................................... = 1000000
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - clip-after ........................................ = 5
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - max-iter .......................................... = 50
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - pin-slope-iters ................................... = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - epsilon ........................................... = 1e-06
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - delta-chi ......................................... = 1e-06
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - conv-quorum ....................................... = 0.99
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - ref-ant ........................................... = None
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - prop-flags ........................................ = default
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - diag-only ......................................... = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - offdiag-only ...................................... = False
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - robust-cov ........................................ = compute
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - robust-scale ...................................... = 1
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - robust-npol ....................................... = 2
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - robust-int ........................................ = 1
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - robust-save-weights ............................... = 0
INFO 13:07:11 - main [0.1 0.1 0.0Gb] - estimate-delays ................................... = False
INFO 13:07:11 - main [0.1 0.1 0.0Gb] Enabling G,DE-Jones
INFO 13:07:11 - main [0.1 0.1 0.0Gb] mode: CorrectResiduals
INFO 13:07:11 - main [0.1 0.1 0.0Gb] solver is apply-only type: True
INFO 13:07:11 - main [0.1 0.1 0.0Gb] solver requires model: True
INFO 13:07:11 - data_handler [0.1 0.1 0.0Gb] reading MS 1561266559_sdp_l0_1024ch_CDFS_2_4.ms
INFO 13:07:11 - data_handler [0.1 0.1 0.0Gb] detected linear (xy) feeds
INFO 13:07:11 - data_handler [0.1 0.1 0.0Gb] fields are *0: CDFS_2_4
INFO 13:07:11 - data_handler [0.1 0.1 0.0Gb] MS contains 1 spectral windows
INFO 13:07:11 - data_handler [0.1 0.1 0.0Gb] max freq chunk size is 256 channels and/or -- MHz
INFO 13:07:11 - data_handler [0.1 0.1 0.0Gb] DDID 0: 1024/1024 channels selected
INFO 13:07:11 - data_handler [0.1 0.1 0.0Gb] found 4 frequency chunks: 0 256 512 768 1024
INFO 13:07:12 - data_handler [0.2 0.2 0.0Gb] applying TAQL query 'FIELD_ID == 0 && DATA_DESC_ID IN [0]' (6148044/6148044 rows selected)
INFO 13:07:12 - data_handler [0.2 0.2 0.0Gb] 62 antennas, 6148044 rows, 1/1 DDIDs, 3148 timeslots, 4 corrs
INFO 13:07:12 - data_handler [0.2 0.2 0.0Gb] DDID central frequencies are at 1.28 GHz
INFO 13:07:12 - data_handler [0.2 0.2 0.0Gb] Input model feed rotation enabled, PA rotation disabled
INFO 13:07:12 - data_handler [0.2 0.2 0.0Gb] Output visibilities derotation enabled
INFO 13:07:12 - parallactic_machin [0.2 0.2 0.0Gb] Initializing new parallactic angle machine for 62 ECEF positions
INFO 13:07:12 - parallactic_machin [0.2 0.2 0.0Gb] Conversion epoch is J2000
INFO 13:07:12 - parallactic_machin [0.2 0.2 0.0Gb] Selecting 'linear' feed bases
INFO 13:07:12 - data_handler [0.2 0.2 0.0Gb] Using 1 model(s) for 2 directions(s)
INFO 13:07:12 - data_handler [0.2 0.2 0.0Gb] model 0 (weight WEIGHT_SPECTRUM):
INFO 13:07:12 - data_handler [0.2 0.2 0.0Gb] direction 0: MODEL_DATA-DIR1_DATA
INFO 13:07:12 - data_handler [0.2 0.2 0.0Gb] direction 1: DIR1_DATA
INFO 13:07:12 - main [0.2 0.2 0.0Gb] subtraction directions set to [1]
INFO 13:07:12 - gain_machine [0.2 0.2 0.0Gb] G solutions will be initialized from peeling.cc-out/peeling-G-field_0-ddid_None.parmdb
INFO 13:07:12 - param_db [0.2 0.2 0.0Gb] reading peeling.cc-out/peeling-G-field_0-ddid_None.parmdb in fragmented mode
INFO 13:07:12 - param_db [0.2 0.2 0.0Gb] loading G:gain, shape 1x224x4x62x2x2
INFO 13:07:12 - param_db [0.2 0.2 0.0Gb] loading G:gain.err, shape 1x224x4x62x2x2
INFO 13:07:12 - gain_machine [0.2 0.2 0.0Gb] dE solutions will be initialized from peeling.cc-out/peeling-dE-field_0-ddid_None.parmdb
INFO 13:07:12 - param_db [0.2 0.2 0.0Gb] reading peeling.cc-out/peeling-dE-field_0-ddid_None.parmdb in fragmented mode
INFO 13:07:12 - param_db [0.2 0.2 0.0Gb] loading dE:gain, shape 2x49x16x62x2x2
INFO 13:07:12 - param_db [0.2 0.2 0.0Gb] loading dE:gain.err, shape 2x49x16x62x2x2
INFO 13:07:12 - main [0.2 0.2 0.0Gb] multi-process mode: 7+1 workers, single thread
INFO 13:07:12 - main [0.2 0.2 0.0Gb] defining chunks (time 72, freq 256, also when SCAN_NUMBER jumps > 1.0)
INFO 13:07:13 - data_handler [0.6 0.6 0.0Gb] read indexing columns (6148044 total rows)
INFO 13:07:14 - data_handler [0.7 0.7 0.0Gb] built timeslot index (3148 unique timestamps)
INFO 13:07:14 - data_handler [0.7 0.7 0.0Gb] max chunk size is 72 timeslots and/or -- seconds
INFO 13:07:22 - data_handler [0.8 0.8 0.0Gb] found 49 time chunks: 0:0:0 1:140616:72 2:281232:144 3:421848:216 4:562464:288 5:703080:360 6:843696:432 7:8768
97:449 8:1017513:521 9:1158129:593 10:1298745:665 11:1439361:737 12:1579977:809 13:1720593:881 14:1753794:898 15:1894410:970 16:2035026:1042 17:2175642:1114 18:2316258:1
186 19:2456874:1258 20:2597490:1330 21:2632644:1348 22:2773260:1420 23:2913876:1492 24:3054492:1564 25:3195108:1636 26:3335724:1708 27:3476340:1780 28:3511494:1798 29:36
52110:1870 30:3792726:1942 31:3933342:2014 32:4073958:2086 33:4214574:2158 34:4355190:2230 35:4390344:2248 36:4530960:2320 37:4671576:2392 38:4812192:2464 39:4952808:253
6 40:5093424:2608 41:5234040:2680 42:5269194:2698 43:5409810:2770 44:5550426:2842 45:5691042:2914 46:5831658:2986 47:5972274:3058 48:6112890:3130 3148
INFO 13:07:22 - data_handler [0.8 0.8 0.0Gb] generated 49 row chunks based on time and DDID
INFO 13:07:22 - data_handler [0.8 0.8 0.0Gb] row chunks yield 49 potential tiles
INFO 13:07:23 - data_handler [1.1 1.1 0.0Gb] coarsening this to 49 tiles (max 4 chunks per tile, based on 7/1 requested)
INFO 13:07:23 - wisdom [1.1 1.1 0.0Gb] Detected a total of 503.57GiB of system memory.
INFO 13:07:23 - wisdom [1.1 1.1 0.0Gb] Per-solver (worker) memory use estimated at 33.18GiB: 6.59% of total system memory.
INFO 13:07:23 - wisdom [1.1 1.1 0.0Gb] Peak I/O memory use estimated at 47.75GiB: 9.48% of total system memory.
INFO 13:07:23 - wisdom [1.1 1.1 0.0Gb] Total peak memory usage estimated at 279.97GiB: 55.60% of total system memory.
INFO 13:07:26 - data_handler [1.1 1.1 0.0Gb] the MS appears to have a properly formed BITFLAG column
INFO 13:07:26 - data_handler [1.1 1.1 0.0Gb] bitflag 'legacy' already exists, will not auto-fill
INFO 13:07:26 - data_handler [1.1 1.1 0.0Gb] BITFLAG column defines the following flagsets: legacy:1 cubical:2
INFO 13:07:26 - data_handler [1.1 1.1 0.0Gb] will exclude flagset cubical
INFO 13:07:26 - data_handler [1.1 1.1 0.0Gb] applying BITFLAG mask 1 to input data
INFO 13:07:26 - data_handler [1.1 1.1 0.0Gb] will save output flags into BITFLAG 'cubical' (2), and into FLAG/FLAG_ROW
INFO 13:07:26 - main [1.1 1.1 0.0Gb] waiting for I/O on tile 0/49
INFO 13:07:26 - main [io] [1.1 1.1 0.0Gb] loading tile 0/49
INFO 13:07:26 - ms_tile [io] [1.1 1.1 0.0Gb] tile 0/49: reading MS rows 0~140615
INFO 13:07:26 - data_handler [io] [1.1 1.1 0.0Gb] reading DATA
INFO 13:07:41 - ms_tile [io] [9.6/10.6 9.1/9.7 0.0Gb] applying solvable baseline cutoff deselects 10630 rows
INFO 13:07:41 - ms_tile [io] [9.6/10.6 9.1/9.7 0.0Gb] 7.56% visibilities deselected via specificed subset and/or baseline cutoffs
INFO 13:07:41 - data_handler [io] [9.6/10.6 9.1/9.7 0.0Gb] reading BITFLAG
INFO 13:07:59 - ms_tile [io] [14.3/15.4 13.9/14.5 0.0Gb] 54.77% input visibilities flagged and/or deselected
INFO 13:08:05 - data_handler [io] [15.7/16.7 15.2/15.9 7.6Gb] reading MODEL_DATA
INFO 13:09:08 - data_handler [io] [24.9/26.0 24.5/25.1 11.9Gb] reading DIR1_DATA
INFO 13:10:19 - main [io] [3.7/4.8 3.3/4.0 16.2Gb] I/O job(s) complete
INFO 13:10:20 - main [io] [3.7/4.8 3.3/4.0 16.2Gb] loading tile 1/49
INFO 13:10:20 - ms_tile [io] [3.7/4.8 3.3/4.0 16.2Gb] tile 1/49: reading MS rows 140616~281231
INFO 13:10:20 - data_handler [io] [3.7/4.8 3.3/4.0 16.2Gb] reading DATA
INFO 13:10:20 - main [1.0/4.8 0.6/4.0 16.2Gb] submitting solver jobs for tile 0/49
INFO 13:10:35 - ms_tile [io] [12.3/119.0 11.6/63.8 16.2Gb] applying solvable baseline cutoff deselects 10379 rows
INFO 13:10:36 - ms_tile [io] [12.3/119.0 11.6/63.8 16.2Gb] 7.38% visibilities deselected via specificed subset and/or baseline cutoffs
INFO 13:10:36 - data_handler [io] [12.3/119.0 11.6/63.8 16.2Gb] reading BITFLAG
INFO 13:10:40 - gain_machine [x03] [11.1/92.2 10.3/73.9 16.2Gb] D0T0F2: loading G:gain from peeling.cc-out/peeling-G-field_0-ddid_None.parmdb
INFO 13:10:40 - gain_machine [x03] [11.1/92.2 10.3/73.9 16.2Gb] D0T0F2: 99.68% valid G:gain slots populated
INFO 13:10:40 - gain_machine [x03] [11.1/92.2 10.3/73.9 16.2Gb] D0T0F2: loading dE:gain from peeling.cc-out/peeling-dE-field_0-ddid_None.parmdb
ERROR 13:10:40 - solver [x03] [11.1/92.2 10.3/73.9 16.2Gb] Solver for tile 0 chunk D0T0F2 failed with exception: 'NoneType' object does not support item
assignment
INFO 13:10:40 - solver [x03] [11.1/92.2 10.3/73.9 16.2Gb] Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/cubical/solver.py", line 894, in run_solver
gm = vdm.gm = gm_factory.create_machine(vdm.weighted_obser, n_dir, n_mod, chunk_ts, chunk_fs, label)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/abstract_machine.py", line 977, in create_machine
gm._load_solutions(self._init_sols, self.grid)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/jones_chain_machine.py", line 166, in _load_solutions
term._load_solutions(init_sols, full_grid)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/abstract_machine.py", line 703, in _load_solutions
self.import_solutions(sols)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/complex_2x2_machine.py", line 92, in import_solutions
super(Complex2x2Gains, self).import_solutions(soldict)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/interval_gain_machine.py", line 404, in import_solutions
self.restrict_solution(self.gains)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/complex_2x2_machine.py", line 255, in restrict_solution
super(Complex2x2Gains, self).restrict_solution(gains)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/interval_gain_machine.py", line 867, in restrict_solution
self.posterior_gain_error[idir, ...] = 0
TypeError: 'NoneType' object does not support item assignment
INFO 13:10:41 - gain_machine [x01] [11.1/59.1 10.3/52.1 16.2Gb] D0T0F0: loading G:gain from peeling.cc-out/peeling-G-field_0-ddid_None.parmdb
INFO 13:10:41 - gain_machine [x01] [11.1/59.1 10.3/52.1 16.2Gb] D0T0F0: loading dE:gain from peeling.cc-out/peeling-dE-field_0-ddid_None.parmdb
ERROR 13:10:41 - solver [x01] [11.1/59.1 10.3/52.1 16.2Gb] Solver for tile 0 chunk D0T0F0 failed with exception: 'NoneType' object does not support item
assignment
INFO 13:10:41 - solver [x01] [11.1/59.1 10.3/52.1 16.2Gb] Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/cubical/solver.py", line 894, in run_solver
gm = vdm.gm = gm_factory.create_machine(vdm.weighted_obser, n_dir, n_mod, chunk_ts, chunk_fs, label)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/abstract_machine.py", line 977, in create_machine
gm._load_solutions(self._init_sols, self.grid)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/jones_chain_machine.py", line 166, in _load_solutions
term._load_solutions(init_sols, full_grid)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/abstract_machine.py", line 703, in _load_solutions
self.import_solutions(sols)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/complex_2x2_machine.py", line 92, in import_solutions
super(Complex2x2Gains, self).import_solutions(soldict)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/interval_gain_machine.py", line 404, in import_solutions
self.restrict_solution(self.gains)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/complex_2x2_machine.py", line 255, in restrict_solution
super(Complex2x2Gains, self).restrict_solution(gains)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/interval_gain_machine.py", line 867, in restrict_solution
self.posterior_gain_error[idir, ...] = 0
TypeError: 'NoneType' object does not support item assignment
INFO 13:10:42 - gain_machine [x02] [11.1/63.9 10.3/56.9 16.2Gb] D0T0F1: loading G:gain from peeling.cc-out/peeling-G-field_0-ddid_None.parmdb
INFO 13:10:42 - gain_machine [x02] [11.1/63.9 10.3/56.9 16.2Gb] D0T0F1: loading dE:gain from peeling.cc-out/peeling-dE-field_0-ddid_None.parmdb
ERROR 13:10:42 - solver [x02] [11.1/63.9 10.3/56.9 16.2Gb] Solver for tile 0 chunk D0T0F1 failed with exception: 'NoneType' object does not support item
assignment
INFO 13:10:42 - solver [x02] [11.1/63.9 10.3/56.9 16.2Gb] Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/cubical/solver.py", line 894, in run_solver
gm = vdm.gm = gm_factory.create_machine(vdm.weighted_obser, n_dir, n_mod, chunk_ts, chunk_fs, label)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/abstract_machine.py", line 977, in create_machine
gm._load_solutions(self._init_sols, self.grid)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/jones_chain_machine.py", line 166, in _load_solutions
term._load_solutions(init_sols, full_grid)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/abstract_machine.py", line 703, in _load_solutions
self.import_solutions(sols)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/complex_2x2_machine.py", line 92, in import_solutions
super(Complex2x2Gains, self).import_solutions(soldict)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/interval_gain_machine.py", line 404, in import_solutions
self.restrict_solution(self.gains)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/complex_2x2_machine.py", line 255, in restrict_solution
super(Complex2x2Gains, self).restrict_solution(gains)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/interval_gain_machine.py", line 867, in restrict_solution
self.posterior_gain_error[idir, ...] = 0
TypeError: 'NoneType' object does not support item assignment
INFO 13:10:42 - gain_machine [x04] [11.1/64.9 10.3/57.9 16.2Gb] D0T0F3: loading G:gain from peeling.cc-out/peeling-G-field_0-ddid_None.parmdb
INFO 13:10:42 - gain_machine [x04] [11.1/64.9 10.3/57.9 16.2Gb] D0T0F3: 99.35% valid G:gain slots populated
INFO 13:10:42 - gain_machine [x04] [11.1/64.9 10.3/57.9 16.2Gb] D0T0F3: loading dE:gain from peeling.cc-out/peeling-dE-field_0-ddid_None.parmdb
INFO 13:10:42 - gain_machine [x04] [11.1/64.9 10.3/57.9 16.2Gb] D0T0F3: 75.00% valid dE:gain slots populated
ERROR 13:10:42 - solver [x04] [11.1/64.9 10.3/57.9 16.2Gb] Solver for tile 0 chunk D0T0F3 failed with exception: 'NoneType' object does not support item
assignment
INFO 13:10:42 - solver [x04] [11.1/64.9 10.3/57.9 16.2Gb] Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/cubical/solver.py", line 894, in run_solver
gm = vdm.gm = gm_factory.create_machine(vdm.weighted_obser, n_dir, n_mod, chunk_ts, chunk_fs, label)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/abstract_machine.py", line 977, in create_machine
gm._load_solutions(self._init_sols, self.grid)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/jones_chain_machine.py", line 166, in _load_solutions
term._load_solutions(init_sols, full_grid)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/abstract_machine.py", line 703, in _load_solutions
self.import_solutions(sols)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/complex_2x2_machine.py", line 92, in import_solutions
super(Complex2x2Gains, self).import_solutions(soldict)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/interval_gain_machine.py", line 404, in import_solutions
self.restrict_solution(self.gains)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/complex_2x2_machine.py", line 255, in restrict_solution
super(Complex2x2Gains, self).restrict_solution(gains)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/interval_gain_machine.py", line 867, in restrict_solution
self.posterior_gain_error[idir, ...] = 0
TypeError: 'NoneType' object does not support item assignment
INFO 13:10:53 - ms_tile [io] [16.6/64.8 15.9/57.8 16.2Gb] 51.13% input visibilities flagged and/or deselected
INFO 13:10:59 - data_handler [io] [18.0/66.2 17.2/59.2 23.9Gb] reading MODEL_DATA
INFO 13:12:01 - data_handler [io] [27.2/75.4 26.5/68.4 28.2Gb] reading DIR1_DATA
INFO 13:13:13 - main [io] [6.0/54.2 5.3/47.2 32.5Gb] I/O job(s) complete
INFO 13:13:13 - main [1.0/4.8 1.0/4.0 32.5Gb] Exiting with exception: TypeError('NoneType' object does not support item assignment)
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3.6/concurrent/futures/process.py", line 175, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/usr/local/lib/python3.6/dist-packages/cubical/solver.py", line 894, in run_solver
gm = vdm.gm = gm_factory.create_machine(vdm.weighted_obser, n_dir, n_mod, chunk_ts, chunk_fs, label)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/abstract_machine.py", line 977, in create_machine
gm._load_solutions(self._init_sols, self.grid)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/jones_chain_machine.py", line 166, in _load_solutions
term._load_solutions(init_sols, full_grid)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/abstract_machine.py", line 703, in _load_solutions
self.import_solutions(sols)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/complex_2x2_machine.py", line 92, in import_solutions
super(Complex2x2Gains, self).import_solutions(soldict)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/interval_gain_machine.py", line 404, in import_solutions
self.restrict_solution(self.gains)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/complex_2x2_machine.py", line 255, in restrict_solution
super(Complex2x2Gains, self).restrict_solution(gains)
File "/usr/local/lib/python3.6/dist-packages/cubical/machines/interval_gain_machine.py", line 867, in restrict_solution
self.posterior_gain_error[idir, ...] = 0
TypeError: 'NoneType' object does not support item assignment
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/cubical/main.py", line 572, in main
stats_dict = workers.run_process_loop(ms, tile_list, load_model, single_chunk, solver_type, solver_opts, debug_opts, out_opts)
File "/usr/local/lib/python3.6/dist-packages/cubical/workers.py", line 217, in run_process_loop
return _run_multi_process_loop(ms, load_model, solver_type, solver_opts, debug_opts, out_opts)
File "/usr/local/lib/python3.6/dist-packages/cubical/workers.py", line 303, in _run_multi_process_loop
stats = future.result()
File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
TypeError: 'NoneType' object does not support item assignment
JSKenyon commented
I believe the culprit is here. self.posterior_gain_error has not been initialized when import_solutions is called. import_solutions then calls restrict_update which attempts to modify self.posterior_gain_error, causing the sky to fall. @IanHeywood You can try out the issue-403 branch. Very rudimentary fix, so @o-smirnov might have a better idea.
o-smirnov commented
Yep, well caught -- looks like an old-standing bug, but only triggered when fix-directions is set -- I guess we haven't combined that with apply-only mode until now.
@JSKenyon's fix is perfectly good. Also, a workaround is to disable fix-directions when applying.
IanHeywood commented
Thanks both. 👍
JSKenyon commented
This should be fixed in the latest release - closing for now.