Cubical stuck in process: waiting for I/O on tile
Closed this issue · 5 comments
viralp commented
Hi, My cubical job is not progressing and halt with the following message
INFO 02:28:08 - solver [14.7/235.8 22.7/689.8 69.1Gb] D0T22F3 (end solve) G: 50 iters, conv 0.00%, g/fl 9.17% [MISSING:17(14.17%) LOWSNR:11(9.17%)], d/fl 14.17%, PGE 0.175 (flagged LOWSNR:11); DD: 68 iters, conv 100.00%, d/fl 16.67%, PGE 0.0 0.0488; G: 50 iters, conv 0.00%, g/fl 9.17% [MISSING:17(14.17%) LOWSNR:11(9.17%)], d/fl 14.17%, PGE 0.172 (flagged LOWSNR:11); DD: 29 iters, conv 100.00%, d/fl 16.67%, PGE 0.0 0.0488, stall 85.78%, chi^2 0.6512 -> 0.6491 (1.011), noise 0.296 -> 0.236
INFO 02:28:18 - solver [14.8/235.8 22.7/689.9 69.2Gb] D0T22F3 has 50826 (0.14%) new data flags: gain flags LOWSNR:11(4.58%) (4.58% total)
INFO 02:28:19 - main [1.1/222.3 7.3/676.4 69.2Gb] finished processing tile 11/24
INFO 02:28:19 - main [1.1/222.3 7.3/676.4 69.2Gb] waiting for I/O on tile 12/24
o-smirnov commented
On stevie or where?
viralp commented
On the ike
bennahugo commented
Its a bug in the application of E-jones according to your log. I'm not sure
under which circumstances this becomes a non-one-dimensional array though?
INFO 02:15:33 - FITSBeamInterpolat [x01] [39.2/329.3 58.1/774.4
74.6Gb] computing beam sample times for 204960 timeslots
INFO 02:15:33 - FITSBeamInterpolat [x01] [39.2/329.3 58.1/774.4
74.6Gb] DtBeamMin=5.00 min results in 3 samples
INFO 02:15:33 - FITSBeamInterpolat [x01] [39.2/329.3 58.1/774.4
74.6Gb] FITSParAngleIncrement=5.00 deg results in 2 samples
INFO 02:15:33 - main [x01] [39.2/329.3 58.1/774.4
74.6Gb] I/O handler for load 12 save 10 failed with exception: operands
could not be broadcast together with shapes (3,) (2,)
INFO 02:15:33 - main [x01] [39.2/329.3 58.1/774.4
74.6Gb] Traceback (most recent call last):
File "/home/parekh/CubiCal/cubical/workers.py", line 469, in _io_handler
tile.load(load_model=load_model)
File "/home/parekh/CubiCal/cubical/data_handler/ms_tile.py", line 1014,
in load
model = subset.load_ddfacet_models(uvwco, loaded_models, model_source,
cluster, imod, idir, model_type)
File "/home/parekh/CubiCal/cubical/data_handler/ms_tile.py", line 163, in
load_ddfacet_models
self.time_col)
File "/home/parekh/CubiCal/cubical/degridder/DDFacetSim.py", line 564, in
simulate
provider=dh.degrid_opts["BeamModel"])
File "/home/parekh/CubiCal/cubical/degridder/DDFacetSim.py", line 364, in
__load_jones
t0 = sample_times - dt
ValueError: operands could not be broadcast together with shapes (3,) (2,)
…On Tue, Jan 5, 2021 at 5:50 AM Viral Parekh ***@***.***> wrote:
On the ike
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#427 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB4RE6VAJWGQKZDXPOYVQIDSYKD7LANCNFSM4VTK7EXA>
.
--
--
Benjamin Hugo
PhD. student,
Centre for Radio Astronomy Techniques and Technologies
Department of Physics and Electronics
Rhodes University
Junior software developer
Radio Astronomy Research Group
South African Radio Astronomy Observatory
Black River Business Park
Observatory
Cape Town
bennahugo commented
Actually I see the bug:
- the timestamp should be modified by an infinitesimal amount if the size
is leq 1, otherwise
- the last dt should be set to
dt = np.zeros_like(sample_times)
dt[:-1] = 0.5 * (sample_times[1:] - sample_times[:-1])
dt[-1] = dt[-2]
Could you make the modification Viral
…On Tue, Jan 5, 2021 at 9:05 AM Benna Hugo ***@***.***> wrote:
Its a bug in the application of E-jones according to your log. I'm not
sure under which circumstances this becomes a non-one-dimensional array
though?
INFO 02:15:33 - FITSBeamInterpolat [x01] [39.2/329.3 58.1/774.4
74.6Gb] computing beam sample times for 204960 timeslots
INFO 02:15:33 - FITSBeamInterpolat [x01] [39.2/329.3 58.1/774.4
74.6Gb] DtBeamMin=5.00 min results in 3 samples
INFO 02:15:33 - FITSBeamInterpolat [x01] [39.2/329.3 58.1/774.4
74.6Gb] FITSParAngleIncrement=5.00 deg results in 2 samples
INFO 02:15:33 - main [x01] [39.2/329.3 58.1/774.4
74.6Gb] I/O handler for load 12 save 10 failed with exception: operands
could not be broadcast together with shapes (3,) (2,)
INFO 02:15:33 - main [x01] [39.2/329.3 58.1/774.4
74.6Gb] Traceback (most recent call last):
File "/home/parekh/CubiCal/cubical/workers.py", line 469, in _io_handler
tile.load(load_model=load_model)
File "/home/parekh/CubiCal/cubical/data_handler/ms_tile.py", line 1014,
in load
model = subset.load_ddfacet_models(uvwco, loaded_models, model_source,
cluster, imod, idir, model_type)
File "/home/parekh/CubiCal/cubical/data_handler/ms_tile.py", line 163,
in load_ddfacet_models
self.time_col)
File "/home/parekh/CubiCal/cubical/degridder/DDFacetSim.py", line 564,
in simulate
provider=dh.degrid_opts["BeamModel"])
File "/home/parekh/CubiCal/cubical/degridder/DDFacetSim.py", line 364,
in __load_jones
t0 = sample_times - dt
ValueError: operands could not be broadcast together with shapes (3,) (2,)
On Tue, Jan 5, 2021 at 5:50 AM Viral Parekh ***@***.***>
wrote:
> On the ike
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <#427 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB4RE6VAJWGQKZDXPOYVQIDSYKD7LANCNFSM4VTK7EXA>
> .
>
--
--
Benjamin Hugo
PhD. student,
Centre for Radio Astronomy Techniques and Technologies
Department of Physics and Electronics
Rhodes University
Junior software developer
Radio Astronomy Research Group
South African Radio Astronomy Observatory
Black River Business Park
Observatory
Cape Town
--
--
Benjamin Hugo
PhD. student,
Centre for Radio Astronomy Techniques and Technologies
Department of Physics and Electronics
Rhodes University
Junior software developer
Radio Astronomy Research Group
South African Radio Astronomy Observatory
Black River Business Park
Observatory
Cape Town