Cubical hanging: CRITICAL "'DD' D0T1F5 Significant gain variance in directions 2"

[Kincaidr]

Hi,

My cubical is hanging again, i have searched for any Runtime error in log file that resulted in the hanging from last time, could not find any. I am suspecting its one of these CRITICAL errors about "'DD' D0T1F5 Significant gain variance in directions 2" but i am not sure.

Full log file is attached below

ddcal_0.log

[o-smirnov]

No your trouble lies here:

INFO      11:59:43 - wisdom             [0.6/0.6 3.5/3.5 0.1Gb] Detected a total of 503.81GiB of system memory.
INFO      11:59:43 - wisdom             [0.6/0.6 3.5/3.5 0.1Gb] Per-solver (worker) memory use estimated at 53.71GiB: 10.66% of total system memory.
INFO      11:59:43 - wisdom             [0.6/0.6 3.5/3.5 0.1Gb] Peak I/O memory use estimated at 347.35GiB: 68.94% of total system memory.
INFO      11:59:43 - wisdom             [0.6/0.6 3.5/3.5 0.1Gb] Total peak memory usage estimated at 1690.13GiB: 335.47% of total system memory.

The cure is the usual, fewer workers, smaller chunks.

I also highly recommend enabling --dist-safe, so that it bails out up front on the memory check.