probcomp/bayeslite

CGPM analysis sometimes hangs on one model; workaround for recovering progress

doctorpangloss opened this issue · 0 comments

Occasionally analysis won't start for one of the initialized models. This means a checkpoint is never issued. As a workaround, I interrupt the multiprocessing execution and serialize the engine myself:

  1. Interrupt the analysis using jupyter's terminate button or killall -INT "python".
  2. Retrieve your generator_idfrom the bayesdb_generator table.
  3. Serialize the engine, while the collected data is still in memory:
generator_id = 1
cgpm = generator.bdb.metamodels['cgpm']
engine = cgpm._engine(generator.bdb, generator_id)
cgpm._serialize_engine(generator.bdb, generator_id, engine, True)