werner-duvaud/muzero-general

ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.

hairinwind opened this issue · 1 comments

Here is the error

Last test reward: 500.00. Training step: 8449/10000. Played games: 13. Loss: 3.642021-12-31 01:53:03,200	WARNING worker.py:1245 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffff5c4f8fe95fa608ecc2d8b86b01000000 Worker ID: dea6505859cd12046772c48e3964b3de7cf585148696c1226c9b42bb Node ID: 9287903437a2cae5bac8e5a217534c9754a749ce7832fbb054cf0b16 Worker IP address: 10.138.0.2 Worker port: 42079 Worker PID: 8537
2021-12-31 01:53:03,438	ERROR worker.py:85 -- Unhandled error (suppress with RAY_IGNORE_UNHANDLED_ERRORS=1): ray::Reanalyse.reanalyse() (pid=8542, ip=10.138.0.2, repr=<replay_buffer.Reanalyse object at 0x7fbc5452cb50>)
  File "/home/yaoappemail/mymuzero/replay_buffer.py", line 332, in reanalyse
    replay_buffer.sample_game.remote(force_uniform=True)
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.
2021-12-31 01:53:03,439	ERROR worker.py:85 -- Unhandled error (suppress with RAY_IGNORE_UNHANDLED_ERRORS=1): ray::Trainer.continuous_update_weights() (pid=8543, ip=10.138.0.2, repr=<trainer.Trainer object at 0x7f7bb02fd890>)
  File "/home/yaoappemail/mymuzero/trainer.py", line 71, in continuous_update_weights
    index_batch, batch = ray.get(next_batch)
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.

How to fix it? Many thanks!

Did you solve this?