System hangs after running a while
rnunziata opened this issue · 1 comments
rnunziata commented
This may be related to memory issues that have been posted. I was running breakout and after a while it just hangs. When I try to restart from last checkpoint it does work but goes right into hang mode. was running RND Montezum but changed to run Breakout. I've attached compressed file of saved models
example output:
[Episode 707(3)] Step: 460 Reward: 16.0 Recent Reward: 17.17 Visited Room: [{}]
[Episode 705(1)] Step: 497 Reward: 17.0 Recent Reward: 16.69 Visited Room: [{}]
[Episode 702(2)] Step: 558 Reward: 12.0 Recent Reward: 16.64 Visited Room: [{}]
[Episode 692(5)] Step: 411 Reward: 15.0 Recent Reward: 16.45 Visited Room: [{}]
[Episode 729(0)] Step: 526 Reward: 18.0 Recent Reward: 16.85 Visited Room: [{}]
[Episode 685(4)] Step: 654 Reward: 16.0 Recent Reward: 16.24 Visited Room: [{}]
[Episode 703(2)] Step: 467 Reward: 16.0 Recent Reward: 16.7 Visited Room: [{}]
[Episode 728(6)] Step: 569 Reward: 19.0 Recent Reward: 17.59 Visited Room: [{}]
[Episode 708(3)] Step: 506 Reward: 17.0 Recent Reward: 17.09 Visited Room: [{}]
[Episode 706(1)] Step: 487 Reward: 16.0 Recent Reward: 16.61 Visited Room: [{}]
[Episode 730(0)] Step: 412 Reward: 11.0 Recent Reward: 16.86 Visited Room: [{}]
[Episode 693(5)] Step: 478 Reward: 16.0 Recent Reward: 16.42 Visited Room: [{}]
[Episode 707(1)] Step: 414 Reward: 15.0 Recent Reward: 16.67 Visited Room: [{}]
[Episode 729(6)] Step: 447 Reward: 16.0 Recent Reward: 17.6 Visited Room: [{}]
[Episode 709(3)] Step: 457 Reward: 16.0 Recent Reward: 17.15 Visited Room: [{}]
[Episode 686(4)] Step: 591 Reward: 20.0 Recent Reward: 16.25 Visited Room: [{}]
[Episode 704(2)] Step: 582 Reward: 19.0 Recent Reward: 16.72 Visited Room: [{}]
[Episode 694(5)] Step: 362 Reward: 10.0 Recent Reward: 16.44 Visited Room: [{}]
[Episode 731(0)] Step: 403 Reward: 15.0 Recent Reward: 16.82 Visited Room: [{}]
[Episode 730(6)] Step: 361 Reward: 6.0 Recent Reward: 17.52 Visited Room: [{}]
[Episode 708(1)] Step: 503 Reward: 17.0 Recent Reward: 16.75 Visited Room: [{}]
Starting Checkpoint:
Num Step: 128
Now Global Step :3136000
Ending Checkpoint:
[Episode 695(5)] Step: 437 Reward: 15.0 Recent Reward: 16.34 Visited Room: [{}]
[Episode 687(4)] Step: 538 Reward: 17.0 Recent Reward: 16.31 Visited Room: [{}]
[Episode 710(3)] Step: 604 Reward: 18.0 Recent Reward: 17.15 Visited Room: [{}]
[Episode 705(2)] Step: 4502 Reward: 6.0 Recent Reward: 16.62 Visited Room: [{}]
[Episode 732(0)] Step: 4502 Reward: 5.0 Recent Reward: 16.7 Visited Room: [{}]
[Episode 731(6)] Step: 4502 Reward: 0.0 Recent Reward: 17.3 Visited Room: [{}]
[Episode 709(1)] Step: 4502 Reward: 2.0 Recent Reward: 16.58 Visited Room: [{}]
[Episode 696(5)] Step: 4502 Reward: 2.0 Recent Reward: 16.2 Visited Room: [{}]
[Episode 688(4)] Step: 4502 Reward: 2.0 Recent Reward: 16.11 Visited Room: [{}]
[Episode 711(3)] Step: 4502 Reward: 2.0 Recent Reward: 16.99 Visited Room: [{}]
[Episode 706(2)] Step: 4502 Reward: 0.0 Recent Reward: 16.45 Visited Room: [{}]
[Episode 733(0)] Step: 4502 Reward: 0.0 Recent Reward: 16.53 Visited Room: [{}]
[Episode 732(6)] Step: 4502 Reward: 0.0 Recent Reward: 17.05 Visited Room: [{}]
[Episode 710(1)] Step: 4502 Reward: 0.0 Recent Reward: 16.42 Visited Room: [{}]
[Episode 697(5)] Step: 4502 Reward: 0.0 Recent Reward: 16.06 Visited Room: [{}]
[Episode 689(4)] Step: 4502 Reward: 0.0 Recent Reward: 16.01 Visited Room: [{}]
[Episode 712(3)] Step: 4502 Reward: 0.0 Recent Reward: 16.92 Visited Room: [{}]
Starting Checkpoint:
Num Step: 128
Now Global Step :3225600
Ending Checkpoint:
[Episode 707(2)] Step: 4502 Reward: 0.0 Recent Reward: 16.24 Visited Room: [{}]
[Episode 734(0)] Step: 4502 Reward: 0.0 Recent Reward: 16.27 Visited Room: [{}]
[Episode 733(6)] Step: 4502 Reward: 0.0 Recent Reward: 16.86 Visited Room: [{}]
[Episode 711(1)] Step: 4502 Reward: 0.0 Recent Reward: 16.27 Visited Room: [{}]
[Episode 698(5)] Step: 4502 Reward: 0.0 Recent Reward: 15.89 Visited Room: [{}]
[Episode 690(4)] Step: 4502 Reward: 0.0 Recent Reward: 15.75 Visited Room: [{}]
[Episode 713(3)] Step: 4502 Reward: 0.0 Recent Reward: 16.73 Visited Room: [{}]
[Episode 708(2)] Step: 4502 Reward: 0.0 Recent Reward: 16.07 Visited Room: [{}]
[Episode 735(0)] Step: 4502 Reward: 0.0 Recent Reward: 16.11 Visited Room: [{}]
[Episode 734(6)] Step: 4502 Reward: 0.0 Recent Reward: 16.71 Visited Room: [{}]
[Episode 712(1)] Step: 4502 Reward: 0.0 Recent Reward: 16.09 Visited Room: [{}]
[Episode 699(5)] Step: 4502 Reward: 0.0 Recent Reward: 15.72 Visited Room: [{}]
[Episode 691(4)] Step: 4502 Reward: 0.0 Recent Reward: 15.58 Visited Room: [{}]
[Episode 714(3)] Step: 4502 Reward: 0.0 Recent Reward: 16.54 Visited Room: [{}]
[Episode 709(2)] Step: 4502 Reward: 0.0 Recent Reward: 15.94 Visited Room: [{}]
[Episode 736(0)] Step: 4502 Reward: 0.0 Recent Reward: 15.97 Visited Room: [{}]
[Episode 735(6)] Step: 4502 Reward: 0.0 Recent Reward: 16.49 Visited Room: [{}]
[Episode 713(1)] Step: 4502 Reward: 0.0 Recent Reward: 15.87 Visited Room: [{}]
[Episode 700(5)] Step: 4502 Reward: 0.0 Recent Reward: 15.43 Visited Room: [{}]
[Episode 692(4)] Step: 4502 Reward: 0.0 Recent Reward: 15.34 Visited Room: [{}]
[Episode 715(3)] Step: 4502 Reward: 0.0 Recent Reward: 16.39 Visited Room: [{}]
Starting Checkpoint:
Num Step: 128
Now Global Step :3315200
rnunziata commented
Closing for now I may have introduced error .. will reopen if not the case