Async training gets stuck in single-agent cases.
Closed this issue · 4 comments
branch: vector-env
See the attached, there is only 1 training epoch for 100 rollout epochs.
script: python examples/async_simple.py
Have fixed it in the latest commit on vector-env branch(b65e5c0). The problem is raised from that updating parameters via aggregation of gradients does not respect settings of learning rates. Since it is a minor fix, the patch is directly pushed to the branch. It will be appreciated if you can recheck the problem.
Have fixed it in the latest commit on vector-env branch(b65e5c0). The problem is raised from that updating parameters via aggregation of gradients does not respect settings of learning rates. Since it is a minor fix, the patch is directly pushed to the branch. It will be appreciated if you can recheck the problem.
I'm curious as to why this can block the training process? It seems to me that the change made in the commit only affects the magnitude of logged gradients.
Fixed it in PR #14. The blocking is caused by unbalanced read/write access to OfflineDatasetServer, which has been alleviated in #5 by using Readers-writer lock.
Have fixed it in the latest commit on vector-env branch(b65e5c0). The problem is raised from that updating parameters via aggregation of gradients does not respect settings of learning rates. Since it is a minor fix, the patch is directly pushed to the branch. It will be appreciated if you can recheck the problem.
I'm curious as to why this can block the training process? It seems to me that the change made in the commit only affects the magnitude of logged gradients.
As told from my test cases, the false gradients generating logic caused the gradients explosion after several rounds of optimization, resulting in abnormal value like "none". These kinds of values ("nan" and "inf") are blocked by the TensorboardX backend, stalling the status update of involved graphs in the frontend webpage.