N_iter is never updated in rainbow agent model
RaviKumarAndroid opened this issue · 8 comments
Thats also the reason why target_model weights are never updated. Its a bug
You should update the n_iter variable value in act function of the agent.
Hello @RaviKumarAndroid ,
Thank You for pointing that out, although I feel it also has to do with Categorical DQN missing in the implementation along with the use of Kullback Leibler divergence loss instead of Huber loss.
Ok, But then where have you updated the target model weight. The target models weight are never updated at all anywhere if you say that n_iter is non useful. Because if you don't update the n_iter. The target model will never learn. because its neither fitted anywhere neither is its weights set from the online model.
There are 2 ways to update the target model. either soft updates using the tau hyper parameter, or using hard updates by copying the weights to target_model from the online model every few thousand iterations.
any update?
Hello @lorrp1
I have been working on some other projects and didn't time to make the improvements mentioned by @RaviKumarAndroid ,
Although I have tried the hard update and it seems to worsen the current benchmark. I anticipate multiple changes required in order to improve the current benchmark and make the agent generalize on the real-time data. I don't have time at the moment. I will start working on this issue soon.
Hello @lorrp1 @RaviKumarAndroid ,
Sorry for the delay, but the changes are made, although the performance of the agent has not improved. I have made agent do hard updates every 10000 iterations. I will soon add categorical dqn and make a few tweaks to the model itself. Will try a lot of hyperparameter tuning to get the best of the agents. Until then I will close this issue.
Thank you, I’ll check it soon.