bias annealing weight updates

[zacwellmer]

I could be wrong but it does not seem that you are annealing the bias with important sampling as suggested in the paper(3.4).

w_i = (1/N * 1/P(i))^beta

I think you would have to multiply this w_i term with your gradients

[zacwellmer]

My apologies, I thought you had included prioritized replay.