Question about batch_size and capacity of queue
jsuit opened this issue · 7 comments
I was reading over this code and I had a simple question that isn't hopefully tied to my lack of familiarity with tensorflow.
The has a capacity of size 1 from line 502: queue = tf.FIFOQueue(1, dtypes, shapes, shared_name='buffer'). However, line 547 dequeues a batch_size of 2: dequeued = queue.dequeue_many(FLAGS.batch_size).
Is there something I am missing? Is it possible to dequeue more elements than the capacity of the queue?
I believe this would just create a dequeue op, and would block if there are not enough elements.
Slightly confused. What did you mean by ‘this’?
dequeued = queue.dequeue_many(FLAGS.batch_size).
But is a reason why they try to dequeue a batchsize of 2 when the capacity of the queue is 1?
The reason for having the queue at a capacity of 1 is that every actor can only contribute one item and then blocks/has to wait until the learner dequeues this item. We have not investigated this decision too much but the underlying motivation is to keep the data in each batch diverse by limiting the amount of (correlated) experience each actor can contribute.
Even for a tf.FIFOQueue of capacity 1, it's possible to dequeue a larger number of elements. Blocked enqueues will be used for the dequeue and thus, will be unblocked.
The reason to use capacity 1 instead of e.g. the number of actors, is that when the queue size is increased, the behaviour policy will become further behind the target policy. The number of updates between the behaviour and target policy is a factor of the number of actors and the queue size. Additionally, as @ogh mentions, this also reduces potential correlation of the experiences in a batch.
Ideally we would use capacity 0, i.e. reducing the tf.FIFOQueue to a blocking "channel". However, it's not supported.
Closing the issue :-) Feel free to reopen if any additional questions comes up.