germain-hug/Deep-RL-Keras

Can not run DDQN with PER

Opened this issue · 4 comments

Xiaoyu006 commented

Hi,

Thanks for this great implementation.
I encountered an error when tried to run "python3 main.py --type DDQN --env CartPole-v1 --batch_size 64 --with_PER"

The error is:
/utils/memory_buffer.py", line 69, in sample_batch
batch.append((*data, idx))
TypeError: 'int' object is not iterable

Have anyone fixed this issue?

thinhlx1993 commented

Hi,

Thanks for this great implementation.
I encountered an error when tried to run "python3 main.py --type DDQN --env CartPole-v1 --batch_size 64 --with_PER"

The error is:
/utils/memory_buffer.py", line 69, in sample_batch
batch.append((*data, idx))
TypeError: 'int' object is not iterable

Have anyone fixed this issue?

I got the same issues. Did anyone fixed it?

surfingkaka commented

trying passing in the data directly instead of the *data I think the author were attempting to make it future proof.

Jeosas commented

Hi, I'm having the same issue.

Changing *data to data bring an error on the next line idx = np.array([i[5] for i in batch]) : tuple index out of range

I've done a little debugging by printing out the the (idx, data) values and something stood out:
"
22014 (array([-0.08095277, 0.61171632, 0.0524595 , -0.81013508]), 1, 1.0, False, array([-0.06871845, 0.80608178, 0.0362568 , -1.08586616]))
22015 (array([-0.06871845, 0.80608178, 0.0362568 , -1.08586616]), 0, 1.0, False, array([-0.05259681, 0.61050081, 0.01453948, -0.78203035]))
22017 (array([-0.04038679, 0.80541993, -0.00110113, -1.07010368]), 1, 1.0, False, array([-0.0242784 , 1.00055643, -0.0225032 , -1.36313198]))
22017 (array([-0.04038679, 0.80541993, -0.00110113, -1.07010368]), 1, 1.0, False, array([-0.0242784 , 1.00055643, -0.0225032 , -1.36313198]))
22018 (array([-0.0242784 , 1.00055643, -0.0225032 , -1.36313198]), 1, 1.0, False, array([-0.00426727, 1.19595297, -0.04976584, -1.66276796]))
22019 (array([-0.00426727, 1.19595297, -0.04976584, -1.66276796]), 1, 1.0, False, array([ 0.01965179, 1.39161785, -0.0830212 , -1.97052703]))
32767 0
"

For some reason, the get function of the MemoryBuffer return "int(0)" as data, that's what's causing the "tuple index out of range" error.
We can also see the faulty idx which jumps to a crazy high value for some reason, and thus out of range of the memory i think (ending up in the data being a 0 ?)
EDIT: idx ends up pointing to a memory slot that has not yet been written (thus returning 0)

I'm still investigating but any help would be appreciated :)

winston-p commented

Hi, has anyone managed to fix this bug? I am new to DDQN and hit into this issue. Hope to get some help. Thanks in advance!