I think Frame skipping hasn't been done correctly.

[kartikpaigwar]

According to this blog which has nicely explained the preprocessing done in actual DQN by Deep Mind, We should skip 3 consecutive frames and consider only 4rth frame. But you have added all these four consecutive frames in the buffer, which I feel is not exactly the frame skipping mentioned in the DQN paper.

for _ in range(self.frame_skipping):
# step 에서 나온 observation은 버림
observation, reward, done, info = self.env.step(action[0, 0])
next_state = self.env.get_screen()
self.add_state(next_state)

You should add the next_state at the end of the above 'for loop' not inside it. This will correct the skipping frame concept. Thanks.