chapter04/gamblers_problem.py line33 to 62 may has a problem
ChenHuaYou opened this issue · 2 comments
ChenHuaYou commented
while True:
old_state_value = state_value.copy()
sweeps_history.append(old_state_value)
for state in STATES[1:GOAL]:
# get possilbe actions for current state
actions = np.arange(min(state, GOAL - state) + 1)
action_returns = []
for action in actions:
action_returns.append(
HEAD_PROB * state_value[state + action] + (1 - HEAD_PROB) * state_value[state - action])
new_value = np.max(action_returns)
state_value[state] = new_value
delta = abs(state_value - old_state_value).max()
if delta < 1e-9:
sweeps_history.append(state_value)
break
# compute the optimal policy
policy = np.zeros(GOAL + 1)
for state in STATES[1:GOAL]:
actions = np.arange(min(state, GOAL - state) + 1)
action_returns = []
for action in actions:
action_returns.append(
HEAD_PROB * state_value[state + action] + (1 - HEAD_PROB) * state_value[state - action])
# round to resemble the figure in the book, see
# https://github.com/ShangtongZhang/reinforcement-learning-an-introduction/issues/83
policy[state] = actions[np.argmax(np.round(action_returns[1:], 5)) + 1]
i think from the line # compute the optimal policy to the end of for loop, it should be in the while loop
ChenHuaYou commented
but i put the policy evaluate and the policy improvement in the bigger while loop , found it run twice evaluate and once improvement makes the result equivalent to your result .. , i dont know if it is a coincidence
ShangtongZhang commented
It's not a bug. It's value iteration not policy iteration, so there is no policy evaluation.
Computing the optimal policy is solely for plotting.