To develop a Python program to evaluate the given policy.
The bandit slippery walk problem is a reinforcement learning problem in which an agent must learn to navigate a 7-state environment in order to reach a goal state. The environment is slippery, so the agent has a chance of moving in the opposite direction of the action it takes.
The environment has 7 states:
- Two Terminal States: G: The goal state & H: A hole state.
- Five Transition states / Non-terminal States including S: The starting state.
The agent can take two actions:
- R: Move right.
- L: Move left.
The transition probabilities for each action are as follows:
- 50% chance that the agent moves in the intended direction.
- 33.33% chance that the agent stays in its current state.
- 16.66% chance that the agent moves in the opposite direction.
For example, if the agent is in state S and takes the "R" action, then there is a 50% chance that it will move to state 4, a 33.33% chance that it will stay in state S, and a 16.66% chance that it will move to state 2.
The agent receives a reward of +1 for reaching the goal state (G). The agent receives a reward of 0 for all other states.
![](https://private-user-images.githubusercontent.com/93427237/267353161-e663bd3d-fc85-41c3-9a5c-dffa57eae250.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM2NjAxNzEsIm5iZiI6MTcyMzY1OTg3MSwicGF0aCI6Ii85MzQyNzIzNy8yNjczNTMxNjEtZTY2M2JkM2QtZmM4NS00MWMzLTlhNWMtZGZmYTU3ZWFlMjUwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE0VDE4MjQzMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTkwNmQ3NGI4MjE4ZjYxNzYyZWQ2ZWRmOTQwNTFlM2NiMjE4MjUxYjdmN2FjY2M1NDI5YmJiYjUyNTJjZTE0NDUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.EUkge-ImQVKvE005z9W8VPlidvRoLIToYFVp-8v4VQg)
def policy_evaluation(pi, P, gamma=1.0, theta=1e-10):
'''Initialize 1st Iteration estimates of state-value function(V) to zero'''
prev_V = np.zeros(len(P), dtype=np.float64)
while True:
'''Initialize the current iteration estimates to zero'''
V=np.zeros(len(P),dtype=np.float64)
for s in range(len(P)):
'''Update the value function for each state'''
for prob,next_state,reward,done in P[s][pi(s)]:
V[s] += prob * (reward + gamma * prev_V[next_state] * (not done))
'''Check for convergence'''
if np.max(np.abs(prev_V-V))<theta:
break
'''Update the previous state-value function'''
prev_V=V.copy()
return V
Thus, a Python program is developed to evaluate the given policy.