| Cliffwalking | Taxi |
|---|---|
 |
 |
This is me experimenting with various algorithms in the Reinforcement Learning Book by Richard Sutton. I chose the Cliffwalking-v0 environment in gymnasium to try out these algorithms. I later modified the training runs slightly to adapt to the Taxi-v3 environment.
I experimented with n-step Q Learning, n-step Sarsa and Monte Carlo methods. I used some basic graphing analysis techniques to guide my hyperparameter search and reward modelling. One interesting thing was that the agent would go in loops; so to disincetivize this, I introduced a large negative reward if the agent revisited a state.
The concepts are crystal clear to me now although the one-off errors when implementing the Q Learning and Sarsa Algorithms were very very annoying.
I don't guarantee that these implementations are fully correct and follow the specficiation. There may be one-off errors but it was enough to solve this toy problem.
I was able to achieve a maximum reward of 18 steps. The optimal trajectory would be 14 steps. With some more hyperparameter search and reward modelling, I could have gotten there.
Install requirements
pip install requirements.txt
To run Cliffwalking, add the --env Cliffwalking flag.
To run Taxi add the --env Taxi flag.
To render training runs, add the --render flag.
Test runs will always render
By default env = Cliffwalking and render flag is off.
Sample
python main.py --env Taxi --render
As you can see all three algorithms converged for Cliffwalking. Although all three algorithms could have been improved, Q Learning was particularly more effective.
