Simulate Markov processes directly into your browser.
Just go to the github page to play with it.
This repo has been inspired by the course of David Silver on Reinforcement learning. As he goes through the basics of this algorithm, he uses Markov processes a lot and I've been struggling to find a nice library to visualize and plays with them online.
So ... I built one!
Creating a nice simulation environment for Markov process (MP), Markov reward process (MRP) and Markov decision process (MDP) including value-function solver, allowing people to have a "feel" about those algorithms in a convenient fashion.
- Fork the repo
- Make your change
- Write a test
- Make a pull request Feedbacks will be fast
The app has two parts:
- An MP configuration part on the left
- A visualization tool on the right
Control which kind of MP you want to simulate.
Control all the possible properties of the current selected MP:
- The number of states
- The transition matrix
- The reward values
- The discounted factor gamma
- A possible finite horizon
- The limit epsilon: the difference between the real solution and our approximation
Note on the horizon:
Whatever you choose for your properties, this will end up as a finite horizon following those rules:
- We calculate a gamma-horizon:
t1
for whichgamma^t1 < epsilon
- We get
t2
the horizon given by the user- We set the final horizon
T
toT = Math.min(t1, t2)
If
t1
andt2
are equal toInfinity
, we just setT
to10000
Control how you approximate the different value-functions
- In one synchronized shot
- Step by step
- Agent based
Control the agent, choose:
- where he starts
- how fast he moves You can also find different properties about the agent in this panel.
We are using SVG to produce the different MP visualizations. Thanks to React for handling SVG nodes and the D3.js force module to help position them in an adequate fashion.
Build a Markov process simulatorBuild a Markov reward process simulatorImplements saving/loading/viewer featureImplement an agentImplements a state-value function solver + an agent based onePort the code to typescript and add eslint- Implements a Markov decision process simulator
- Implements a policy iteration algorithm and an action-value function solver
MIT
Check the LICENCE file