Pensieve

Pensieve is a system that generates adaptive bitrate algorithms using reinforcement learning. http://web.mit.edu/pensieve/

Prerequisites

Create a virtual envirnment with python 3.8
Install prerequisites (tested with Ubuntu 18.04, python3.8, Tensorflow v2.7.0, TFLearn v0.5.0 but not tested on Selenium)

pip install tensorflow tflearn matplotlib selenium
python setup.py

Training

To train a new model, put training data in sim/cooked_traces and testing data in sim/cooked_test_traces, then in sim/ run python get_video_sizes.py and then run

python multi_agent.py

The reward signal and meta-setting of video can be modified in multi_agent.py and env.py. Monitoring the testing curve of rewards, entropy and td loss can be done by launching tensorboard from the terminal as follows:

tensorboard --logdir=path/to/results

Where path/to/results is in dir sim. More details can be found in sim/README.md.

Testing

To test the trained model in simulated environment, first copy over the model to test/models and modify the NN_MODEL field of test/rl_no_training.py , and then in test/ run python get_video_sizes.py and then run

python rl_no_training.py

Similar testing can be performed for buffer-based approach (bb.py), mpc (mpc.py) and offline-optimal (dp.cc) in simulations. More details can be found in test/README.md.

Running experiments over Mahimahi

To run experiments over mahimahi emulated network, first copy over the trained model to rl_server/results and modify the NN_MODEL filed of rl_server/rl_server_no_training.py, and then in run_exp/ run

python run_all_traces.py

This script will run all schemes (buffer-based, rate-based, Festive, BOLA, fastMPC, robustMPC and Pensieve) over all network traces stored in cooked_traces/. The results will be saved to run_exp/results folder. More details can be found in run_exp/README.md.

Real-world experiments

To run real-world experiments, first setup a server (setup.py automatically installs an apache server and put needed files in /var/www/html). Then, copy over the trained model to rl_server/results and modify the NN_MODEL filed of rl_server/rl_server_no_training.py. Next, modify the url field in real_exp/run_video.py to the server url. Finally, in real_exp/ run

python run_exp.py

The results will be saved to real_exp/results folder. More details can be found in real_exp/README.md.

Training and cross-validation (testing) Curve using tensorboard

The RL-model converges after 3 days of continuous training using training data in sim/cooked_traces and testing data in sim/cooked_test_traces.

TD Loss & Total Reward of sim/multi-agent.py: