I used this code in the paper Exploiting locality and physical invariants to design effective Deep Reinforcement Learning control of the unstable falling liquid film
that you can check out here - https://aip.scitation.org/doi/10.1063/1.5132378
If you find this work useful and / or use it in your own research, please cite our works:
V. Belus, J. Rabault, J. Viquerat, Z. Che, E. Hachem, and U. Reglade
Exploiting locality and physical invariants to design effective Deep
Reinforcement Learning control of the unstable falling liquid film,
ArXiv (2019)
This README will be in three parts :
- The first part will be about the code in itself
- The second part will be about getting an environment ready to run the code (either in a docker container, or in Ubuntu)
- The third part will be about the details of how to train a model
The code is in drl_fluid_film_python3/gym-film
. You find the following files:
param.py
contains the parameters that your next training will use (all the parameters of the project are centralized here)train.py
is the script you will use to train or render a model
In gym_film
, you will find the following directories:
envs
, where our basic environment classfilm_env.py
is defined. It is a custom gym environment,and it interacts with the C++ simulation that is defined insimulation_solver
.model
, where the script to retrieve models is defined. These models are imported from the library stable_baselines, and only the PPO2 implementation has been used in the paper.results
, where previously trained models are stored along with the parameters used for the training, and tensorboard logs to have insights on how the model performs during training
Because the simulation is built from scratch in C++ and linked with the Python API with the library Boost.Python, you need to have the library installed for the simulation to run.
The easiest way is to run the code in a docker container, built from a Ubuntu 18.04 image. This way you can run the code whether you are on Windows, MacOS or Linux as long as you can run docker.
Docker is a convenient tool to use containers. If you're not familiar with the concept, everything you need to run code from this project will be inside the container. In this case, it will be an Ubuntu 18.04 distribution with Python3 and other necessary packages.
The details of how I built my docker image from ubuntu:latest
are in this repository. You can find all these steps in the Dockerfile
.
-
Once you can use Docker, get the image :
- from the
Dockerfile
by running the following command in the same directory as theDockerfile
(this can take several minutes or more depending on your internet connection, and the docker image will take 2.4GB on your disk):
$ docker build . -t falling-liquid-film:latest
- by downloading it from the following adress : https://folk.uio.no/jeanra/Research/falling-liquid-film_latest.tar.gz
- from the
-
You can then run a container from this image with the following command :
$ docker run -it falling-liquid-film:latest
- You should now be in the
gym-film
folder, ready to launch trainings XX: make display possible
You can install the dependencies on your own Ubuntu distribution, as is done when building the Docker image.
The details are in this repository, after the "Packages I installed step by step" section.
The basic command is python3 train.py
but you will need to add some flags :
--train
or-t
to train your model--render
or-r
if you want to render the simulation
If you use this flag with -t
, it will render during the training as well. You can change how often the rendering is done with the RENDER_PERIOD
parameter
--training_name
or-tn
is the name of the training, default value is "test"--load
or-l
followed by an integern
to load a trained model. This will look for a model trained for n episodes under the training name you specified--load_best
or-lb
will load the "best" model under the training name you specified, "best" meaning the model trained on k episodes, where the maximum episode reward of the entire training was obtained during episode k--port
or-p
, followed by an integer between 1024 and 49151, which will be the port used when using the multi-environment method (Method M3 in the paper).
If you want to train a model with the default parameters and render it, you can do it with python train.py --train --render --training_name my-first-training
All the parameters relevant to the training and simulation are in param.py
. You will have to change the parameters in this file.
We will use the following syntax: parameter
(type, typical low value - typical high value)
Some important parameters here are :
- The number of jets:
n_jets
(int, 1 - 20) - The position of the jets:
jets_position
(array) ORposition_first_jet
(float) andspace_between_jets
(float, 5 - 20) - The maximum power of the jets:
JET_MAX_POWER
(float, 1 - 15) and their width:JET_WIDTH
(float, 2 - 10) -> As it is done, the action of the agent is always between-1
and1
and is later normalized and multiplied byJET_MAX_POWER
to be applied in the simulation. The default values are letting the jets modify q with the same amplitude as the waves naturally forming, so it should be enough for control. - The size of the observation
size_obs
(float, 10 - 40), input of the agent, and the size of the observation on which the reward is calculatedsize_obs_to_reward
(float, 5 - 20) - The non-dimensional duration of one episode
simulation_time
(float, 5 - 20) - The duration of the simulation before we begin any training
initial_waiting_time
(float). Default value is 200, letting the simulation get to a converged state before we do any training (the state is stored and not computed each time) simulation_step_time
is the non-dimensional duration of one step of the simulation. In one such step, we don_step
steps of the numerical scheme- Whether to render the simulation with matplotlib with
render
(bool). If True, render it everyRENDER_PERIOD
(int) environment steps
And that is exactly what you can do here: LINK TO THE NOTEBOOK REPO You need Docker to run this
This notebook will demonstrate how the three implemented methods build on top of each other, and you will be able to look at what the different parameters do on the simulation interactively.
I made it for a course on Deep Reinforcement Learning at my school (Mines ParisTech, PSL University).