/BSM1_gym

Transform a Modelica model (BSM1) into a Python OpenAI Gym environment, and optimize operation cost using reinforcement learning.

Primary LanguageHTMLMIT LicenseMIT

BSM1_gym

Transform a Modelica model (BSM1) into a Python OpenAI Gym environment, and optimize operation cost using reinforcement learning agents.

This README provides the general instructions on how to:

  1. Run a Modelica model using Open Modelica from Python (OMPython)
  2. Convert the model to the FMU format and simulate it directly within Python using PyFMI
  3. Create an OpenAI environment for the FMU, and use a Reinforcement Learning agent to interact with the model

The code here was developed to simulate a Modelica implementation of the Wastewater Treatment Plant Benchmark Simulation Model No. 1 (BSM1), that can be found in the ./WasteWaterResearch/BSM1/ directory within this repository, and that was adapted from the work of Félix Hernández del Olmo. Furthremore, its compiled FMU has been made available in this repository (BSM1.BSM1.fmu), which can be used directly under a Windows 64bits system (in which it was compiled) without the need of recompiling it (in any case, the instrucions for generating the FMU have been included in this README). Finally, even if the BSM1 is of no interest to the user, this repository provides with the tools and instructions to adapt any Modelica model to be integrated within the OpenAI Gym API and be used for training reinforcement learning agents on it.

0. Preparation steps

  • Download and install Miniconda for Python 3.7: https://docs.conda.io/en/latest/miniconda.html
  • Once installed, open an Anaconda prompt, and create a Modelica environmet: conda create -n modelica python=3.7
  • Activate it: conda activate modelica
  • Install some libraries that will be neeed:
conda install numpy matplotlib psutil pyzmq scipy future git pandas seaborn scikit-learn tqdm
conda install -c conda-forge ffmpeg
  • Clone this repository to your computer and navigate to it:
git clone https://github.com/OscarPellicer/BSM1_gym.git
cd BSM1_gym

1. Interacting with OpenModelica models through OMPython

  • Download and install OpenModelica: https://www.openmodelica.org/
  • To run Modelica models from Python (non-interactively), we can install and use the OMPython library:
  • Install: python -m pip install -U https://github.com/OpenModelica/OMPython/archive/master.zip
  • To see any plots generated by the code in the following sections, it is recommended to run the code within Jupyter Notebook / Jupyter lab.
  • To install Jupyter lab: conda install jupyter jupyterlab.
  • Then, run jupyter notebook in the console, and navigate to http://localhost:8888/lab. Then click on File > New > Notebook, and start coding!
  • We have created a simplified runner for OpenModelica commands in Python, Runner, which can be found in utils.py. Testing on the bouncing ball model:
from utils import Runner
R= Runner()
R.run("loadModel(Modelica)")
R.run("loadFile(getInstallationDirectoryPath() + \"/share/doc/omc/testmodels/BouncingBall.mo\")")
R.run("instantiateModel(BouncingBall)")
R.run("simulate(BouncingBall, stopTime=3.0)")
  • This will generate a .mat output file that can be read with DyMat, which has been included in this repository. Furthermore, class MatHandler uses DyMat to simplify the process of reading and plotting simulation results:
from utils import MatHandler
m= MatHandler('BouncingBall_res.mat')
print('Variable names:', m.names)
m.plot(['h', 'v', 'der(v)'], plot_indep=False)

2. Creating a FMU and simulating it using PyFMI

  • PyFMI allows to simulate FMUs in Python
  • Install pyfmi: conda install -c conda-forge pyfmi (note: if running jupyter lab, you will now need to open a new Anaconda terminal and actiavte the modelica environment, or close Jupyter)
  • Then, create the FMU from Python using OMPython:
model_name= "BouncingBall"
R.run("loadModel(Modelica)")
R.run("loadFile(getInstallationDirectoryPath() + \"/share/doc/omc/testmodels/BouncingBall.mo\")")
R.run("instantiateModel(BouncingBall)")
R.run('translateModelFMU(%s, version="2.0", fmuType = "me")'%model_name)
  • And simualte it using PyFMI:
from pyfmi import load_fmu
model= load_fmu('%s.fmu'%model_name)
res= model.simulate(final_time=3)
  • We can do the same for our custom model
# Create a FMU using OMPython
base_path= 'WasteWaterResearch'
model_name= 'BSM1.BSM1'
R.run('loadModel(Modelica,{"3.2.3"},true,"",false)')
R.run('loadFile("%s/WasteWater/package.mo","UTF-8",true,true,false)'%base_path)
R.run('loadFile("%s/BSM1/package.mo","UTF-8",true,true,false)'%base_path)
R.run('disableNewInstantiation()')
R.run('translateModelFMU(%s, version="2.0", fmuType = "me")'%model_name)

# Import PyFMI, set the options so that sparse solver is not used, and simulate
from pyfmi import load_fmu
model = load_fmu('%s.fmu'%model_name)
opts = model.simulate_options() 
opts["CVode_options"]["linear_solver"] = "DENSE"
res = model.simulate(final_time=1, options=opts)
  • pyFMI allows an easy interface to the results, without having to read the generated .mat file:
# To see the name of all variables
res.keys()
# To access the values of one of them
res['time']
# To access final value
res.final('time')
# To get the description
res.result_data.description[res.keys().index('time')]

3. Creating an OpenAI Gym environment and training a Reinforecement Learning agent on it

  • Install OpenAI Gym: conda install -c conda-forge gym
  • An example of setting up an environment (BSM1), and training an agent to optimize operation cost on it can be found in Training.ipynb
  • In order to use deep learning agents, you need to install pytorch too. It is also needed for running Training.ipynb. You can follow the instructions here: https://pytorch.org/get-started/locally/
  • Obviously, to run the Notebook, Jupyter Notebook must be first installed (see above).
  • Please note that BSM1 simulations can sometimes fail due to the agents making a series of actions that yield to division by zero within the model. This is a rare occourance that unfortunately has not yet been solved. If this happens, please rerun the simulation.
  • If you want to adapt the library to your own needs, you should first implement a class similar to BSM1Env found in BSM1Envs/bsm1_env.py. This class implements the ModelicaEnv class, which, in turn inherits from gym.Env, which is the parent class for all Gym environments.
  • You should also modify some of the variables in Training.ipynb, such as base_path, model_name, action_names, env_name, entry_point, output_names to adapt them to your own model.
  • The rest of the Python files in the repository are used by Training.ipynb, and provide the following functionality:
    • agents.py: This file contains the implementation of some of the most common Q-like agents, using pytorch for the deep agents.
    • wrappers.py: This contains some useful wrappers for OpenAI environments to help integrate them with different kinds of agents.
    • training.py: This file contains all the code for letting an agent interact and learn from the evironment as well as to plot the evolution of the values of the variables over time.

This work is part of my end-of-degree project for my Computer Science degree at UNED University, which can be found in this repository (in Spanish).