/MARL-UDS

Source code for the MARL-enabled real-time control of urban drainage system

Primary LanguagePythonGNU General Public License v3.0GPL-3.0

MARL-UDS

Source code, data and figures for the MARL-enabled real-time control of urban drainage systems. Refer to the published article:

  • Zhang Z., Tian, W. & Liao, Z., 2023.Towards coordinated and robust real-time control: a decentralized approach for combined sewer overflow and urban flooding reduction based on multi-agent reinforcement learning. Water Research, 229(119498). https://doi.org/10.1016/j.watres.2022.119498.

UDS Environments

  1. Astlingen: A benchmark SWMM model Astlingen of a combined sewer system with 6 storage tanks and 4 controllable outflow orifices. Thanks to Dr. Sun and other contributors for developing this model.

    • Sun C, Lorenz Svensen J, Borup M, Puig V, Cembrano G, Vezzaro L. An MPC-Enabled SWMM Implementation of the Astlingen RTC Benchmarking Network. Water. 2020; 12(4):1034. https://doi.org/10.3390/w12041034
    • Schütze, M.; Lange, M.; Pabst, M.; Haas, U. Astlingen—A benchmark for real time control (RTC). Water Sci. Technol. 2017, 2, 552–560. https://doi.org/10.2166/wst.2018.172

  2. Chaohu: A real-case model of a combined sewer system with 2 pump stations and forebays. Papers using this model:

    • Liao, Z., Gu, X., Xie, J., Wang, X., & Chen, J. (2019). An integrated assessment of drainage system reconstruction based on a drainage networkmodel. Environmental Science and Pollution Research, 26(26), 26563–26576. https://doi.org/10.1007/s11356-019-05280-1
    • Zhi, G., Liao, Z., Tian, W., Wang, X., & Chen, J. (2019). A 3D dynamic visualization method coupled with an urban drainage model. Journal ofHydrology, 577, 123988. https://doi.org/10.1016/j.jhydrol.2019.123988
    • Tian, W., Liao, Z., Zhi, G., Zhang, Z.&Wang, X., 2022b. Combined Sewer Overflow and Flooding Mitigation Through a Reliable Real-Time Control Based on Multi-Reinforcement Learning and Model Predictive Control. Water Resources Research, 58(7): e2021WR030703. https://doi.org/10.1029/2021WR030703
    • Tian, W., Liao, Z., Zhang, Z., Wu, H.&Xin, K., 2022a. Flooding and Overflow Mitigation Using Deep Reinforcement Learning Based on Koopman Operator of Urban Drainage Systems. Water Resources Research, 58(7): e2021WR030939. https://doi.org/10.1029/2021WR030939

Algorithms

  1. DQN: Deep Q-learning (double & dueling network)
  2. IQL: Independent Q-learning
  3. VDN: Value Decomposition Network
  4. QMIX: Monotonic Value Function Factorisation
  5. A2C & MAA2C: advantage actor-critic
  6. PPO & MAPPO:
  7. Behavior Cloning (demo)
  8. GAIL (demo): Generative adversarial imitation learning

Tricks

  • Multiple input & Multiple output: Used in DQN when if_mac == True
  • Recurrent network (demo): if_recurrent DRQN
    • Use GRU to deal with timeseries data
  • Graph convolution network (demo): global_state
    • Info of all the elements (nodes or links)
    • Use GCN to convolve before MLP
    • Share convolution layers: share_conv_layer

Requirements

  • tensorflow >= 2.3
  • tensorflow_probability >= 0.11.1
  • spektral == 1.2.0
  • pyswmm >= 0.6.2
  • pystorms >= 1.0.0
  • swmm-api == 0.2.0.18
  • pymoo == 0.6.0