/bma-package

Replication package for SEAMS 2022 submission 14

Primary LanguagePythonMIT LicenseMIT

DOI License: MIT

SEAMS 2022 submission 14

Replication Package

This package contains the raw data as well as scripts used to carry out the empirical evaluation presented in the following research paper:

"Taming Model Uncertainty in Self-adaptive Systems Using Bayesian Model Averaging"

submitted to the 17th Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2022).

Authors:

  • Matteo Camilli, Free University of Bozen-Bolzano, Italy
  • Raffaela Mirandola, Politecnico di Milano, Italy
  • Patrizia Scandurra, University of Bergamo, Italy

Getting started

The package contains software and data to replicate the controlled experiments described in the paper. The software is a set of scripts implemented by using python3 and R.

The following python libraries are required. Install them by running pip3 install <library> (e.g., pip3 install numpy):

  • mpmath
  • numpy
  • pandas
  • statsmodels
  • geneticalgorithm
  • sklearn

The following R packages are required. Install them by running install.packages('<package>') from a R terminal (e.g., install.packages('BMA')):

  • BMA
  • MASS
  • BAS
  • argparse
  • R.utils
  • ggplot2

This package is structured as follows:

  • root contains the scripts to run the experiments.
  • data/ contains the raw observations generated by using the oracle model of the rescue robot.
  • logs/ contains raw logs pre-generated by running the scripts as well as clean data files that can be used to draw the plots.
  • plots/ contains .pdf plots pre-generated starting from the data in the logs/ folder.

Running the experiments

RQ1 (prediction accuracy). Experiments can be executed from the root folder as follows:

python3 bma_prec_recall.py > log01_precision_recall

The script requires the following files in the folder data/:

  • data/training_rescueRobot_450.csv training set
  • data/validation_rescueRobot_450.csv validation set

The log log01_precision_recall contains the results of precision, recall, and F1 score, for each model in the model space as well as the average model.

RQ2 (effectiveness of the adaptation decisions). Experiments can be executed as from the root folder follows:

python3 bma_adaptation.py > log02_adaptation

The script requires the following files in the folder data/:

  • data/training_rescueRobot_450.csv training set
  • data/validation_rescueRobot_450.csv validation set

The log log02_adaptation contains the results of RE and success (True/False), for each adaptation decision guided by all the models in the model space as well as the average model.

WARNING: the execution may take long time since it computes almost 1k adaptation decisions in total.

RQ3 (cost of BMA estimates). Experiments can be executed from the root folder as follows:

python3 bma_cost.py > log03_cost

The script requires the following files in the folder data/:

  • data/training_rescueRobot_25600_64.csv training set

The log log03_cost contains the wall-clock time required to build the average model by varying number of variables and observations.

WARNING: the execution may take long time since it computes around 4.8k BMA estimates (around 2.4k of them with very high-dimensional model spaces).

Cleaning the data

RQ1 (prediction accuracy). The log produced by the execution of bma_prec_recall.py can be cleaned as follows:

bash clean_precision_recall.sh log01_precision_recall > logs/precision_recall.log

RQ2 (effectiveness of the adaptation decisions). The log produced by the execution of bma_adaptation.py can be cleaned as follows:

bash clean_adaptation.sh log02_adaptation > logs/re_mba_logit.log

RQ2 (cost of BMA estimates). The log produced by bma_cost.py is ready to be precessed. Copy it in the right folder as follows:

cp log03_cost logs/bma_cost.log

Generating the plots

All the plots (RQ1, RQ2, and RQ3) can be generated as follows:

Rscript bma_plots.R

The script requires the pdfcrop command line tool and the following files in the folder logs/:

  • logs/precision_recall.log clean precision-recall data
  • logs/re_mba_logit.log clean relative-error data
  • logs/bma_cost.log clean cost data

All the plots are generated in .pdf format in the root folder.