Microservices Performance Assesment

This repository contains dataset and scripts to replicate the experiments reported in the manuscript: "Actor-driven Decomposition of Microservices through Multi-level Scalability Assessment" submitted for possible publication to the Transactions on Software Engineering and Methodology, ACM (2022).

Smart Mobility App Dataset

Structure of the package

Raw Data

Each set of experiments is contained in a directory that follows this naming convention:

<architecture>_iter<i>_<behavior-mix>

architecture: one of the three alternative software architecture Monolith (Mono), CQRS (Cqrs), RS (Role);
i: #iteration (either 1 or 2) which determines the deployment_pm configuration;
behavior-mix behavior mix models adopted in the experiments either Ω (bal) or Ω' (unbal).

Each directory contains, for each workload intensity, the data collected during three load testing sessions.

Example (CQRS, 1st iteration, 100 users, bal profile):

Cqrs_iter1_bal
├── Cqrs_100th_try0_iter1_bal
│   └── test_Cqrs_100th_try0_iter1_bal.jtl
├── Cqrs_100th_try1_iter1_bal
│   └── test_Cqrs_100th_try1_iter1_bal.jtl
├── Cqrs_100th_try2_iter1_bal
│   └── test_Cqrs_100th_try2_iter1_bal.jtl

Clean Data

In the folder PerformanceComparison/DATA there are four cleaned datasets that contain the result of a set of experiments, ready to be analyzed through the performance assessment scripts. Each dataset is saved in an spreadsheet file with the following naming convention:

iter<i>_<type>-profile_<date>.xlsx

where <i> indicates the iteration number (1 or 2), <type> is the behavior mix model (either bal or unbal) and <date> is a timestamp. Each file contains the average and standard deviation of the response time per individual operation exposed by microservices as well as the frequency of associated with requests (Mix).

The content is as follows:

column 1: number of concurrent users (load intensity);
columns 2-6: deployment configurations;
columns 7+: measures for each microservice operation.

R Scripts

Preliminaries

Clean: remove unnecessary files and keep the .jtl logs generated by JMeter.

Rscript clean.R <dir>

Exapmple: Rscript clean.R Cqrs_iter1_bal

summary: clean the data and extract a summary from a set of experiments.

Rscript summary.R <dir> <users> <out>

Example: summary.R Cqrs_iter1_bal 100 Cqrs_iter1_bal_report.csv

multiple summary: extract the final summary for further analysis from a set of experiments.

Rscript multi_summary.R <dir> <out>

Example: summary.R Cqrs_iter1_bal Cqrs_iter1_bal_report.csv

statistical difference: Detect statistical difference (Mann-Whitney U test) of response-time values collected from alternative architectures.

Rscript stat_difference_exp.R <dir> <dir2> <users>

Example: Rscript stat_difference_exp.R Mono_iter1_bal Cqrs_iter1_bal 100

effect size: Produce the summary of effect-size (Vargha-Delaney A12) and p-value (Mann-Whitney U-Test) from a set of jtl files collected from the three alternative architectures.

Rscript effect_size.R <dir1> <dir2> <dir3> <users> <out_dir>

Example: Rscript effect_size.R Mono_iter1_bal Cqrs_iter1_bal 100 effectsize_iter1_bal

multiple effect-size summary: Run the computation of the effect size multiple times (for each #user value).

Rscript multi_effectsize_summary.R Mono_iter1_bal Cqrs_iter1_bal effectsize_iter1_bal

Performance assessment and comparison (Smart Mobility App)

There are five scripts in the folder PerformanceComparison/RScripts used to perform the multi-level assessment analysis:

start.R;
computeThreshold.R;
computeDM_FUNCTIONS.R;
NewSpiders.R;
WilcoxonAnalysis.R.

There are few ways to run the scripts, as follows:

Command Line

Uncomment the following lines in start.R:

#args = commandArgs(trailingOnly=TRUE)
#workingDirectory<-args[1]
#RScript start.R <path to workingDirectory>

and then run with command:

RScript start.R <path to workingDirectory>

R Studio

Run either start.R or Start.Rmd in RStudio.

Jupiter notebook

Run start.ipynb from the folder PerformanceComparison as follows:

Jupiter notebook start.ipynb

Output

The output is contained in the folder PerformanceComparison/Results and it includes:

the failing services per architecture deployment and iteration;
the plots (Radar, Ridge, Ploygons); and
the Wicoxon tests per individual component.

Performance assessment and comparison (TrainTicket)

There are two scripts in the folder PerformanceComparison/RScripts used to perform the multi-level assessment analysis specifically to TrainTicket:

computeThreshold_train.R;
NewSpiders_train.R;

It is possible to run directly Start_TrainTicket.Rmd in PerformanceComparison folder to plot analysis result Script behavior is the same of previous application and operate on results on data present in PerformanceComparison/DATA/TrainTicket folder

zerodayshack/Microservices-Performance-Assessment