/Benchmark

A benchmark repository for TLR between (textual) Software Architecture Documentation and Software Architecture Models

MIT LicenseMIT

DOI

Benchmark

This repository contains a benchmark for traceability link recovery (TLR) between textual Software Architecture Documentation (SAD) and Software Architecture Models (SAM).

Each project of the benchmark is structured as follows:

  • The README of each project contains some information about the used languages and lines of code created with cloc.
  • The folder pcm contains a Palladio Component Model (PCM) of the system. It has at least the repository view (components) of the system.
  • The folder uml contains a Papyrus UML model. It is created from the PCM Repository using PCM2UML.
  • The text file(s) in the project folder contains the SAD of the projects as plain text.
  • The file goldstandard.csv contains the traceability links between SAD and SAM. It links the model elements by id with the sentences by their number (index starting at 1).

Using the benchmark

In order to provide an easy approach to use the benchmark, we provide an example TLR approach called Simple Tracelink Discovery (STD) that uses this benchmark in its evaluation. Therefore, the benchmark is linked to the STD repository via a git subtree.

References

Note

Our LICENSE does only apply to the PCM models and the Gold Standards (CSV files). The texts are licensed w.r.t. to the actual projects.

BigBlueButton

BigBlueButton (BBB) is a non-scientific application that provides a web conferencing system with the focus on creating a "global teaching platform". The documentation of BBB is licensed under LGPL. Therefore, the text we extracted from their documentation is licensed according to the license of BBB under LGPL.

MediaStore

MediaStore is a "model application built after the iTunes Store". Its architecture was used for exemplary performance analyses on software architecture models. The text we extracted from their documentation originates from the publication Modeling and Simulating Software Architectures: The Palladio Approach.

Teammates

TEAMMATES is an open-source "online tool for manageing peer evaluations and other feedback paths of your students". The documentation of TEAMMATES is part of their repository. Therefore, the text we extracted from their documentation is licensed according to the license of TEAMMATES under GPL-2.0.

Teastore

Teastore is a scientific application that is used as a "micro-service reference test application". The documentation of Teastore was part of their repository. Therefore, the text we extracted from their documentation is licensed according to the license of Teastore under Apache-2.0.