This repository contains the code to evaluate the capability of the CiRA tool to automatically generate test case descriptions from causal natural language requirements. To this end, the requirements of the Corona Warn App were manually classified as either causal or non-causal, and for all causal sentences a set of test cases were generated. The analysis in this repository compares the manually generated test cases with the output of CiRA.
This repository contains the following files:
data: data necessary for the evaluationcwa-acceptance-criteria: list of sentences constituting the acceptance criteria of the Corona Warn App, manually classified as either causal or non-causalground-truth: a json files containing a manually generated set of test cases following the CiRA test suite syntax for every causal sentence
src\evaluation.ipynb: Jupyter notebook for performing the calculationrequirements.txt: required Python libraries for executing the evaluation
In order to reproduce the evaluation, follow the steps below:
- Make sure both Python 3.10 and pip are available on your system.
- Install the requirements by running
pip install -r requirements.txt. - Make sure the CiRA tool is available on your system.
- Follow the installation instructions to set up the CiRA tool on your system.
- Ensure that the REST API, which the CiRA tool provides, is currently running by pinging
http://localhost:8000/api/health, which should return a json object containing at least a version string in response.
- Execute the evaluation notebook.
Copyright © 2023 Julian Frattini
This work (source code) is licensed under MIT License.