The SBFT Workshop offers a challenge for software testers who want to work with self-driving cars in the context of the usual tool competition.
The deadline to submit your tool is: January 13th 2023 January 20th 2023
The results of the evaluation will be communicated to participants on: February 21st 2023
The camera-ready paper describing your tool is due to: March 17th 2023
The competitors should propose a test generator that produces virtual roads to test a lane keeping assist system. The aim of the generation is to produce diverse failure-inducing tests, i.e., roads that make the lane keeping assist system drive out of the lane.
The ranking of the tools is based on coverage which measures the number of failed tests and their diversity. In fact, it is in the interest of any tester to provide as diverse failures as possible. This facilitates better root cause analysis.
The generated roads are evaluated in the BeamNG.tech driving simulator. This simulator is ideal for researchers due to its state-of-the-art soft-body physics simulation, ease of access to sensory data, and a Python API to control the simulation.
In the competition two lane keeping assist systems are used: BeamnNG.AI provided by the BeamnNG.tech simulator and DAVE-2 trained by the competition organizers (with a speed-limit of 25 km/h).
Note: BeamNG GmbH, the company developing the simulator, kindly offers it for free for researcher purposes upon registration (see Installation).
We make available a code pipeline that will integrate your test generator with the simulator by validating, executing and evaluating your test cases. Moreover, we offer some sample test generators to show how to use our code pipeline.
Deciding which test generator is the best is far from trivial and, currently, remains an open challenge. In the 2023 edition of the competition, we rank the test generators according to a coverage metric. In particular, we select relevant features, build a feature map for them, and compute how much each tool covers this map. Possible features to be used include:
- Direction Coverage (DirCov).
- Standard Deviation of the Steering Angle (StdSA).
- Maximum Curvature (MaxCurv).
- Mean Lateral Position (MLP).
We expect that the submitted tools are stochastic in nature, so we compute the coverage as the total coverage over several repetitions of the tool.
More information can be found on the SBFT tool competition website: https://sbft23.github.io/tools/. See also the tool report of the previous competition edition: https://ieeexplore.ieee.org/document/9810771
Code pipeline: code that integrates your test generator with the simulator.
Self driving car testing library: library that helps the integration of the test input generators, our code pipeline, and the BeamNG simulator.
Scenario template: basic scenario used in this competition.
Documentation: contains the installation guide, detailed rules of the competition, and the frequently asked questions.
- Installation Guide: information about the prerequisites and how to install the code pipeline.
- Guidelines: goal and rules of the competition.
- FAQ: answers to the most frequent asked questions.
Sample test generators: sample test generators already integrated with the code pipeline for illustrative purposes.
Requirements: contains the list of the required packages.
The software we developed is distributed under GNU GPL license. See the LICENSE.md file.
Dr. Matteo Biagiola - Università della Svizzera italiana, Lugano, Switzerland - matteo.biagiola@usi.ch
Dr. Stefan Klikovits - National Institute of Informatics, Tokyo, Japan - klikovits@nii.ac.jp
Dr. Jarkko Peltomäki - Åbo Akademi University, Turku, Finland - jarkko.peltomaki@abo.fi
Dr. Vincenzo Riccio - Università di Udine, Udine, Italy - vincenzo.riccio@uniud.it