/speleo-test-suite

An automated C++ build/test suite for the Speleo project

Primary LanguageShellMIT LicenseMIT

Speleo MT-2019 Test Suite

🔥 An automated build/test script for the EPFL C++ Speleo MT-2019 project. Written purely in bash 🔥
Formally known as the Boulic-Proof Test Suite

EPFL   Bash   Release   Compliance


An example of usage

Quick Links
Prerequisites  •  Installation  •  Usage  •  Performance Analysis  •  Updating

Preface

This test suite is derived from the C++ ColoReduce MT-2018 project. It is strongly recommended to understand the C++ build/debug process, and use this only as a helper to double-check your work. Experiment with your own build/test scripts!

While providing out-of-the-box support for the Speleo project, the build/test scripts can easily be modified to support most simple C++ projects.

✅ What this does

  • 🛠️ Compiles your program with the official C++ compilation tool and compilation flags
  • 🤔 Compares the output and execution time of your program against the reference program
  • 🤨 Judges you.

❌ What this doesn't do

  • 🐛 Fix your bugs automagically
  • ✏️ Write your program for you
  • 😎 Turn you into a pro-coder
  • 🧮 Solve the P versus NP problem
  • 🦄 Make unicorns real

Prerequisites

This tool is purely written in Bash, and works universally on most Linux systems (and probably also on WSL/emulators). The only required dependencies are the g++, script and time commands.

$ sudo apt install g++
This is most likely already installed on your system

Installation

The recommended way to grab a copy of the test suite is to run git clone.

$ git clone https://github.com/MarcusCemes/speleo-test-suite.git
Type this in a terminal (without the "$") to download the suite

If you don't have git, you can also use the Clone or download → Download ZIP button at the top of this page. This will not support updates.

Usage

The test suite comes with all the standard Speleo tests and the reference demo program by default. To run the test suite, provide your source code to the run.sh script like so:

$ ./run.sh ../path/to/source.cpp

You may need to add execute permissions with chmod +x run.sh


This will compile your C++ code with the official compilation flags and stricter syntax enforcement, before executing the binary against each test and running the performance analysis.

You may add your own tests to the tests/ directory. Each file represents a single test, and is piped to the program via stdin. The execution results, for manual inspection, will be generated under the results/ directory.

Performance Analysis

The performance analysis will score your program against a reference program. The Speleo binary is bundled by default, but this may be replaced with any other binary.

Each test is worth 3 points. A negative score for your test will deduct from your total.

Output correctness

The most important factor in the examination, verifying if your algorithm provides correct results.

  • +3 Correct - Your program outputs byte-perfect results.
  • +1 Similar - Your program has the same output with all spacing and new-lines removed. Maybe you're missing a new-line at the end?
  • +0 Incorrect - Your output did not match the reference output.

Execution time

While not explicitly required by the regulations, a long execution time probably means there's something quite wrong and may result in your submission being terminated pre-maturely for safety reasons.

  • +0 Negligible - If the reference program is faster than 100ms.
  • +0 Faster - Your program was faster than the reference program!
  • +0 Acceptable - Your program was 1-2x slower than the reference program.
  • -2 Too slow - Your program was more than 2x slower than the reference program!

Updating

The run.sh script can automatically update itself. If prompted, the script will automatically download new commits from GitHub and pull the changes into the repository using git pull.

In order for the auto-update function to work, you must have downloaded the repository using git clone. This will correctly initialize the the remote branch.

If you have messed something up, you can revert all local changes and re-synchronize yourself with GitHub by running the following:

$ git reset --hard origin/master
⚠ This will delete ALL modifications you may have made! Be warned! ⚠

To skip updates altogether, such as when making your own changes to avoid merge conflicts, you can do pass a second parameter as shown below. You can manually update by running git pull.

$ ./run.sh code.cpp --no-update

Contributing

This was written as my own collection of automation scripts, which I decided unify, document and share with others as a result of the amount of work I put into it.

Got an idea? Create an issue or submit a pull request.

Disclaimer

This is in no way an official evaluation, just a handy helper to make your life a bit easier. Don't rely solely on this.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Author