This repository contains an early-stage solution to compare the output of different implementations in the algorithm archive. Comparing the output is done by executing each implementation and retrieving the processes output and checking it against a given file containing the expected values.
As this project was written in C#, dotnet core 5.X is needed to run it.
Installation instructions can be found here.
You can confirm your installation by executing dotnet --version and checking that it shows a version greater than 5.0.
Installing is as simple as cloning the repository into a local folder:
git clone https://github.com/stormofice/AAA-Implementation-Verifier.git
Navigate into the newly created directory and execute the following:
dotnet run <content path> <config path> <output path>
content path is the path to either the contents directory of the algorithm archive to test all chapters or the path to a single chapter to only test its implementations
config path is the path to the directory containing the language configuration files, by default it should be execution_config/
output path is the path to a temporary folder containing files generated by compilation and executing the implementations (it will be automatically created and cleared after every run)
The behaviour of the verifier can be changed by modifying the internal.json file in the execution_config/ directory.
The following configuration values are available:
- LogLevel: Controls the verbosity of the output (Debug = 0, Info = 1, Warn = 2, Error = 3)
- FileExtensions: Changes which files get considered based on their extension. If you include "all" as an extension, every file gets checked (only if a language config is available)
- ShowExecutionErrorOutput: Prints out data on the error stream of created processes; this includes compiler errors or runtime errors of programs
- ShowExecutionStandardOutput: Prints out data on the standard output stream of created processes; this basically just dumps the output of every executed implementation (good for comparing outputs manually, if there is no file for expected values yet)
- StopOnExecutionError: Controls whether the program should be aborted if compilation or execution of a single implementation fails
- RedirectJsonToFile: If this is given, then the test results will be printed to the given path
- IgnoreMissingExpectedValues: If this is enabled, the execution will not abort if no suitable
expected.jsonfile could be found; this setting should be enabled until every chapter has been standardized
Information on how to run a given language is kept in language configuration files in the execution_config/ directory.
While naming this file to reflect the language it configures is not necessary, it is highly recommended (so the file containing the java configuration should be named java.json).
The following configuration values are necessary:
- Language: Human friendly name of the language
- Extension: The file extension corresponding to the language; this should be unique among all languages
- Description: This is the section to put explanatory comments on the steps below, if needed
- Steps: Contains all steps to execute
An execution step can be configured as follows:
- Runtime: The program which should be executed; it is strongly recommended to assume that the executable is in the
PATHenvironment variable and to not hardcode the path (usegccinstead of/usr/bin/gcc) - Command: A formatted strings which contains the arguments passed to the executable given in the
runtimeargument. It is possible to use placeholders like{0}..{n}and replace them with values given below (in theargsargument) - Args: An array which contains values fitting to the placeholders mentioned above; this array must be the same length as the amount of placeholders given above
There are some magic values, which can be used in the configuration above. They only work in the Runtime and Args parameter, but it is recommend to use the Args array extensively, to keep it readable.
Keep in mind that every process gets started with the working directory set to output path argument as described above.
- ALL_FILES_IN_DIR: - This gets replaced by every file in the directory of the code file
- FILE_NAME_WEX: - The chapterPath of the implementation without its extension
- FILE_PATH: - The full path to the code file
- WORKING_DIR_FULL: - The full path to the working directory
- WORKING_DIR: - The path to the working directory as given by user input
{
"Language": "asm x86_64", // The name of the language, ideally
// this is the same as it gets displayed in the AAA
"Extension": "s", // The file extension of the language, in this case "s";
// the dot must be ommited
"Description": "", // You could describe the flags used below here
"Steps": [
{ // Compilation is the first step
"Runtime": "gcc", // Path to the compiler to use
"Command": " -no-pie -o {0} {1} -lm", // The argument string similar to how
// you would run it in your terminal
"Args": [ // As we used to placeholders, we need two arguments
"FILE_NAME_WEX", // We want to output the file into the <output path> directory
// with a predicable name, in this case it's the file name
// without the extension (verlet.s -> verlet)
"ALL_FILES_IN_DIR" // These are the input files to use; this gets replaced
// by all files in the same directory as the original asm file.
// This is done as there *may* be given headers or object files
// to use for compilation
]
},
{ // Now that the compilation is finished, we can go ahead
// and execute the program
"Runtime": "WORKING_DIR/FILE_NAME_WEX", // As the compiler was started with its working
// directory set to the <output path> directory
// and we changed the name, we know where the
// produced executable is
// This is similar to runnig ./verlet in your terminal
"Command": "", // As we don't need to pass any arguments, this can stay empty
"Args": [] // As there are no placeholders, no replacements are needed
}
]
}