SOFA Regression Documentation

This repository contains:

The tests to be perform on the scene for non regression: /Regression_test/
All the reference file of the tested scenes: /References

Building Regression_test program

This repository should be included inside the main SOFA repository as /applications/projects/Regression folder.
Then while configuring SOFA cmake project, check the option: APPLICATION_REGRESSION_TEST and be sure to also have SOFA_BUILD_TESTS.

1 - Non Regression Test general mechanism

1.a - Non regression-tests files

The Regression_test program will search for *.regression-tests files in the target directory (SOFA directory). This file contains a list of scene to be tested. Each line of the list file must contain:

A local path to the scene
The number of simulation steps to run
A numerical epsilon for comparison
Set this value to 1 if mechanicalObject inside a mapped Node need to be tested. Otherwise 0.
Set this value to 1 if only last iteration need to be dumped and tested. Otherwise 0.

See for example: SOFA_DIR/examples/RegressionStateScenes.regression-tests

### Demo scenes ###
Demos/caduceus.scn 100 1e-3 0 1
Demos/TriangleSurfaceCutting.scn 100 1e-4 1 1
Demos/liver.scn 100 1e-4 1 1

See https://github.com/sofa-framework/regression/blob/ca388cf402244e5196d2b46da69ed2f5a92fbdb1/Regression_test/RegressionSceneList.cpp#L73 for the parsing code.

The class RegressionSceneList is used to parse folders and look for those files. Right now we have:

State tests: RegressionStateScenes.regression-tests
Topology tests: RegressionTopologyScenes.regression-tests See section 4 to add a new type of test.

1.b - References

Reference files are stored in this repository under the References/ folder, with the same folder hiearchy as the target scenes. *For example, for Demo scenes inside SOFA, references will be stored inside References/Demos/ *

The reference files are generated when running the test for the first time on a scene and must be manually added to this repository. See section 3.a.

If improvements of SOFA break some regression tests, the reference files of those tests need to be replaced by the new ground truth. To do that, the reference files must be locally deleted so they can be regenerated by running the tests. Then the new referenced files need to be pushed onto this repository to update the CI.

1.c - Non regression tests types

For the moment non regression tests class are:

StateRegression_test: At each step, the state (position/velocity) of every independent dofs is compared to values in reference files.
TopologyRegression_test: At each step, the number of topology element of every topology container is compared to values in reference files.

2 - How to run the regression tests

2.a - Full solution: By setting up environment

The program Regression_test can not take arguments because of its internal mechanism. Thus to run it correctly, several Environment variables need to be set :

REGRESSION_SCENES_DIR : path to the root folder containing the scenes (usually SOFA_SRC_DIR/examples)
REGRESSION_REFERENCES_DIR : path to the root folder containing the references (this repository/References)
SOFA_ROOT : path to sofa build directory.
SOFA_PLUGIN_PATH : path to the plugin library to be loaded by "required plugin" in the scenes (usually SOFA_ROOT/lib)

Then just launch the Regression_test programm and it will parse the REGRESSION_SCENES_DIR folder to search for .regression-tests files.

2.b - Quick solution: By using python script

To avoid setting up Env. Variables, a python script regression_test.py, is available in this repository. You need to execute it by passing the Environment variables and paths as arguments of the script:

#arguments:
# 0 - script name
# 1 - regression_test binary path
# 2 - regression scenes dir
# 3 - regression references dir
# 4 - sofa build dir

Example of use on linux:

python regression_test.py /home/epernod/projects/regression_test_sofa/build/bin/Regression_test ~/projects/sofa-src ~/projects/regression_test_sofa/src/references ~/projects/sofa-build/

Example of use on windows:

python ./sofa-src/applications/projects/Regression/regression_test.py /c/projects/sofa-build/bin/Release/Regression_test.exe /c/projects/sofa-src/ /c/projects/sofa-src/applications/projects/Regression/references/ /c/projects/sofa-build/

3 - How to add new tests

3.a - Add a new SOFA scene to be tested

If the scene is already in SOFA repository:

Just add its path inside the corresponding file: Examples/RegressionStateScenes.regression-tests for state test or Examples/RegressionTopologyScenes.regression-tests for topology test.
Create the reference file by runing the Regression_test program.
Important Push first the reference files into the sofa-framework/regression repository so the CI has the new refs for your SOFA PR.
Push the updated .regression-tests file into your PR in sofa-framework/sofa repository

3.b - Add scene to be tested in a plugin

If the scene is inside a plugin: Create first a new .regression-tests file inside your plugin and do the steps mentionned in 3.a.

4 - Add a new non-regression test type

4.a - Tests internal mechanism

For each line of the regression-tests file, the class RegressionSceneList store all the parameters, as well as the scene path and its reference path, inside the a structure RegressionSceneData .

The result of this process is thus a vector of RegressionSceneData structures: std::vector<RegressionSceneData> m_scenes;

Then, for each RegressionSceneData a gtest is created and the method runTest from the Regression_test class will be called to really perform the test (or create the reference).

4.b - Add a new type of test

To add a new type of test you will need to add:

Inside the RegressionSceneList file:
- Update the RegressionSceneData structure to store, per scene, the data needed for the test.
Inside the Regression_test file:
- Add a new xxRegressionSceneList to specify the targeted name file. For example static struct StateRegressionSceneList : public RegressionSceneList is looking for "RegressionStateScenes.regression-tests" files.
- Add a new class xxRegression_test inheriting from BaseRegression_test. And implement the method void runTestImpl(RegressionSceneData data, sofa::simulation::Node::SPtr root, bool createReference = false); with the wanted test. This method is called by BaseRegression_test::runTest which will check for the scene.
See for example class StateRegression_test : public BaseRegression_test where runTestImpl check At each step, the state (position/velocity) of every independent dofs is compared to values in reference files and if the mean difference per node exceed threshold this will be reported as an error.
Finally in Regression_test.cpp you need to add the code to create a gtest for each RegressionSceneData of your list with your xxRegression_test::runTest method.

See for example the code of StateRegression_test:

/// Create one instance of StateRegression_test per scene in stateRegressionSceneList.m_scenes list
/// Note: if N differents TEST_P(StateRegression_test, test_N) are created this will create M x N gtest. M being the number of values in the list.
INSTANTIATE_TEST_CASE_P(Regression_test,
    StateRegression_test,
    ::testing::ValuesIn( stateRegressionSceneList.m_scenes ),
    StateRegression_test::getTestName);

// Run state regression test on the listed scenes
TEST_P(StateRegression_test, sceneTest)
{
    runTest(GetParam());
}

alxbilger/Regression