- The tests to be perform on the scene for non regression: /Regression_test/
- All the reference file of the tested scenes: /References
- This repository should be included inside the main SOFA repository as /applications/projects/Regression folder.
- Then while configuring SOFA cmake project, check the option: APPLICAITON_REGRESSION_TEST and be sure to also have SOFA_BUILD_TESTS.
The Regression_test program will search for *.regression-tests files in the target directory (SOFA directory). This file contains a list of scene to be tested. Each line of the list file must contain:
- A local path to the scene
- The number of simulation steps to run
- A numerical epsilon for comparison
- Set this value to 1 if mechanicalObject inside a mapped Node need to be tested. Otherwise 0.
- Set this value to 1 if only last iteration need to be dumped and tested. Otherwise 0.
See for example: SOFA_DIR/examples/RegressionStateScenes.regression-tests
### Demo scenes ###
Demos/caduceus.scn 100 1e-3 0 1
Demos/TriangleSurfaceCutting.scn 100 1e-4 1 1
Demos/liver.scn 100 1e-4 1 1
See https://github.com/sofa-framework/regression/blob/ca388cf402244e5196d2b46da69ed2f5a92fbdb1/Regression_test/RegressionSceneList.cpp#L73 for the parsing code.
The class RegressionSceneList
is used to parse folders and look for those files.
Right now we have:
- State tests: RegressionStateScenes.regression-tests
- Topology tests: RegressionTopologyScenes.regression-tests See section 4 to add a new type of test.
Reference files are stored in this repository under the References/ folder, with the same folder hiearchy as the target scenes. *For example, for Demo scenes inside SOFA, references will be stored inside References/Demos/ *
The reference files are generated when running the test for the first time on a scene and must be manually added to this repository. See section 3.a.
If improvements of SOFA break some regression tests, the reference files of those tests need to be replaced by the new ground truth. To do that, the reference files must be locally deleted so they can be regenerated by running the tests. Then the new referenced files need to be pushed onto this repository to update the CI.
For the moment non regression tests class are:
- StateRegression_test: At each step, the state (position/velocity) of every independent dofs is compared to values in reference files.
- TopologyRegression_test: At each step, the number of topology element of every topology container is compared to values in reference files.
The program Regression_test can not take arguments because of its internal mechanism. Thus to run it correctly, several Environment variables need to be set :
- REGRESSION_SCENES_DIR : path to the root folder containing the scenes (usually SOFA_SRC_DIR/examples)
- REGRESSION_REFERENCES_DIR : path to the root folder containing the references (this repository/References)
- SOFA_ROOT : path to sofa build directory.
- SOFA_PLUGIN_PATH : path to the plugin library to be loaded by "required plugin" in the scenes (usually SOFA_ROOT/lib)
Then just launch the Regression_test programm and it will parse the REGRESSION_SCENES_DIR folder to search for .regression-tests files.
To avoid setting up Env. Variables, a python script regression_test.py, is available in this repository. You need to execute it by passing the Environment variables and paths as arguments of the script:
#arguments:
# 0 - script name
# 1 - regression_test binary path
# 2 - regression scenes dir
# 3 - regression references dir
# 4 - sofa build dir
python regression_test.py /home/epernod/projects/regression_test_sofa/build/bin/Regression_test ~/projects/sofa-src ~/projects/regression_test_sofa/src/references ~/projects/sofa-build/
python ./sofa-src/applications/projects/Regression/regression_test.py /c/projects/sofa-build/bin/Release/Regression_test.exe /c/projects/sofa-src/ /c/projects/sofa-src/applications/projects/Regression/references/ /c/projects/sofa-build/
If the scene is already in SOFA repository:
- Just add its path inside the corresponding file: Examples/RegressionStateScenes.regression-tests for state test or Examples/RegressionTopologyScenes.regression-tests for topology test.
- Create the reference file by runing the Regression_test program.
- Important Push first the reference files into the sofa-framework/regression repository so the CI has the new refs for your SOFA PR.
- Push the updated .regression-tests file into your PR in sofa-framework/sofa repository
If the scene is inside a plugin: Create first a new .regression-tests file inside your plugin and do the steps mentionned in 3.a.
For each line of the regression-tests file, the class RegressionSceneList
store all the parameters, as well as the scene path and its reference path, inside the a structure RegressionSceneData
.
The result of this process is thus a vector of RegressionSceneData
structures: std::vector<RegressionSceneData> m_scenes;
Then, for each RegressionSceneData
a gtest is created and the method runTest
from the Regression_test class will be called to really perform the test (or create the reference).
To add a new type of test you will need to add:
-
Inside the RegressionSceneList file:
- Update the
RegressionSceneData
structure to store, per scene, the data needed for the test.
- Update the
-
Inside the Regression_test file:
- Add a new xxRegressionSceneList to specify the targeted name file. For example
static struct StateRegressionSceneList : public RegressionSceneList
is looking for "RegressionStateScenes.regression-tests" files. - Add a new class xxRegression_test inheriting from BaseRegression_test. And implement the method
void runTestImpl(RegressionSceneData data, sofa::simulation::Node::SPtr root, bool createReference = false);
with the wanted test. This method is called byBaseRegression_test::runTest
which will check for the scene.
See for example
class StateRegression_test : public BaseRegression_test
where runTestImpl check At each step, the state (position/velocity) of every independent dofs is compared to values in reference files and if the mean difference per node exceed threshold this will be reported as an error. - Add a new xxRegressionSceneList to specify the targeted name file. For example
-
Finally in Regression_test.cpp you need to add the code to create a gtest for each RegressionSceneData of your list with your xxRegression_test::runTest method.
/// Create one instance of StateRegression_test per scene in stateRegressionSceneList.m_scenes list
/// Note: if N differents TEST_P(StateRegression_test, test_N) are created this will create M x N gtest. M being the number of values in the list.
INSTANTIATE_TEST_CASE_P(Regression_test,
StateRegression_test,
::testing::ValuesIn( stateRegressionSceneList.m_scenes ),
StateRegression_test::getTestName);
// Run state regression test on the listed scenes
TEST_P(StateRegression_test, sceneTest)
{
runTest(GetParam());
}