FortranTestGenerator

FortranTestGenerator (FTG) is a tool for automatically generating unit tests for subroutines of existing Fortran applications based on an approach called Capture & Replay.

One of the main effort for creating unit tests is the set-up of consistent input data. When working with legacy code, we can make use of the existing infrastructure and extract test data from the running application. FTG generates code for serializing and storing a subroutines input data and inserts this code temporarily into the subroutine (capture code). In addition, FTG generates a basic test driver which loads this data and runs the subroutine (replay code). Meaningful checks and test data modification needs to be added by the developer. All the code generated by FTG is based on customizable templates. So you are able to adapt it to your software environment.

FTG shall work with any kind of Fortran code up from Fortran90, but has not yet been tested with every single feature from every single standard. It is written in Python and the principles of FTG are described in the following paper:

C. Hovy and J. Kunkel, "Towards Automatic and Flexible Unit Test Generation for Legacy HPC Code," 2016 Fourth International Workshop on Software Engineering for High Performance Computing in Computational Science and Engineering (SE-HPCCSE), Salt Lake City, UT, 2016, pp. 1-8. DOI: 10.1109/SE-HPCCSE.2016.005 (Download PDF)

So far, the documentation is not complete. If your interested in using FortranTestGenerator, please feel free to contact me:
Christian Hovy <hovy@informatik.uni-hamburg.de>

ATTENTION: The latest version uses Serialbox2 instead of Serialbox-ftg.

In general it works as follows

You identify an existing subroutine in your Fortran application and a certain execution of this subroutine that you want to run for test purposes in isolation, that is without the surrounding application.
You run FTG to insert the capture code into the subroutine. This code is responsible for storing all input variables to you hard drive when the capturing is active. Thanks to a built-in static source code analysis, only those variable are captured that are actually needed by the subroutine or by one of its directly or indirectly called routines.
Then you define the event on which the capturing should take place. By default, it's the first execution of the subroutine.
You compile and run your application with the capture code.
You run FTG to create the replay code, that means a basis test driver which loads the captured data and calls the subroutine by passing the captured data as input.

Variables that are considered to be input data:

Arguments of intrisic types
Components of derived type arguments that are actually used by the subroutine
Module variables of the same module or imported by USE statements that are actually used by the subroutine

For the source code analysis, FTG uses the tool FortranCallGraph, which needs assembler files generated by gfortran for the analysis.

Prerequisites

To run FTG, you will need the following software packages:

Python 2.7 or 3.x
Cheetah Template Engine: https://github.com/CheetahTemplate3/cheetah3
Serialbox2 (>= 2.5.1): https://github.com/eth-cscs/serialbox2
FortranCallGraph (>= 1.4.0): https://github.com/fortesg/fortrancallgraph

Quick Start Guide

1. Get and install `Serialbox2`

... from here: https://github.com/eth-cscs/serialbox2 and learn how to build your application with it. Make sure that you build Serialbox2 with CMake options SERIALBOX_ENABLE_FORTRAN and SERIALBOX_ENABLE_FTG switched on.

2. Get and install the `Cheetah Template Engine`

...from here: https://github.com/CheetahTemplate3/cheetah3 or just look if your OS provides a package (e.g. Ubuntu does).

3. Get `FortranCallGraph`

...from here: https://github.com/fortesg/fortrancallgraph

4. Configure and try `FortranCallGraph`

...according to its documentation.

5. Clone this repo

$> git clone https://github.com/fortesg/fortrantestgenerator.git
$> cd fortrantestgenerator

6. Fill out the configuration file config_fortrantestgenerator.py

The meaning of the variables is documented in the sample configuration file.

7. Create assembler files

Compile your Fortran application with gfortran and the options -S -g -O0 or -save-temps -g -O0 to generate assembler files.

8. Create capture code

Let's assume your subtroutine under test is the subroutine my_subroutine from the module my_module. Just run:

$> ./FortranTestGenerator.py -c my_module my_subroutine

9. Define capture event

Have a look at the generated code in the module file where the subroutine under test (my_subroutine) is located. When using one of the provided templates, there are now the two functions: ftg_my_subroutine_capture_input_active and ftg_my_subroutine_capture_output_active. Those functions define when the time is come to capture the subroutines' input and output.

By default, both functions just compare the variable ftg_my_subroutine_round, in which the subroutine executions are counted, with the variable ftg_my_subroutine_round. By default, ftg_my_subroutine_capture_round is set to 1, which means that the capturing takes place in the first execution of my_subroutine.

If you want the capturing to happen for example in the 42nd execution of my_subroutine, just set ftg_my_subroutine_capture_round to 42, but you can also change the functions to what ever you like. If you want to make the time for capturing dependent on the status of another variable, you can also add arguments to those functions. Of course, then you need to add the arguments also at the places where the functions are called.

10. Create folders for the captured data

Create the directory defined in your configuration file in the variable TEST_DATA_BASE_DIR.

11. Compile and run your application with the capture code

This will only work if you have added the includes and libraries of Serialbox2 to your build configuration, see step 1.

When the capturing is taking place, there will be messages printed to stdout beginning with FTG....

When each MPI process has printed FTG FINALIZE OUTPUT DATA my_subroutine, capturing has finished and you can kill your application.

12. Create replay code

Run:

$> ./FortranTestgenerator.py -r my_module my_subroutine

13. Compile and run the generated test driver (replay code)

You have to run the test with the same numbers of MPI processes as you have done for capturing.

14. Compare the original output with the output from the test

The original output is located in TEST_DATA_BASE_DIR/ftg_my_subroutine_test/output and the test output was put into TEST_DATA_BASE_DIR/ftg_my_subroutine_test/output_test.

To compare the data, you can use the Python tool compare.py included in Serialbox2.

Do the following:

$> cd TEST_DATA_BASE_DIR/ftg_my_subroutine_test
$> <serialbox2_install_path>/python/compare/compare.py output/ftg_my_subroutine_output_0.json output_test/ftg_my_subroutine_output_0.json

This compares the output for the first MPI process. Replace _0 by _1, _2, etc. for comparing the output of the other processes.

If deviations are shown, it's up to you to figure out what went wrong, for example if one variable was missed by the source code analysis or if there is some kind of non-determinism in your code.

When using the template IconCompare, results are checked automatically by the test driver after running the subroutine under test. You don't have to use the compare tool.

15. Make a real test out of the generated test driver

For example add some checks, modify the loaded input data and run the subroutine under test again etc.

You should also remove the dependencies to the capture code, so that you can remove that stuff from the subroutine and its module.

If you want to load the original output data for your checks, just have a look how this is done for the input data.

Some basic checks will be added to the provided templates in the future.

Please Note

FortranTestGenerator.py -c and -r not only generate capture and replay code, but also add PUBLIC statements in every module that contains a module variable that is needed by the test and not yet public (export code). This only works for module variables that are private because the whole module is private and they are not explicitly set to public. If a variable is private because it has the private keyword in its declaration, this procedure won't work and you have to manually make them public. The compiler will tell you if there is such a problem. Similar problems can occure elsewhere. With -e you can only create the export code.
For each module that is modified a copy of the original version is created with the file ending .ftg-backup. You can restore these backups by running
```
$> ./FortranTestGenerator.py -b
```

$> ./FortranTestGenerator.py -a

The latter will only restore the capture code backups and the export code backups.

You can combine the options -a, -b, -c, -e and -r in any combination. When running FortranTestGenerator.py with -a or -b option, restoring the backups will always be the first action, and when running with -r, generating the replay code will come at last.
If you want any generated code to stay, just remove the corresponding .ftg-backup file, so it want be considered by -a or -b. It then might make sense to add some preprocessor directives around the generated code (e.g. something like #ifdef __FTG_TESTS_ENABLED__ ... #endif). If you want to have such directives always be there, just add them to the template you are using.
As long as there is a backup file, any analysis is done on this instead of the original file.
As mentioned before, the static source code analysis is done by FortranCallgraph which combines an analysis of assembler files with an analysis of the original source code. Actually, it first creates a call graph with the subroutine under test as root by parsing the assembler files and then it traverses this call graph while analysing the original (unpreprocessed) source files. This procedure can lead to problems if your code contains too much preprocessor acrobatics.

And there are also other cases where the assembler code differs from the orginal source code. Example:
```
LOGICAL, PARAMETER check = .TRUE.
IF (check) THEN
  CALL a()
ELSE
  CALL b()
END IF
```
Even when compiled with -O0, the ELSE block in this example won't be in the assembler/binary code. But usually this is not a problem, there will just be a warning during the analysis that the subroutine b is not found in the call graph.
When you change your code, you will have to compile again with -S -g -O0 to generate new assembler files. For example, when you have generated capture and export code and removed some backup files to make the code permanent, you have to compile again.
The static source code analysis has the same limitations as every static analysis, it can only find out what can be found out by parsing the source code. So mainly, it can not handle runtime polymorphism. That means, the use of, for example, function pointers or inheritance can lead to wrong results.
The FortranTestGenerator frontend of the SerialBox2 library contains code like
```
IF (ASSOCIATED(ptr)) THEN
   ! write ptr
THEN
```
to prevent unassociated pointer variables and alike from being captured.

Unfortunately, the ASSOCIATED function only works properly for pointer variables that have either been nullified or actually associated with some variable, but not with variables that have never been initialized.

So, if a segmentation fault occures during capturing, please check first if an uninitialized pointer has been passed. It is good practice to add => NULL() to every declaration of a pointer variable.
If any problem occurs, please feel free to contact me:
Christian Hovy <hovy@informatik.uni-hamburg.de>

Modifying the templates

The templates are based on the Cheetah Template Engine and an API which has no documentation so far. Please ask me, if you need help with adapting the generated code to your needs:
Christian Hovy <hovy@informatik.uni-hamburg.de>

Notes for ICON developers

1. Build ICON with Serialbox2

For including the libraries, you can just use the OTHER_LIBS variable in your mh-linux file:

OTHER_LIBS  = ${OTHER_LIBS} -L$SERIALROOT/lib  -lSerialboxFortranStatic -lSerialboxCStatic  -lSerialboxStatic -lstdc++ -lstdc++fs

For including the includes, there is no such variable, so I have just addded them to the FFLAGS variable:
```
FFLAGS = $FFLAGS -I$SERIALROOT/include
```

2. Create assembler files

I have done it like this:

In my mh-linux file I have added $FFLAGS itself to FFLAGS under the gcc section:
```
FFLAGS      = $FFLAGS $FCPP $FLANG $FWARN $INCLUDES
```

Then, when I want to create the assembler files, I just run:

$> make clean
$> export FFLAGS='-save-temps -g -O0' && ./configure && make
$> find build/x86_64-unknown-linux-gnu -name *.f90 -delete
$> export FFLAGS='' && ./configure && make

3. Configuration

My FortranCallGraph configuration file for ICON

My FortranTestGenerator configuration file for ICON

My FCG configuration chooses the *_orig subtypes of t_comm_pattern and t_comm_pattern_collection as the one and only implementations. If you are working with yaxt, please change the configuration accordingly:

ABSTRACT_TYPE_IMPLEMENTATIONS = {'t_comm_pattern':'t_comm_pattern_orig',
                                 't_comm_pattern_collection':'t_comm_pattern_collection_orig'}

4. Compiling the tests

When using the icon_standalone template, just put the generated test files into the src/tests directory:
```
TEST_SOURCE_DIR = ICON_DIR + '/src/tests'
```
./configure will then automatically add the test to the Makefile and you will get a binary in build/.../bin.
You can also generate tests for the testbed by using the icon_testbed templates, but this will be a bit more complicated. You will then have to integrate the generated modules manually into the testbed environment and create proper run scripts.

5. Uninitialized pointer variables

Please see the note on that above. Unfortunately, ICON contains a lot of such variables, especially in the derived types. Just add => NULL() to all of the declarations if possible.

6. JSBACH

Due to its dynamic data structures JSBACH cannot be analyzed properly by FortranCallGraph. Therefore, I have created a mock interface that captures and replays the output of the original JSBACH interface: https://github.com/fortesg/jsbach-mock. The template IconJsbachMock makes use of this interface.

License

GNU General Public License v3.0

fortesg/fortrantestgenerator