/cdss-app-statedmi-test

Functional tests for StateDMI using command files.

Primary LanguageCAP CDSCreative Commons Attribution 4.0 InternationalCC-BY-4.0

cdss-app-statedmi-test

This repository contains functional tests for the StateDMI software. StateDMI is software developed for Colorado's Decision Support Systems (CDSS).

StateDMI functional tests consist of StateDMI command files and test data used in automated regression tests. The purpose of these tests is to validate StateDMI command functionality for default command parameters and combinations of command parameter values. The tests are also helpful for understanding how to use StateDMI commands and can therefore be used as a reference. See the following online resources:

See the following sections in this page:

Cloning the Repository

Because the tests can be run from any folder, the repository can be cloned into any folder.

If the tests are being included in a StateDMI Eclipse development workspace, then clone the repository into a folder parallel to other StateDMI component repositories so that the tests can be conveniently run from the developer environment.

Warning - there are hundreds of tests so cloning the full repository will result in many files. Evaluate whether individual test files should be downloaded instead.

Repository File Structure

The important files in the repository are as follows (other files can generally be ignored):

test/                                       Top-level folder for all tests.
  regression/                               Top-level folder for automated regression tests.
    commands/                               Folder containing command tests.
      CommandName/                          Command name, as in StateDMI.
        Data/                               Data used for command tests, usually small datasets.
        ExpectedResults/                    Expected output from test when files are involved,
                                             generated, inspected, and saved for continued use.
        Results/                            Results generated by the current test runs.
        Test_CommandName_TestName.StateDMI  Individual test command files to run.
    scripts/                                Supporting scripts.
    src/                                    Source could used to implement UI tests
                                            (old can can be deleted after review).
    TestSuites/                             Folder for saving test suites, when the above command
                                             tests are run in groups.
      commands/                             Folder for general command test suite, to
                                             run all commands from above in bulk.
        create/                             Includes StateDMI command files to create the test suite.
        run/                                Includes auto-generated test suite to run many command tests.

It is undesirable to store dynamic test results in the repository. Therefore, each Results folder includes a .gitignore file to ignore all files in the folder, except for the .gitignore file itself. Other approaches were evaluated and may be implemented in the future, such as wildcards; however, using individual .gitignore files is fail-safe. The empty Results folders will be available if the repository is cloned. This is helpful because StateDMI commands do not generally auto-generate output folders.

See the Configuring a Test section for more information about a test.

Configuring a Test

Each StateDMI test is a command file that is intended to test the functionality of a single command or some reasonably small combination of commands. Command files typically include the following steps involving StateDMI commands:

  • Setup step(s), such as opening a log file and defining synthetic time series data with Set* and other commands. In some cases, data are read from files using various Read... commands (input is generally stored in the Data folder under the command test folder).
  • Execution step(s), in particular running the command to be tested, which will modify in-memory data and/or write output to file(s).
  • Test/comparison step(s), for example using the CompareFiles or other command. These are equivalent to unit test "assert" tests commonly used in software testing.
  • Teardown/cleanup step(s), such as removing files or freeing in-memory data objects. However, because dynamic test result output is not saved to the Git repository, there is generally not a need to do much tear-down. A specific example where tear-down is appropriate is when testing the SetDebugLevel command it is necessary to turn debugging off after the test so that subsequent commands don't run in debug mode, which generates a lot of output and is slow.

Refer to the command tests for examples of how the above template is implemented. Newer tests include better comments.

Running the Tests

The functional tests are run by the StateDMI software from the development environment or a normal install. Tests can be run from the StateDMI user interface or in batch mode. This approach allows StateDMI to test itself as well as being used to test essentially any workflow process that can be run by StateDMI, including running external programs. The advantage of this approach is that StateDMI users can create automated tests without having to install and be proficient with the StateDMI software development environment.

Run a Single Test

Running a single test is the approach taken when doing test-driven development. This is particularly efficient given that StateDMI provides a list of previously-opened command files and the top choice can be selected when StateDMI is started. Although unit tests in the code could be implemented, function tests validate the full software functionality and ensure that tests are created that can be run in the operational environment, if needed.

To run a single test in the StateDMI user interface, use the StateDMI File / Open / Command File menu and browse to the location of one of the command *.StateDMI test files. Once loaded into the StateDMI interface, use the Run All Commands button. If the test is successful, no red or yellow markers will be shown. Note that some tests are structured to generate an error or warning on purpose in which case an #@expectedStatus comment will usually be found at the top of the test command file. Whether or not the test is considered a pass or fail depends on the expected and actual result of running the command file.

Run a Test Suite

Running the general commands test suite is the approach taken to verify software functionality before a software release. Because some tests require a database connection or other setup (done outside of the individual test), it can be difficult to run the full test suite for all tests. In some cases a subset will be run based on specific system setup. These are details that StateDMI maintainers are familiar with.

To run the full test suite for all commands, first use the StateDMI File / Open / Command File menu and browse to the TestSuites/commands/create folder and select Create_RunTestSuite_commands.StateDMI command file. Run the commands, which will generate the command file that runs all the tests, as described below.

To run the full test suite for all commands, use the StateDMI File / Open / Command File menu and browse to the TestSuites/commands/run folder and select RunRegressionTest_commands.StateDMI. Once loaded into the StateDMI interface, use the Run All Commands button. This will run all the tests.

  • Some tests always fail - these are tests that need additional attention and are left in so that they are not forgotten. Generally they are seldom used or legacy commands that need more attention.

  • Some tests are expected to fail and will generate warning and/or failure indicators. However, the true indication of whether a test passed is whether the output results match the expected results.

  • Text report, csv, and Excel output files are generated to facilitate reviewing the results.

The test suite can also be run in batch mode with StateDMI -commands xxx.StateDMI. However, this has not traditionally been done because the StateDMI development has not used continuous integration and tests that currently always fail would need to be cleaned up.

License

The license for the tests is the Creative Commons Attribution International 4.0 (CC BY 4.0) license.

Contributing

Contribute to the documentation as follows:

  1. Use GitHub repository issues to report minor issues. Fill out the template issue.
  2. Use GitHub pull requests.
  3. A member of the core development team will follow up to issues and pull requests.

Maintainers

This repository is maintained by the OpenCDSS team.