Clean and Simple Test
=====================
Author
------
Randall White
Copyright (C) 2015, 2017
Released under the MIT license.
(Please see LICENSE file in top-level directory.)
ranwhite@gmail.com
===============================================================================
Introduction
------------
casTest is a system of software tools (make files, scripts, library, and the
casTest executable) which is designed to make test driven development (or test
development in general) in C++ a bit easier. Hopefully, quite a bit easier.
casTest is distributed with all source code. It is available in an easy to use
self-extracting tarball from one of my dropbox folders. Just point your browser
to:
https://www.dropbox.com/sh/2wtklxt4tyqoant/AAA54F3BVCyvVCvJiYVz6WNPa?dl=0
or use wget:
wget https://www.dropbox.com/s/jv54lga0f57zooy/installCast.bin
wget https://www.dropbox.com/s/v0daaowfwjke3ej/installCast.bin.sha1
You can also download from GitHub:
https://github.com/obibuffett/casTest/releases
If you like, you can clone the project from GitHub:
- via browser:
https://github.com/obibuffett/casTest
- via commandline:
git clone https://github.com/obibuffett/casTest.git
casTest started out as a personal project just to explore questions related
to testing frameworks. As someone once said, to figure out how something
works, sometimes you have to implement it yourself. (I seem to recall reading
this in one of Scott Meyers' books, but I can't find the quote. If I have this
wrong, my apologies to both Scott Meyers and the actual author.) The casTest
framework is the result of this exploration and, at least to some extent, was
developed using Test Driven Development principles.
A primary goal of the project is to keep the code clean and simple by applying
the SOLID principles of software design as described by Robert C. Martin
(https://en.wikipedia.org/wiki/SOLID_(object-oriented_design)).
Another primary goal of casTest is to allow a developer to create tests
quickly. An initial (default) sanity test using casTest can be setup by
executing a single command. I use casTest for performing all my C++ code
katas.
===============================================================================
Installing from the Self-extracting tarball
--------------------------------------
Installing casTest from the self-extracting tarball is probably the easiest
method of installation. This is especially true for installing to a system
directory such as /usr/local or /opt as root.
To install casTest using this method, just copy installCast.bin to a directory
for which you have access and execute it.
Note: Currently, you must cd to the directory contatining installCast.bin
and execute it from there.
If you'd like to install as root, just use sudo (or su or login as root) to
run installCast.bin.
> ./installCast.bin
or
> sudo ./installCast.bin
Whether you install as a normal user or as root, you will be prompted for
a top-level installation location. To accept the displayed default location,
just hit [enter]. The default installation location for normal users is
$HOME/casTest. For root it is /usr/local/casTest.
After entering (or accepting) the installation location, installCast.bin will
do the following:
- extract itself to the selected location
- create resource scripts (castEnv.sourceMe.bash and castEnv.sourceMe.csh)
in the installation directory
- do a full build of all targets (libcasTest.a, casTest, and several
self-tests)
- run the self tests located in <castParentDir>/casTest/src/tests
NOTE: You can re-run the self-tests by executing:
$> casTest <castParentDir>/casTest/test/*.test
If you are installing as a normal user, installCast.bin will create a link
to castEnv.sourceMe.bash and castEnv.sourceMe.csh in your HOME directory.
You should source the file appropriate for your shell before beginning a test
creation session. Each file sets a few environment variables and adds the
path to the casTest executable to your PATH variable. If you don't use bash,
csh, or tcsh you'll need to create your own script to set the casTest
environment variables. Sorry for any inconvenience, one of these days I hope
to remove the need to source one of these files. If you are installing
as root, these links will not be created. In this case, users wishing to,
may create the link themselves:
#Creating a link to /usr/local/casTest/castEnv.sourceMe.bash
$> cd $HOME
$> ln -s /usr/local/casTest/castEnv.sourceMe.bash .
or for tcsh or csh
#Creating a link to /usr/local/casTest/castEnv.sourceMe.csh
$> cd $HOME
$> ln -s /usr/local/casTest/castEnv.sourceMe.csh .
Whether installed as a normal user or root, castEnv.sourceMe.[bash|csh] can
be sourced directly from the installation directory:
$> . <castParentDir>/casTest/castEnv.sourceMe.bash
or, of course,
$> source <castParentDir>/casTest/castEnv.sourceMe.csh
Installing from Git Repository Clone
-------------------------------
If you obtained casTest by cloning a Git repository, just cd to the top level
project directory and make the project:
$> cd <casTestParentDir>/casTest
$> make install
The top-level make file is written to automatically generate the required
environment scripts (for bash and csh), source the appropriate one (assuming
you are using bash or csh), and proceed to execute all make scripts
in the project tree. When executing "make install" the make targets are
installed as follows:
- casTest (executable): <castParentDir>/casTest/bin
- tpp2cpp (executable): <castParentDir>/casTest/bin
- libcasTest.a: <castParentDir>/casTest/lib
- casTest.sourceMe.bash: <castParentDir>/casTest
- casTest.sourceMe.csh: <castParentDir>/casTest
- casTest.sourceMe.bash: linked to $HOME
- casTest.sourceMe.csh: linked to $HOME
- All self-tests
-- aboutCmdTest.test: <castParentDir>/casTest/test
-- addTestSuiteCmdTest.test: <castParentDir>/casTest/test
-- castCmdExecTest.test: <castParentDir>/casTest/test
-- castCmdTest.test: <castParentDir>/casTest/test
-- castUtilTest.test: <castParentDir>/casTest/test
-- cmdFactoryTest.test: <castParentDir>/casTest/test
-- cmdLineTest.test: <castParentDir>/casTest/test
-- runTestsTest.test: <castParentDir>/casTest/test
-- testCaseTest.test: <castParentDir>/casTest/test
-- testLibTest.test: <castParentDir>/casTest/test
-- testSuiteMakefileTest.test: <castParentDir>/casTest/test
-- testSummaryTest.test: <castParentDir>/casTest/test
-- tpp2cppTest.test: <castParentDir>/casTest/test
-- usageCmdTest.test: <castParentDir>/casTest/test
-- verCmdTest.test: <castParentDir>/casTest/test
Before using casTest, source the appropriate resource script for
your shell (casTest.sourceMe.bash or casTest.sourceMe.csh).
===============================================================================
Quick Start
--------------------------------------
After installing casTest, just source castEnv.sourceMe.bash (or
castEnv.sourceMe.csh for csh or tcsh) and you'll be ready to create your
first test.
- Source castEnv.sourceMe.bash.
$> . ~/castEnv.sourceMe.bash
or
$> source ~/castEnv.sourceMe.csh
- Create a test dir and cd to it.
$> mkdir ~/myTestDir
$> cd ~/myTestDir
- Initialize a new test suite.
$> casTest -addTestSuite myTest
This step produces three files within the new directory:
-- Makefile (which calls other casTest generated make files within the
directory)
-- myTest.mak (make file for myTest.test)
-- myTest.tpp (source file for myTest.test)
Note that the test source uses ".tpp" as an appendix. This is done to make
it easier to generate the appropriate code to create the tests. A utility,
tpp2cpp, is used to process the .tpp file into a .cpp which is identical to
its corresponding .tpp file except for the following:
-- Any @SKIP directives (explained below) are commented out and the
following test is set to be skipped when executed.
-- A factory function is appended to the .cpp file which creates one of
each test defined in the .tpp file. The created factory function is
called by the framework when casTest is executed.
Line numbers of a .tpp file and its corresponding .cpp are identical with the
exception of the appended factory function.
With the exception of any @SKIP directives, myTest.tpp is a C++ source file. It
illustrates the use of the macros (defined in testCase.h) DEFINE_TEST
and END_DEF. These macros are provided to allow tests to be defined
quickly and painlessly. If you open myTest.tpp, you will see the
test, "DoNothing", defined as follows:
DEFINE_TEST(DoNothing)
void run()
{
}
END_DEF
- Run the DoNothing test using make.
$> make test
Make processes myTest.tpp, creates myTest.cpp, and builds, then runs
the test (myTest.test). The output will be similar to the following:
| /<CAST_DIR>/tools/tpp2cpp myTest.tpp
| tpp2cpp: myTest.tpp -> myTest.cpp
| src: myTest.tpp dst: myTest.cpp
| /usr/bin/g++ -o myTest.o -I/<CAST_DIR>/include -fpic\
| -D_XOPEN_SOURCE=700 -g -fpic -DCAS_TEST -c myTest.cpp
| /usr/bin/g++ -o myTest.test -I/<CAST_DIR>/include \
| -shared -L /<CAST_DIR>/lib myTest.o -lcasTest
| /<CAST_DIR>/bin/casTest myTest.test
|
| casTest: Running tests from: myTest.test
| 1..1
| ok 1 - DoNothing
|
| Summary:
| Tests ran: 1
| Passed: 1
| Skipped: 0
| Failed: 0
The purpose of the DoNothing test is ensure that we can get a test suite
executing quickly.
- After proving that all is well with the DoNothing test, you can delete it and
add the first test of the new test suite. For example, if we add the following
test (after deleting the DoNothing test) we will have a simple sanity check
of the test suite itself.
DEFINE_TEST(SanityTest)
void run()
{
bool val(false);
CK(val == true);
}
END_DEF
casTest uses a single evaluation function, CK(). Well, actually it is
a macro that calls TestCase::Assert(), passing the condition, an error
string created from the condition source, and the file/line number.
As you can see, CK() should fail when the line is executed as val is
initially defined as false. When writing your tests, just pass an
expression that should evaluate to true.
Creating a SanityTest like this is, of course, unnecessry. You could simply
begin writing your first test of the code you want to create. At least you
can do this once you have confidence that casTest will perform correctly.
- Running make again will produce something like the following:
$> make test
| /<CAST_DIR>/tools/tpp2cpp myTest.tpp
| tpp2cpp: myTest.tpp -> myTest.cpp
| src: myTest.tpp dst: myTest.cpp
| /usr/bin/g++ -o myTest.o -I/<CAST_DIR>/include -fpic\
| -D_XOPEN_SOURCE=700 -g -fpic -DCAS_TEST -c myTest.cpp
| /usr/bin/g++ -o myTest.test -I/<CAST_DIR>/include \
| -shared -L /<CAST_DIR>/lib myTest.o -lcasTest
| /<CAST_DIR>/bin/casTest myTest.test
|
| /<CAST_DIR>/tools/tpp2cpp myTest.tpp
| tpp2cpp: myTest.tpp -> myTest.cpp
| src: myTest.tpp dst: myTest.cpp
| /usr/bin/g++ -o myTest.o -I/<CAST_DIR>/include -fpic\
| -D_XOPEN_SOURCE=700 -g -fpic -DCAS_TEST -c myTest.cpp
| /usr/bin/g++ -o myTest.test -I/<CAST_DIR>/include \
| -shared -L /<CAST_DIR>/lib myTest.o -lcasTest
| /<CAST_DIR>/bin/casTest myTest.test
|
| casTest: Running tests from: myTest.test
| 1..1
| not ok 1 - SanityTest
| ---
| Caught cas::Test::Error: Assertion(true == val) FAILED: statTest.cpp:8
| ---
|
| Summary:
| Tests ran: 1
| Passed: 0
| Skipped: 0
| Failed: 1
|
| make: *** [test] Error 1
NOTE1: Of course if you prefer, instead of running "make test", you could
run make, then execute casTest passing in myTest.test in a second
step.
$> make
$> casTest myTest.test
You can also run multiple test suites from a single invocation.
$> casTest myTest.test myOtherTest.test anotherTest.test
NOTE2: The .test suffix is used to identify a shared library containing
a collection of one or more tests (ie a suite of tests). The
casTest executable loads each .test file specified on the command
line, maps appropriate symbols, then creates and executes the
tests within the shared library. The .test suffix is only a
convention. If you write your own make files, rather than using
the -addTestSuite command option to casTest, you can name them
anything you like.
- Next, edit myTest.tpp changing the line "bool val(false);" to read
"bool val(true);"
- Rerun the test.
$> make test
The output will now indicate success:
| /<CAST_DIR>/tools/tpp2cpp myTest.tpp
| tpp2cpp: myTest.tpp -> myTest.cpp
| src: myTest.tpp dst: myTest.cpp
| /usr/bin/g++ -o myTest.o -I/<CAST_DIR>/include -fpic\
| -D_XOPEN_SOURCE=700 -g -fpic -DCAS_TEST -c myTest.cpp
| /usr/bin/g++ -o myTest.test -I/<CAST_DIR>/include -shared\
| -L /<CAST_DIR>/lib myTest.o -lcasTest
| /<CAST_DIR>/bin/casTest myTest.test
|
| casTest: Running tests from: myTest.test
| 1..1
| ok 1 - SanityTest
|
| Summary:
| Tests ran: 1
| Passed: 1
| Skipped: 0
| Failed: 0
- Other tests can be added to myTest.tpp to create an entire suite of tests.
Typically, several related tests will be added to the same .tpp file.
Note that we use the term test suite in a generic sense. There is no
"TestSuite" class in the framework.
Hopefully, this short intro will get you started. For more detailed
information about the working of casTest, see the manual (well, once its
complete...it's still under construction).
Convenience Macros
------------------
As mentioned in the Quick Start section above, a test can be created with
the macro, DEFINE_TEST(<testName>). This macro takes a single argument
which is the name of the test class you wish to define.
For example:
//myTest.tpp
DEFINE_TEST(MyTest)
void run()
{
bool val(false);
CK(true == val);
}
END_DEF
expands to:
struct MyTest : cas::TestCase
{
MyTest()
cas::TestCase("MyTest")
{}
void run()
{
bool val(false);
CK(true == val);
}
};
cas::TestCase is an abstract class with run() being pure virtual.
cas::TestCase also defines the virtual methods setUp() and tearDown(). setUp()
is automatically called before run(); tearDown() after run() returns.
setUp(), as the name implies, is used to set up test prerequisites.
tearDown() is used to clean up resources that may have been allocated during
setUp(), or, in rare cases, during the running of the test. Of course,
you are free to use the functions as you see fit.
Two other macros are provided. DEFINE_BASE(<testName>) is used to define a
test case class that is intended to be used as a base class for other test
classes rather than executed as a test in and of itself.
DEFINE_TEST_FROM(<derivedTestName>, <baseTestName>) is used to derive a test
from another test. There are examples of the use of these macros in
the self-test source directory at <casParentDir>/casTest/src/test/.
As you can see, the macros are simply convenient tools to define a class and
provide a constructor that calls the base class, "cas::TestCase", passing the
name of the test as a string. Any C++ code that is appropriate within a class
definition can be added between the DEFINE_<type>() macro and the END_DEF
macro. For instance, perhaps you need to allocate an object dynamically for
your test. One way to do this might look something like:
//myTest.tpp
#include "myGame.h"
DEFINE_TEST(myGameSetPlayerNameTest)
void setUp()
{
newPlayer_ = new MyGame::Player;
}
void tearDown()
{
delete new_Player_;
}
void run()
{
std::string name("Uncle Bob");
newPlayer_.setName(name);
CK(name == newPlayer.name());
}
private:
MyGame::Player* newPlayer_;
END_DEF
In the above, a MyGame::Player object is created dynamically in setUp().
The setName() function is tested in run(). Then, the object created for the
test is destroyed in tearDown(). A member variable, newPlayer_* is also
added to the class definition with private access.
Skipping Tests
---------------------
A test can be skipped by adding the string "@SKIP" just above the line that
defines a test. For instance the following test will be skipped.
//myTest.tpp
@SKIP
DEFINE_TEST(MyTest)
void run()
{
bool val(false);
CK(true == val);
}
END_DEF
The string "@SKIP" must appear on the line before either the DEFINE_TEST or
the DEFINE_TEST_FROM macros.
Generated make files
--------------------
As mentioned above, the first time you create a new test suite, three files are
generated. The test source file (.tpp file) and two make files. The first,
makefile, named Makefile, is a top-level make file and just calls the other
make file that was created. The second makefile is a single target make file
with a .mak suffix. Subsequent uses of -addTestSuite within the same
directory, will generate additional test sources and single target make files,
and will add appropriate entries to the top-level make file (Makefile).
NOTE: This method of make file management was chosen simply because it is
similar to other projects on which I have worked. By collecting common
rules in project make files and including them, we are (in theory)
avoiding a lot of duplication in the make file heirarchy. Of course,
I readily admit that I'm no make expert, but this seems to work pretty
well. If you have a better idea, I'm all ears.
NOTE2: We use the term test suite in a general sense. There is no TestSuite
class defined by the project. In typical use, a single test file
(test suite) will house several related tests.
The single target make files (.mak suffix) are meant to be edited by the user.
Each includes comments indicating where it may be appropriate to customize.
For instance, if you are creating a test to test a class that depends on librt.so,
just add "-lrt" to the LIBS macro:
##### TODO #####
# If your target must be linked to
# additional libraries, list them here.
#
# EX:
# -lrt
################
LIBS := -lrt
Of course, for unit tests, you should probably consider mocking any call
outside of the class under test. One method I have used in production is to
create a mocks subdirectory within the test directory. By setting the INCLUDES
variable to include the mocks directory before others, the code under test can
use mocked functions and classes without being aware that they are just
standins. When mocking classes and functions, I usually just define the
mock entity fully within a header file named identically to one included by
the code under test.
If you need additional sources, add them to the SRCS macro. (Don't add any
sources to the TSTSRC macro as only one test source can be used.) Instructions
for most edits you may need to make to your .mak files are included as comments
in the generated .mak file.
===============================================================================
Test Output
-----------
casTest's output is very similar to the format specified for the Test Anything
Protocol (TAP). (See https://testanything.org/tap-specification.html.) After
reproting the test suite being executed, casTest prints the test plan. Using the
addTestSuiteCmdTest.test test suite from casTest's self tests as an example, this
will look something like this:
casTest: Running tests from: test/addTestSuiteCmdTest.test
1..4
This shows that there are 4 tests in the suite numbered 1 through 4.
Next, each test is executed. The result of the test (ok, not ok, or skipped)
is output with the test number and the name of the test. Again, using the
addTestSuiteCmdTest.test suite, casTest reports:
ok 1 - AddTestSuiteCmdCreatesTppSource
ok 2 - AddTestSuiteCmdThrowsIfCAST_DIRIsNotDefined
ok 3 - AddTestSuiteCmdThrowsIfTooFewArgs
ok 4 - AddTestSuiteCmdThrowsIfMakeFileTemplateCopyFails
Finally, casTest reports the summary of the tests that were run:
Summary:
Tests ran: 4
Passed: 4
Skipped: 0
Failed: 0
Here, all 4 tests passed.
When executing multiple test suites, the summary covers all tests run from
all test suites passed on the command line.
For failed tests, casTest will provide the condition that failed:
not ok 1 - SanityTest
---
Caught cas::Test::Error: Assertion(true == val) FAILED: stackTest.cpp:8
---
===============================================================================
Coverage
--------
The casTest make files provide a built-in coverage capability. To generate a
test which will produce gcov artifacts on execution, define the TAG variable to be
"-cov" when invoking make:
$> make TAG="-cov"
This will produce gcov instrumented test libraries for each test suite in the
directory. Each of these instrumented libraries will have an appendix of
".test-cov". For example, running this command in $<casTestParentDir>/src/test
results in the following test suite libraries being produced:
- aboutCmdTest.test-cov
- addTestSuiteCmdTest.test-cov
- castCmdExecTest.test-cov
- castCmdTest.test-cov
- castUtilTest.test-cov
- cmdFactoryTest.test-cov
- cmdLineTest.test-cov
- runTestsTest.test-cov
- testCaseTest.test-cov
- testLibTest.test-cov
- testSuiteMakefileTest.test-cov
- testSummaryTest.test-cov
- usageCmdTest.test-cov
- verCmdTest.test-cov
In addition, a gcov notes file (.gcno) is produced for each test library:
- aboutCmdTest-cov.gcno
- addTestSuiteCmdTest-cov.gcno
- castCmdExecTest-cov.gcno
- castCmdTest-cov.gcno
- castUtilTest-cov.gcno
- cmdFactoryTest-cov.gcno
- cmdLineTest-cov.gcno
- runTestsTest-cov.gcno
- testCaseTest-cov.gcno
- testLibTest-cov.gcno
- testSuiteMakefileTest-cov.gcno
- testSummaryTest-cov.gcno
- usageCmdTest-cov.gcno
- verCmdTest-cov.gcno
When executed, a gcov data file (.gcda) will be produced. gcov uses these
files to collect and report code coverage.
$> casTest *test-cov
------------------------------------------------------------------------
- Of course the two steps described above can be combined into one -
- using make to run the test: -
- -
- $> make TAG="-cov" test -
------------------------------------------------------------------------
After executing the instrumented test(s), use gcov to obtain the coverage
report:
$> gcov *gcda
gcov prints a summary of each source file that was executed and generates an
annotated source listing showing detailed coverage information for each.
Examining the annotated source will reveal the number of times each line
executed.
NOTE: I have found it very convenient to actually include the source .cpp file
under test in the .tpp file rather than adding it the the SRCS variable
in the .mak file. There are actually two reasons for this.
- First, I've found that when I add the .cpp file under test to the
SRCS variable it is hard to mock other objects/functions. For some
reason the directory containing the .cpp (usually the parent of the
test directory) is added to the the include directories automatically.
This is something I don't understand, but have encountered multiple
times on different linux variants. (As I said before, I'm not a make
expert.)
- The second reason is just a matter of making it easier to collect
coverage information. When adding the .cpp file under test to the
SRCS variable, the gcov files are deposited in the .cpp file's
directory (again, usually the parent directory of the test directory).
But, when simply including the .cpp file in the .tpp (as you would
typically include a header file), all gcov files are deposited in the
test directory. If you examine the .tpp files in
$<casTestDir>/src/test, you will observe that each .tpp file includes
the source file it is meant to test and that it is not listed in the
SRCS variable in its .mak file.
Note to the note: The including of a .cpp file within another source
file is not really that unusual. I once used a test suite under
Solaris which required this. While not required under casTest, I've
found this practice to be very useful.
NOTE2: I don't really like the convention of defining TAG on the commandline
to generate the coverage files. I don't think it will be that hard to
adopt another method, I just haven't taken the time to do so. This is
another item for the TODO list.
===============================================================================
Feedback is always welcome
--------------------------
Comments, question, and/or suggestions are welcome and would be greatly
appreciated. Feel free to contact me at GitHub (obibuffett), or send email to:
ranwhite@gmail.com
Thanks very much for trying casTest. I hope you find it useful.
Randy