/cmockery

A lightweight library to simplify and generalize the process of writing unit tests for C applications.

Primary LanguageShellApache License 2.0Apache-2.0

Cmockery Unit Testing Framework

Cmockery is a lightweight library that is used to author C unit tests.

Contents

Motivation

There are a variety of C unit testing frameworks available however many of them are fairly complex and require the latest compiler technology. Some development requires the use of old compilers which makes it difficult to use some unit testing frameworks. In addition many unit testing frameworks assume the code being tested is an application or module that is targeted to the same platform that will ultimately execute the test. Because of this assumption many frameworks require the inclusion of standard C library headers in the code module being tested which may collide with the custom or incomplete implementation of the C library utilized by the code under test.

Cmockery only requires a test application is linked with the standard C library which minimizes conflicts with standard C library headers. Also, Cmockery tries avoid the use of some of the newer features of C compilers.

This results in Cmockery being a relatively small library that can be used to test a variety of exotic code. If a developer wishes to simply test an application with the latest compiler then other unit testing frameworks maybe preferable.

Overview

Cmockery tests are compiled into stand-alone executables and linked with the Cmockery library, the standard C library and module being tested. Any symbols external to the module being tested should be mocked - replaced with functions that return values determined by the test - within the test application. Even though significant differences may exist between the target execution environment of a code module and the environment used to test the code the unit testing is still valid since its goal is to test the logic of a code modules at a functional level and not necessarily all of its interactions with the target execution environment.

It may not be possible to compile a module into a test application without some modification, therefore the preprocessor symbol UNIT_TESTING should be defined when Cmockery unit test applications are compiled so code within the module can be conditionally compiled for tests.

Test Execution

Cmockery unit test cases are functions with the signature void function(void *``*state). Cmockery test applications initialize a table with test case function pointers using unit_test*() macros. This table is then passed to the run_tests() macro to execute the tests.

run_tests() sets up the appropriate exception / signal handlers and other data structures prior to running each test function. When a unit test is complete run_tests() performs various checks to determine whether the test succeeded.

Using run_tests()

run_tests.c

#include <stdarg.h>
#include <stddef.h>
#include <setjmp.h>
#include <cmockery.h>

// A test case that does nothing and succeeds.
void null_test_success(void **state) {
}

int main(int argc, char* argv[]) {
    const UnitTest tests[] = {
        unit_test(null_test_success),
    };
    return run_tests(tests);
}

Exception Handling

Before a test function is executed by run_tests(), exception / signal handlers are overridden with a handler that simply displays an error and exits a test function if an exception occurs. If an exception occurs outside of a test function, for example in Cmockery itself, the application aborts execution and returns an error code.

Failure Conditions

If a failure occurs during a test function that's executed via run_tests(), the test function is aborted and the application's execution resumes with the next test function.

Test failures are ultimately signalled via the Cmockery function fail(). The following events will result in the Cmockery library signalling a test failure...

Assertions

Runtime assert macros like the standard C library's assert() should be redefined in modules being tested to use Cmockery's mock_assert() function. Normally mock_assert() signals a test failure. If a function is called using the expect_assert_failure() macro, any calls to mock_assert() within the function will result in the execution of the test. If no calls to mock_assert() occur during the function called via expect_assert_failure() a test failure is signalled.

Using mock_assert()

assert_module.c

#include <assert.h>

// If unit testing is enabled override assert with mock_assert().
#if UNIT_TESTING
extern void mock_assert(const int result, const char* const expression, 
                        const char * const file, const int line);
#undef assert
#define assert(expression) \
    mock_assert((int)(expression), #expression, __FILE__, __LINE__);
#endif // UNIT_TESTING

void increment_value(int * const value) {
    assert(value);
    (*value) ++;
}

void decrement_value(int * const value) {
    if (value) {
        (*value) --;
    }
}

assert_module_test.c

#include <stdarg.h>
#include <stddef.h>
#include <setjmp.h>
#include <cmockery.h>

extern void increment_value(int * const value);

/* This test case will fail but the assert is caught by run_tests() and the
 * next test is executed. */
void increment_value_fail(void **state) {
    increment_value(NULL);
}

// This test case succeeds since increment_value() asserts on the NULL pointer.
void increment_value_assert(void **state) {
    expect_assert_failure(increment_value(NULL));
}

/* This test case fails since decrement_value() doesn't assert on a NULL
 * pointer. */
void decrement_value_fail(void **state) {
    expect_assert_failure(decrement_value(NULL));
}

int main(int argc, char *argv[]) {
    const UnitTest tests[] = {
        unit_test(increment_value_fail),
        unit_test(increment_value_assert),
        unit_test(decrement_value_fail),
    };
    return run_tests(tests);
}

Assert Macros

Cmockery provides an assortment of assert macros that tests applications should use use in preference to the C standard library's assert macro. On an assertion failure a Cmockery assert macro will write the failure to the standard error stream and signal a test failure. Due to limitations of the C language the general C standard library assert() and Cmockery's assert_true() and assert_false() macros can only display the expression that caused the assert failure. Cmockery's type specific assert macros, assert_{type}equal() and assert{type}_not_equal(), display the data that caused the assertion failure which increases data visibility aiding debugging of failing test cases.

<a name="UsingAssertEqualMacros>Using assert_{type}_equal() macros

assert_macro.c

#include <string.h>

static const char* status_code_strings[] = {
    "Address not found",
    "Connection dropped",
    "Connection timed out",
};

const char* get_status_code_string(const unsigned int status_code) {
    return status_code_strings[status_code];
};

unsigned int string_to_status_code(const char* const status_code_string) {
    unsigned int i;
    for (i = 0; i < sizeof(status_code_strings) /
                    sizeof(status_code_strings[0]); i++) {
        if (strcmp(status_code_strings[i], status_code_string) == 0) {
            return i;
        }
    }
    return ~0U;
}

assert_macro_test.c

#include <stdarg.h>
#include <stddef.h>
#include <setjmp.h>
#include <cmockery.h>

extern const char* get_status_code_string(const unsigned int status_code);
extern unsigned int string_to_status_code(
    const char* const status_code_string);

/* This test will fail since the string returned by get_status_code_string(0)
 * doesn't match "Connection timed out". */
void get_status_code_string_test(void **state) {
    assert_string_equal(get_status_code_string(0), "Address not found");
    assert_string_equal(get_status_code_string(1), "Connection timed out");
}

// This test will fail since the status code of "Connection timed out" isn't 1
void string_to_status_code_test(void **state) {
    assert_int_equal(string_to_status_code("Address not found"), 0);
    assert_int_equal(string_to_status_code("Connection timed out"), 1);
}

int main(int argc, char *argv[]) {
    const UnitTest tests[] = {
        unit_test(get_status_code_string_test),
        unit_test(string_to_status_code_test),
    };
    return run_tests(tests);
}

Dynamic Memory Allocation

To test for memory leaks, buffer overflows and underflows a module being tested by Cmockery should replace calls to malloc(), calloc() and free() to test_malloc(), test_calloc() and test_free() respectively. Each time a block is deallocated using test_free() it is checked for corruption, if a corrupt block is found a test failure is signalled. All blocks allocated using the test_*() allocation functions are tracked by the Cmockery library. When a test completes if any allocated blocks (memory leaks) remain they are reported and a test failure is signalled.

For simplicity Cmockery currently executes all tests in one process. Therefore all test cases in a test application share a single address space which means memory corruption from a single test case could potentially cause the test application to exit prematurely.

Using Cmockery's Allocators

allocate_module.c

#include <malloc.h>

#if UNIT_TESTING
extern void* _test_malloc(const size_t size, const char* file, const int line);
extern void* _test_calloc(const size_t number_of_elements, const size_t size, 
                          const char* file, const int line);
extern void _test_free(void* const ptr, const char* file, const int line);

#define malloc(size) _test_malloc(size, __FILE__, __LINE__)
#define calloc(num, size) _test_calloc(num, size, __FILE__, __LINE__)
#define free(ptr) _test_free(ptr, __FILE__, __LINE__)
#endif // UNIT_TESTING

void leak_memory() {
    int * const temporary = (int*)malloc(sizeof(int));
    *temporary = 0;
}

void buffer_overflow() {
    char * const memory = (char*)malloc(sizeof(int));
    memory[sizeof(int)] = '!';
    free(memory);
}

void buffer_underflow() {
    char * const memory = (char*)malloc(sizeof(int));
    memory[-1] = '!';
    free(memory);
}

allocate_module_test.c

#include <stdarg.h>
#include <stddef.h>
#include <setjmp.h>
#include <cmockery.h>

extern void leak_memory();
extern void buffer_overflow();
extern void buffer_underflow();

// Test case that fails as leak_memory() leaks a dynamically allocated block.
void leak_memory_test(void **state) {
    leak_memory();
}

// Test case that fails as buffer_overflow() corrupts an allocated block.
void buffer_overflow_test(void **state) {
    buffer_overflow();
}

// Test case that fails as buffer_underflow() corrupts an allocated block.
void buffer_underflow_test(void **state) {
    buffer_underflow();
}

int main(int argc, char* argv[]) {
    const UnitTest tests[] = {
        unit_test(leak_memory_test),
        unit_test(buffer_overflow_test),
        unit_test(buffer_underflow_test),
    };
    return run_tests(tests);
}

Mock Functions

A unit test should ideally isolate the function or module being tested from any external dependencies. This can be performed using mock functions that are either statically or dynamically linked with the module being tested. Mock functions must be statically linked when the code being tested directly references external functions. Dynamic linking is simply the process of setting a function pointer in a table used by the tested module to reference a mock function defined in the unit test.

Return Values

In order to simplify the implementation of mock functions Cmockery provides functionality which stores return values for mock functions in each test case using will_return(). These values are then returned by each mock function using calls to mock().

Values passed to will_return() are added to a queue for each function specified. Each successive call to mock() from a function removes a return value from the queue. This makes it possible for a mock function to use multiple calls to mock() to return output parameters in addition to a return value. In addition this allows the specification of return values for multiple calls to a mock function.

Using will_return()

database.h

typedef struct DatabaseConnection DatabaseConnection;

/* Function that takes an SQL query string and sets results to an array of
 * pointers with the result of the query.  The value returned specifies the
 * number of items in the returned array of results.  The returned array of
 * results are statically allocated and should not be deallocated using free()
 */
typedef unsigned int (*QueryDatabase)(
    DatabaseConnection* const connection, const char * const query_string,
    void *** const results);

// Connection to a database.
struct DatabaseConnection {
    const char *url;
    unsigned int port;
    QueryDatabase query_database;
};

// Connect to a database.
DatabaseConnection* connect_to_database(const char * const url,
                                        const unsigned int port);

customer_database.c

#include <stddef.h>
#include <stdio.h>
#include <database.h>
#ifdef _WIN32
#define snprintf _snprintf
#endif // _WIN32

// Connect to the database containing customer information.
DatabaseConnection* connect_to_customer_database() {
    return connect_to_database("customers.abcd.org", 321);
}

/* Find the ID of a customer by his/her name returning a value > 0 if
 * successful, 0 otherwise. */
unsigned int get_customer_id_by_name(
        DatabaseConnection * const connection,
        const char * const customer_name) {
    char query_string[256];
    int number_of_results;
    void **results;
    snprintf(query_string, sizeof(query_string),
             "SELECT ID FROM CUSTOMERS WHERE NAME = %s", customer_name);
    number_of_results = connection->query_database(connection, query_string,
                                                   &results);
    if (number_of_results != 1) {
        return -1;
    }
    return (unsigned int)results[0];
}

customer_database_test.c

#include <stdarg.h>
#include <stddef.h>
#include <setjmp.h>
#include <cmockery.h>
#include <database.h>


extern DatabaseConnection* connect_to_customer_database();
extern unsigned int get_customer_id_by_name(
    DatabaseConnection * const connection, const char * const customer_name);

// Mock query database function.
unsigned int mock_query_database(
        DatabaseConnection* const connection, const char * const query_string,
        void *** const results) {
    *results = (void**)mock();
    return (unsigned int)mock();
}

// Mock of the connect to database function.
DatabaseConnection* connect_to_database(const char * const database_url,
                                        const unsigned int port) {
    return (DatabaseConnection*)mock();
}

void test_connect_to_customer_database(void **state) {
    will_return(connect_to_database, 0x0DA7ABA53);
    assert_true(connect_to_customer_database() ==
                (DatabaseConnection*)0x0DA7ABA53);
}

/* This test fails as the mock function connect_to_database() will have no
 * value to return. */
void fail_connect_to_customer_database(void **state) {
    will_return(connect_to_database, 0x0DA7ABA53);
    assert_true(connect_to_customer_database() ==
                (DatabaseConnection*)0x0DA7ABA53);
}

void test_get_customer_id_by_name(void **state) {
    DatabaseConnection connection = {
        "somedatabase.somewhere.com", 12345678, mock_query_database
    };
    // Return a single customer ID when mock_query_database() is called.
    int customer_ids = 543;
    will_return(mock_query_database, &customer_ids);
    will_return(mock_query_database, 1);
    assert_int_equal(get_customer_id_by_name(&connection, "john doe"), 543);
}

int main(int argc, char* argv[]) {
    const UnitTest tests[] = {
        unit_test(test_connect_to_customer_database),
        unit_test(fail_connect_to_customer_database),
        unit_test(test_get_customer_id_by_name),
    };
    return run_tests(tests);
}

Checking Parameters

In addition to storing the return values of mock functions, Cmockery provides functionality to store expected values for mock function parameters using the expect_*() functions provided. A mock function parameter can then be validated using the check_expected() macro.

Successive calls to expect_*() macros for a parameter queues values to check the specified parameter. check_expected() checks a function parameter against the next value queued using expect_*(), if the parameter check fails a test failure is signalled. In addition if check_expected() is called and no more parameter values are queued a test failure occurs.

Using expect_*()

product_database.c

#include <database.h>

// Connect to the database containing customer information.
DatabaseConnection* connect_to_product_database() {
    return connect_to_database("products.abcd.org", 322);
}

product_database_test.c

#include <stdarg.h>
#include <stddef.h>
#include <setjmp.h>
#include <cmockery.h>
#include <database.h>

extern DatabaseConnection* connect_to_product_database();

/* Mock connect to database function.
 * NOTE: This mock function is very general could be shared between tests
 * that use the imaginary database.h module. */
DatabaseConnection* connect_to_database(const char * const url,
                                        const unsigned int port) {
    check_expected(url);
    check_expected(port);
    return (DatabaseConnection*)mock();
}

void test_connect_to_product_database(void **state) {
    expect_string(connect_to_database, url, "products.abcd.org");
    expect_value(connect_to_database, port, 322);
    will_return(connect_to_database, 0xDA7ABA53);
    assert_int_equal(connect_to_product_database(), 0xDA7ABA53);
}

/* This test will fail since the expected URL is different to the URL that is
 * passed to connect_to_database() by connect_to_product_database(). */
void test_connect_to_product_database_bad_url(void **state) {
    expect_string(connect_to_database, url, "products.abcd.com");
    expect_value(connect_to_database, port, 322);
    will_return(connect_to_database, 0xDA7ABA53);
    assert_int_equal((int)connect_to_product_database(), 0xDA7ABA53);
}

/* This test will fail since the mock connect_to_database() will attempt to
 * retrieve a value for the parameter port which isn't specified by this
 * test function. */
void test_connect_to_product_database_missing_parameter(void **state) {
    expect_string(connect_to_database, url, "products.abcd.org");
    will_return(connect_to_database, 0xDA7ABA53);
    assert_int_equal((int)connect_to_product_database(), 0xDA7ABA53);
}

int main(int argc, char* argv[]) {
    const UnitTest tests[] = {
        unit_test(test_connect_to_product_database),
        unit_test(test_connect_to_product_database_bad_url),
        unit_test(test_connect_to_product_database_missing_parameter),
    };
    return run_tests(tests);
}

Test State

Cmockery allows the specification of multiple setup and tear down functions for each test case. Setup functions, specified by the unit_test_setup() or unit_test_setup_teardown() macros allow common initialization to be shared between multiple test cases. In addition, tear down functions, specified by the unit_test_teardown() or unit_test_setup_teardown() macros provide a code path that is always executed for a test case even when it fails.

Using unit_test_setup_teardown()

key_value.c

#include <stddef.h>
#include <stdlib.h>
#include <string.h>

typedef struct KeyValue {
    unsigned int key;
    const char* value;
} KeyValue;

static KeyValue *key_values = NULL;
static unsigned int number_of_key_values = 0;

void set_key_values(KeyValue * const new_key_values,
                    const unsigned int new_number_of_key_values) {
    key_values = new_key_values;
    number_of_key_values = new_number_of_key_values;
}

// Compare two key members of KeyValue structures.
int key_value_compare_keys(const void *a, const void *b) {
    return (int)((KeyValue*)a)->key - (int)((KeyValue*)b)->key;
}

// Search an array of key value pairs for the item with the specified value.
KeyValue* find_item_by_value(const char * const value) {
  unsigned int i;
    for (i = 0; i < number_of_key_values; i++) {
        if (strcmp(key_values[i].value, value) == 0) {
            return &key_values[i];
        }
    }
    return NULL;
}

// Sort an array of key value pairs by key.
void sort_items_by_key() {
    qsort(key_values, number_of_key_values, sizeof(*key_values),
          key_value_compare_keys);
}

key_value_test.c

#include <stdarg.h>
#include <stddef.h>
#include <setjmp.h>
#include <string.h>
#include <cmockery.h>

/* This is duplicated here from the module setup_teardown.c to reduce the
 * number of files used in this test. */
typedef struct KeyValue {
    unsigned int key;
    const char* value;
} KeyValue;

void set_key_values(KeyValue * const new_key_values,
                    const unsigned int new_number_of_key_values);
extern KeyValue* find_item_by_value(const char * const value);
extern void sort_items_by_key();

static KeyValue key_values[] = {
    { 10, "this" },
    { 52, "test" },
    { 20, "a" },
    { 13, "is" },
};

void create_key_values(void **state) {
    KeyValue * const items = (KeyValue*)test_malloc(sizeof(key_values));
    memcpy(items, key_values, sizeof(key_values));
    *state = (void*)items;
    set_key_values(items, sizeof(key_values) / sizeof(key_values[0]));
}

void destroy_key_values(void **state) {
    test_free(*state);
    set_key_values(NULL, 0);
}

void test_find_item_by_value(void **state) {
    unsigned int i;
    for (i = 0; i < sizeof(key_values) / sizeof(key_values[0]); i++) {
        KeyValue * const found  = find_item_by_value(key_values[i].value);
        assert_true(found);
        assert_int_equal(found->key, key_values[i].key);
        assert_string_equal(found->value, key_values[i].value);
    }
}

void test_sort_items_by_key(void **state) {
    unsigned int i;
    KeyValue * const kv = *state;
    sort_items_by_key();
    for (i = 1; i < sizeof(key_values) / sizeof(key_values[0]); i++) {
        assert_true(kv[i - 1].key < kv[i].key);
    }
}

int main(int argc, char* argv[]) {
    const UnitTest tests[] = {
        unit_test_setup_teardown(test_find_item_by_value, create_key_values,
                                 destroy_key_values),
        unit_test_setup_teardown(test_sort_items_by_key, create_key_values,
                                 destroy_key_values),
    };
    return run_tests(tests);
}

Example

A small command line calculator calculator.c application and test application that full exercises the calculator application calculator_test.c are provided as an example of Cmockery's features discussed in this document.