`goldtest`

Automagically generate tests to verify API responses or DB state.

Install

pip install goldtest

Usage

You write goldtests as if you already have a "gold" file which contains the expected output of your program. This file will actually be generated by the test itself when you run it with the GOLDTEST_GEN environment variable set. Under normal operation, goldtest will compare the output of your program to the gold file, and will fail the test if there are any differences.

Writing your tests

import goldtest


class TestClass(goldtest.TestCase):
    def test_testcase(self):
        gt = goldtest.Goldtest(self)

        resp = call_api()  # function which returns a JSON-serializable object

        gt.test('calling_api', resp)

Generating gold files

Run the test with the GOLDTEST_GEN environment variable set.

Example: GOLDTEST_GEN=1 python test_goldtest.py

Or, if you use pytest: GOLDTEST_GEN=1 py.test

Checking the state of the database

Currently, it only works if you use SQLAlchemy. It has also only been tested with postgres.

# These 2 lines should not be in your test - they are often found in the model.
# They're only here to illustrate where db_metadata and db_engine come from.
engine = create_engine(...)
Base = declarative_base()

gt = goldtest.Goldtest(self,
                       db_metadata=Base.metadata,
                       db_engine=engine)

modify_db_state()  # a function which modifies the state of the DB

gt.test_db('after_modification', tables=['users', 'accounts'])

Non-deterministic outputs

Sometimes, parts of your output will be non-deterministic. Examples include timestamps, randomly generated tokens, etc. Don't worry, goldtest handles this!

It does so by running each test twice when generating a gold file. If a field changes in between runs, it will be replaced with "\*". This is invalid JSON, so it will never conflict with legitimate output. This "wildcard" lets goldtest know that anything is acceptable for this field when it is verifying output.

vadimg/goldtest