test-pipeline
is a small (very small!) library that can be used to improve your Clojure (or ClojureScript)
test suite.
The central idea is that tests can be implemented as a series of reusable, composable, cooperating steps.
A context map is passed to each step function, which may add or change keys in the context map before passing it to the next step. This may be familiar: the same pattern shows up in Ring as middleware, where each function passes a request map to the next function.
Allowing steps to communicate allows for clearer code, and better code reuse between tests.
This can be explained in the context of an example test from our application's code base; like many of our tests, it mocks up our primary backend service (order services, or OS) with a fixed response from a previously stored file:
(deftest cancel-order
(let [order-path "os-responses/curbside-pickup-and-fc-delivery.json"
cid (order-path->cid order-path)]
(with-test-system [system (new-test-system {:os-client (mocks/mock-os-client
(tc/read-resource-as-os-payload order-path))})]
(with-redefs [validate-auth (mock-validate-auth cid)]
(with-now "2020-08-24T20:00:00.000-0800"
(let [transaction-id (gen/uuid)
q "
mutation($input: CancelOrderInput) {
cancelOrder(input: $input) {
result: cancellationResult {
status
statusText
statusSubText
omsCode
}
}
}"
variables {:input {:orderId "8508200004824"
:transactionId transaction-id
:subReasonCode "209"
:orderLines [{:lineId "1"
:quantity 1}]
:cancelAction "CANCEL_NOW"}}
response (process-request system q variables)]
(reporting response
(is (= 200 (:status response)))
(is (= {:data
{:cancelOrder
{:result
{:status "SUCCESS"
:statusText "Canceled"
:statusSubText "You canceled this item on Aug 24"
:omsCode nil}}}}
(:body response))))))))))
This test is organized around a GraphQL request and response; we have to provide mock components, start and stop a Component system, override the current date/time, send the request into the system, and finally make assertions about the response.
Our premise is that the test is somewhat difficult to understand and maintain; the code reflects how to do the work of the test, rather than what behavior the test is designed to verify.
By contrast, a rewrite of the same test to use test-pipeline
(as alias p
)
is less busy, less deeply nested, and easier to read and maintain:
(deftest cancel-order
(p/execute
default-system
(force-os-response "os-responses/curbside-pickup-and-fc-delivery.json")
(force-now "2020-08-24T20:00:00.000-0800")
start-system
(send-request
"mutation($input: CancelOrderInput) {
cancelOrder(input: $input) {
result: cancellationResult {
status
statusText
statusSubText
omsCode
}
}
}"
{:input {:orderId "8508200004824"
:transactionId (gen/uuid)
:subReasonCode "209"
:orderLines [{:lineId "1"
:quantity 1}]
:cancelAction "CANCEL_NOW"}})
expect-success
(expect-data {:cancelOrder
{:result
{:status "SUCCESS"
:statusText "Canceled"
:statusSubText "You canceled this item on Aug 24"
:omsCode nil}}})))
This style of test reads more like a recipe, with more clearly deliniated steps that all work together to accomplish the final result.
Again, most of these functions are specific to our application. default-system
is a step function that initializes the base Component system map and places it in
the :system key of the context. force-os-response
reads the JSON file and
mocks the Component responsible for communicating with the external system; it can
also mock the authentication function.
Because start-system
is an explicit step, it is easy to inject mocks into the system map
in whatever order is convenient, prior to the system being started.
send-request
sends the request and captures
the response as context key :response. expect-success
asserts that the response
is status 200 and no GraphQL errors are present. expect-data
asserts that the :data key
of the body matches the provided value.
What we've done is establish a convention for how test data is stored into the context, so
that individual steps can read or update that data; thus expect-success
knows that a prior step
has recorded a :response key into the context, and expect-data
can use that same :response key.
Each step function takes a context
map as its only parameter, and then
invokes p/continue
to continue to the next step.
Obviously, a real step function will do something useful first, such as override a component in a component system,
or redefine a function with a mock, make an assertion with clojure.test/is
, or anything else that's needed.
For example, the start-system
step from the above example is coded as:
(defn start-system
[context]
(let [system (-> context :system component/start-system)]
(try
(p/continue (assoc context :system system)
(finally
(component/stop-system system))))))
This is a function that accepts a context, operates on it, and passes it to the
continue
function (which, in turn, finds the next step function, and passes the context to
that). The try
ensures that the system is stopped, regardless of what happens
in later steps.
Often, a step requires data specific to a particular test; in that case, a step function builder can create a
step function that is passed to p/execute
.
For example, the expect-data
function isn't a step itself; it is a builder that returns a step function:
(defn expect-data
[data]
(fn [context]
(is (= data (get-in context [:response :body :data])))
(p/continue context)))
Although the step function returned by expect-data
is typically the final step in the pipeline, it should still
call continue
just in case it isn't. For example, a test might use expect-data
to assert an expected
response, but then have further steps to make other assertions, such as checking how data was persisted to a database.
The execute
function will throw an exception if the final step function never gets invoked (due
to a prior step not calling continue
), as this is almost certainly a bug in the step function
implementation.
The then
macro is a step builder that evaluates to a step function that itself
evaluates some expressions before continuing;
this is handy for performing some assertions when the context is not directly needed, or triggering some
code for side effects.
...
(p/then (is (= 42 (calculate-final-answer))))
The above example has its own special case and can be rewritten as:
...
(p/is (= 42 (calculate-final-answer)))
A final clojure.test
helper is testing
, which is just a wrapper around clojure.test/testing
.
...
(p/testing "Deep Thought")
(p/is (= 42 (calculate-final-answer)))
"Deep Thought" will be the innermost test context (in clojure.test/*testing-contexts*
) when checking the
result of calculate-final-answer
.
The pipeline execution can be terminated with halt
; halt
exists to avoid the above check that all
steps executed.
This is useful when an early failure (say, an incorrect HTTP response from a server) will lead to a crowd of meaningless failures further on (such as validating the HTTP response).
The function halt-on-failure
is often more useful, it ensures that after
any step where test failures or errors occur, the pipeline execution is terminated.
Typically, this is often used when initially developing the code and tests, but can
be discarded once everything is stable.
The library itself is quite small; here's the key functions and macros:
execute
is the primary entrypointcontinue
is invoked by a step function to continue to the next stepmock
is used to override a function with a mock implementationspy
is used to capture arguments passed to a function, and optionally mock it at the same timecalls
is used to obtain the captured arguments to a spied functionupdate-in-context
andassoc-in-context
are used to modify the context during executioncapture-logging
captures log events; a wrapper aroundclojure.tools.logging.test/with-log
halt
terminates pipeline executionhalt-on-failure
terminates execution if any test failures occurthen
evaluate expressions during executionis
,context-is
,testing
wrappers around clojure.testcleanup
expressions execute after pipeline execution completesproviding
,validating
provide more powerful mocking capabilities, care of nubank/mockfnmatches?
which builds on matcher-combinators
Please refer to the API documentation for more details.
Copyright © 2022-Present Howard Lewis Ship
Distributed under the Apache License, Version 2.0.