Integrations tests are about functionality while contract tests are about the interface.
These tests are not component tests. They do not test the behaviour of the service deeply but that the inputs and outputs of service calls contain required attributes and that response latency and throughput are within acceptable limits. Ideally, the contract test suites written by each consuming team are packaged and runnable in the build pipelines for the producing services. In this way, the maintainers of the producing service know the impact of their changes on their consumers.
- Pact is a contract testing tool that further improves the contract testing workflow by allowing users to implement consumer-driven contracts.
- Contract testing is an alternative approach to traditional integration testing.
Pact on its own allows you to create and verify contracts. Pact + Pact Broker allows you to integrate contract testing into your CI/CD pipeline to allow you to release code faster.
Contract testing is a technique for testing an integration point by checking each application in isolation to ensure the messages it sends or receives conform to a shared understanding that is documented in a “contract”.
- During the consumer tests, each request made to a Pact mock provider is recorded into the contract file (the Pact file), along with its expected response. The Pact mock provider can be a substituted with an adapter, for instance if WireMocks or MSW.
- A Pact simulated consumer then replays each request against the real provider, and compares the actual and expected responses.
- If they match, we have verified that the simulated applications behave the same way as the real applications.
- When contract testing is in place we use Pact Broker to: share contract across teams, manage contracts across branches, orchestrate builds.
https://pactflow.io/how-pact-works/#slide-1
Pact is consumer-driven, which means it’s the consumer who defines the contract. In the test, you define the pact, preferably using a library which then sets up a mock server based on the pact. You can then use your consumer code against the mock server and verify that it works as expected. This does not verify that the request the client makes will work against the producer or that the mocked response is anything like what the producer will return as it’s all mocked by you. Producer tests actually verify the contract. Pact may be consumer-driven but only the producer (provider) can actually verify if requests and responses match. The pact files created by the consumer tests are run against your application. Your application is booted up, preferably using mocked data, and the requests defined in the pacts are fired at it. The responses are then verified against the expected responses in the pact. If they match the tests pass.
- The Pact Broker is an application for sharing consumer driven contracts and verification results.
- It is open source, there is a plug-and-play version payed called PactFlow.
- Pact Broker can be used via Docker https://hub.docker.com/r/pactfoundation/pact-broker
- Contract testing gives you tests that are quicker to execute. One down side of the approach is that the important information that would be available all in one place at the end of an integration test suite execution. The Pact Broker is a tool that brings all this information back together again.
- Pacts are uploaded by the consumers to the broker, allowing producers to download the pacts and verify them. When producers have verified a pact they upload the status (pass or fail) back to the broker for future use.
- Pacts have a version which changes when the pact changes. Multiple application versions may have the same version of their pact if it does not change.
- Webhooks are a feature of the pact broker which makes HTTP requests to configurable endpoints for pact lifecycle changes, for instance if a pact has been changed; When this happens a webhook can trigger a pipeline that will run the relevant producer tests to verify the changed pact.
Pact Broker is useful for:
- Integration with automate contract testing in CI/CD and allow deploying service indipendently
- Solve the problme how to share contracts and verify the results between consumer and providers
- Versions management
- Allows you to ensure backwards compatibility between multiple consumer and provider versions
- Alows you to visualise and document the relationships between your services, digrams and charts
Main features:
- RESTful API for publishing and retrieving pact files
- API browser to navigate the API
- Autogenerated docs
- Autogenerated diagrams for microservice networks
- Provides a "matrix" of compatible consumer and provider versions
- Webhooks to trigger actions when pacts change or verification results are published eg. run provider build, notify a Slack channel.
- Vew diffs between Pact versions
- CLI encorporating Pact workflow
How it is working:
Step 1
- The consumer project runs its tests using the Pact library to provide a mock service (or use an adapter like for WireMocks or MSW)
- While the tests run, the mock service writes the requests and the expected responses to a JSON "pact" file - this is the consumer contract.
- The generated pact file is then published to the Pact Broker (the easiest way to do this is to publish the pact file using the Pact Broker Client CLI)
- When a pact file is published, a webhook in the Pact Broker kicks off a build of the provider project if the pact content has changed since the previous version and requires verification.
Step 2
- The provider has a verification task that is configured to retrieve the relevant pacts between itself and its consumer.
- The provider build runs the pact verification task, which retrieves the pact(s) from the Pact Broker, replays each request against the provider, and checks that the responses match the expected responses.
- If the pact verification fails, the build fails.
- The results of the verification are published back to the Pact Broker by the pact verification tool, so the consumer team will know if the code they have written will work in real life.
- The Provider CI determines if the provider is compatible with its consumers in a particular environment.
- If the pact has been verified successfully, the deployment can proceed.
- When the provider is deployed they record the deployment with Pact.
Step 3. Back to the Consumer CI build
- The Consumer CI determines if the pact has been verified.
- If the pact has been verified successfully, the deployment can proceed.
- When the consumer is deployed they record the deployment with Pact .
https://docs.pact.io/pact_broker
https://martinfowler.com/articles/microservice-testing
It is a command-line tool that checks if it’s safe for you to release your application. If you’re deploying something that will break a contract, or the contract has not been verified, it will fail. Without using this, you may deploy something which is broken.
can_i_deploy should be used in your pipeline, before you deploy to each environment and tag pacts with the environment after it is finished so we can avoid having a PR merged which can brack our contracts.
Bi-Directional Contract Testing is a type of static contract testing where two contracts - one representing the consumer expectations, and another representing the provider capability - are compared to ensure they are compatible.
The comparisons is based on their schema.
BDCT is a feature exclusive to PactFlow and is not available in the Pact OSS project (it is a paid solution).
- Contract first provider driven workflows.
- Using OpenAPI.
- Integration with MSQ, Cypress, Wiremock BYO (Bring Your Own mocks).
- Support contract verification Postman, Dredd.
- For white/black box testing.
- Wide audience contract testers: dev, qas, SDET (Software Development Engineer in Test).
Pact is a record and replay style contract using specification by example to prove correctness. Bi-Directional Contract Testing instead using a schema comparison approach.
Packflow provides continuous api compliance by comparison the consumer contract with the provider contract such as an API contract. It allows to generate of contracts from existing mocks (Generated by tooling like WireMocks, MSW or others). Developers can use their plug-and-play adapter. This is good because you can leverage existing tools.
Pros:
- Low barrier to entry.
- Scale faster, reuse existing tests.
- Upscale re-using current processed and tolling.
- Consumer tests behaviour against a mock (write a test that checks our consumer behavior against a mock), we could use WireMocks or MSW in this part
- From the mock used to test the consumer, we use it to generate to contract (pact file, aka the consumer contract).
- Test consumer and provider independently.
- We start with a provider contract (for instance OpenAPI spec, generated by hand or from source code).
- We do provider testing, make sure the provider code base is compatible with that OpenAPI spec (verify provider contract).
- The end result we have 2 contracts, one what the consumer needs and another what provider can do.
- Both contracts are uploaded on Pactflow, the consumer and provider have separated pipelines.
- After a contract comparison is made, to make sure their consumer is a valid subset of the provider contract (schema comparison).
- The consumer tests its behavior against a mock such Wiremock, MSW or Pact (Pact or an adapter will generate the pact file).
- Consumer contract is produced in the form of pact file, there only the interaction with the consumer is captured.
- Consumer contract is uploaded to Pactflow.
- We check the compatibility with the provider.
- The provider starts with its own specification (OpenAPI), this is the provider contract (can be generated by hand or by source code), if OpenAPI is generated by code we can skip the next point.
- The provider contract is tested against the provider using a funcional API testing tool such RestAssured, Dredd, or Postman.
- Provider contract i uploaded to Pactflow.
- We check the compatibility with the consumer.
With BDCT, the key difference is that a Provider uploads its own provider contract advertising its full capacity which is statically compared to the expectations in the consumer contract - the consumer contract is never replayed against the provider code base. This creates a much simpler and decoupled workflow. See the trade-offs for more.
- Contract tests focus on the messages that flow between a consumer and provider (schemas).
- Functional tests also ensure that the correct side effects have occurred (for instance a new entry is created in db).
- A consumer contract is a collection of interactions that describe how the Consumer expects the Provider to behave. Each Consumer will have its own unique consumer contract for each of its Providers.
- A provider contract specifies the capability of the Provider, this could take the form of an OpenAPI document, but may be other formats such as a GraphQL schema, a SOAP XSD, a protobuf definition and so on.
https://pactflow.io/how-pact-works/#slide-1
https://www.youtube.com/watch?v=j0xe2dfAI-I
https://www.youtube.com/watch?v=VmluCw6a6IE
https://www.youtube.com/watch?v=V-OV6lRwhYA
https://betterprogramming.pub/your-contract-tests-are-not-protecting-you-563a5d6cdfef
A note for Pact users: when contract testing with Pact, you need to write and maintain a separate set of (Pact) tests that are responsible for ensuring systems are compatible. The tests on the consumer side produce a consumer contract containing the example scenarios which must be supported for the consumer to work, which are then replayed against an actual running provider in a record and replay style interaction. If the Provider responds correctly, the contract is valid.
With BDCT, the key difference is that a Provider uploads its own provider contract advertising its full capability which is statically compared to the expectations in the consumer contract - the consumer contract is never replayed against the provider code base. This creates a much simpler and decoupled workflow. See the trade-offs for more.
BDCT is not intended to replace Pact testing, but to provide an alternative in the cases where Pact may not be best suited.