Have you ever verified your integration with an external service using a mocked response, only to realize later that things work quite differently in a production environment? If only you knew some approach to ensure mocks are just fine and up to date, right?
Contract testing to the rescue!
Introduction
Distributed systems these days are complex, there’s no doubt about that. With more and more moving parts — new services being created, old services being deprecated, new API versions published, or existing API versions being extended — it’s difficult to keep up with the changes. It becomes especially tedious to ensure that all test doubles (mocks) reflect the actual behavior of the API of external services that you depend on.
Contract testing is a technique that might help you ease the pain. In a nutshell, it’s an approach to testing the integration between two parties — a provider and a consumer — in isolation. Each side must ensure it fulfills the agreed requirements written in a contract. Typically the integration is either an HTTP request/response communication or some kind of messaging.
The idea of verifying API contracts isn’t new — you can easily find articles dating back to 2011, or even older, mentioning this approach. However, nowadays contract testing is gathering more and more popularity in the world struggling with the integration of an ever-growing number of microservices.
Contract tests nomenclature
Before going any further let’s establish an understanding of common nomenclature used for contract tests.
- a contract — a document that defines the expected structure of an AP: request, response, path, query parameters, headers, etc. Usually written in a text file in a specific format, depending on the technology used.
- a consumer — a side of a contract that consumes, or simply uses a given API. Sometimes referred to as a client.
- a provider — a side of a contract that provides, or simply owns a given API. Sometimes referred to as a server.
Provider-driven contract testing
In a provider-driven approach, it’s the provider that drives the process of API evolution, publishing new contracts (and associated test doubles) whenever there’s a significant change to it. A typical flow of actions looks as follows:
- The provider defines a contract in his codebase
- The provider runs a build and tests to ensure the contract is fulfilled
- If tests pass the provider publishes a contract and associated test doubles (i.e. WireMock mappings) into an artifact repository
- When the consumer runs his build, it gets the latest test doubles published by the provider and runs some in-memory HTTP server to mock the provider
- Then the consumer may run tests to verify the integration with the provider
The contracts’ purpose is to ensure the provider implementation fulfills the request/response specification completely, regardless of the fact if his clients use all or just a subset of data in a response. Or to put it in a simpler way, the provider defines a here-is-what-I-can-do kind of contract for his clients, and it’s up to them to pick whatever is interesting for them.
The provider publishes a response stub including all the fields available in /v1/person/{id} API (❶). However, the needs of each client are different — Consumer A requires id and name(❷), Consumer B uses onlyage (❸), and Consumer C needs all of the fields (❹).
Pros
The provider-driven approach to contract testing leaves full control of the evolution of the API to the provider, meaning he can publish the new endpoint versions, or select the naming by himself. That’s quite useful if you’re in a team that develops a provider service with a well-defined business domain, as it’s usually you, not your clients, that know the upcoming new features and their impact on the API first.
This approach is also well-applicable for a provider that exposes a public API and can’t work closely with his clients. It enables new clients to get to know the capabilities of an API quickly just by looking at the published stubs.
Cons
On the other hand, as this approach pretty much ignores the consumer perspective, it has an obvious drawback — it doesn’t stop the provider from producing a hard-to-use API. It’s easy to fell into a trap of assuming how your clients perceive your domain, or what are their expectations towards your API.
Another downside of not knowing how your API is used by your clients is that whenever you want to deprecate or modify it in a breaking way, you must assume that all your consumers may need your whole response. Or, alternatively, ask around and double-check with each one of your clients — if only you know them all of course.
Consumer-driven contract testing
In a consumer-driven approach to contract testing, it’s the consumer that drives the changes to the provider’s API. What does it mean exactly? A typical flow of actions looks as follows:
- The consumer defines a contract in his codebase
- The consumer publishes the contract into a dedicated contract repository (i.e. Pact Broker)
- The provider gets notified about the new contract
- The provider run tests that verify if all the requirements are met by firing consumer-defined requests against the provider API and checking if the responses match with consumer-defined responses
- If tests pass the integration between the consumer and the provider is marked as verified
Please note that, depending on the tool used, there are other possible ways to organize consumer-driven contract testing flow, i.e. have a look at Spring Cloud Contract documentation for more details.
The contracts are focused on consumer integrations with the provider, rather than specifying the whole request/response structure. For a provider, it’s a valuable perspective — each of its clients defines exactly what fields they use, and which they simply ignore. Simply speaking, it’s the consumer that defines here-is-what-I-need kind of contract for the provider to fulfill.
All consumers define their needs separately — Consumer A requires id and name(❷), Consumer B uses onlyage (❸), and Consumer C needs all of the fields (❹). Putting all the requirements together the Provider can see what needs to be provided in /v1/person/{id} API and to whom.
Does it mean the provider is no longer in charge of his API and has to blindly follow all the requirements coming from all his clients? Not necessarily. It’s not that consumers have the final decision on everything, including naming or data structures. Instead, they define the initial proposal of changes, i.e. as a pull request in the consumer’s codebase, which should later be discussed involving both sides — a provider and a consumer. As a provider, it’s perfectly fine to disagree with the initial proposal, and suggest some changes — it’s your API after all. However, the important bit is the discussion itself which is usually an eye-opener. As a provider you get to know your client’s needs, you can see how they intend to use your service’s API, you can simply find out how they see your service from the outside world.
Pros
Using the consumer-driven approach to contract tests typically leads to better-designed APIs which are easier to use for the clients. From the provider perspective, the feedback loop on the quality of your API definition is much shorter as it involves the consumers as soon as at the design stage!
Additionally, the provider knows exactly which endpoints or fields from the response are actively used and required by its clients. It allows for lower maintenance costs, i.e. makes API/field deprecation easier — no more log scrapping or questionnaires send out to your clients to find out if anyone uses the abc field from your response. As a provider, you can simply run a set of tests against the contracts defined by your clients and get the answer immediately.
Cons
What are the drawbacks then? First of all, there’s no single view of all the capabilities of an API — they are scattered between different clients’ contracts. This, as opposed to the provider-driven approach, may put some additional effort to understand the full picture of the API on consumer teams aiming to start using it without prior knowledge.
Secondly, the client-specific tests on the provider side might be cumbersome to maintain if there’s no alignment on how to define your consumer-driven contracts. How is that possible? Consider the following example. Assuming the provider uses UUID as a type of the id field in his response DTO class and all his clients expect this field in their contracts. However, some of them specified it as UUID(with example value ofd8c203e1-987a-4e3d-9256-6c79982f5bd6), and others as String (with example value of this-is-some-id). The latter creates expectations about the response from the Provider that cannot be fulfilled and verified in his tests because of the way his DTOs are implemented, not the actual difference in the JSON structure. This is why it’s important to remember that creating contracts is supposed to be a collaboration between the consumer and provider, despite the fact that the actual code ends up in the consumer’s codebase.
Wrap up
There are two approaches to contract tests: provider-driven and consumer-driven. The final result — a verifiable contract between the consumer and provider and test doubles that are always up to date — is the same for both.
It’s the way we create them, the flow of communication, that differs.
Whatever approach you take remember that contracts should be created in close cooperation of both sides — the consumer and provider. Even the most comprehensive tools are not able to replace the discussion between the members of two teams. A well-defined contract that is up-to-date brings a lot of value to both parties, which makes them both responsible for it, no matter whose codebase actually contains the code describing it.
