Maximise your test coverage by combining property-based testing with scenarios

Would you trade stories for properties? [Photo credit: Steve Miller]

You’d think they are mutually exclusive. Scenario-based testing and property-based testing present entirely different approaches to what is essentially the same problem: you want to detect bugs in your code and prevent regressions later.

But as I’ll demonstrate in this article, they complement each other remarkably well.

First, the terminology…

Scenarios are specific examples of a story: “Mary’s login details were incorrect”, “Bob passed Go so he collects $200”, “Insufficient apples in stock” etc. So it seems natural to base your test scenarios on these — and indeed this is usually the right choice. In effect, you’re writing example-based specifications:

“If there are insufficient apples in stock, I want the following to happen…”

Property-based tests provide an alternative approach. They involve generating random or semi-random examples while the test runs, to verify that certain properties of the code hold true:

“If I eat an apple, I expect to have one less apple available to sell”

The same test can then be applied across multiple randomly generated input values, to ensure the property is true in all cases. In the above example, the random values would be the initial quantity of apples and the number that you eat.

Both approaches do have their own drawbacks, which we should consider:

• 😦 Property-based tests can be hard to write, at least at first; and “newbies” on the team might take a while to understand them. (Of course, problems like these improve over time). And sometimes, clauses like “insufficient apples in stock” are just better expressed as scenarios. Meanwhile…
• 😦 Scenario-based tests must each be manually thought of and then written. To cover the requirements comprehensively, you might want ten or more scenarios per story. Each scenario must be specified individually, and there’s always a chance some edge cases were forgotten.

So why use either approach? Luckily they have formidable strengths too:

• 😃 Property-based tests are efficient at providing multiple data-based scenarios, including many that we…