Code coverage vs test coverage

Congrats, your code coverage got the high score! But what has the project really gained? [Photo: Yucel Moran]

What’s up with the following assertion?

Our team must improve its test coverage so that all our code is covered by tests.

Test coverage and code coverage are often used interchangeably, but they mean very different things:

• Code coverage is a metric showing the proportion of code covered by tests
• Test coverage is about the proportion of a story or feature covered by tests

Opinions on which is “better” do vary, but — shocking disclosure alert — I’m a great fan of test coverage and not so keen on code coverage (at least, when used as a stick to hit developers with). They both have their uses though.

So with that in mind, let’s dive in.

Code Coverage

There’s a multitude of tools for measuring code coverage: Cobertura for Java and Istanbul for JavaScript, to name a couple. IDEs such as Intellij have code coverage built in. Simply “run tests with coverage” instead of “run tests” to get a coverage report.

What these tools have in common is that they monitor the call stack while the tests run, to see where the “long finger of the tests” goes poking. They then produce an overall percentage of code that was executed… the theory being that if a line of code is run during a test, then it must be covered by the test.

This type of analysis is statement coverage. Many such tools also analyze branch coverage, i.e. given a branching condition in the code, are all the evaluated possibilities reached by a test?

To quickly demo code coverage in action, here’s a rather contrived example.

Given these three functions—funcA calls funcB, which in turn calls funcC:

We create 3 tests, each of which simply calls one of the functions. Then run each test individually, with coverage enabled: