Measuring Coverage Accurately with Coveralls.io

If you've ever developed an open source project on Github you probably have heard about coveralls.io already but you, like me until recently are probably not measuring code coverage accurately enough if you have multiple testsuites (e.g.. unit, functional, integration).

How did I reach this conclusion? Keep reading.

What Is Coveralls

Coveralls is a web service to help you track your code coverage over time, and ensure that all your new code is fully covered.

There is but one prerequisite: Your code must be hosted on Github.

The Coveralls service is language-agnostic and CI-agnostic.

The Incident

At my day job we started  parallelizing our builds on Travis-Ci so that we will know faster if the unit and functional tests have passed in the meanwhile until the integration testsuite finishes up.

When the first parallelized build completed and succeeded we got the a notification from coveralls that our coverage decreased by 50%. That's a very drastic change. It's seems like a bug no? But it's an unintended feature and a great one.

The Cause

When you split the build by testsuites each testsuite's code coverage is being measured separately . Apparently, our integration tests covered much more code than our unit tests. It seems intuitive but it took me a minute or two to figure it out.

Conclusion

Parallelize your build. It's important not only because the build process is more responsive but because it helps you measure code coverage correctly.