Defining High-quality Unit Tests - Codecov (2024)

Unit testing is one of the most important ways to ensure code quality and reliability, as it allows developers to verify that the individual units of their code are working as intended. High-quality unit tests not only ensure that code is working correctly but also serve as living documentation. Good tests help to prevent future bugs and regressions that might occur from a code change and make it easier to maintain and improve the codebase over time.

At Codecov, we work to provide code coverage data to developers. Code coverage is useful in that it helps identify parts of the codebase that aren’t being tested. For organizations, this information can guide the development of unit tests so that issues can be mitigated before they reach production.

But code coverage only helps identify untested code. For code that is tested, how do we know that the unit tests are any good?

Attributes of High-Quality Unit Tests

Here are a few attributes of unit tests that are considered high-quality.

• Thorough
• Isolated
• Performant
• Maintainable
• Reliable

Thorough Unit Tests

A good unit test suite should be thorough. This means that it should cover all relevant cases and edge cases. They look out for many types of inputs and ensure the code outputs the correct result or errors accordingly. They should use asserts to check against correct values and make developers aware when those assertions fail. This helps to ensure that the code is robust and can handle a wide range of inputs.

Isolated Unit Tests

Isolated unit tests will not depend on other APIs or services. Their dependencies will be as minimal as possible so as to only be testing one function at a time. A high-quality unit test should not depend on the results of other tests or external factors such as the current time or the state of the database. This ensures that the test can be run in isolation and the results will be consistent.

A well-isolated test should also be repeatable. A unit test should produce the same results every time it is run, provided that the code being tested has not changed. This makes it easy to diagnose issues and ensure that the code is working correctly.

Performant Unit Tests

Running unit tests should be fast, the sooner an issue is found the quicker the turnaround time. On average, a unit test should take a few milliseconds to run. This will help ensure that tests are narrow and isolated.

Maintainable Unit Tests

The test should be maintainable. As the codebase changes over time, unit tests may need to be updated to reflect these changes. It’s important to write tests in a way that makes them easy to understand and maintain so that they can continue to provide value as the codebase evolves.

This means that a unit test should be well-named and should be documented for any unclear edge case. Good unit tests are short and easy to understand. Grouping together in obvious places in the directory tree allows them to be easily discovered. This will also help ensure that developers know which behaviors are already tested preventing wasted resources.

Reliable Unit Tests

A good unit test suite should produce the same results no matter how many times it’s run. Flaky unit tests, those that return as passed or failed unexpectedly, degrade trust in the test suite. Sometimes this is due to third-party APIs or randomness. A good unit test will try to mock these cases out to focus on testing the codebase.

Mutation Testing: Testing the Tests

Mutation testing is a technique that involves making small changes, called “mutations”, to the code being tested and then running the test suite to see if the tests detect the changes. The idea is to determine whether the tests are able to effectively detect the changes, or whether they are “killed” by the tests.

Mutation testing can be a useful way to improve the quality of your unit tests because it helps to ensure that your tests are actually testing the code, rather than just passing because of some external factor. For example, if a test is passing because it is checking the wrong thing, a mutation test will likely reveal this by causing the test to fail.

To use mutation testing, you will need to use a mutation testing tool that is compatible with your programming language and test framework. The tool will generate mutations in your code and then run the tests to see if they detect the changes. If a mutation is not detected, it means that the test is not sufficient for detecting that particular change. This can help you identify weaknesses in your test suite and improve the coverage of your tests.

However, the downside to using mutation testing is that it can be time-consuming, as it requires running the test suite multiple times with different mutations. This is why writing unit tests that can run quickly is so important if mutation testing is implemented. For this reason, mutation testing is to be used as a complement to traditional unit testing as opposed to a replacement.