Automated software tests are a requirement for ensuring we are delivering a product with quality to our users. It helps in finding bugs and requirements not fulfilled at development time, but also decreases the cost of maintenance by making the future changes to our code safer. Besides, the act of writing testable code alone increases the quality of the code we are writing because testable code has to be decoupled.
In this last post of this series, I’ll show how to analyze and enforce a minimum code coverage in our applications, and how to use integration tests to increase our testing surface.
Code coverage is a software metric that shows how much of our code is executed (covered) by our automated tests. It is shown as a percentage and can be calculated with different formulas, based on the number of lines or branches, for example. The higher the percentage, more of our code is being tested.
In this example, we have an ASP.NET Core API with a simple use case class that checks an input number and returns a string telling if the number is even or odd:
For now, we have only one unit test for this use case class:
To analyze the code coverage of our application, first we need to install Coverlet’s MSBuild integration using the
coverlet.msbuild nuget package in our test project:
Then, run the
dotnet test command with the Coverlet options on the solution or project folder:
dotnet test -p:CollectCoverage=true -p:CoverletOutputFormat=opencover -p:CoverletOutput=../TestResults
We are using the following options:
- CollectCoverage: Inform dotnet test to use coverlet to collect the code coverage data;
- CoverletOutputFormat: The format of the report that coverlet will generate (opencover, cobertura, json). More here;
- CoverletOutput: The path where the coverage report will be saved in. This path is relative to the test project;
This will print the code coverage result in a table and generate a report file named
⚠️ We can also run coverlet on the command line with the
coverlet.collectornuget package, but it has limited options and doesn’t print the results in the command line. More details here;
Coverlet generates the report in formats that are not easily readable by humans, so we need to generate an HTML report based on Coverlet report. To do it, we’ll usa a tool called ReportGenerator.
ReportGenerator is installed as a .NET global tool.
To do this, we run the following command:
dotnet tool install --global dotnet-reportgenerator-globaltool --version 4.8.6
To generate an HTML report based on a Coverlet report, we run the following command:
reportgenerator "-reports:TestResults.opencover.xml" "-targetdir:coveragereport" -reporttypes:Html
We are using the following options:
- reports: The path to the coverage report;
- targetdir: The path where the HTML report will be saved in;
- reporttypes: The format the report will be generated in.
The command output will tell the relative path to the generated report:
coveragereport\index.html file we can see the project Line and Branch coverage:
CodeCoverageSample.UseCases.IsEvenUseCase we can see details of the code coverage by method (in the table) and the line and branch coverage for the class:
But what is line coverage and branch coverage?
- Line coverage: Indicates the percentage of lines that are covered by the tests;
- Branch coverage: Indicates the percentage of logical paths that are covered by the tests (if, else, switch condition, etc).
In the example below, we can see that two lines in the
else branch are not covered by the tests.
This will result in:
- 50% of branch coverage, because only the
ifbranch is covered;
- 71.4% of line coverage, because only 5 of the 7 lines are covered.
The is an extension called Run Coverlet Report that integrates Coverlet and ReportGenerator with Visual Studio.
- First, we need to install the
coverlet.collectornuget package in our test projects. Xunit template already has this package installed by default.
- Then, navigate to
Extensions > Manage extensionsand install the
Run Coverlet Reportextension.
- Navigate to the new option
Tools > Run Code Coverage. This will generate the ReportGenerator HTML report that will be open in Visual Studio.
Also, after running the code coverage tool, Visual studio will read the coverlet report and show the coverage in our source file:
Now we will implement the
OddNumber_ReturnsOdd method to test the logical path we didn’t test before:
This will increase our average coverage to 50% of branch and 22.58% of line:
And 100% for the
Integration tests using the
WebApplicationFactory class (More here) are also considered in the code coverage reports. Let’s look at our
Program classes coverage:
Let’s implement a simple integration test. It will just instantiate our API and make a call passing a number and validate the results:
Now we run the code coverage report again and the
Program classes are covered by the tests:
If we want to remove a class or method from the code coverage analysis, we can decorate it with the
ℹ️ We can also create custom attributes to exclude from coverlet code coverage. Details here.
Coverlet has the
SkipAutoProps option to exclude the auto-properties from the coverage report.
For example, this class doesn’t have any logic and doesn’t need that the
set methods of its properties be tested:
Just set the
true when running the code coverage from the command line:
dotnet test -p:CollectCoverage=true -p:CoverletOutputFormat=opencover -p:CoverletOutput=TestResults -p:SkipAutoProps=true
⚠️ Unfortunately, the Run Coverage Report extension still doesn’t allow us to configure the coverlet parameters. There is an open pull request with this feature awaiting for approval here.
Just like code style and code quality rules, that I talked about in my previous post, we need to enforce a minimum code coverage in our build pipeline to maintain a level of quality in our code. Coverlet has the
Threshold option that we can set to a minimum percentage and it will fail the tests if our code coverage is below this percentage:
dotnet test -p:CollectCoverage=true -p:CoverletOutputFormat=opencover -p:CoverletOutput=TestResults -p:SkipAutoProps=true -p:Threshold=80
We can also use the
ThresholdType option to set the type of coverage to enforce. Not specifying will enforce all types of coverage (Line, Branch and Method). Details here.