Introduction

Automated software tests are a requirement for ensuring we are delivering a product with quality to our users. It helps in finding bugs and requirements not fulfilled at development time, but also decreases the cost of maintenance by making the future changes to our code safer. Besides, the act of writing testable code alone increases the quality of the code we are writing because testable code has to be decoupled.

In this last post of this series, I’ll show how to analyze and enforce a minimum code coverage in our applications, and how to use integration tests to increase our testing surface.

What is code coverage?

Code coverage is a software metric that shows how much of our code is executed (covered) by our automated tests. It is shown as a percentage and can be calculated with different formulas, based on the number of lines or branches, for example. The higher the percentage, more of our code is being tested.

Analyzing the code coverage of our application

In this example, we have an ASP.NET Core API with a simple use case class that checks an input number and returns a string telling if the number is even or odd:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

using CodeCoverageSample.Interfaces;

namespace CodeCoverageSample.UseCases;

public class IsEvenUseCase : IIsEvenUseCase
{
    public string IsEven(int number)
    {
        if (number % 2 == 0)
        {
            return "Number is even";
        }
        else
        {
            return "Number is odd";
        }
    }
}

For now, we have only one unit test for this use case class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

using CodeCoverageSample.UseCases;

namespace CodeCoverageSample.UnitTests;

public class IsEvenUseCaseTests
{
    [Fact]
    public void EvenNumber_ReturnsEven()
    {
        //Arrange
        var isEvenUseCase = new IsEvenUseCase();

        //Act
        var result = isEvenUseCase.IsEven(2);

        //Assert
        Assert.Equal("Number is even", result);
    }
}

To analyze the code coverage of our application, first we need to install Coverlet’s MSBuild integration using the coverlet.msbuild nuget package in our test project:

Install-Package coverlet.msbuild

Then, run the dotnet test command with the Coverlet options on the solution or project folder:

dotnet test -p:CollectCoverage=true -p:CoverletOutputFormat=opencover -p:CoverletOutput=../TestResults

We are using the following options:

• CollectCoverage: Inform dotnet test to use coverlet to collect the code coverage data;
• CoverletOutputFormat: The format of the report that coverlet will generate (opencover, cobertura, json). More here;
• CoverletOutput: The path where the coverage report will be saved in. This path is relative to the test project;

This will print the code coverage result in a table and generate a report file named TestResults.opencover.xml:

⚠️ We can also run coverlet on the command line with the coverlet.collector nuget package, but it has limited options and doesn’t print the results in the command line. More details here;

Generating HTML reports

Coverlet generates the report in formats that are not easily readable by humans, so we need to generate an HTML report based on Coverlet report. To do it, we’ll usa a tool called ReportGenerator.

Installing ReportGenerator

ReportGenerator is installed as a .NET global tool.

To do this, we run the following command:

dotnet tool install --global dotnet-reportgenerator-globaltool --version 4.8.6

Generating an HTML report of the opencover report

To generate an HTML report based on a Coverlet report, we run the following command:

reportgenerator "-reports:TestResults.opencover.xml" "-targetdir:coveragereport" -reporttypes:Html

We are using the following options:

• reports: The path to the coverage report;
• targetdir: The path where the HTML report will be saved in;
• reporttypes: The format the report will be generated in.

The command output will tell the relative path to the generated report: coveragereport\index.html.

Opening the coveragereport\index.html file we can see the project Line and Branch coverage:

Clicking on CodeCoverageSample.UseCases.IsEvenUseCase we can see details of the code coverage by method (in the table) and the line and branch coverage for the class:

Line vs Branch coverage

But what is line coverage and branch coverage?

• Line coverage: Indicates the percentage of lines that are covered by the tests;
• Branch coverage: Indicates the percentage of logical paths that are covered by the tests (if, else, switch condition, etc).

In the example below, we can see that two lines in the else branch are not covered by the tests.

This will result in:

• 50% of branch coverage, because only the if branch is covered;
• 71.4% of line coverage, because only 5 of the 7 lines are covered.

Code coverage on Visual Studio

The is an extension called Run Coverlet Report that integrates Coverlet and ReportGenerator with Visual Studio.

1. First, we need to install the coverlet.collector nuget package in our test projects. Xunit template already has this package installed by default.

Install-Package coverlet.collector

2. Then, navigate to Extensions > Manage extensions and install the Run Coverlet Report extension.

3. Navigate to the new option Tools > Run Code Coverage. This will generate the ReportGenerator HTML report that will be open in Visual Studio.

Also, after running the code coverage tool, Visual studio will read the coverlet report and show the coverage in our source file:

Improving our code coverage

Fixing the unit tests

Now we will implement the OddNumber_ReturnsOdd method to test the logical path we didn’t test before:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

using CodeCoverageSample.UseCases;

namespace CodeCoverageSample.UnitTests;

public class IsEvenUseCaseTests
{
    [Fact]
    public void EvenNumber_ReturnsEven()
    {
        //Arrange
        var isEvenUseCase = new IsEvenUseCase();

        //Act
        var result = isEvenUseCase.IsEven(2);

        //Assert
        Assert.Equal("Number is even", result);
    }

    [Fact]
    public void OddNumber_ReturnsOdd()
    {
        //Arrange
        var isEvenUseCase = new IsEvenUseCase();

        //Act
        var result = isEvenUseCase.IsEven(3);

        //Assert
        Assert.Equal("Number is odd", result);
    }
}

This will increase our average coverage to 50% of branch and 22.58% of line:

And 100% for the IsEvenUseCase class:

Implementing integration tests

Integration tests using the WebApplicationFactory class (More here) are also considered in the code coverage reports. Let’s look at our IsEvenController and Program classes coverage:

Let’s implement a simple integration test. It will just instantiate our API and make a call passing a number and validate the results:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

using Microsoft.AspNetCore.Mvc.Testing;
using System.Net;

namespace CodeCoverageSample.UnitTests.IntegrationTests;

public class IsEvenIntegrationTest : IClassFixture<WebApplicationFactory<Program>>
{
    private readonly WebApplicationFactory<Program> _factory;

    public IsEvenIntegrationTest(WebApplicationFactory<Program> factory)
    {
        _factory = factory;
    }

    [Theory]
    [InlineData(2, "Number is even")]
    [InlineData(3, "Number is odd")]
    public async Task Number_ReturnsCorrectAndOk(int number, string expectedResult)
    {
        var HttpClient = _factory
            .CreateClient();

        //Act
        var HttpResponse = await HttpClient.GetAsync($"/iseven/{number}");

        //Assert
        Assert.Equal(HttpStatusCode.OK, HttpResponse.StatusCode);

        var ResponseStr = await HttpResponse.Content.ReadAsStringAsync();

        Assert.Equal(expectedResult, ResponseStr);
    }
}

Now we run the code coverage report again and the IsEvenController and Program classes are covered by the tests:

Removing code from the code coverage analysis

If we want to remove a class or method from the code coverage analysis, we can decorate it with the ExcludeFromCodeCoverage attribute:

1
2
3
4
5
6
7
8
9

using System.Diagnostics.CodeAnalysis;

namespace CodeCoverageSample;

[ExcludeFromCodeCoverage]
public class DoNotTestMe
{
    ...
}

ℹ️ We can also create custom attributes to exclude from coverlet code coverage. Details here.

Ignoring auto-properties

Coverlet has the SkipAutoProps option to exclude the auto-properties from the coverage report.

For example, this class doesn’t have any logic and doesn’t need that the get and set methods of its properties be tested:

1
2
3
4
5
6
7

namespace CodeCoverageSample;

public class NoLogicHere
{
    public int Id { get; set; }
    public int Name { get; set; }
}

Just set the SkipAutoProps to true when running the code coverage from the command line:

dotnet test -p:CollectCoverage=true -p:CoverletOutputFormat=opencover -p:CoverletOutput=TestResults -p:SkipAutoProps=true

⚠️ Unfortunately, the Run Coverage Report extension still doesn’t allow us to configure the coverlet parameters. There is an open pull request with this feature awaiting for approval here.

Enforcing a minimum code coverage on the build pipeline

Just like code style and code quality rules, that I talked about in my previous post, we need to enforce a minimum code coverage in our build pipeline to maintain a level of quality in our code. Coverlet has the Threshold option that we can set to a minimum percentage and it will fail the tests if our code coverage is below this percentage:

dotnet test -p:CollectCoverage=true -p:CoverletOutputFormat=opencover -p:CoverletOutput=TestResults -p:SkipAutoProps=true -p:Threshold=80

We can also use the ThresholdType option to set the type of coverage to enforce. Not specifying will enforce all types of coverage (Line, Branch and Method). Details here.

References and Links

• https://github.com/coverlet-coverage/coverlet
• https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/MSBuildIntegration.md
• https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md
• https://github.com/danielpalme/ReportGenerator
• https://marketplace.visualstudio.com/items?itemName=ChrisDexter.RunCoverletReport
• https://github.com/the-dext/RunCoverletReport/blob/master/README.md