Introduction to Mutation Testing

Application development is an ever-evolving process. As a result, there are high chances that the code which was already written, would be revisited later to be enhanced to achieve additional functionalities. In such cases, it is critical for the developer who is going to make the changes to understand the existing code and make changes without affecting the existing functionalities. And to confirm if the changes are working in the old and the new code perfectly, it is essential to write unit tests through which the developer can identify the issues.

A100% code coverage with unit testing might not be sufficient for the new developer to ensure that the code will work perfectly. This is because there are chances that the existing unit tests did not cover all the scenarios of the code. In other words, even though the existing test case would have covered all the lines but it wouldn’t have validated whether all the expected results are met.

With this assumption, developers would do the enhancement of the existing functionality and will end up in a scenario where the unit test cases would pass. But if there is a mistake in the newly developed code, then the code would fail in the executing environment.

And it is practically impossible for developers to go line by line and validate whether all the lines of code are validated. So, the important question is how can we ensure that the unit test cases that pre-exist had validated all the necessary scenarios in the code? We have a type of testing just to solve this and it is called mutation testing. In this post, let us understand what is mutation testing along with a practical implementation of it in C#.

Table of contents

1. What is Mutation testing?
2. Mutants and their types
3. Mutators and mutation score
4. Implementing mutation testing in C#
5. Conclusion

What is Mutation testing?

Mutation testing is a type of software testing where the fault is injected deliberately and the test results are validated against the expected result of the test data set. This is to determine the effectiveness of the test case and test set that already exists.

As the name suggests, mutation testing is all about making a small change or a mutation. Small changes are introduced in the source code to check whether the existing test cases can capture the errors. In ideal cases, the test cases should not pass after introducing the error.

Mutation testing is also known as fault-based testing, program mutation, error based testing, and mutation analysis. In summary, it helps to determine the quality of defined test cases.

What are Mutants and what are their different types?

The subtly modified source code is called the mutants. It is expected that when the test data is run through the mutant, it should ideally give a different result than expected.

There are three types of mutants –

1. Survived mutants,
2. Killed mutants, and
3. Equivalent mutants.

Survived mutants

Mutants that are not detected as errors or the code that did not fail after the modification is injected are survived mutants. Ideally, these mutants must be killed. They are also known as live mutants.

Killed mutants

Mutants that are detected or the mutants that lead to a different result than the original version of the source code are called killed mutants. When killed mutants are more in number, it indicates that the unit test and test set are well written.

Equivalent mutants

Mutants that have the same meaning as the source code and are alive even after running the test data through them are known as equivalent mutants. Equivalent mutants convey the same meaning as the original code but through a different syntax or approach.

What are Mutators and Mutation score?

Mutators are responsible for the code mutation, as they define the kind of change to make, to have the mutant version of the source code. On the other hand, Mutation Scores are calculated based on the number of live and killed mutants

The formula to calculate the mutation score is -

Mutation score = (number of killed mutants / total number of mutants) * 100

Higher the mutation score, the higher the correctness of the unit test and test set. Equivalent mutants are not considered for calculating mutation score.

Implementing Mutation testing in C#

To implement mutation testing in C#, we need to create an application that implements functionality and a unit test case to test it. After completing the implementation of unit testing, to achieve mutation testing, we are going to use Stryker as an example in this blog. Stryker will take care of mutating the code and running the mutated code against the test set and providing the killed and survived mutants. Remember - it is a cumbersome task to mutate the entire set of code and figure out the killed and survived mutants manually.

Here are the steps to create the project -

Step 1

Create a console app project in visual studio.

Open the program.cs file and create a new method. It could be simply a functionality to check whether the input number is odd or even. You can implement any other functionality of your choice.

Step 2

Add a Nunit test project to the solution.

Open the Unittest.cs file and implement the logic to unit test the code in the program.cs file.

Step 3

Open the package manager console or command prompt and install Stryker globally using the below command.

dotnet tool install -g dotnet-stryker

After installing Stryker, go to the directory of the unit test project inside the solution and execute the below command to run the mutation testing.

dotnet-stryker

On running the above command, Stryker will run the mutation testing and provide the result. The same will be created as an HTML report for the users to view in the browser.

Some of the important results produced by Stryker include –

• Total number of mutation testing that has been run
• Killed mutants
• Survived mutants
• Mutation score

Along with these, Stryker provides an HTML report, through which users can figure out the exact mutation which Stryker has carried out and the code which has failed to kill.

Clicking on Program.cs file on this report will open the detailed error results as shown below.

Conclusion

Mutation testing comes in as a rescue to strengthen the unit testing and test set of a given code. It makes unit testing more reliable for the developers to make changes to any existing code. So go ahead and perform mutation testing for your next code.