> For the complete documentation index, see [llms.txt](https://cover-docs.diffblue.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cover-docs.diffblue.com/features/tutorials/how-to-measure-test-quality.md).

# How to measure test quality

<figure><img src="/files/rnQcgj6hgTZDwB5vQ3Wi" alt=""><figcaption></figcaption></figure>

Traditional test coverage (i.e line, statement, branch, etc.) measures what code is executed by your tests. It does not check that your tests are actually able to detect faults in the executed code.

Mutation testing fills this gap. Faults (or **mutations**) are seeded into your code, then your tests are run. If your tests fail then the mutation is **killed**, if your tests pass then the mutation **lived**. The quality of your tests can be gauged from the percentage of mutations killed.

Source code is changed or mutated in three ways – [value](#value-mutations), [decision](#decision-mutations) and [statement](#statement-mutations).

#### Value mutations

Change a small value to a larger value or vice versa.

```c
/* Initial code */

int mod = 1000000007;
int a = 12345678;
int b = 98765432;
int c = (a + b) % mod;

/* Changed code */

int mod = 1007;
int a = 12345678;
int b = 98765432;
int c = (a + b) % mod;
```

#### Decision mutations

Change logical or arithmetic operators to detect errors.

```c
/* Initial code */

if (a < b) {
  c = 10;
} else {
  c = 20;
}

/* Changed code */

if (a > b) {
  c = 10;
} else {
  c = 20;
}
```

#### Statement mutations

Delete a statement or replace it with some other statement

```c
/* Initial code */

if (a < b) {
  c = 10;
} else {
  c = 20;
}

/* Changed code */

if (a < b) {
  d = 10;
} else {
  d = 20;
}
```

### Advantages of mutation testing

* Brings a good level of error detection in the program
* Discovers ambiguities in the source code
* Helps testers write better tests and measure effectiveness of tests written
* Makes source code more efficient

### Mutation Testing Plugins

* [PIT](https://pitest.org/) *(recommended)*
* [Judy](http://www.mutationtesting.org/judy2/)
* [Jester](https://jester.sourceforge.net/)
* [Jumble](https://jumble.sourceforge.net/)

***

### Setting up PIT on your project

#### Maven

[Documentation](https://pitest.org/quickstart/maven/) | [Plugin](https://central.sonatype.com/artifact/org.pitest/pitest-maven)

Add this plugin to your `pom.xml`:

```xml
<plugin>
    <groupId>org.pitest</groupId>
    <artifactId>pitest-maven</artifactId>
    <version>1.17.4</version>
</plugin>
```

Then run: `mvn test-compile org.pitest:pitest-maven:mutationCoverage`, which will output an HTML report to `target/pit-reports/<YYYYMMDDHHMI>`.

For multi-module projects, please see [Aggregating tests across modules](https://pitest.org/aggregating_tests_across_modules/).

#### Gradle

[Documentation](https://gradle-pitest-plugin.solidsoft.info/) | [Plugin](https://central.sonatype.com/artifact/info.solidsoft.gradle.pitest/gradle-pitest-plugin)

Add this plugin to your `build.gradle`:

```gradle
plugins {
    id 'info.solidsoft.pitest' version '1.15.0'
}

// To aggregate Pitest results for a multi-module projects
apply plugin: 'info.solidsoft.pitest.aggregator'

// For multi-module projects
subprojects {
    apply plugin: 'java'
    apply plugin: 'info.solidsoft.pitest'

    pitest {
        threads = 4
        // Export mutations.xml and line coverage for aggregation
        outputFormats = ["XML"]
        exportLineCoverage = true
        timestampedReports = false

        if (project.name in ['module-without-any-test']) {
            failWhenNoMutations = false
        }

        reportAggregator {
            // aggregation configuration (optional)
        }
    }
}
```

Run `gradle pitest pitestReportAggregate` task to have an aggregated report be produced at `${PROJECT_DIR}/build/reports/pitest`.