# Mutation testing
[](https://github.com/domohuhn/mutation-test/actions/workflows/dart.yml)
[](https://codecov.io/gh/domohuhn/mutation-test)
When writing test cases for software, QA engineers often rely on metrics like
code coverage to verify that your test cases actually test your program.
However, you cannot quantify the quality of your tests with such a simple metric. It is possible to reach high line and branch coverage, while you are only testing a fraction of the observable behavior of your units. The worst case are tests that only call all methods to reach a high line coverage, but do not contain any assertions.
Sometimes you forget to add an assertion statement in a test case or you removed some assertions during in your development branch so that the continuous integration build succeeds. Ideally, this should be caught during the code review, but any manual process is error prone.
So how can we evaluate the quality of our software tests if line coverage is not a good metric? What is a "good" test?
In short, a good test should fail if there are changes of the observable behavior of the tested procedures.
You can evaluate the quality of your tests by modifying a single line of your program and then verify that your tests are sensitive to that change. This process is called [Mutation testing](https://en.wikipedia.org/wiki/Mutation_testing). After a certain number of mutations, the fraction of detected to undetected mutations is an indication of the quality of your tests.
Performing this procedure manually on a whole program is extremely tedious.
This repository contains a command line program that automates this procedure for code in any programming language. It can be customized to
your needs, because all rules modifying the source code and how to
run the tests can be defined in XML documents. The program is fully self contained, so you can just grab the binary and start testing!
## Quick start
If you are working on a dart project, you can run the binary without any arguments at the root of your project.
The application will the assume that "dart test" is the test command and that all files ending with ".dart" in
the directory lib/ are input files.
```bash
# Adds the package
dart pub add --dev mutation_test
dart run mutation_test
```
Running this command may take a long time (hours depending on the size of your code). The output will be written to the directory ./mutation-test-report.
The default report format is html. A top-level [report](https://domohuhn.github.io/mutation-test/doc/output/mutation-test-report.html) will be generated listing all input files:
From there, you can follow the links to the [reports for the individual input files](https://domohuhn.github.io/mutation-test/doc/output/example/source.dart.html).
These reports show all lines of the source files, and undetected mutations are marked as red lines. You can view the undetected changes by clicking
on the respective line:
The application also supports several command line options:
```bash
# Prints a summary of all command line options:
dart run mutation_test --help
# Run the tests defined in "example/config.xml":
dart run mutation_test example/config.xml
# Or a fully customized test run with a rules file and 3 input files:
# The rules contained in mutation-rules.xml are always used when testing files.
# inputset1.xml may define special rules for some files that
# are also listed in the same xml document.
# The input files source1.cpp and source2.cpp
# are just tested with the rules from mutation-rules.xml (--rules).
# The output is written to directory output (-o) and the
# report is generated as markdown file (-f md).
dart run mutation_test -f md -o output --rules mutation-rules.xml inputset1.xml \
source1.cpp source2.cpp
```
The first command in the section above would produce the following [report](https://domohuhn.github.io/mutation-test/doc/output/mutation-test-report.html).
Check also the [examples folder](https://github.com/domohuhn/mutation-test/tree/main/example), as it contains the inputs to produce this report.
The API documentation generated by dart can be found on [pub](https://pub.dev/documentation/mutation_test/latest/).
## Running an incremental analysis (CI)
Performing a mutation test on your whole code base will take very long, and in most cases it is not needed. Often you will only want to check the
difference between two commits, e.g. to review a pull request. This is especially helpful to run the analysis as part of a continuous integration pipeline.
On linux, you can run an incremental analysis on the changes between the current and last commit by using this command:
```bash
dart run mutation_test $(echo $(git diff --name-only HEAD HEAD~1 | grep -v "^test" | grep ".dart$" | tr '\n' ' '))
```
The command lists all changed files, removes paths starting with test and all files not ending with '.dart', and then runs the mutation test on all remaining files.
Similar versions of this command should work on windows or mac os.
### Speeding up the analysis
In order to reduce the time needed for a full project analysis, you can provide coverage data in the [lcov](https://github.com/linux-test-project/lcov) format when calling the program.
The dart sdk supports generating the coverage information with a few commands:
```bash
dart pub global activate coverage
dart pub global run coverage:test_with_coverage
dart run mutation_test --coverage coverage/lcov.info
```
The algorithm to exclude tests uses a conservative approach: It will only exclude mutants that are marked as instrumented and without any hits in the lcov file.
Lines or files that are not present in the coverage database are assumed to be part of the covered statements.
There is also an experimental option to exclude strings without interpolation as mutation candidates:
```bash
dart run mutation_test --exclude-strings
```
## Features
- Fully configurable mutation rules via XML documents and regular expressions
- Sections of files can be whitelisted on a per file basis
- Only mutants whose statements are covered will be executed
- You can add global exclusion rules for e.g. comments, loop conditions via regular expressions
- Different report formats are supported: html, xunit/junit, markdown and XML
## A brief description of the program
mutation_test is a program that mutates your source code and verifies that the test commands
specified in the input xml files are sensitive to those changes. Mutations
are done as simple text replacements with regular expressions, so any text
file can be mutated. Once one of the files has been mutated, all provided
test commands are run as a separate process. The exit code of these
commands is used to verify that the mutation was detected. If all tests
return the expected return value, then the mutation was undetected and is
added to the results. After all mutations were done, the results will be
written to the terminal and a report file is generated.
mutation_test is free software, as in "free beer" and "free speech".
mutation_test contains a set of builtin rules, that allow you to start
testing right away. However, all rules defining the behavior of this program
can be customized. They are defined in XML documents, and you can change:
- input files and whitelist lines for mutations
- compile/test commands, expected return codes and timeouts
- provide exclusion zones via regular expressions
- mutation rules as simple text replacement or via regular
expressions including capture groups
- the quality gate and quality ratings
You can view a complete example with every possible XML element parsed by
this program by invoking "mutation_test -s". This will print a XML document to
the standard output. The displayed document also contains comments explaining
the syntax of the XML file. You can provide multiple input documents for a
single program start. The inputs are split into three categories:
- xml rules documents: The mutation rules for all other files are parsed
from these documents and added globally. Rules are specified via "--rules".
- xml documents: These files will be parsed like the rules documents, but
anything defined in them applies only inside this document.
- all other input files
If a rules file is provided via the command line flag "--rules", then the
builtin rules are disabled, unless you specifically add them by passing "-b".
You can provide as many rule sets as you like, and all of them will be added
globally. The rest of the input files is processed individually. If the file
extension is ".xml", then the file will be parsed like an additional rules file.
However, this document must have a element that lists all mutation
targets. Any other file is interpreted as mutation target and processed with
the rules from the documents provided via "--rules".
The rules documents and the input xml files use the same syntax, so both
files may define mutation rules, inputs, exclusions or test commands.
However, a quality threshold may only be defined once.
## Reports
After a input file is processed, a report is generated. You can choose multiple output formats for the reports. As default, a html file is generated, but you can also choose xunit/junit, markdown or XML. You can see examples of the outputs in the [example folder](example/config-report.md).
## Input XML documents
This chapter explains the structure of the input XML documents. They must conform to the following schema:
```Xml
...
...
...
...
...
...
```
You can see an example for an input document in the example folder, or the application can generate one by running one of these commands:
```bash
# Shows a XML document with the complete syntax:
mutation_test -s
# Shows the builtin mutation rules and exclusions:
mutation_test -g
```
The generated documents also contain some helpful comments on how to create your own rules. You should usually provide two different documents: one with the mutation rules given as argument to "-r" and another one with the input files. The reason why mutation_test always loads two files (unless you disable the builtin rule set via "--no-builtin" and don't provide your own rules file) is that you can reuse the same set of rules for many different input files.
### Files
The children of "files" elements are individual files:
```Xml
example/source.dartexample/source2.dart
```
The application will perform the mutation tests in sequence on the listed files. All mutations that are not in an exclusion or inside a whitelisted area will be applied.
### Directories
The children of "directories" elements are directories where files are searched:
```Xml
srclib
```
The application will perform the mutation tests on all files found in the directories.
### Commands
The commands block lets you specify the command line programs to verify that a mutation is detected. The commands are run in document sequence and must be each a single command line call.
```Xml
make -j8ctest -j8
```
### Exclude
You can create rules to exclude portions of source files or the full file from the mutation testing:
```Xml
path/to/exclude.dart
```
Explicit exclusions have precedence over inclusions.
### Rules
This element is the most important part of the document. It defines what is mutated, and how it is changed.
```Xml
```
### Threshold
The threshold element allows you to configure the limit for a successful analysis and the quality ratings.
Below is the built-in configuration:
```Xml
```
When setting a failure limit, remember that some mutations may be impossible to detect (e.g. converting "0" to "-0").
### Table of XML elements
Here is a table of all XML elements that are parsed by this program:
| Element | Children | Attributes | Description |
| --------- | ------------------------------- | ----------- | ----------- |
| mutations | files, rules, exclude, commands | version | Top level element |
| files | file | | Holds the list of files to mutate |
| directories | directory | recursive | Holds the list of directories to search for files |
| exclude | token, regex, lines | | Holds the list of exclusions from mutations. |
| commands | command | | Holds the list of commands to run |
| rules | literal, regex | | Holds the list of mutation rules |
| file | lines | | Contains the path the to file as text. If there are lines children present, only the given lines are mutated. |
| lines | | begin, end | Specifies an interval of lines \[begin,end\] in the source file. |
| matching | | pattern | Specifies the pattern for the file names in the directory.
| command | | name, group, expected-return, timeout | Contains the command to execute as text. All attributes are optional. |
| token | | begin, end | A range in the source file delimited by the begin and end tokens. |
| literal | mutation | id, text | Matches the string in attribute text and replaces it with its children. |
| regex | mutation | id, pattern, dotAll | A pattern for a regular expression. The expression is always multiline and processes the complete file. You can use "." to match newlines if the optional attribute dotAll is set to true. |
| mutation | | text | A replacement for a match. If this element is a child of a regex node, then capture groups can be used in the text via $i. |
| threshold | rating | failure | Configures the limit for a failed analysis and the quality ratings |
| rating | | over, name | A quality rating. Attribute over is the lowest percentage for this rating. |
## Command line arguments
```bash
mutation_test
```
The program accepts the following command line arguments:
| Short | Long | Description |
| -------------- | ------------------------- | --------------------------------------------------------------------------------------------------------- |
| -h | --help | Displays the he ... ...