Skip to main content
Antal István

Guides6 min read

How mutation testing works

Mutation testing breaks your code on purpose and checks that a test notices. A worked example with 100% coverage and two surviving bugs, and what it costs to run on a real project.

A screen-print poster of a hand reaching in with tweezers to swap one small yellow gear inside an oversized open clockwork mechanism, while a red twin-bell alarm clock wired to the mechanism rings.

A test suite tells you when the code is wrong. Nothing tells you when the suite is wrong. A test that asserts too little passes just like a good one, and it keeps passing after the code it covers breaks.

Mutation testing checks the tests. A tool makes one small change to your code, such as turning a >= into a >, and runs the suite. If a test fails, the suite noticed the change. If everything stays green, you have found a bug that your tests would let through.

I gave it two paragraphs in Testing and static analysis. This is the longer version, with an example you can run.

A function with 100% coverage

Here is a function that works out what an order costs. Members get 10% off, and orders of £50 or more ship for free. Amounts are in pence.

export function orderTotal(subtotal: number, isMember: boolean): number {
    let total = subtotal;
    if (isMember) {
        total = total - subtotal / 10;
    }
    if (total >= 5000) {
        return total;
    }
    return total + 499;
}

And here are its tests:

test('adds shipping to a small order', () => {
    expect(orderTotal(2000, false)).toBe(2499);
});

test('ships a large order for free', () => {
    expect(orderTotal(10000, false)).toBe(10000);
});

test('gives members a discount', () => {
    expect(orderTotal(10000, true)).toBeLessThan(10000);
});

All three pass, and the coverage report has nothing to complain about. Every statement runs and every branch is taken both ways.

----------|---------|----------|---------|---------|
File      | % Stmts | % Branch | % Funcs | % Lines |
----------|---------|----------|---------|---------|
 order.ts |     100 |      100 |     100 |     100 |
----------|---------|----------|---------|---------|

What the tool does

I ran Stryker on it. A mutation testing tool works in a loop:

  1. Run the suite on the untouched code, to check it is green and to record which tests reach which lines.
  2. Make one small change to the code. The changed copy is called a mutant.
  3. Run the tests that reach the changed line.
  4. If a test fails, the mutant is killed. If they all pass, it survived.
  5. Undo the change and repeat with the next one.

The changes come from a fixed list of mutators. They swap arithmetic and comparison operators, force a condition to true or false, empty a block, blank a string, and so on. Each one imitates a mistake people really make. For this function Stryker came up with twelve:

The twelve mutants of orderTotal and whether the tests caught each one
OriginalMutantResult
the function body{}Killed
if (isMember)if (true)Killed
if (isMember)if (false)Killed
the discount block{}Killed
total - subtotal / 10total + subtotal / 10Killed
subtotal / 10subtotal * 10Survived
if (total >= 5000)if (true)Killed
if (total >= 5000)if (false)Killed
total >= 5000total > 5000Survived
total >= 5000total < 5000Killed
the free shipping block{}Killed
total + 499total - 499Killed

Ten killed out of twelve is a mutation score of 83%. The score is less interesting than the two that got away.

The two survivors

Stryker prints each survivor as a diff. The first:

[Survived] EqualityOperator
src/order.ts:7:9
- if (total >= 5000) {
+ if (total > 5000) {

No test places an order of exactly £50, so nothing pins down which side of the line it falls on. If someone changed that operator during a refactor, a customer spending exactly £50 would start paying for shipping and the suite would stay green.

The second is worse:

[Survived] ArithmeticOperator
src/order.ts:5:25
- total = total - subtotal / 10;
+ total = total - subtotal * 10;

With this change a member who orders £100 of goods is charged minus £895.01. The shop pays them. The test called "gives members a discount" still passes, because it only asserts that the total is less than £100, and minus £895.01 is.

Coverage could not have told us either of these. Coverage records which lines ran. It does not know whether anything looked at the result. That weak discount test ran the discount line, so the line counts as covered.

Killing them

Each survivor points at a missing or weak test, so the fix is usually obvious. Here it is one new test and one sharper assertion:

test('ships an order of exactly £50 for free', () => {
    expect(orderTotal(5000, false)).toBe(5000);
});

test('gives members 10% off', () => {
    expect(orderTotal(10000, true)).toBe(9000);
});

Run it again and all twelve die. Stryker also reports which test killed how many mutants, which is a quick way to see what each test is worth:

✓ adds shipping to a small order (killed 5)
✓ ships a large order for free (killed 2)
✓ ships an order of exactly £50 for free (killed 1)
✓ gives members 10% off (killed 4)

A test that kills nothing is worth a look too. It may check something the mutators cannot reach, or it may check nothing.

Why it is worth having

The result is specific. A coverage figure of 85% tells you that something is untested somewhere. A surviving mutant gives you a file, a line and a concrete change that nobody would notice. You can decide in a minute whether it matters.

It measures assertions. As far as I know it is the only common tool that does. Linters, type checkers and coverage all look at the code under test. Mutation testing is the one that fails when a test runs a line without checking what it produced.

It finds code that does nothing. Sometimes a mutant survives because the line makes no difference: a default that is never used, or a check that an earlier check already made. The right fix is then to delete the line, and the codebase gets smaller.

Here is one that looks like careful code. The suite even has a test for the empty cart:

export function cartTotal(prices: number[]): number {
    if (prices.length === 0) {
        return 0;
    }
    return prices.reduce((sum, price) => sum + price, 0);
}
[Survived] ConditionalExpression
src/cart.ts:2:9
- if (prices.length === 0) {
+ if (false) {

With the guard switched off, the empty cart test still passes. reduce starts from the 0 it is given, so an empty array already comes out as 0. The guard is probably there because reduce throws on an empty array when it has no starting value. Here it has one, and the three lines can go.

It keeps AI-written tests honest. An agent asked for tests will write plenty, and many assert whatever the code happens to do, or very little. I can't read them all closely. A mutation run doesn't care who wrote the tests or how many there are, and a surviving mutant is a task an agent handles well: the target is exact, and the tool says when it is done.

What it costs

The price is time. Every mutant needs its own test run, and a real project has thousands of mutants. MechaGremlin, the small agent harness I wrote, has 7,414 of them, and the full run takes about four and a half hours on a GitHub-hosted runner. So it runs as a separate CI job after the ordinary checks pass. It is not part of the loop while I work.

Two things keep that manageable:

  • Per-test coverage. Stryker records which tests reach each line and runs only those for a mutant. In the example that came to 1.5 tests per mutant, not 4.
  • Mutating only what changed. You can point the tool at a file or a range of lines, so a change gets its own short run and the full one is left to CI.
npx stryker run --mutate "src/order.ts:4-6"

The second cost is mutants that no test can kill, because the changed code behaves the same as the original. These are called equivalent mutants. MechaGremlin sorts MCP servers by name, and Stryker turns the < in that comparison into <=. No two servers share a name, so the result is identical and no test could tell the difference.

Most projects accept a score below 100% for this reason. On MechaGremlin I went the other way: the build fails if any mutant survives, and an equivalent mutant is switched off with a comment that has to say why.

// Names are unique, so no two compare equal, and `<=` sorts as `<` does.
// Stryker disable next-line EqualityOperator
const inNameOrder = (first: string, second: string): number => (first < second ? -1 : 1);

There are ten of those in the whole codebase. That is much easier on a project that starts that way. On an existing codebase, read the survivors in the code you are changing and leave the overall score alone.

The third cost is less obvious. A mutant can turn a loop that ends into one that never does. The tool has a timeout for that, but one of MechaGremlin's mutants looped while allocating memory, at around 300 MB a second, and a few of them side by side filled the CI runner's 16 GB before the timeout fired. The mutation run now sits under a small guard that kills any test process holding more than 2 GiB.

Trying it

For a JavaScript or TypeScript project:

npm init stryker@latest
npx stryker run

The first command asks which test runner you use and writes a config file. The second prints the survivors and writes an HTML report where you can browse them file by file.

Other languages have their own tools: cargo-mutants for Rust, PIT for Java and mutmut for Python. Stryker also has versions for C# and Scala.

Start with one module you care about, such as pricing or permissions, and read what survives.

Related reading