Mutation testing with infection

A few months ago I read about mutation testing for the first time. At that time I started my current pet project with a focus on proper testing, and the idea hooked me immediately. However, back then my overall testing state was not at a level that such advanced mechanisms would make sense. But now, during my christmas vacation, the time was right 🙂

What is mutation testing? In my own words: with mutation testing one can check the quality of unit tests for code, by automatically and intentionally mutating the code and comparing the unit test results with the original ones. The mutated code is called a “mutant” and if your test cases recognize most of them your tests may be considered sufficient (a lot ifs and whens apply here).

For my php-based pet project, infection was the natural choice; I couldnt find any other mutation testing framework which is similarly well maintained and documented.

The initial setup, however, was not fully smooth, I had to overcome some issues in my environment:

First, I had some trouble to run my unit tests locally (so far I only executed them on my server and in CI, but I realized I can shorten my turnaround times significantly when I also enable them locally). As I – shame on me – only have Windows on my desktop, I had to setup php (with extensions such as xdebug), mariadb and Papercut (SMTP service) locally. I wanted to do this already for a while, so now I digged into it. Cost me some hours to sort everything out.

The second issue was more tricky. Some history: Due to the way how PHP and PHPUnit work (some say its even a bug in PHP?), outputting http headers via the header() function is conflicting with PHPUnit. A helpful answer on stackoverflow lists the options to solve this, and I chose the second option long time ago. The first option is not working for me, the third option didnt seem feasible either as I actually need the header output for my functionality. So I chose to run PHPUnit with the “–stderr” parameter, worked for me. With infection however, this resulted in issues. Outputting all PHPUnit results on stderr made infection, which parses PHPUnit’s output for errors, fail. I almost got mad in resolving this “deadlock”, but then stumbled over a github issue in which some compatibility topics were discussed. As far as I understand, their issue was another one, but I tried the given workaround, which seperates the initial PHPUnit run from the mutated runs:

vendor/bin/phpunit --coverage-xml=build/logs/coverage-xml --log-junit=build/logs/junit.xml
vendor/bin/infection --skip-initial-tests --threads=$(nproc) --coverage=build/logs --show-mutations --no-interaction 

This gave me the first output of infection:

The metrics at the bottom looked quite good, according to the infection documentation those values are good-ish. The hint in the second-to-last line is also important to acknowledge: Not all slipped-through mutants are in fact problems. Anyhow, it also shows 26 not covered mutants, i.e. code mutations which were not uncovered by my existing test cases.

In the infection.log file you can see which of those there are. Example (redacted):

2) ...\TaskController.php:137    [M] NewObject

--- Original
+++ New
@@ @@
         if (empty($taskId)) {
-            return new Response('HTTP/1.1 500 Internal Server Error ');
+            new Response('HTTP/1.1 500 Internal Server Error ');
+            return null;

Seems my test cases are not checking the return-types sufficiently or my code is not further using the returned objects here. Interesting! As a next step I will investigate how I can improve my unit tests based on the mutation test results.