Kategorien
Coding

Comparison of Python mutation testing modules

When I discovered the existence of Mutation Testing, it was a revelation. It showed me an almost effortless way to „test my tests“ with an absolutely clever way. Recently I did a lot of coding in Python and when I looked for a mutation testing framework for that language I found many results, but no comparison. So I did my own 🙂

The following table is a very subjective and superficial evaluation, without investing extensive digging into each tool’s configuration options. I just took the most straightfoward way to run it on a current company project.

ModuleActively maintained?Ease of useRaw outputEvaluation
mutmutyesok

run & generate html
„Killed 640 out of 954 mutants“

HTML file
314 non killed mutants of which 280 were in a config.py file, so can be ignored.
MutPynoneeded local fix

run
all: 146
killed: 0 (0.0%)
survived: 144 (98.6%)
incompetent: 2 (1.4%)
timeout: 0 (0.0%)
The run is very fast, so I have the feeling that the unit tests for some reasons are not really executed (even though they are listed on startup)
mutatestnosimpleSURVIVED: 5
DETECTED: 16
TOTAL RUNS: 21
RUN DATETIME: 2021-10-10 13:46:20.181892
It seems to run only a random set of mutations
Cosmic Rayyescomplex

create config file, init, baseline, execute, generate html
HTML file
199 non-killed mutants

No summary/expand for each file, only long list of findings

From the above, the maintained mutmut and cosmic ray found the most surviving mutants in my example project, while both other tools found much less. MutPy left a dubious feeling if it actually worked as intended. Both MutPy and mutatest are not maintained anymore. In the future I will look closer into mutmut and cosmic ray and see if I can tweak them towards actually using the results, as the initial ones are still a bit „too raw“ to directly act upon (which lies in the nature of my test project, too).

Eine Antwort auf „Comparison of Python mutation testing modules“

I also compared and tested mutation testing tools for python. I agree that mutatest is very random and not very applicatable for an automated use, it also only works for python <= 3.8. Same for MutPy only works on Python 3.4 to 3.7. Whith mutmut i did not have any issues and it worked out of the box. But I think Cosmic Ray is a little out of place, the latest release is from the 29.Nov.2017 …. I think the only recommendable tool is mutmut.
Thank you for your overview it was a good start for choosing a mutation tool for python

Schreibe einen Kommentar

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert

Diese Website verwendet Akismet, um Spam zu reduzieren. Erfahre mehr darüber, wie deine Kommentardaten verarbeitet werden.