I used mutmut yesterday to evaluate the unit test quality of some of my projects. I tried cosmic-ray before, but it was unusable (bugs, performance). I also tried mut.py, but I also couldn’t get that one startd.
One issue when using mutmut I found is that it gives too many false-positives: Results where – according to mutmut – at least one unit test should have failed, but it didn’t. Most of the cases I observed were string changes:
# mutmut adds 'XX' to the following strings:
__version__ = "0.1.2"
ValueError("Explanation about the cause of the error")
logging.info("Starting to execute xy.")
All of those should not get a mutation in the first place. I don’t want to add # pragma: no mutate to all of those lines. I simply want to delete / skip / ignore this type of mutation in mutmut (which should also speed the execution up, I hope).
Is this possible?
There are a few different answers to this:
__version__is whitelisted in mutmut since a while back.- You can mark a line with
# pragma: no mutateto not mutate it. - There’s a new system to have advanced whitelisting that might solve this for you: https://github.com/boxed/mutmut/#advanced-whitelisting-and-configuration
(For the speed, try if you can use hammett, it’s WAY faster)
There is no system to disable an entire class of mutations though. I don’t really think that’s a good tool, because it’s super important to test your error cases like exceptions because I’ve found that’s where most bugs mutmut finds are! Logs are a bit more mixed bad I’d agree but that can be handled with the two whitelisting systems.
Btw, I’m the author of mutmut. Next time and for anyone coming from google, ask in the github issue tracker! I just stumbled on this by mistake, and 7 months later.