[SystemSafety] 7% of mistakes in Coq proofs go undetected

[Derek M Jones]

Dewi,

> I had no idea there was so much tool support for mutation testing. It

A recent survey:
https://orbilu.uni.lu/handle/10993/31612

> sounds like it could be a useful activity to carry out in conjnunction with
> (or even instead of) structural coverage analysis.

The random nature of mutation analysis means there are likely to be
some gaps in coverage.

> The paper does state, "It is important to note that mutation scores are
> much higher than traditionally seen in mutation testing research". Does
> anyone know what mutation scores are achieved by typical test suites?

This difference probably has more to do with the maturity of the
mutation operators used.
Researchers get a paper published if they invent a mutation operator
that generates a higher percentage of undetected mutants, compared to
existing operators.  Over time the percentage has increased.

Also researchers tend to use the same test suites, so there is some
over learning.

I'm sure the failure to detect rate will 'improve', as researchers
learn more about Cow mutation operators.

-- 
Derek M. Jones           Software analysis
tel: +44 (0)1252 520667  blog:shape-of-code.coding-guidelines.com