[SystemSafety] Data on Proof effectiveness from real projects

[Steve Tockey]

Bev wrote:

"This is indeed interesting. But what was the the definition of “efficiency of fault detection” here? Finding the most faults for a given outlay of effort? Maximising the chance of finding all faults? I can imagine other definitions."

Generally speaking, efficiency is looking at investment per unit of work done. In other words, increasing efficiency means that the same work is done at lower investment—the same (number of?) defects are found for a lower investment. Maximizing the chance of finding defects would be an issue of effectiveness. Effectiveness is looking at the rate at which the job is done correctly (I.e., a defect is found, not missed). One needs to look at both efficiency and effectiveness of processes to make a fair comparison.


-- steve



From: systemsafety <systemsafety-bounces at lists.techfak.uni-bielefeld.de<mailto:systemsafety-bounces at lists.techfak.uni-bielefeld.de>> on behalf of "Littlewood, Bev" <Bev.Littlewood.1 at city.ac.uk<mailto:Bev.Littlewood.1 at city.ac.uk>>
Date: Wednesday, March 30, 2016 6:48 AM
To: David MENTRE <dmentre at linux-france.org<mailto:dmentre at linux-france.org>>
Cc: "systemsafety at lists.techfak.uni-bielefeld.de<mailto:systemsafety at lists.techfak.uni-bielefeld.de>" <systemsafety at lists.techfak.uni-bielefeld.de<mailto:systemsafety at lists.techfak.uni-bielefeld.de>>
Subject: Re: [SystemSafety] Data on Proof effectiveness from real projects

Hi David

On 30 Mar 2016, at 07:39, David MENTRE <dmentre at linux-france.org<mailto:dmentre at linux-france.org>> wrote:

And there is also King et al. paper "Is proof More Cost Effective than Testing?" on SHOLIS project.

Interestingly, for SHOLIS the efficiency of fault detection was, in decreasing order, "Z proof" (i.e. spec proof), "System Validation" (i.e. System tests), "Integration Test", Code proof and "Acceptance" (client tests?) and Unit test. This illustrates well that the best approach is a mix of test (especially for integration and validation) and proof (especially at spec level, very efficient, but code proof is also more efficient that unit test). Of course, as Matthew said, I should not generalize too much from some samples.

This is indeed interesting. But what was the the definition of “efficiency of fault detection” here? Finding the most faults for a given outlay of effort? Maximising the chance of finding all faults? I can imagine other definitions.

Some of these definitions of efficiency might not correspond with what a potential user of the system would want. If, for example, they (quite reasonably, I think) wanted their system to be reliable, they might define efficiency, ideally, in terms of how much the reliability improved for a given outlay of fault-detection effort.

Maximising the number of faults found (for a given outlay) is, of course, not the same as maximising the reduction in unreliability. There is data from Ed Adams in IBM, many years ago (early 80s), which showed that, for the systems he studied, faults differed enormously in their sizes (i.e. in the contributions they made to the system’s unreliability). The most famous observation from this work was that 30% of the faults found during operation were 5000-year faults (i.e. such a fault would occur in operational use - over thousands of sites - no more than once every 5000 years of exposure).

Clearly, finding lots of 5000-year faults will not improve reliability a great deal. You’d prefer to find the larger faults. So efficiency defined just in terms of numbers of faults found for a given outlay of effort could be misleading unless we know something about the  “seriousness” of the faults - i.e. if your interest is in reliability, what their contributions to unreliability are. It might be nice if the “more serious” faults were always found with higher probability than the “less serious” ones.

The only known procedure that has this property - that the chances of discovery are exactly ranked with their seriousness (in terms of contributions to unreliability) - is operational testing. However, it may not be very efficient, even at reducing unreliability, compared with other approaches… (and that may be an understatement!)

I’m not, of course, suggesting that “testing it in” is a good means of achieving reliability (I’d better say that explicitly before readers of this list jump on me). But it seems there are some subtleties here that may not be captured in some simple notions of “efficiency of fault detection".

Cheers

Bev



_______________________________________________

Bev Littlewood
Professor of Software Engineering
Centre for Software Reliability
City University London EC1V 0HB

Phone: +44 (0)20 7040 8420  Fax: +44 (0)20 7040 8585

Email: b.littlewood at csr.city.ac.uk<mailto:b.littlewood at csr.city.ac.uk>

http://www.csr.city.ac.uk/
_______________________________________________

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20160402/dd91381d/attachment.html>