[SystemSafety] Statistical Assessment of SW ......

[Derek M Jones]

Peter,

> Methods of assessing reliability of SW are normally predicated on *no failures having occurred for a
> certain number of trials*. Providing that no failures have been observed, the conclusion that the
> failures have a specified low occurrence rate may be drawn with a specified level of confidence,
> dependent on the number of trials observed. I mean, this is just basic statistical methodology, is
> it not?

Indeed and here is one of the best papers I have ever read on software
reliability discussing this issue:
http://ntrs.nasa.gov/search.jsp?R=19820013026

The elephant in the room that everybody (except the authors of the above
report and a few others) ignores is the distribution of the input data.

I can have very reliable software, having tested it with English input,
that fails miserably when presented with input in other languages.

Nobody likes to talk about the distribution of the input values because
it is so very hard to quantify.

Software reliability analysis, with a few exceptions like the above,
assumes that the input distribution is unchanging.  Ok, now figure out
the probability that the input will remain unchanging......

-- 
Derek M. Jones           Software analysis
tel: +44 (0)1252 520667  blog:shape-of-code.coding-guidelines.com