[SystemSafety] A Critical-System Assurance Manifesto

[Peter Bernard Ladkin]

On 2017-12-11 15:47 , Derek M Jones wrote:
> Peter,
> 
>> * Statistical Evaluation
> Chapter 1, the Urn model.
> 
> The empirical evidence is that the Urn model is not a good
> model of faults discovery.

[Or, as modified, "model of defective input space"]

I always find it helpful to read a paper before drawing conclusions about its contents.

Probability is one of the most misunderstood technical subjects, even among those who have to use
probability calculations in their daily work.

Compare: the empirical evidence is that 2+2 is not always 4.

How would you gather such empirical evidence? By looking at examples of where 2+2 occurs and seeing
what it comes to. How would you do that? Looking in books; observing people doing the calculation in
their daily lives; asking people to perform the calculation.

The point is that there is a conceptual mistake. You would actually be measuring the number of times
2+2 is miscalculated, not what the actual value of 2+2 is.

I would suggest that a similar conceptual mistake lies in the above claim about the Urn model.

> The Urn model gives equal weight to each item it contains.

What does that mean? What is "weight"? Actually, the Urn model says precisely nothing about how
inputs are selected.

I was once in e-mail conversation with a university professor who worked in reliability. She
claimed, if I remember rightly, that a Bernoulli Process required that its inputs were uniformly
distributed. Let us look at what that might mean.

Suppose I have a stochastic process with 100,000 possible inputs. Some of these inputs result
(deterministically) in incorrect behaviour ("failure") and some (deterministically) in correct
behaviour ("success"). According to the statement above, if I give 5 inputs to the process, it can't
be Bernoulli. For five inputs have occurred, and 99,995 have not. That is not consistent with what
is meant by the "uniform distribution" of inputs. Similarly, if I have given 99,995 inputs, each of
them exactly once, and have 5 more to go: it's not uniform, so according to the above can't be
Bernoulli. Then, magically, 5 inputs later I give all 100,000 inputs exactly once, and can now claim
it is Bernoulli! And ditto for every multiple of 100,000. So apparently I have a process which is
Bernoulli every multiple of 100,000 inputs (providing each occurs exactly the same number of times),
and in between (including, say, 199,999 inputs given) is not. Nonsense, isn't it?

So maybe we can say that it can be a Bernoulli process with other than multiples of 100,000 inputs,
providing that the *inputs to come will satisfy the "uniformity" condition*. But then, we still
can't derive anything useful about the stochastic process we are looking at until we discover, in
the future, that it has indeed turned out to have been Bernoulli (as we reach that magic multiple of
100,000 inputs). According to this view, it is a wonder that anyone has ever found a Bernoulli
Process good for modelling anything at all. Which would bely its history over the last three hundred
years.

So, back to the current conversation:

On 2017-12-11 18:06 , Derek M Jones wrote:
> While in theory a visit to any part of the input space might have
> the same probability, in practice some inputs occur much more
> frequently than others.  .........
> 
> The Urn model is not a good description of visits to the input space,
> for many problems.

The inputs can be scattered any which way you like. The Urn model says nothing about how balls are
chosen. And for that reason, you are right that the "Urn model [does not give] a good description of
visits to the input space. It doesn't give *any description* of visits to the input space. That is
why they can be as you like.

PBL

Prof. Peter Bernard Ladkin, Bielefeld, Germany
MoreInCommon
Je suis Charlie
Tel+msg +49 (0)521 880 7319  www.rvs-bi.de





-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20171212/e09d2e4f/attachment.sig>