[SystemSafety] Difference between software reliability and astrology

[Paul Sherwood]

On 2024-08-20 21:13, Prof. Dr. Peter Bernard Ladkin wrote:
> On 2024-08-20 20:40 , Paul Sherwood wrote:
>> 
>> These memoryless distributions seem chosen to model relatively simple 
>> software
> 
> They are not "chosen", they exist. Either your software behaviour 
> fulfils the property, or it doesn't.

Hmmm. As I understand it they are models, chosen by the modeller(s)? In 
some sense these distributions actually describe what is observed in 
some classes of software behaviour, but your papers eloquently describe 
various limitations.

It's obvious that some software behaviour doesn't fulfil the property, 
so if we hope to make progress there we would need to 
identify/devise/choose different models?

>> "We conclude that establishing the reliability of RTOS practically 
>> using the Bernoulli/Poisson mathematics in this manner looks close to 
>> infeasible. Yet Annex D currently states in its second sentence “This 
>> approach is considered particularly appropriate as part of the 
>> qualification of operating systems, [etc.]” !
>> 
>> It seems to me that for complex software in general, we'll need 
>> something better?
> 
> Better? Like what?

https://mathworld.wolfram.com/BayesianAnalysis.html perhaps?

> If the behaviour of your software doesn't fulfil the constraints of 
> particular stochastic processes, all you can say is that you can't 
> evaluate the software using those stochastic processes.

True.

> If you want to try to magic up stochastic processes that fit the 
> behaviour of your software, be my guest.

I'm an engineer - magic wouldn't do the trick, so to speak. However I 
and my colleagues have spent some years on this now, and I believe we 
have made progress.

> Until that point, statistical evaluation of your software would not be 
> possible. That, in particular, would invalidate all claims you might 
> want to make via "statistics".

True, until we establish a model or models for the behaviour we actually 
observe.

>> - the standards have been oriented towards simpler software (with 
>> justification, because complexity makes safety more difficult), and 
>> simple software (particularly software designed for safety-critical 
>> use running on simple hardware) can be considered practically 
>> deterministic.
> 
> It is necessary to be more specific. My point concerned the IEC 61508 
> standard, and you might find people working with that standard who 
> wouldn't necessarily agree that it is "oriented towards simpler 
> software".

I'm sure you are correct, and to be fair it's possible to develop 
complex software even for a single core microcontroller.

> I think what we would all agree is that, if your software doesn't 
> fulfil the requirements of a standard such as 61508, then it doesn't. 
> If that is so for all standards concerning safety-related software 
> (such as for rail, or for civil aviation), then you are out of luck in 
> claiming your software is somehow nevertheless appropriate for use in 
> that area.

As I understand it some standards (including 61508) allow for the 
possibility of tailoring and new methods, so perhaps things are not 
quite as black-and-white as that.

>> - for more complex software the Bernoulli/Poisson model may be 
>> applicable in some cases, but not generally.
> 
> I don't know why you would conflate Bernoulli processes and Poisson 
> processes.

Only because they were the two models you mentioned in your papers.

> There are a couple of basic facts here. You are lacking any useful 
> statistical evaluation procedure for something like the Linux kernel. 
> Furthermore, it wasn't developed according to any of the standards, in 
> any industry branch, for safety-related software. So you can't use it 
> in any of those applications. Nothing that has been said during 
> discussion changes either of those two facts.

So far, that is also true. I am making sure I understand the 
"state-of-the-art" as I said in my opening email. As always I am 
grateful for your willingness to explain, and your clarity of 
communication.

br
Paul