[SystemSafety] What do we know about software reliability?

Wed Sep 16 11:46:38 CEST 2020

If A is dependent on some temporal event, and the testing is conducted prior to that event – what does the testing tell you about the outcome after that event?

From: systemsafety <systemsafety-bounces at lists.techfak.uni-bielefeld.de> On Behalf Of Peter Bishop
Sent: 16 September 2020 10:21
To: systemsafety at lists.techfak.uni-bielefeld.de
Subject: Re: [SystemSafety] What do we know about software reliability?

I guess that is the nub of the argument with regard to systematic faults.

- as a standalone artefact, the software is "faulty" rather than reliable

As a component in some operating environment, E, "reliability" becomes meaningful.

i.e. the probability of failure of the software component is

    Pr_E( not A)

i.e. In environment E , the probability of an input value that is (not A).

Now suppose we can specify which input values in a program are correct or faulty (i.e. we know A and not A precisely).

could you tell me exactly when the program will fail in operation ?

- Obviously (apart from the special case where it always fails,  Pr_E( not A)=1), 
   you cannot do this unless you have a crystal ball that predicts future inputs in environment E with total precision.

But you can express the future occurrence of A in probabilistic terms, i.e. how often (not A) will arise in environment E.

In practice of course you do not actually have information about (not A)
- these are the bugs you have not found yet.

But testing in operational environment E allows you to set a confidence bound on Pr_E(not A)
- even if you do not know where the bugs are located in the program input space.

And that is what statistical testing does,
but as others have already said, the reliability bound is conditional on the operating environment E.

Peter

PS

There has been some work on the worst case change in the bound that can be claimed for a different operational profile E'
https://www.researchgate.net/publication/307555914_Deriving_a_frequentist_conservative_confidence_bound_for_probability_of_failure_per_demand_for_systems_with_different_operational_and_test_profiles

where idea of extra "padding tests" is introduced to maintain the claimed bound for specified departures from E.

There is also an alternative strategy - "fair" testing - where testing is based on the likelihood of a particular input A being faulty 
(based on code path length rather than E), which can reduce sensitivity of reliability bounds to profile change.

https://www.researchgate.net/publication/234802669_Rescaling_reliability_bounds_for_a_new_operational_profile

On 16/09/2020 01:02, hugues.bonnin at free.fr <mailto:hugues.bonnin at free.fr>  wrote:

Hi all,

I have an alternative "toy" to propose: do you think that this software is reliable (written in ada-like code)?

begin

if A then 
  do_nothing 
else
  fail --potentially hurt and kill people
end if

end

The specification of the software is to do nothing; 
NB: I'm not asking if it is the best implementation, whatever the criteria are, but just : "is it reliable?"

regards,

Hugues

----- Mail original -----

De: "Peter Bernard Ladkin"  <mailto:ladkin at causalis.com> <ladkin at causalis.com>
À: systemsafety at lists.techfak.uni-bielefeld.de <mailto:systemsafety at lists.techfak.uni-bielefeld.de> 
Envoyé: Mardi 15 Septembre 2020 19:58:45
Objet: Re: [SystemSafety] What do we know about software reliability?

Bev and I and Dewi have a colleague who poses the following question.

"We have clients who have installed hundreds of [examples of our kit]
over the last ten years, and
have never seen any failure. They want to use it in further systems
that they build. What arguments
do we/they need to provide in order validly to justify such further
use?"

So, what is the answer to that question?

PBL

Prof. Peter Bernard Ladkin, Bielefeld, Germany
Styelfy Bleibgsnd
Tel+msg +49 (0)521 880 7319  www.rvs-bi.de <http://www.rvs-bi.de> 

_______________________________________________
The System Safety Mailing List
systemsafety at TechFak.Uni-Bielefeld.DE <mailto:systemsafety at TechFak.Uni-Bielefeld.DE> 
Manage your subscription:
https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety

_______________________________________________
The System Safety Mailing List
systemsafety at TechFak.Uni-Bielefeld.DE <mailto:systemsafety at TechFak.Uni-Bielefeld.DE> 
Manage your subscription: https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety

-- 

Peter Bishop
Chief Scientist
Adelard LLP
24 Waterside, 44-48 Wharf Road, London N1 7UX

Email: pgb at adelard.com <mailto:pgb at adelard.com> 
Tel:  +44-(0)20-7832 5850

Registered office: 5th Floor, Ashford Commercial Quarter, 1 Dover Place, Ashford, Kent TN23 1FB
Registered in England & Wales no. OC 304551. VAT no. 454 489808

This e-mail, and any attachments, is confidential and for the use of
the addressee only. If you are not the intended recipient, please
telephone 020 7832 5850. We do not accept legal responsibility for
this e-mail or any viruses.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/pipermail/systemsafety/attachments/20200916/38ac0bf2/attachment.html>