<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Consolas",serif;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-GB link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='mso-fareast-language:EN-US'>If A is dependent on some temporal event, and the testing is conducted prior to that event – what does the testing tell you about the outcome after that event?<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=EN-US>From:</span></b><span lang=EN-US> systemsafety <systemsafety-bounces@lists.techfak.uni-bielefeld.de> <b>On Behalf Of </b>Peter Bishop<br><b>Sent:</b> 16 September 2020 10:21<br><b>To:</b> systemsafety@lists.techfak.uni-bielefeld.de<br><b>Subject:</b> Re: [SystemSafety] What do we know about software reliability?<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p>I guess that is the nub of the argument with regard to systematic faults.<br><br>- as a standalone artefact, the software is "faulty" rather than reliable<o:p></o:p></p><p>As a component in some operating environment, E, "reliability" becomes meaningful.<o:p></o:p></p><p>i.e. the probability of failure of the software component is<o:p></o:p></p><p> Pr_E( not A)<o:p></o:p></p><p>i.e. <b>In environment E</b> , the probability of an input value that is (not A).<o:p></o:p></p><p>Now suppose we can specify which input values in a program are correct or faulty (i.e. we know A and not A precisely).<o:p></o:p></p><p>could you tell me exactly when the program will fail in operation ?<o:p></o:p></p><p>- Obviously (apart from the special case where it always fails, Pr_E( not A)=1), <br> you cannot do this unless you have a crystal ball that predicts future inputs in environment E with total precision.<o:p></o:p></p><p>But you can express the future occurrence of A in probabilistic terms, i.e. how often (not A) will arise in environment E.<o:p></o:p></p><p>In practice of course you do not actually have information about (not A)<br>- these are the bugs you have not found yet.<o:p></o:p></p><p>But testing in operational environment E allows you to set a confidence bound on Pr_E(not A)<br>- even if you do not know where the bugs are located in the program input space.<o:p></o:p></p><p>And that is what statistical testing does,<br>but as others have already said, the reliability bound is conditional on the operating environment E.<o:p></o:p></p><p>Peter<o:p></o:p></p><p>PS<o:p></o:p></p><p>There has been some work on the worst case change in the bound that can be claimed for a different operational profile E'<br><a href="https://www.researchgate.net/publication/307555914_Deriving_a_frequentist_conservative_confidence_bound_for_probability_of_failure_per_demand_for_systems_with_different_operational_and_test_profiles">https://www.researchgate.net/publication/307555914_Deriving_a_frequentist_conservative_confidence_bound_for_probability_of_failure_per_demand_for_systems_with_different_operational_and_test_profiles</a><o:p></o:p></p><p>where idea of extra "padding tests" is introduced to maintain the claimed bound for specified departures from E.<o:p></o:p></p><p>There is also an alternative strategy - "fair" testing - where testing is based on the likelihood of a particular input A being faulty <br>(based on code path length rather than E), which can reduce sensitivity of reliability bounds to profile change.<o:p></o:p></p><p><a href="https://www.researchgate.net/publication/234802669_Rescaling_reliability_bounds_for_a_new_operational_profile">https://www.researchgate.net/publication/234802669_Rescaling_reliability_bounds_for_a_new_operational_profile</a><o:p></o:p></p><div><p class=MsoNormal>On 16/09/2020 01:02, <a href="mailto:hugues.bonnin@free.fr">hugues.bonnin@free.fr</a> wrote:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><pre>Hi all,<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>I have an alternative "toy" to propose: do you think that this software is reliable (written in ada-like code)?<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>begin<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>if A then <o:p></o:p></pre><pre> do_nothing <o:p></o:p></pre><pre>else<o:p></o:p></pre><pre> fail --potentially hurt and kill people<o:p></o:p></pre><pre>end if<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>end<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>The specification of the software is to do nothing; <o:p></o:p></pre><pre>NB: I'm not asking if it is the best implementation, whatever the criteria are, but just : "is it reliable?"<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>regards,<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>Hugues<o:p></o:p></pre><pre><o:p> </o:p></pre><pre><o:p> </o:p></pre><pre>----- Mail original -----<o:p></o:p></pre><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><pre>De: "Peter Bernard Ladkin" <a href="mailto:ladkin@causalis.com"><ladkin@causalis.com></a><o:p></o:p></pre><pre>À: <a href="mailto:systemsafety@lists.techfak.uni-bielefeld.de">systemsafety@lists.techfak.uni-bielefeld.de</a><o:p></o:p></pre><pre>Envoyé: Mardi 15 Septembre 2020 19:58:45<o:p></o:p></pre><pre>Objet: Re: [SystemSafety] What do we know about software reliability?<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>Bev and I and Dewi have a colleague who poses the following question.<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>"We have clients who have installed hundreds of [examples of our kit]<o:p></o:p></pre><pre>over the last ten years, and<o:p></o:p></pre><pre>have never seen any failure. They want to use it in further systems<o:p></o:p></pre><pre>that they build. What arguments<o:p></o:p></pre><pre>do we/they need to provide in order validly to justify such further<o:p></o:p></pre><pre>use?"<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>So, what is the answer to that question?<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>PBL<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>Prof. Peter Bernard Ladkin, Bielefeld, Germany<o:p></o:p></pre><pre>Styelfy Bleibgsnd<o:p></o:p></pre><pre>Tel+msg +49 (0)521 880 7319 <a href="http://www.rvs-bi.de">www.rvs-bi.de</a><o:p></o:p></pre><pre><o:p> </o:p></pre><pre><o:p> </o:p></pre><pre><o:p> </o:p></pre><pre><o:p> </o:p></pre><pre><o:p> </o:p></pre><pre><o:p> </o:p></pre><pre>_______________________________________________<o:p></o:p></pre><pre>The System Safety Mailing List<o:p></o:p></pre><pre><a href="mailto:systemsafety@TechFak.Uni-Bielefeld.DE">systemsafety@TechFak.Uni-Bielefeld.DE</a><o:p></o:p></pre><pre>Manage your subscription:<o:p></o:p></pre><pre><a href="https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety">https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety</a><o:p></o:p></pre></blockquote><pre>_______________________________________________<o:p></o:p></pre><pre>The System Safety Mailing List<o:p></o:p></pre><pre><a href="mailto:systemsafety@TechFak.Uni-Bielefeld.DE">systemsafety@TechFak.Uni-Bielefeld.DE</a><o:p></o:p></pre><pre>Manage your subscription: <a href="https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety">https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety</a><o:p></o:p></pre></blockquote><pre>-- <o:p></o:p></pre><pre><o:p> </o:p></pre><pre>Peter Bishop<o:p></o:p></pre><pre>Chief Scientist<o:p></o:p></pre><pre>Adelard LLP<o:p></o:p></pre><pre>24 Waterside, 44-48 Wharf Road, London N1 7UX<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>Email: <a href="mailto:pgb@adelard.com">pgb@adelard.com</a><o:p></o:p></pre><pre>Tel: +44-(0)20-7832 5850<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>Registered office: 5th Floor, Ashford Commercial Quarter, 1 Dover Place, Ashford, Kent TN23 1FB<o:p></o:p></pre><pre>Registered in England & Wales no. OC 304551. VAT no. 454 489808<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>This e-mail, and any attachments, is confidential and for the use of<o:p></o:p></pre><pre>the addressee only. If you are not the intended recipient, please<o:p></o:p></pre><pre>telephone 020 7832 5850. We do not accept legal responsibility for<o:p></o:p></pre><pre>this e-mail or any viruses.<o:p></o:p></pre></div></body></html>