<div dir="ltr">Peter wrote:<div><br></div><div>"And statistical testing is used in the UK nuclear industry fore safety critical systems,"</div><div><br></div><div>This is true, but note that the statement is safety critical 'systems'; not 'software'. While the UK standards do talk about 'on demand' and some numbers, it is actually focussed at the system level, not the software. There is (at last) a distinct but still tacit acceptance that it is a nonsense to accept software has a 'reliability' and hence to be treated in the same way that hardware fails. It is much better to give a reason why the software should be trusted rather than rely (!) on some outdated research for which the foundations are highly questionable. This is now gaining acceptance rather than a statistical approach. </div><div><br></div><div>As I recall, I have said before on this list, software has no wear out mechanism so software reliability is somewhat meaningless. I was widely abused (some even said bullied) for suggesting that software reliability was not the right way of thinking about software assurance. It is therefore with some trepidation that I dive into this thread. </div><div><br></div><div><br></div><div><br></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><span></span><span></span>Nick Tudor<div>Tudor Associates Ltd</div><div>Mobile: +44(0)7412 074654</div><div><a href="http://www.tudorassoc.com" target="_blank">www.tudorassoc.com</a></div><div><img src="http://www.tudorassoc.com/wpimages/wpb4e71a5c_0f.jpg" width="200" height="40"></div><div><font color="#00144d" face="Arial, Helvetica, sans-serif" size="1"><b><br></b></font></div><div><span style="color:rgb(0,20,77);font-family:Arial,Helvetica,sans-serif"><font size="1"><b><span></span><span></span>77 Barnards Green Road</b></font></span></div><div><span style="color:rgb(0,20,77);font-family:Arial,Helvetica,sans-serif"><font size="1"><b>Malvern</b></font></span></div><div><span style="color:rgb(0,20,77);font-family:Arial,Helvetica,sans-serif"><font size="1"><b>Worcestershire</b></font></span></div><div><span style="color:rgb(0,20,77);font-family:Arial,Helvetica,sans-serif"><font size="1"><b>WR14 3LR</b><strong><br>Company No. 07642673</strong></font></span></div><div><span style="color:rgb(0,20,77);font-family:Arial,Helvetica,sans-serif"><font size="1"><strong>VAT No:116495996</strong></font></span></div><div><span style="color:rgb(0,20,77);font-family:Arial,Helvetica,sans-serif"><font size="1"><strong><br></strong></font></span></div><div><strong style="color:rgb(0,20,77);font-family:Arial,Helvetica,sans-serif;font-size:x-small"><a href="http://www.aeronautique-associates.com" target="_blank">www.aeronautique-associates.com</a></strong>
</div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 15 Sep 2020 at 09:46, Peter Bishop <<a href="mailto:pgb@adelard.com">pgb@adelard.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>On 14/09/2020 15:04, Martyn Thomas
wrote:<br>
</div>
<blockquote type="cite">
<p>Why are you completely dismissing software reliablity? <br>
</p>
<p>Is it not the case that if you can tolerate a failure rate of
once in 1000 hours, 99% confidence through testing would take
about 200 days to demonstrate (so long as the test environment
is "sufficiently" like the future operating environment and you
are able to detaect every failure correctly)? <br>
</p>
</blockquote>
<p>And statistical testing is used in the UK nuclear industry fore
safety critical systems, so it is not just abstract theory,</p>
<p>Re your characterisation of confidence based statistical testing
on P153 (with no reference), I do not think it is fair to dismiss
this because "p can vary by orders of magnitude". Testing presumes
a fixed operational profile and a constant probability of failure.
<br>
</p>
<p>There has also been some work on the impact of profile change on
the bound that can be claimed.</p>
<p><a href="https://www.researchgate.net/publication/307555914_Deriving_a_frequentist_conservative_confidence_bound_for_probability_of_failure_per_demand_for_systems_with_different_operational_and_test_profiles" target="_blank">https://www.researchgate.net/publication/307555914_Deriving_a_frequentist_conservative_confidence_bound_for_probability_of_failure_per_demand_for_systems_with_different_operational_and_test_profiles</a><br>
</p>
<p>BTW, re, your summary of my paper on the same page, I think you
missed the main point. This is a<b> predictive</b> theory to
derive a worst case bound for some time in the future, i.e. <br>
</p>
<p>Given N faults what is the worst possible reliability at some
future time T?<br>
- it assumes fault fixing will occur during that time.<br>
</p>
<p>You also only presented the theory of N=1, and you seem to assume
the T has already happened with zero failures (not a requirement
for this model)<br>
</p>
<p>Might have been better to reference the original worst case bound
version (which makes it clear that it is a long term forward
prediction)<br>
</p>
<p><a href="https://www.researchgate.net/publication/3152200_A_conservative_theory_for_long-term_reliability-growth_prediction" target="_blank">https://www.researchgate.net/publication/3152200_A_conservative_theory_for_long-term_reliability-growth_prediction</a><br>
</p>
<blockquote type="cite">
<p> </p>
<p>Of course, the testing would have to be repeated following a
change to the software, unless you have enough formality to show
that the change cannot affect reliability.</p>
<p>In specific circumstances, you can do better than this. Bev
Littlewood's published papers provide strong evidence and a rich
bibliography. Bev's paper on "How reliable is a program that has
never failed?" offers a useful rule-of-thumb: that aften n hours
of fault free operation, there is about 50% chance of a failure
in the following n hours (subject to some obvious constraints).<br>
</p>
<p>The difficulties rapidly escalate when you need 10^-4 or better
at >90% confidence. <br>
</p>
<p>Martyn<br>
</p>
<div>On 14/09/2020 14:14, SPRIGGS, John J
wrote:<br>
</div>
<blockquote type="cite">
<div>
<p class="MsoNormal" style="margin-top:6pt"><span>In
my experience, if Software Reliability is mentioned at a
conference, at least one member of the audience will
laugh, and if it is mentioned in a work discussion, at
least one member of the group will get angry.<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-top:6pt"><span>Interestingly,
some of the same people who say it is impossible to
quantify software failure rates will set numerical
requirements for Software Availability – if you get one of
those, ask the Customer how (s)he wants you to demonstrate
satisfaction of the requirement.<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-top:6pt"><span><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-top:6pt"><span>John<u></u><u></u></span></p>
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> systemsafety <a href="mailto:systemsafety-bounces@lists.techfak.uni-bielefeld.de" target="_blank"><systemsafety-bounces@lists.techfak.uni-bielefeld.de></a>
<b>On Behalf Of </b>Derek M Jones<br>
<b>Sent:</b> 14 September 2020 12:54<br>
<b>To:</b> <a href="mailto:systemsafety@lists.techfak.uni-bielefeld.de" target="_blank">systemsafety@lists.techfak.uni-bielefeld.de</a><br>
<b>Subject:</b> [SystemSafety] What do we know about
software reliability?<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">All,<br>
<br>
What do we know about software reliability?<br>
<br>
The answer appears to be, not a lot:<br>
<a href="http://shape-of-code.coding-guidelines.com/2020/09/13/learning-useful-stuff-from-the-reliability-chapter-of-my-book" target="_blank">http://shape-of-code.coding-guidelines.com/2020/09/13/learning-useful-stuff-from-the-reliability-chapter-of-my-book/</a><br>
<br>
-- <br>
Derek M. Jones Evidence-based software engineering<br>
tel: +44 (0)1252 520667
blog:<a href="http://shape-of-code.coding-guidelines.com" target="_blank">shape-of-code.coding-guidelines.com</a><br>
_______________________________________________<br>
The System Safety Mailing List<br>
<a href="mailto:systemsafety@TechFak.Uni-Bielefeld.DE" target="_blank">systemsafety@TechFak.Uni-Bielefeld.DE</a><br>
Manage your subscription: <a href="https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety" target="_blank">
https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety</a><u></u><u></u></p>
</div>
<br>
<br>
<hr width="100%"> <font style="font-size:9pt" face="Verdana">If
you are not the intended recipient, please notify our Help
Desk at Email <a href="mailto:Information.Solutions@nats.co.uk" target="_blank">Information.Solutions@nats.co.uk</a>
immediately. You should not copy or use this email or
attachment(s) for any purpose nor disclose their contents to
any other person. <br>
<br>
NATS computer systems may be monitored and communications
carried on them recorded, to secure the effective operation of
the system. <br>
<br>
Please note that neither NATS nor the sender accepts any
responsibility for viruses or any losses caused as a result of
viruses and it is your responsibility to scan or otherwise
check this email and any attachments. <br>
<br>
NATS means NATS (En Route) plc (company number: 4129273), NATS
(Services) Ltd (company number 4129270), NATSNAV Ltd (company
number: 4164590) or NATS Ltd (company number 3155567) or NATS
Holdings Ltd (company number 4138218). All companies are
registered in England and their registered office is at 4000
Parkway, Whiteley, Fareham, Hampshire, PO15 7FL.</font>
<hr> <br>
<fieldset></fieldset>
<pre>_______________________________________________
The System Safety Mailing List
<a href="mailto:systemsafety@TechFak.Uni-Bielefeld.DE" target="_blank">systemsafety@TechFak.Uni-Bielefeld.DE</a>
Manage your subscription: <a href="https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety" target="_blank">https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety</a></pre>
</blockquote>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
The System Safety Mailing List
<a href="mailto:systemsafety@TechFak.Uni-Bielefeld.DE" target="_blank">systemsafety@TechFak.Uni-Bielefeld.DE</a>
Manage your subscription: <a href="https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety" target="_blank">https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety</a></pre>
</blockquote>
<pre cols="72">--
Peter Bishop
Chief Scientist
Adelard LLP
24 Waterside, 44-48 Wharf Road, London N1 7UX
Email: <a href="mailto:pgb@adelard.com" target="_blank">pgb@adelard.com</a>
Tel: +44-(0)20-7832 5850
Registered office: 5th Floor, Ashford Commercial Quarter, 1 Dover Place, Ashford, Kent TN23 1FB
Registered in England & Wales no. OC 304551. VAT no. 454 489808
This e-mail, and any attachments, is confidential and for the use of
the addressee only. If you are not the intended recipient, please
telephone 020 7832 5850. We do not accept legal responsibility for
this e-mail or any viruses.</pre>
</div>
_______________________________________________<br>
The System Safety Mailing List<br>
<a href="mailto:systemsafety@TechFak.Uni-Bielefeld.DE" target="_blank">systemsafety@TechFak.Uni-Bielefeld.DE</a><br>
Manage your subscription: <a href="https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety" rel="noreferrer" target="_blank">https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety</a></blockquote></div>