<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">On 14/09/2020 15:04, Martyn Thomas
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:226737c4-708c-8a79-d0b5-394fb6516781@72f.org">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p>Why are you completely dismissing software reliablity? <br>
</p>
<p>Is it not the case that if you can tolerate a failure rate of
once in 1000 hours, 99% confidence through testing would take
about 200 days to demonstrate (so long as the test environment
is "sufficiently" like the future operating environment and you
are able to detaect every failure correctly)? <br>
</p>
</blockquote>
<p>And statistical testing is used in the UK nuclear industry fore
safety critical systems, so it is not just abstract theory,</p>
<p>Re your characterisation of confidence based statistical testing
on P153 (with no reference), I do not think it is fair to dismiss
this because "p can vary by orders of magnitude". Testing presumes
a fixed operational profile and a constant probability of failure.
<br>
</p>
<p>There has also been some work on the impact of profile change on
the bound that can be claimed.</p>
<p><a class="moz-txt-link-freetext" href="https://www.researchgate.net/publication/307555914_Deriving_a_frequentist_conservative_confidence_bound_for_probability_of_failure_per_demand_for_systems_with_different_operational_and_test_profiles">https://www.researchgate.net/publication/307555914_Deriving_a_frequentist_conservative_confidence_bound_for_probability_of_failure_per_demand_for_systems_with_different_operational_and_test_profiles</a><br>
</p>
<p>BTW, re, your summary of my paper on the same page, I think you
missed the main point. This is a<b> predictive</b> theory to
derive a worst case bound for some time in the future, i.e. <br>
</p>
<p>Given N faults what is the worst possible reliability at some
future time T?<br>
- it assumes fault fixing will occur during that time.<br>
</p>
<p>You also only presented the theory of N=1, and you seem to assume
the T has already happened with zero failures (not a requirement
for this model)<br>
</p>
<p>Might have been better to reference the original worst case bound
version (which makes it clear that it is a long term forward
prediction)<br>
</p>
<p><a class="moz-txt-link-freetext" href="https://www.researchgate.net/publication/3152200_A_conservative_theory_for_long-term_reliability-growth_prediction">https://www.researchgate.net/publication/3152200_A_conservative_theory_for_long-term_reliability-growth_prediction</a><br>
</p>
<blockquote type="cite"
cite="mid:226737c4-708c-8a79-d0b5-394fb6516781@72f.org">
<p> </p>
<p>Of course, the testing would have to be repeated following a
change to the software, unless you have enough formality to show
that the change cannot affect reliability.</p>
<p>In specific circumstances, you can do better than this. Bev
Littlewood's published papers provide strong evidence and a rich
bibliography. Bev's paper on "How reliable is a program that has
never failed?" offers a useful rule-of-thumb: that aften n hours
of fault free operation, there is about 50% chance of a failure
in the following n hours (subject to some obvious constraints).<br>
</p>
<p>The difficulties rapidly escalate when you need 10^-4 or better
at >90% confidence. <br>
</p>
<p>Martyn<br>
</p>
<div class="moz-cite-prefix">On 14/09/2020 14:14, SPRIGGS, John J
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:LOYP265MB1888063E4BE5C54850A48119A6230@LOYP265MB1888.GBRP265.PROD.OUTLOOK.COM">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Roboto Light";
panose-1:2 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle20
{mso-style-type:personal-compose;
font-family:"Roboto Light";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal" style="margin-top:6.0pt"><span
style="font-family:"Roboto
Light";color:#365F91;mso-fareast-language:EN-US">In
my experience, if Software Reliability is mentioned at a
conference, at least one member of the audience will
laugh, and if it is mentioned in a work discussion, at
least one member of the group will get angry.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-top:6.0pt"><span
style="font-family:"Roboto
Light";color:#365F91;mso-fareast-language:EN-US">Interestingly,
some of the same people who say it is impossible to
quantify software failure rates will set numerical
requirements for Software Availability – if you get one of
those, ask the Customer how (s)he wants you to demonstrate
satisfaction of the requirement.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-top:6.0pt"><span
style="font-family:"Roboto
Light";color:#365F91;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-top:6.0pt"><span
style="font-family:"Roboto
Light";color:#365F91;mso-fareast-language:EN-US">John<o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span
lang="EN-US"> systemsafety <a
class="moz-txt-link-rfc2396E"
href="mailto:systemsafety-bounces@lists.techfak.uni-bielefeld.de"
moz-do-not-send="true"><systemsafety-bounces@lists.techfak.uni-bielefeld.de></a>
<b>On Behalf Of </b>Derek M Jones<br>
<b>Sent:</b> 14 September 2020 12:54<br>
<b>To:</b> <a class="moz-txt-link-abbreviated"
href="mailto:systemsafety@lists.techfak.uni-bielefeld.de"
moz-do-not-send="true">systemsafety@lists.techfak.uni-bielefeld.de</a><br>
<b>Subject:</b> [SystemSafety] What do we know about
software reliability?<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">All,<br>
<br>
What do we know about software reliability?<br>
<br>
The answer appears to be, not a lot:<br>
<a
href="http://shape-of-code.coding-guidelines.com/2020/09/13/learning-useful-stuff-from-the-reliability-chapter-of-my-book"
moz-do-not-send="true">http://shape-of-code.coding-guidelines.com/2020/09/13/learning-useful-stuff-from-the-reliability-chapter-of-my-book/</a><br>
<br>
-- <br>
Derek M. Jones Evidence-based software engineering<br>
tel: +44 (0)1252 520667
blog:shape-of-code.coding-guidelines.com<br>
_______________________________________________<br>
The System Safety Mailing List<br>
<a href="mailto:systemsafety@TechFak.Uni-Bielefeld.DE"
moz-do-not-send="true">systemsafety@TechFak.Uni-Bielefeld.DE</a><br>
Manage your subscription: <a
href="https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety"
moz-do-not-send="true">
https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety</a><o:p></o:p></p>
</div>
<br>
<br>
<hr width="100%"> <font style="font-size: 9pt" face="Verdana">If
you are not the intended recipient, please notify our Help
Desk at Email <a class="moz-txt-link-abbreviated"
href="mailto:Information.Solutions@nats.co.uk"
moz-do-not-send="true">Information.Solutions@nats.co.uk</a>
immediately. You should not copy or use this email or
attachment(s) for any purpose nor disclose their contents to
any other person. <br>
<br>
NATS computer systems may be monitored and communications
carried on them recorded, to secure the effective operation of
the system. <br>
<br>
Please note that neither NATS nor the sender accepts any
responsibility for viruses or any losses caused as a result of
viruses and it is your responsibility to scan or otherwise
check this email and any attachments. <br>
<br>
NATS means NATS (En Route) plc (company number: 4129273), NATS
(Services) Ltd (company number 4129270), NATSNAV Ltd (company
number: 4164590) or NATS Ltd (company number 3155567) or NATS
Holdings Ltd (company number 4138218). All companies are
registered in England and their registered office is at 4000
Parkway, Whiteley, Fareham, Hampshire, PO15 7FL.</font>
<hr> <br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
The System Safety Mailing List
<a class="moz-txt-link-abbreviated" href="mailto:systemsafety@TechFak.Uni-Bielefeld.DE" moz-do-not-send="true">systemsafety@TechFak.Uni-Bielefeld.DE</a>
Manage your subscription: <a class="moz-txt-link-freetext" href="https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety" moz-do-not-send="true">https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety</a></pre>
</blockquote>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
The System Safety Mailing List
<a class="moz-txt-link-abbreviated" href="mailto:systemsafety@TechFak.Uni-Bielefeld.DE">systemsafety@TechFak.Uni-Bielefeld.DE</a>
Manage your subscription: <a class="moz-txt-link-freetext" href="https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety">https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety</a></pre>
</blockquote>
<pre class="moz-signature" cols="72">--
Peter Bishop
Chief Scientist
Adelard LLP
24 Waterside, 44-48 Wharf Road, London N1 7UX
Email: <a class="moz-txt-link-abbreviated" href="mailto:pgb@adelard.com">pgb@adelard.com</a>
Tel: +44-(0)20-7832 5850
Registered office: 5th Floor, Ashford Commercial Quarter, 1 Dover Place, Ashford, Kent TN23 1FB
Registered in England & Wales no. OC 304551. VAT no. 454 489808
This e-mail, and any attachments, is confidential and for the use of
the addressee only. If you are not the intended recipient, please
telephone 020 7832 5850. We do not accept legal responsibility for
this e-mail or any viruses.</pre>
</body>
</html>