<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">[This is a resend of my posting from yesterday, which did not appear on the list because I sent it from the wrong email address đ. It was sent in response to Derekâs posting yesterday at 13.10. Derek did receive it, however, and his response also did appear on the list at 17.12. Apologies for the confusion! Bev]<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">Begin forwarded message:</div><br class="Apple-interchange-newline"><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;" class=""><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif; color:rgba(0, 0, 0, 1.0);" class=""><b class="">From: </b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;" class="">Bev Littlewood <<a href="mailto:bevlittlewood@icloud.com" class="">bevlittlewood@icloud.com</a>><br class=""></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;" class=""><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif; color:rgba(0, 0, 0, 1.0);" class=""><b class="">Subject: </b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;" class=""><b class="">[SystemSafety] Comparing reliability predictions with reality</b><br class=""></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;" class=""><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif; color:rgba(0, 0, 0, 1.0);" class=""><b class="">Date: </b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;" class="">24 February 2025 at 16:08:49 GMT<br class=""></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;" class=""><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif; color:rgba(0, 0, 0, 1.0);" class=""><b class="">To: </b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;" class="">Derek M Jones <<a href="mailto:derek@knosof.co.uk" class="">derek@knosof.co.uk</a>>, <a href="mailto:systemsafety@lists.techfak.uni-bielefeld.de" class="">systemsafety@lists.techfak.uni-bielefeld.de</a><br class=""></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;" class=""><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif; color:rgba(0, 0, 0, 1.0);" class=""><b class="">Cc: </b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;" class="">Bev Littlewood <<a href="mailto:bevlittlewood@icloud.com" class="">bevlittlewood@icloud.com</a>><br class=""></span></div><br class=""><div class=""><meta http-equiv="Content-Type" content="text/html; charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class="">Hi Derek</div><br class=""><blockquote type="cite" class=""><div dir="ltr" class=""><br class="">Having spent some time reading lots of papers on<br class="">models of software reliability, based on<br class="">cpu time between fault experiences, I have not found<br class="">one that measures the accuracy of the models. By<br class="">accuracy I mean comparison of the prediction of time<br class="">to next fault against actual time to next fault.<br class=""><br class="">There is something of a cottage industry of papers<br class="">that compare probability distributions fitted to historical<br class="">data. Researchers seem shy about taking the next step of<br class="">comparing predictions of future faults. Perhaps because<br class="">the results are not very good?</div></blockquote><div class=""><br class=""></div>Perhaps you have not been looking in the right places, Derek! Itâs about forty years since I published ways of doing these kinds of comparisons (i.e. of predictions against (later seen) observations). This work was reported in, e.g., IEEE Trans Software Engineering (hardly an obscure journal, I think you will agree), and various conferences, etc.<div class=""><br class=""></div><div class="">The easiest read for this stuff is in Michael Lyuâs old book (old, but still one of the best introductory accounts of this kind of material I believe):</div><div class=""><br class=""></div><div class=""><span class="docData;DOCY;v5;1794;BQiAAgAAEYQCAAAGiAIAAAO5BAAABccEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAALQIAAAAoAgAAASQAAAAVBgYAAAA5ADcAMgABBgAAAAAJBgAAAAAaBgAAAAAbBgAAAAAC+gEAAAXFAAAAAQAAAAAIuwAAAAC2AAAAGCBUAGUAYwBoAG4AaQBxAHUAZQBzACAAZgBvAHIAIABwAHIAZQBkAGkAYwB0AGkAbwBuACAAYQBuAGEAbAB5AHMAaQBzACAAYQBuAGQAIAByAGUAYwBhAGwAaQBiAHIAYQB0AGkAbwBuABkgIAAoAHcAaQB0AGgAIABTACAAQgByAG8AYwBrAGwAZQBoAHUAcgBzAHQAKQAsACAAQwBoAGEAcAB0AGUAcgAgADQAIABvAGYAIAAFcgAAAAEDAAAAAQEBCGUAAAAAYAAAAFQAaABlACAASABhAG4AZABiAG8AbwBrACAAbwBmACAAUwBvAGYAdAB3AGEAcgBlACAAUgBlAGwAaQBhAGIAaQBsAGkAdAB5ACAARQBuAGcAaQBuAGUAZQByAGkAbgBnAAWHAAAAAQAAAAAIfQAAAAB4AAAAIAAoAEUAZAAuACAATQBpAGMAaABhAGUAbAAgAEwAeQB1ACkALAAgAE0AYwBHAHIAYQB3AC0ASABpAGwAbAAsACAATgBlAHcAIABZAG8AcgBrACwAIAAxADkAOQA1ACwAIABwAHAAIAAxADEAOQAtADEANgA2AC4ABQoAAAABAAAAAAgAAAAABQoAAAABAAAAAAgAAAAABQoAAAAIBQAAAA0AAAAACgAAAAAAAAAAEQAAAAA3AgAAABIAAAABBgAAAAAJBgAAAAAbBgAAAAABAAAAAAIWAgAAABECAAABBgAAADkANwAyAAIMAAAATgBvAHIAbQBhAGwACQEAAAADAwYAAAA5ADQAOQAEBgAAADkANwAyABIGAAAAOQA0ADkABeUAAAAAAQABAQACAQsDAQAEBgoAAABUAGkAbQBlAHMABQYKAAAAVABpAG0AZQBzAAcGHgAAAFQAaQBtAGUAcwAgAE4AZQB3ACAAUgBvAG0AYQBuAAYGHgAAAFQAaQBtAGUAcwAgAE4AZQB3ACAAUgBvAG0AYQBuAAgEGAAAAAoBAAwBACYEAAAAAA8BABABABEBACcEAAAAABQBABUBABYEFAAAABcBABgBABkGCgAAAGUAbgAtAEcAQgAaBgoAAABhAHIALQBTAEEAGwYKAAAAZQBuAC0AVQBTABwGAgAAAAAAHgEALgEABt8AAAAAAQABBhIAAAAkBAAAAAAlBAAAAAAmBAAAAAAFAQMGAQAHAQAIAQAJBhsAAAAnBPAAAAALAQEcAQAoBAAAAAAdAQApBAAAAAAOBggAAAAAAQEBA////xkBARsGfQAAAAAUAAAAAAMAAAAFBAAAAAAGBAQAAAADAQABFAAAAAADAAAABQQAAAAABgQEAAAAAwEAAhQAAAAAAwAAAAUEAAAAAAYEBAAAAAMBAAMUAAAAAAMAAAAFBAAAAAAGBAQAAAADAQALFAAAAAADAAAABQQAAAAABgQEAAAAAwEALAEA">âTechniques for prediction analysis and recalibrationâ (with S Brocklehurst), Chapter 4 of </span><i class="">The Handbook of Software Reliability Engineering</i> (Ed. Michael Lyu), McGraw-Hill, New York, 1995, pp 119-166. (I think it is still available for free download)</div><div class=""><br class=""></div><div class="">More techie accounts are the original reports:</div><div class=""><br class=""></div><div class=""><span class="docData;DOCY;v5;1772;BQiAAgAAEYQCAAAGiAIAAAOjBAAABbEEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFwIAAAASAgAAASQAAAAVBgYAAAA5ADcAMgABBgAAAAAJBgAAAAAaBgAAAAAbBgAAAAAC5AEAAAX1AAAAAQAAAAAI6wAAAADmAAAAGCBFAHYAYQBsAHUAYQB0AGkAbwBuACAAbwBmACAAYwBvAG0AcABlAHQAaQBuAGcAIABzAG8AZgB0AHcAYQByAGUAIAByAGUAbABpAGEAYgBpAGwAaQB0AHkAIABwAHIAZQBkAGkAYwB0AGkAbwBuAHMAGSAgACgAdwBpAHQAaAAgAEEAIABBACAAQQBiAGQAZQBsAC0ARwBoAGEAbAB5ACwAIABQACAAWQAgAEMAaABhAG4ALAAgAGIAbwB0AGgAIABDAGkAdAB5ACAAVQBuAGkAdgBlAHIAcwBpAHQAeQApACwAIAAFVAAAAAEDAAAAAQEBCEcAAAAAQgAAAEkARQBFAEUAIABUAHIAYQBuAHMALgAgAFMAbwBmAHQAdwBhAHIAZQAgAEUAbgBnAGkAbgBlAGUAcgBpAG4AZwAsAAVfAAAAAQAAAAAIVQAAAABQAAAAIABTAEUALQAxADIALAAgADkALAAgAFMAZQBwAHQAZQBtAGIAZQByACwAIAAxADkAOAA2ACwAIABwAHAALgAgADkANQAwAC0AOQA2ADcALgAFCgAAAAEAAAAACAAAAAAFCgAAAAEAAAAACAAAAAAFCgAAAAgFAAAADQAAAAAKAAAAAAAAAAARAAAAADcCAAAAEgAAAAEGAAAAAAkGAAAAABsGAAAAAAEAAAAAAhYCAAAAEQIAAAEGAAAAOQA3ADIAAgwAAABOAG8AcgBtAGEAbAAJAQAAAAMDBgAAADkANAA5AAQGAAAAOQA3ADIAEgYAAAA5ADQAOQAF5QAAAAABAAEBAAIBCwMBAAQGCgAAAFQAaQBtAGUAcwAFBgoAAABUAGkAbQBlAHMABwYeAAAAVABpAG0AZQBzACAATgBlAHcAIABSAG8AbQBhAG4ABgYeAAAAVABpAG0AZQBzACAATgBlAHcAIABSAG8AbQBhAG4ACAQYAAAACgEADAEAJgQAAAAADwEAEAEAEQEAJwQAAAAAFAEAFQEAFgQUAAAAFwEAGAEAGQYKAAAAZQBuAC0ARwBCABoGCgAAAGEAcgAtAFMAQQAbBgoAAABlAG4ALQBVAFMAHAYCAAAAAAAeAQAuAQAG3wAAAAABAAEGEgAAACQEAAAAACUEAAAAACYEAAAAAAUBAwYBAAcBAAgBAAkGGwAAACcE8AAAAAsBARwBACgEAAAAAB0BACkEAAAAAA4GCAAAAAABAQED////GQEBGwZ9AAAAABQAAAAAAwAAAAUEAAAAAAYEBAAAAAMBAAEUAAAAAAMAAAAFBAAAAAAGBAQAAAADAQACFAAAAAADAAAABQQAAAAABgQEAAAAAwEAAxQAAAAAAwAAAAUEAAAAAAYEBAAAAAMBAAsUAAAAAAMAAAAFBAAAAAAGBAQAAAADAQAsAQA=">âEvaluation of competing software reliability predictionsâ (with A A Abdel-Ghaly, P Y Chan), </span><i class="">IEEE Trans. Software Engineering,</i> SE-12, 9, September, 1986, pp. 950-967.</div><div class=""><br class=""></div><div class=""><span class="docData;DOCY;v5;1710;BQiAAgAAEYQCAAAGiAIAAANlBAAABXMEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA2QEAAADUAQAAASQAAAAVBgYAAAA5ADcAMgABBgAAAAAJBgAAAAAaBgAAAAAbBgAAAAACpgEAAAW9AAAAAQAAAAAIswAAAACuAAAAGCBSAGUAYwBhAGwAaQBiAHIAYQB0AGkAbgBnACAAcwBvAGYAdAB3AGEAcgBlACAAcgBlAGwAaQBhAGIAaQBsAGkAdAB5ACAAbQBvAGQAZQBsAHMAGSAsACAAKAB3AGkAdABoACAAUwAgAEIAcgBvAGMAawBsAGUAaAB1AHIAcwB0ACwAIABQACAAWQAgAEMAaABhAG4ALAAgAEoAIABTAG4AZQBsAGwAKQAsACAABVAAAAABAwAAAAEBAQhDAAAAAD4AAABJAEUARQBFACAAVAByAGEAbgBzACAAUwBvAGYAdAB3AGEAcgBlACAARQBuAGcAaQBuAGUAZQByAGkAbgBnAAVdAAAAAQAAAAAIUwAAAABOAAAALAAgAFYAbwBsACAAMQA2ACwAIABOAG8AIAA0ACwAIABwAHAAIAA0ADUAOAAtADQANwAwACwAIABBAHAAcgBpAGwAIAAxADkAOQAwAC4ABQoAAAABAAAAAAgAAAAABQoAAAABAAAAAAgAAAAABQoAAAAIBQAAAA0AAAAACgAAAAAAAAAAEQAAAAA3AgAAABIAAAABBgAAAAAJBgAAAAAbBgAAAAABAAAAAAIWAgAAABECAAABBgAAADkANwAyAAIMAAAATgBvAHIAbQBhAGwACQEAAAADAwYAAAA5ADQAOQAEBgAAADkANwAyABIGAAAAOQA0ADkABeUAAAAAAQABAQACAQsDAQAEBgoAAABUAGkAbQBlAHMABQYKAAAAVABpAG0AZQBzAAcGHgAAAFQAaQBtAGUAcwAgAE4AZQB3ACAAUgBvAG0AYQBuAAYGHgAAAFQAaQBtAGUAcwAgAE4AZQB3ACAAUgBvAG0AYQBuAAgEGAAAAAoBAAwBACYEAAAAAA8BABABABEBACcEAAAAABQBABUBABYEFAAAABcBABgBABkGCgAAAGUAbgAtAEcAQgAaBgoAAABhAHIALQBTAEEAGwYKAAAAZQBuAC0AVQBTABwGAgAAAAAAHgEALgEABt8AAAAAAQABBhIAAAAkBAAAAAAlBAAAAAAmBAAAAAAFAQMGAQAHAQAIAQAJBhsAAAAnBPAAAAALAQEcAQAoBAAAAAAdAQApBAAAAAAOBggAAAAAAQEBA////xkBARsGfQAAAAAUAAAAAAMAAAAFBAAAAAAGBAQAAAADAQABFAAAAAADAAAABQQAAAAABgQEAAAAAwEAAhQAAAAAAwAAAAUEAAAAAAYEBAAAAAMBAAMUAAAAAAMAAAAFBAAAAAAGBAQAAAADAQALFAAAAAADAAAABQQAAAAABgQEAAAAAwEALAEA">âRecalibrating software reliability modelsâ, (with S Brocklehurst, P Y Chan, J Snell), </span><i class="">IEEE Trans Software Engineering</i>, Vol 16, No 4, pp 458-470, April 1990.</div><div class=""><br class=""></div><div class="">Very briefly, there are two main tools: u-plots, and prequential likelihood comparisons. The first of them is a kind of âabsoluteâ assessment of a single model's accuracy, based on a sequence of (prediction, outcome)-pair comparisons. It turns out that you can use this - somewhat surprisingly - to improve a modelâs predictions, essentially by allowing it to learn from its past âerrorsâ. The second is a means of comparing competing models to select the best (for a particular data set - i.e. sequence of inter-failure times). </div><div class=""><br class=""></div><div class="">So solutions to the problem you pose have been around for years. I think my recalibration idea based on one of them is particularly neat - even if I say so myselfâŚ.đ</div><div class=""><br class=""></div><div class="">I have not worked on reliability growth models for many years. It became a cottage industry producing new models with small tweaks. In fact you will still find papers on it in, e.g. IEEE Trans Reliability. But it long ago seemed to me that only very minor benefits in model accuracy were being obtained. And the techniques I have described above could often be used to check accuracy by applying several of the existing models.</div><div class=""><br class=""></div><div class="">I agree with you, though, that there continues to be a dearth of published data.</div><div class=""><br class=""></div><div class="">Cheers</div><div class=""><br class=""></div><div class="">Bev<br class=""><div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><span class="Apple-style-span" style="border-collapse: separate; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: normal; border-spacing: 0px; -webkit-text-decorations-in-effect: none;"><div class=""><br class="Apple-interchange-newline">_______________________________________________</div><div class=""><br class=""></div><div class="">Bev Littlewood</div><div class="">Emeritus Professor of Software Engineering</div><div class="">City St George's, University of London </div><div class=""><br class=""></div><div class="">_______________________________________________</div><div class=""> </div></span></div></div></div></div>
</div>
<br class=""></div></div></div></blockquote></div><br class=""><div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-ligatures: normal; font-variant-position: normal; font-variant-caps: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-stroke-width: 0px;"><div class=""><br class="Apple-interchange-newline">_______________________________________________</div><div class=""><br class=""></div><div class="">Bev Littlewood</div><div class="">Emeritus Professor of Software Engineering</div><div class="">City St George's, University of London </div><div class=""><br class=""></div><div class="">_______________________________________________</div><div class=""> </div></span></div></div></div></div>
</div>
<br class=""></body></html>