<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Just for information, we did some code verification on an industrial real-time product<br>
and the majority of faults that mattered were concurrency bugs.<br>
<br>
Not sure how you would test for that.<br>
<br>
Mind you this was verification - "have you built the system right?" <br>
Not validation - "have you built the right system?" </div>
<ul data-editing-info="{"orderedStyleType":1,"unorderedStyleType":2}" style="margin-top: 0px; margin-bottom: 0px;">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); list-style-type: "- ";">
<div class="elementToProof"> where software requirement-based tests might give you a false sense of security</div>
</li></ul>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
Peter Bishop<br>
Adelard (now part of NCC Group)</div>
<div id="appendonsend"></div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr style="display: inline-block; width: 98%;">
<div id="divRplyFwdMsg" dir="ltr"><span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>From:</b> systemsafety <systemsafety-bounces@lists.techfak.uni-bielefeld.de> on behalf of Dewi Daniels <dewi.daniels@software-safety.com><br>
<b>Sent:</b> 08 August 2024 09:31<br>
<b>To:</b> Steve Tockey <steve.tockey@construx.com><br>
<b>Cc:</b> systemsafety@lists.techfak.uni-bielefeld.de <systemsafety@lists.techfak.uni-bielefeld.de><br>
<b>Subject:</b> Re: [SystemSafety] State of the art for "safe Linux"</span>
<div> </div>
</div>
<div style="direction: ltr;">On Wed, 7 Aug 2024 at 19:03, Steve Tockey <<a href="mailto:steve.tockey@construx.com" id="OWAfb34643a-7ead-59f0-592a-6762517923fd" class="OWAAutoLink">steve.tockey@construx.com</a>> wrote:</div>
<blockquote style="margin: 0px 0px 0px 0.8ex; padding-left: 1ex; border-left: 1px solid rgb(204, 204, 204);">
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;">Paul,</div>
<div style="direction: ltr;">I should have jumped on this from the start …</div>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;"><i>“DO-178C doesn't require you to achieve 100% structural coverage</i></div>
<div style="direction: ltr;"><i>through unit tests. It requires that your requirements-based tests</i></div>
<div style="direction: ltr;"><i>(which can be hardware/software integration tests, software</i></div>
<div style="direction: ltr;"><i>integration tests or low-level tests) cover all the software</i></div>
<div style="direction: ltr;"><i>requirements and exercise all the code.”</i></div>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;">That’s not at all how I read DO-178C. Yes, DO-178C does mandate what I would call “100% Requirements Coverage”. But in addition, Table A-7 on Structural Coverage Analysis clearly does mandate (quote):</div>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;">Row 5 "Test coverage of software structure (modified condition/decision) is achieved” for DAL A</div>
<div style="direction: ltr;">Row 6 “Test coverage of software structure (decision coverage) is achieved” for DAL A , B</div>
<div style="direction: ltr;">Row 7 “Test coverage of software structure (statement coverage) is achieved” for DAL A, B, C</div>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;">While DO-178C may not explicitly mandate that said Structural Coverage be demonstrated at the unit test level, it is often impossible in practice to achieve it in any other way.</div>
</blockquote>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;">What I wrote is correct. I agree that DO-178C requires test coverage of software structure at Levels A, B and C. However, this can be achieved through a combination of hardware/software integration tests, software integration tests
and low-level tests as shown in DO-178C figure 6-1. I also agree that achieving structural coverage will invariably require at least some unit tests. This is acknowledged in the note to DO-178C §6.4.1, which reads "In many cases, the requirements-based coverage
and structural coverage necessary can be achieved only with more precise control and monitoring of the test inputs and code execution than generally possible in a fully integrated environment. Such testing may need to be performed on a small software component
that is functionally isolated from other software components".</div>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;">I was responding to Paul's response to my question</div>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;"><span style="color: rgb(80, 0, 80);">> Even Level C isn't that hard. Why wouldn't</span><br>
<span style="color: rgb(80, 0, 80);">> you want to achieve statement coverage?</span></div>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;">in which he wrote</div>
<div style="direction: ltr;"><br>
</div>
<blockquote style="margin: 0px 0px 0px 0.8ex; padding-left: 1ex; border-left: 1px solid rgb(204, 204, 204);">
<div style="direction: ltr;">Actually, I wouldn't, because of my personal bias. I've successfully<br>
delivered small amounts of critical code (in the few thousands of LoC<br>
range) without unit tests, for systems which then worked for years<br>
without problems. Usually I've had more success with system tests than<br>
unit tests, on many projects.</div>
</blockquote>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;">Paul seemed to be assuming that statement coverage is achieved through unit testing alone. That was my point in writing "DO-178C doesn't require you to achieve 100% structural coverage through unit tests". I should have added "alone".</div>
<div style="direction: ltr;"><br>
</div>
<blockquote style="margin: 0px 0px 0px 0.8ex; padding-left: 1ex; border-left: 1px solid rgb(204, 204, 204);">
<div style="direction: ltr;">In fact, “Statement Coverage” as a Structural Coverage criterion is the absolute weakest of all Control-Flow based Structural Coverage criteria and it has logical holes in it so big that one could drive a proverbial double decker
London City bus through them.</div>
</blockquote>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;">Absolutely. My original comment was "I don't understand why it's so hard for Linux to just comply with standards such as RTCA DO-178C/EUROCAE ED-12C? DO-178C Level D is just Software Engineering 101. Even Level C isn't that hard.
Why wouldn't you want to achieve statement coverage?". I find it very hard to understand why it's so difficult to create a 'Safe Linux' that complies with DO-178C Level C, let alone DO-178C Level D. As Prof. Peter Ladkin wrote, "This business about Linux kernel
for safety-related systems has been going on for so long. Other companies have written kernel-function OSs for safety-critical systems, and have assessment certificates for them from recognised assessors, all within that time. What's wrong with trying that
route?".</div>
<div style="direction: ltr;"><br>
</div>
<blockquote style="margin: 0px 0px 0px 0.8ex; padding-left: 1ex; border-left: 1px solid rgb(204, 204, 204);">
<div style="direction: ltr;">And the vast majority of organizations I work with only aim for 60% to 70% Statement Coverage for even their most critical code. Shocking.</div>
</blockquote>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;">There's a very big gap between safety-critical software and other kinds of software. There always has been. I don't entirely understand why. </div>
<div style="direction: ltr;"><br>
</div>
<div style="direction: ltr;">Yours,</div>
<div style="direction: ltr;">Dewi </div>
<hr>
<p style="font: 10px arial, sans-serif">This email is sent for and on behalf of NCC Group. NCC Group is the trading name of NCC Services Limited (Registered in England CRN: 2802141). The ultimate holding company is NCC Group plc (Registered in England CRN:
4627044). This email may be confidential and/or legally privileged. </p>
</body>
</html>