[SystemSafety] State of the art for "safe Linux"

[Steve Tockey]

Dewi,
Thank you very much for the clarification. I appreciate it. I agree 100% with you.


Best,

— steve



On Aug 8, 2024, at 1:31 AM, Dewi Daniels <dewi.daniels at software-safety.com> wrote:

On Wed, 7 Aug 2024 at 19:03, Steve Tockey <steve.tockey at construx.com<mailto:steve.tockey at construx.com>> wrote:

Paul,
I should have jumped on this from the start …

“DO-178C doesn't require you to achieve 100% structural coverage
through unit tests. It requires that your requirements-based tests
(which can be hardware/software integration tests, software
integration tests or low-level tests) cover all the software
requirements and exercise all the code.”

That’s not at all how I read DO-178C. Yes, DO-178C does mandate what I would call “100% Requirements Coverage”. But in addition, Table A-7 on Structural Coverage Analysis clearly does mandate (quote):

Row 5 "Test coverage of software structure (modified condition/decision) is achieved” for DAL A
Row 6 “Test coverage of software structure (decision coverage) is achieved” for DAL A , B
Row 7 “Test coverage of software structure (statement coverage) is achieved” for DAL A, B, C

While DO-178C may not explicitly mandate that said Structural Coverage be demonstrated at the unit test level, it is often impossible in practice to achieve it in any other way.

What I wrote is correct. I agree that DO-178C requires test coverage of software structure at Levels A, B and C. However, this can be achieved through a combination of hardware/software integration tests, software integration tests and low-level tests as shown in DO-178C figure 6-1. I also agree that achieving structural coverage will invariably require at least some unit tests. This is acknowledged in the note to DO-178C §6.4.1, which reads "In many cases, the requirements-based coverage and structural coverage necessary can be achieved only with more precise control and monitoring of the test inputs and code execution than generally possible in a fully integrated environment. Such testing may need to be performed on a small software component that is functionally isolated from other software components".

I was responding to Paul's response to my question

> Even Level C isn't that hard. Why wouldn't
> you want to achieve statement coverage?

in which he wrote

Actually, I wouldn't, because of my personal bias. I've successfully
delivered small amounts of critical code (in the few thousands of LoC
range) without unit tests, for systems which then worked for years
without problems. Usually I've had more success with system tests than
unit tests, on many projects.

Paul seemed to be assuming that statement coverage is achieved through unit testing alone. That was my point in writing "DO-178C doesn't require you to achieve 100% structural coverage through unit tests". I should have added "alone".

In fact, “Statement Coverage” as a Structural Coverage criterion is the absolute weakest of all Control-Flow based Structural Coverage criteria and it has logical holes in it so big that one could drive a proverbial double decker London City bus through them.

Absolutely.  My original comment was "I don't understand why it's so hard for Linux to just comply with standards such as RTCA DO-178C/EUROCAE ED-12C? DO-178C Level D is just Software Engineering 101. Even Level C isn't that hard. Why wouldn't you want to achieve statement coverage?". I find it very hard to understand why it's so difficult to create a 'Safe Linux' that complies with DO-178C Level C, let alone DO-178C Level D. As Prof. Peter Ladkin wrote, "This business about Linux kernel for safety-related systems has been going on for so long. Other companies have written kernel-function OSs for safety-critical systems, and have assessment certificates for them from recognised assessors, all within that time. What's wrong with trying that route?".

And the vast majority of organizations I work with only aim for 60% to 70% Statement Coverage for even their most critical code. Shocking.

There's a very big gap between safety-critical software and other kinds of software. There always has been. I don't entirely understand why.

Yours,
Dewi

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/pipermail/systemsafety/attachments/20240808/52ea1cad/attachment.html>