[SystemSafety] State of the art for "safe Linux"

[Steve Tockey]

Paul,
I should have jumped on this from the start …

“DO-178C doesn't require you to achieve 100% structural coverage
through unit tests. It requires that your requirements-based tests
(which can be hardware/software integration tests, software
integration tests or low-level tests) cover all the software
requirements and exercise all the code.”

That’s not at all how I read DO-178C. Yes, DO-178C does mandate what I would call “100% Requirements Coverage”. But in addition, Table A-7 on Structural Coverage Analysis clearly does mandate (quote):

Row 5 "Test coverage of software structure (modified condition/decision) is achieved” for DAL A
Row 6 “Test coverage of software structure (decision coverage) is achieved” for DAL A , B
Row 7 “Test coverage of software structure (statement coverage) is achieved” for DAL A, B, C

While DO-178C may not explicitly mandate that said Structural Coverage be demonstrated at the unit test level, it is often impossible in practice to achieve it in any other way.


“If your tests haven't achieved statement coverage, then there's code that
you've never executed, not even once, during your testing.”

Yes, that’s certainly true. But merely executing any one line of code once during testing does not at all guarantee that:

a) all defects in that one line of code have been revealed
b) all defects in how that one line of code relates to the other lines of code have been revealed

In fact, “Statement Coverage” as a Structural Coverage criterion is the absolute weakest of all Control-Flow based Structural Coverage criteria and it has logical holes in it so big that one could drive a proverbial double decker London City bus through them. It is trivial to show how easy-to-make defects in code would be missed (i.e., unrevealed) by Statement Coverage. IMHO, that anyone would ever be satisfied with even 100% Statement Coverage for anything but the most harmless of software is rather upsetting. It’s quite trivial to show how even Modified Condition/Decision Coverage (MC/DC) leaves the door open to potentially unexposed defects (subtle nod to Data-flow based Structural Coverage criteria here).

And the vast majority of organizations I work with only aim for 60% to 70% Statement Coverage for even their most critical code. Shocking.


— steve



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/pipermail/systemsafety/attachments/20240807/535b2afa/attachment-0001.html>