[SystemSafety] State of the art for "safe Linux"

[Prof. Dr. Peter Bernard Ladkin]

On 2024-08-06 19:35 , Paul Sherwood wrote:
>
>> [Dewi Daniels] DO-178C doesn't require you to achieve 100% structural coverage
>> through unit tests. It requires that your requirements-based tests
>> (which can be hardware/software integration tests, software
>> integration tests or low-level tests) cover all the software
>> requirements and exercise all the code. If your tests haven't covered
>> all the requirements, there's functionality you haven't tested. If
>> your tests haven't achieved statement coverage, then there's code that
>> you've never executed, not even once, during your testing.
>
> I understand the argument, but this last sentence is flawed.


How is the last sentence "flawed"? It seems to me a clear statement of the obvious (which I imagine 
is what Dewi intended).


> The fact that some statements are not covered by tests does not necessarily
> mean that they have never been tested. For example the Linux community
> benefits from the fact that their software is tested on billions of
> devices every day.

That is unfortunately an equivocation on the meaning of "tested". Using Linux "every day" is 
operational experience, which is different from the software engineering development process known 
as testing. The various standards committees on statistical evaluation of software (at the German 
DKE and the IEC and some people I know in rail who deal with it via CENELEC) have discussed for many 
years key the differences between evaluating operational experience and performing statistical 
testing. Everyone involved in those discussions, with one exception (which I consider eccentric), 
acknowledges the differences.

Also, we emphasise the issue of knowing your operational profile, and do so in the proposed 61508 
revised guidance (Part 7 Annex D; the current version is highly misleading). You can only draw valid 
conclusions (to a given level of confidence) about future use of software from operational 
experience when (a) you know what your operational profile has been, and (b) you are sure that your 
future use retains essentially the same profile.

An operational profile is the set of: inputs L to software along with the proportion of the total 
inputs that each L represents. The total inputs to any piece of software in the past is a finite set 
so this is a well-defined notion.

The reason for (a) and (b) has been presented rather well by Steve Tockey's example of Heartbleed. 
If future use of software pre-Heartbleed had been restricted to pre-Heartbleed use, then to a rather 
high level of confidence Heartbleed would not have been discovered.

A colleague at Exida, formerly on DKE and IEC committees, encountered quite often reasoning from 
clients who said "Windows (or what have you) has millions, even billions of hours of use on millions 
of computers. According to Part 7 Annex D Table D.1 I can have 95% confidence that it won't break in 
the next several-million hours of use in the use we are proposing for it. So it's SIL 2/SIL 3/SIL 4 
capable." He encountered this often enough that his firm position was (and, I imagine, still is) 
that Part 7 Annex D needed to be removed. He had a point. The argument in quotes is a rotten and 
naive argument. I am rather surprised to see it surfacing again in this community. I thought we, at 
least, had well and truly laid it to rest.

>
>> Peter has pointed out
>> that using statistical techniques to measure confidence in software as
>> complex as an operating system is a pipe dream.
>
>  I don't agree with you, or with him.

There is really not much intellectual scope for disagreement here. It's easy to write the words "I 
don't agree", but it is much harder to construct an intellectually valid disagreement. The maths 
tells you the hard constraints.

When you buy stuff in a supermarket, the till person asks you for a certain amount of money. You can 
say "I don't agree". The till person prints out a bill. You can go through each item and agree that, 
yes you presented it for purchase and, yes, the price as shown is correct. You can get out your 
calculator and tot up the total yourself. The supermarket supervisor can get out her calculator and 
tot up the total. There is very little scope for persisting disagreement on the total. The maths of 
arithmetic sets a hard constraint. You can still say "I don't agree" but you will have been shown, 
definitively, to be right or to be wrong.

It is similar with stateval of software. Assuming you can get a grasp on (a) and (b) above, the 
maths of renewal processes place hard constraints (or, for our US readers, "the math ... places") on 
the amount of operational experience without failure (meaning: actual failure independently of 
whether you observed it or not) you have to have recorded in order to reach what is currently 
considered an acceptable level of confidence in a specific reliability of, say, 1 in 10^(-x) per 
operating hour. Just as customer and till person and supermarket supervisor are bound by the maths 
of arithmetic, any software statistical evaluator is bound by the maths of renewal processes (and 
indeed rather a lot more).

I have never seen a believable operational history of use of a general purpose operating system such 
as Linux such that it could be viably used for statistical evaluation for use of that OS in high-SIL 
safety functions. And, believe me, I've asked.

There are plenty of limited-functionality devices used for specific functions as part of a safety 
function. There are also commercial operating systems which have obtained certificates, or 
certification, for specific uses in specific safety-related systems. These are typically much more 
restricted, and very much smaller, than the Linux kernel. They have not typically been 
full-functionality statistically evaluated, although they might well have been statistically tested 
for a specific function. But testing, whether or full or of limited functionality, will not have 
been restricted to purely statistical testing.

PBL

Prof. Dr. Peter Bernard Ladkin
Causalis Limited/Causalis IngenieurGmbH, Bielefeld, Germany
Tel: +49 (0)521 3 29 31 00