[SystemSafety] State of the art for "safe Linux"

[Paul Sherwood]

Steve, thanks for piling in! :)

On 2024-08-06 20:47, Steve Tockey wrote:
> “_I understand the argument, but this last sentence is flawed._
> The fact that some statements are not covered by tests does not
> necessarily
> mean that they have never been tested. For example the Linux community
> benefits from the fact that their software is tested on billions of
> 
> _devices every day._”
> 
> Sorry, I’m not buying that argument. Counter-example? Heartbleed. It
> was reported to have been introduced into OpenSSL's TLS in 2012 and
> apparently not discovered by “the good guys” until 2014. That no
> “normal” code triggered the buffer overrun certainly did not at
> all mean that the buffer overrun defect wasn’t there. Who knows how
> many times that defect was maliciously exploited before it was finally
> discovered by "the good guys” and eliminated?

I accept what you are saying, but your I think your point is tangential 
to my statement.

100% test coverage obviously does not guarantee (nor even remotely 
suggest) that all input value combinations are tested.

A colleague shares the story of an organisation hired to create unit 
tests who chose zero as the input value for all unit tests, and were 
happily achieving 100% coverage until they triggered a crash in code 
that divided by one of their input values.

I expect that many (perhaps **very** many) projects with unit test 
coverage as a metric were caught out by heartbleed. As I mentioned, so 
was SEL4 (unless my memory is completely faulty... i've not been able to 
track down the blog post)

Better "coverage" for this kind of topic would be obtained by fuzzing, 
for example.

> And Heartbleed was in software that WAS supposedly tested—the issue
> being that the testing that was done was obviously inadequate to
> reveal that defect.

Agreed, but blindly focusing on statement coverage would not improve 
matters.

> Given the rank amateur-ish manner in which the vast majority of
> software is produced these days

Sadly I agree

> —particularly in the open source
> community—

I think this is unfair... most open source project teams I have engaged 
with are doing great work - arguably better than many corporate 
organisations I've dealt with.

"the open source community" is not one community. It is thousands (more 
accurately tens or hundreds of thousands) of communities, each with 
their own characters norms, rules, strengths, weaknesses. There is a 
well-established open source community focusing on GCC, for example - I 
hope you don't believe them to be amateurish?

> there can be no substitute for _truly comprehensive_
> testing to have any trust at all in that code.

I agree with "truly comprehensive", but remain unconvinced by "statement 
coverage" as a metric.

Thanks again for your comments.

br
Paul