[SystemSafety] State of the art for "safe Linux"

[Steve Tockey]

Paul Sherwood wrote:

“I've seen way too much supposedly "tested" software behave unexpectedly in operation.”

With all due respect, please indulge me in a slight rephrasing of that statement into:

“I've seen way too much amateurishly-developed and inadequately-tested" software behave unexpectedly in operation.”

My point is that it’s not necessarily only one particular form of Structural Coverage software testing that’s at fault here. There are many other amazingly common sources of software defects that all forms of testing can be blind to:

*) The requirements could have easily been incomplete, inconsistent, or just plain wrong

*) The design could have easily been incomplete, inconsistent, just plain wrong, or (shockingly commonly) ignored altogether

*) The code could have been hacked together without regard to basic correctness notions of, for example, Simplicity, Cleanliness, Design by Contract, Provability, Assertions, and so on


That’s why, in particular, DO-178C mandates certain behaviors on software projects:

Table A-1 on Software Planning—go into software projects with a definite, deliberate, and well thought-out plan of attack rather than just slinging code from day 1

Table A-2 on Software Development, Table A-3 on Software Requirements, Table A-4 on Software Design, Table A-6 on Software Coding & Integration—be deliberate and methodical in requirements, design, code, & integration, and ensure consistency between them all

Table A-6 on Testing Outputs of Integration Process and Table A-7 on Verification of Verification Process Results—have confidence that the software that got built & integrated does all (and only) what it is supposed to do

Table A-8 on Software Configuration Management—be sure that the software consists of the right pieces in the right places at the right time so that versions don’t get inadvertently out of synch

Table A-9 on Software Quality Assurance—be confident that what was planned to happen definitely and deliberately is what actually happened and that we are delivering what we said would be delivered


Dare I say that of DO-178C's 71 objectives for DAL A software, the vast majority of them—oh, on the order of 60 or so—are nothing more and nothing less than basic, sound Software Engineering professional practice. And that’s not even including valuable things like appropriate Static Analysis as mentioned by Martyn.

Development of reliable software takes a broad spectrum, professional approach across the entire discipline of Software Engineering, not merely a single-dimensional, “We’ll try to test the crap out of it at the end” (without even coming close to actually achieving that).


And that’s the root of my concern for the state of the software practice today:

Not only do I see precious little evidence of any kind of broad spectrum, professional approach in those who ARE BEING PAID to develop software, I’m seeing even less evidence in those who are not necessarily being paid (read: the Open Source community).


— steve





On Aug 7, 2024, at 11:29 AM, Paul Sherwood <paul.sherwood at codethink.co.uk> wrote:

On 2024-08-07 17:33, andrew at andrewbanks.com wrote:
On Wednesday, August 7, 2024 10:38 AM, Paul Sherwood wrote:
Because we can **test**, without creating **tests**.
We may have executed the code, but not created tests for it.
Indeed... mega-hours of nominal operation is fine, but serves no purpose if
1. the test object is not appropriately specified, or inappropriately
configured
2. the test environment is not specified, configured, nor repeatable
3. the test scenario is not specified, configured, nor repeatable
4. the expected result is not specified to an appropriate accuracy,
precision, specificity and sensitivity
5. the achieved result is not favourably compared to the expected result
Etc
There is a lot more to testing than simply randomly executing it.

I agree with your points 1-5 as good practice (best practice? who really knows what's best?).

But to say that operation without those practices "serves no purpose" doesn't align with my experience. I've seen way too much supposedly "tested" software behave unexpectedly in operation.
_______________________________________________
The System Safety Mailing List
systemsafety at TechFak.Uni-Bielefeld.DE
Manage your subscription: https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/pipermail/systemsafety/attachments/20240807/7e2dc794/attachment-0001.html>