[SystemSafety] [System Safety] FOSDEM talk by Paul Sherwood

[Phil Koopman]

Paul,

Thanks for sharing this. Hot take: felt more like a sales pitch than a 
really concrete explanation.  Old ways dismissed based more on "nobody 
wants to do it" and "new is better".  I get why that argument has 
appeal, and surely some (not all) people don't do the old ways as well 
as they should. But it was pretty light on why "new will be sufficient 
for acceptable safety" vs. deciding that doing the new ways really well 
will automatically be fine.   I just recently saw a live talk that had a 
lot of the same arguments and was similarly left with more questions 
than answers on this topic. Maybe those in the trenches on the new 
technology have a different viewpoint. To be sure, this is not me trying 
to die on the hill of the old ways, but rather expressing what I think 
is appropriate caution on jumping to new system design approaches on 
life critical systems.

We can justify the old way in hindsight in that it seems to work, even 
if we struggle to rigorously explain why. Do we want to jump to a new 
way without understanding why it is expected to work and spend decades 
of mishaps climbing the hill to getting to where it really needs to be?  
Or is there a way to have some confidence about it before then?

Assuming they make good on everything they plan, what would help me a 
lot is understanding the basis for the safety case for their approach in 
a general sense.  What parts of assurance does it provide, and what 
parts does it not provide?  What underlying assumptions does it make?  
As a simple example, supply chain attacks will be a huge issue compared 
to a proprietary OS + custom application. This is not news to them, but 
is an example of the type of new challenge that might surprise us with 
an mishap news headline.

Discussion here is fine, but this is not something we are likely to 
resolve in an e-mail chain.  This is a whole discussion that needs to 
happen over many years in many forums.  And it is not just about FOSS in 
general. There are these concerns plus additional specific concerns 
about using machine learning technology, tool chains and libraries as 
well, in which we already have multi-ton machines hurtling down public 
roads by companies who have decided (most, but not all of them) that 
core safety standards -- including ones specifically for their 
technology -- are irrelevant because they stifle innovation.

We might not be happy with old-school safety practices, but there are 
lessons there learned the hard way over decades. We should be reluctant 
to throw them out wholesale without taking some time to figure out what 
lessons we need to learn with the new approaches.

My own take on this topic in a somewhat more abstract discussion is 
here, in which I point out the gaps in understanding how/why current 
approaches work and how we might close those gaps going forward for any 
approach:

  * Johansson, R. & Koopman, P., "Continuous Learning Approach to Safety
    Engineering
    <https://users.ece.cmu.edu/~koopman/pubs/Johansson2022_CARS_ContinuousLearningSafety.pdf>,"
    Critical Automotive Applications: Robustness & Safety / CARS at EDCC2022.
  * https://users.ece.cmu.edu/~koopman/pubs/Johansson2022_CARS_ContinuousLearningSafety.pdf


Kind regards,
Phil


On 2/8/2025 10:03 PM, paul_e.bennett at topmail.co.uk wrote:
> I cannot recall if this might have been linked in recently, but from
> the date iof the talk, probably not.
>
> <https://fosdem.org/2025/schedule/event/fosdem-2025-6204-the-trustable-software-framework-a-new-way-to-measure-risk-in-continuous-delivery-of-critical-software/>
>
> Paul gives a view on the attempt to evaluate Free Open Source
> Software for trustability. I offer the link with no further comment
> or claim from my side. I'll let people make up their own minds.
>
> Regards
>
> Paul E. Bennett IEng MIET
> Systems Engineer
> Lunar Mission One Ambassador

-- 
Prof. Phil Koopmankoopman at cmu.edu   
(he/him)https://users.ece.cmu.edu/~koopman/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/pipermail/systemsafety/attachments/20250210/77a95314/attachment.html>