[SystemSafety] ISO and IEC Technical Specifications on Functional Safety and AI

[Les Chambers]

Michael

Thanks for your concern regarding the integration of AI with Safety-Critical 
systems. 

Tesla full self driving/supervised is now available in Australia. There have 
been some crazy behaviours. 

Refer: https://bit.ly/CrazyTeslaFSD

It seems the classical engineering definition of the term, "technology" 
remains axiomatic. "Technology is a thing that was invented after you were 
born and 
 remains a thing that doesn't quite work yet."
Elon is yet to get his functional safety act together.

To me, it is annoying that these vehicles, guided as they are with immature 
technology, were allowed on Australian roads by naive regulators.

I'm yet to discover any AI driven guidance system that can deliver the SIL 
4/ASIL D Performance that these vehicles should provide. It seems to me that 
attempts to develop functional safety standards for this class of system are 
fruitless given: 
1. No one, including their developers, fully understands why or how they work 
at all.
2. The black box that is AI is a demonstrably unreliable agent in the service 
of public safety. Further, the most effective risk management strategy in 
assuring SIL 4/ASIL D performance is to remove the AI black box altogether.

I have faith that, in the fullness of time, this problem will be solved with 
Gold Standard neural networks that can be trusted. Waymo is heading in this 
direction but with a US$250,000 vehicle geofenced to territory thoroughly 
LIDAR-mapped and bristling with LIDAR sensors. 

In the meantime, shareholder value and the "but-it's-so-cool!!" zeitgeist has 
trumped public safety guiding vehicles with neural networks that class 
somewhere south of bronze standard. 

After 50 years engineering safety-critical systems, beginning with direct 
software control of potentially explosive chemical reactions in the 1970s, I'm 
watching my profession seemingly discard decades of lessons we paid for dearly
—in nervous breakdowns, blood and treasure. 

A sad state of affairs.

If anyone on this list could cheer me up, I would be most grateful.

Cheers
Les

> Hello everybody!
> 
> Generally I like to stay out of the discussion here although I appreciate 
this resource very much. As my company works on automation devices and 
especially small proximity sensors I would get a quite narrow viewing angle so 
it is important to get information like I get it here, from avionics through 
railway through other systems.
> 
> Unfortunately I don't have further information on the self-driving car 
industry and what they do to justify what they put on the market. In Germany 
you would have the institution of the Kraftfahrtbundesamt as an institution 
and I am very sure that they are quite conservative about that. But this shows 
why everybody is eager on rulesets regarding AI in functional safety so we are 
working on that.
> 
> Very interesting that we standardize safety and AI while there still is 
debate about what AI really is. But actually there seem to be some systems 
that might save lives so we need to look under which circumstances that can be 
allowed. I think if I would need to defend the use of AI in safety at the 
moment people would ask why I took that risk and used AI. Somewhere in the 
future it could be that I need to defend myself because I didn't use AI 
because it would have prevented the accident. Some say that moment will never 
come. Some say this moment is in the past, maybe not for something as complex 
as self-driving cars but perhaps in other applications with perhaps less 
dimensional inputs.
> 
> Some would say risk reduction below SIL 1 should not be regulated too 
strongly. But when I think about how people put some data into a system and 
cheer if it showed nice behaviour it might be good to ask about requirement 
management, configuration management and solid proof of validity of the 
concept - so speak about functional safety management to the AI audience. 
That's the direction now that the ISO/IEC group will meet in Sydney this week 
for TS 22440.
> 
> But not to bore with this stuff - maybe not a safety application but a case 
where people are detected using WLAN routers - even if they don't have WLAN 
devices with them. A little scary but they say in a different article that 
they could distinguish people walking into a room even when they carried a 
container with bottles of beer. That showed me that this is a realistic 
University scenario here in Germany. This is the only resource I found in 
English but it is actually also distributed from other sources.
> https://interestingengineering.com/innovation/wifi-tech-can-identify-
individuals
> 
> Curious where AI and safety will go. Have a good week!
> Michael
> 
> --
> Michael KINDERMANN (he/him)
> Head of Functional Safety
> Team Leader Safety & Security
> Dpt. Global Compliance
> Pepperl+Fuchs SE, Mannheim
> 
> Pepperl+Fuchs SE, Mannheim
> Vorstand/Board member: Dr. Wilhelm Nehring (Vors.), Tobias Blöcher, Lutz 
Liebers, Reiner Müller, Florian Ochs, Martin Walter
> Vorsitzende des Aufsichtsrats/Chairwoman of the supervisory board: Monika 
Müller-Michael
> Registergericht/Register Court: AG Mannheim HRB 737016 ∙ UST-ID Nr. DE 
143877372
> 
> Wichtiger Hinweis:
> Diese E-Mail einschliesslich ihrer Anhaenge enthaelt vertrauliche und 
rechtlich geschuetzte Informationen, die nur fuer den Adressaten bestimmt 
sind. Sollten Sie nicht der bezeichnete Adressat sein, so teilen Sie dies 
bitte dem Absender umgehend mit und loeschen Sie diese Nachricht und ihre 
Anhaenge. Die unbefugte Weitergabe, das Anfertigen von Kopien und jede 
Veraenderung der E-Mail ist untersagt. Der Absender haftet nicht fuer die 
Inhalte von veraenderten E-Mails.
> 
> Important Information:
> This e-mail message including its attachments contains confidential and 
legally protected information solely intended for the addressee. If you are 
not the intended addressee of this message, please contact the addresser 
immediately and delete this message including its attachments. The 
unauthorized dissemination, copying and change of this e-mail are strictly 
forbidden. The addresser shall not be liable for the content of such changed 
e-mails.
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
> Manage your subscription: https://lists.techfak.uni-
bielefeld.de/mailman/listinfo/systemsafety



--
Les Chambers
les at chambers.com.au

https://www.chambers.com.au
https://www.systemsengineeringblog.com

+61 (0)412 648 992