[SystemSafety] AI and the virtuous test Oracle - AI properties
Les Chambers
les at chambers.com.au
Mon Jul 3 21:53:36 CEST 2023
Hi Michael
Good to hear that some structured thought is occurring on AI in the Safety-
Critical domain. When will ISO TR 5469 be available to the engineering
community?
Is anyone publishing thoughts, elsewhere, for example, on properties that need
to be established.
I have the sense that, in the light of AI, we're going to have to rethink
systems engineering sharpish like.
Its good advice: In order to discover new lands, one must be willing to lose
sight of the shore for a very long time. - André Gide
Les
> A very interesting thread as well. Hello everybody!
>
> I have worked with a group of people on an ISO TR 5469 regarding functional
safety and AI. Some people who didn't read it but comment it uttered it would
standardise AI. Actually the idea was different: there were AI specialists and
safety specialists meeting to talk about the use of AI in FS applications and
the first thing we found to be needed was that we search for a common language
to speak to each other. As some know there are different concepts of "risk" in
different disciplines and it is good to align first that the view of the groups
is the same. Now the result is for me rather that this TR explains basic safety
concepts to readers who want to fulfil safety standards with systems
incorporating AI. It doesn't give a clear metric for expected integrity but it
enables the communities to talk to each other and find solutions.
>
> Why was this started? Because the AI community knows that the trust in what
the systems do isn't there. So the logics is that if a standard would prescribe
rules - maybe linked to certain technologies where evidence can be provided for
certain properties that need to be established. This is why I must say that
these catalogs of criteria are already heavily discussed. The biggest group
involved into the creation of the TR at least in Germany were test houses. At
the same time you also say that there are several resources that give rulesets
and I also say I have seen six or seven different concepts. The fun part about
that is that I rarely see the same concept in two of them, they are all using a
different base. I also didn't find one ruleset that would give criteria for
when the system really is working safe enough. One standard seems to be in
favour at the moment that uses a catalog of criteria provided by one of the
Fraunhofer institutes here in Germany as basis. So for me the time is right for
the discussion.
>
> About philosophy lessons. I fear future generations of engineers will use the
future chatGPTs for their research, and if they are intelligent enough they
will look for proof that what was provided is correct. I am sometimes sceptical
where this leads. I had a discussion about numbers and figures days ago and
told that I don't accept any diagnostic coverage value if it diagnoses only bus
errors but not the problem - the drift of the sensor value. It was a little
scary for me that I needed more than two minutes to convince that this is the
right way. As logics also is a part of philosophy I'd start with that one. All
the rest seems to be a very good idea but secondary (by the way I have read two
books by technology philosopher Kornwachs). A similar problem seems to me that
all teachers need to study their profession but a good background in education
or didactics is underrepresented in the curriculum.
>
> Besten GruÃ!
> Michael
>
> --
> Michael KINDERMANN (he/him)
> Head of Functional Safety
> Team Leader Safety & Security
> Dpt. Global Compliance
> Phone: +49 621 776-2608
>
> Pepperl+Fuchs SE, Mannheim
>
> >
>
> Pepperl+Fuchs SE, Mannheim
> Vorstände/Board members: Dr.-Ing. Gunther Kegel (Vors.), Werner Guthier
(stellv. Vors.), Mehmet Hatiboglu, Lutz Liebers, Reiner Müller, Florian Ochs
> Vorsitzende des Aufsichtsrats/Chairwoman of the supervisory board: Monika
Müller-Michael
> Registergericht/Register Court: AG Mannheim HRB 737016 - UST-ID Nr. DE
143877372
>
> -----Ursprüngliche Nachricht-----
> > Von: systemsafety [mailto:systemsafety-bounces at lists.techfak.uni-
> > bielefeld.de] Im Auftrag von Peter Bernard Ladkin
> > Gesendet: Dienstag, 27. Juni 2023 10:15
> > An: systemsafety at lists.techfak.uni-bielefeld.de
> > Betreff: Re: [SystemSafety] AI and the virtuous test Oracle - action now!
> >
> > ### EXTERNAL EMAIL ###
> >
> >
> > Les,
> >
> > On 2023-06-27 06:15 , Les Chambers wrote:
> > > ..... international bodies
> > > that currently regulate software-intensive Safety-Critical systems -
> > > who cling to regulating processes that have ceased to exist - are
> > > likely to be overrun and made redundant.
> >
> > I don't see how this makes much sense. There are no international bodies
> > that regulate software-intensive Safety-Critical systems (SISCS for short),
> > except for IMO as far as I can tell.
> > Except for IMO, regulation occurs at the level of nation-states, or the EU
> > (whose member states have delegated certain regulatory activities to the EU
> > in the sense that the EU writes directives that are then taken into
national
> > law by the members).
> >
> > And as far as IMO goes, the level of SISCS in large ocean-going vessels
seems
> > to be of somewhat limited effect on the hazards of shipping (though I am
> > open to reconsidering).
> >
> > I don't know what "processes that have ceased to exist" you might be
> > referring to; can you say?
> >
> > Hazard and risk analysis (HRA) is regarded by IEC and ISO as key to
standards
> > involving safety considerations - that is explicitly what Guide 51 says -
and
> > Guide 51 says HRA shall be required in such standards, and tells us what it
is.
> > The regulation in many states of SISCS depends upon adherence to such
> > standards. I don't see that the emergence of ML-based subsystems affects a
> > requirement for HRA much at all - but I do see that traditional HRA is put
in a
> > quandary by how to evaluate systems with ML-based subsystems. The
> > informal development standards applied by ML subsystem developers
> > (often called "AI safety") don't work in traditional HRA assessments -
rather,
> > they do nominally work and rule ML-based subsystems out because
> > reliability calculations are not possible.
> >
> > However, I do see that there is considerably commercial pressure to approve
> > safety-critical software which essentially uses ML-based subsystems for
> > pervasive use, in particular in the road-vehicle sector, despite the lack
of
> > reliability assessment. But here there are, yes, regulatory hurdles. As
well as
> > considerable scepticism amongst many engineers. Not helped, of course, by
> > revelations such as those by Handelsblatt, which suggests that Tesla knows
> > of way more problems with its "Autopilot"
> > SW than have been made public (Handelsblatt got hold of gigabytes of
> > customer reports).
> >
> > > In favour of organisations such as:
> > >
> > > - The Center for Human-Compatible AI at UC Berkeley
> > > - The Future of Life Institute
> > > - The Center for AI Safety (CAIS)
> > > - Stanford Center for AI Safety
> >
> > Can you name any reports on the reliability assessment of, say, control
> > systems involving ML-based subsystems that any of those institutions have
> > published? (There are quite a few such reports around, but those
institutions
> > are not where they come from.)
> >
> > > .... This is a major
> > > inflection point in the evolution of intelligence. Carbon hosts will
> > > always be limited; silicon is unbounded.
> > Well, ChatGPT and its emergent behaviour certainly made the headlines
> > recently. It's not new to me.
> > I've been working on two projects since 2017 with language models based on
> > word embedding (invented by Google ten years ago: Mikolov, Chen, Corrado
> > and Dean). OpenAI and Google and Meta upped the scale and changed the
> > application somewhat in 2021-2022, and then OpenAI puts a conversation bot
> > on the open Internet and everybody goes bonkers. Because, rather than just
> > a few devoted people (say, at the institutions you name) thinking about
> > issues with chatbots, millions of people suddenly are.
> >
> > It does seem worth emphasising that Chatbots based on word-embedding
> > technology and control systems designed around ML-based environment-
> > interpretation subsystems are two almost completely different technologies.
> > What they have in common is ML technology.
> >
> > The reason that word-embedding technology made what seems to be a
> > quantum leap is the existence of huge corpora. You can train these things,
if
> > you wish, on more or less all the language that has ever been written down.
> > And OpenAI (and maybe Google and Meta) did. Reported to have cost nine-
> > figure sums of money. The CEO of OpenAI has said openly (and I believe him)
> > that that is not a sustainable development model. Not necessarily for the
> > cost, for there is lots of that kind of money in the world, but for the
effort
> > involved and the very special case of the entire environment being
available
> > (a universal corpus, as it were). Whereas the environment for road vehicle
> > operation is not similarly available. It is also vastly more complex, as
far as
> > anyone can tell. We can't even say what it is. (Whereas conceptualising a
> > corpus is something people have been able to do for
> > millenia.) Apple and Google and who knows else have been training their
> > road vehicle technology on public roads for well over the decade it took
from
> > the invention of word-embedding technology to the emergence of ChatGPT,
> > and they are nowhere near "prime time" yet.
> >
> > Further, I think you're wrong on the silicon level. There are hard physical
> > limits to the development of current digital-computational processing
units.
> > Moore's Law cannot be open-ended.
> > Many HW developers have pointed out we are reaching limits. I would be
> > much more inclined to consider an "inflection point" when/if people get
> > quantum computing to work. (I won't reiterate that well-rehearsed
> > reasoning here.)
> >
> > What does interest me is the political inflection point, if I may term it
that. FLI
> > put out its Slaughterbot video some years ago, and people such as Stuart
> > Russell tried to get everyone to take it very seriously. We can thank our
lucky
> > stars that no capable national militaries seem to have taken it
particularly
> > seriously, for if they had we could well be in a world-wide political
crisis in
> > which no influential politician or national executive in any country could
ever
> > be seen in the open air ever again. Slaughterbot and similar threats have
little
> > to do with "intelligence", just with the capabilities of technology
developed
> > by people whom society has put in category of "AI research". But put a
> > Chatbot on the Internet and all of a sudden the sky is falling.
> >
> > PBL
> >
> > Prof. i.R. Dr. Peter Bernard Ladkin, Bielefeld, Germany
> > Tel+msg +49 (0)521 880 7319
> > Tel+http://atpscan.global.hornetsecurity.com/index.php?atp_str=dan4noM
> > CUfBIyfdve2Doy2zefN4Lu0TRKaS1U-XN27cRDB9W0CdYA-
> > rLa2izwfYaFuQYHLQxszB8hGsFUZecRJGFWwebrxzSJTQ0ilwKKy2Q0EffIKw139
> > HTdBdS1hqPhHNz8Pt4wOTAVEuWUjHlFmnsuI3eYdhLSTnedu7oPmhEbobtov
> > kq9ENIP40iO53uhfyw6ZZJqRCR4-LuCegp-
> > 8ZDntuWs0bMAA8ONpKIFpMLnR5qIKsWiXWkhzMzdlsAvLmm1gHZXz44DN-
> > ueYpQp6ZrH6MD-
> > 0HNY0nYt8kUSS0GFugtlxWO0t0PnfbWtPEGlY9bSYWy0i3_zbQjOjojR-
> > De2PidghgzsbRyIzo6Ix_JKl133GFDt4afXjqtIn0
> >
> >
> >
>
> Wichtiger Hinweis:
> Diese E-Mail einschliesslich ihrer Anhaenge enthaelt vertrauliche und
rechtlich geschuetzte Informationen, die nur fuer den Adressaten bestimmt sind.
Sollten Sie nicht der bezeichnete Adressat sein, so teilen Sie dies bitte dem
Absender umgehend mit und loeschen Sie diese Nachricht und ihre Anhaenge. Die
unbefugte Weitergabe, das Anfertigen von Kopien und jede Veraenderung der E-
Mail ist untersagt. Der Absender haftet nicht fuer die Inhalte von veraenderten
E-Mails.
>
> Important Information:
> This e-mail message including its attachments contains confidential and
legally protected information solely intended for the addressee. If you are not
the intended addressee of this message, please contact the addresser
immediately and delete this message including its attachments. The unauthorized
dissemination, copying and change of this e-mail are strictly forbidden. The
addresser shall not be liable for the content of such changed e-mails.
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
> Manage your subscription: https://lists.techfak.uni-
bielefeld.de/mailman/listinfo/systemsafety
--
Les Chambers
les at chambers.com.au
+61 (0)412 648 992
More information about the systemsafety
mailing list