[SystemSafety] AI and the virtuous test Oracle - action now!

[Steve Tockey]

Les,

“silicon is unbounded”

I disagree. Silicon is bounded, just in different ways than humans. For one, Turing Computability. Turing Machines, which all modern silicon computers are a kind of, are 100% deterministic. Human intelligence is non-deterministic. Thus, I argue, human intelligence is not something that a computer could ever actually be capable of. See, for example:

https://youtu.be/i2trJEIFIvY

I agree that, “We are embarked; we need to deal with it”. But we also have to be aware of the limits of so-called AI.


— steve



On Jun 26, 2023, at 9:15 PM, Les Chambers <les at chambers.com.au> wrote:

Hi

I put it to you and the list in general, a "discuss later" mentality is foolish
in the extreme. The process of deploying intelligence in automated systems has
, and will,  fundamentally change. Among other things, international bodies
that currently regulate software-intensive Safety-Critical systems - who cling
to regulating processes that have ceased to exist - are likely to be overrun
and made redundant.

In favour of organisations such as:

- The Center for Human-Compatible AI at UC Berkeley
- The Future of Life Institute
- The Center for AI Safety (CAIS)
- Stanford Center for AI Safety

My view is that this is not a steady-as-she-goes situation. This is a major
inflection point in the evolution of intelligence. Carbon hosts will always be
limited; silicon is unbounded. We are embarked; we need to deal with it.

Suggested reading: Max Tegmark, Life 3.0

Les

C2 - Confidential

Hello,

For the moment, I don't see in industry any attempt to realise E/EPE safety
related function with On-line AI. All what I see is focused on off-line AI,
meaning that the training is done specified training data sets, and the
validation is done on specified test datasets. We don't see any performance
better than roughly 3.3 10-3 in automotive. So let's wait for the achievement
of 10-8 to 10-5 error rate on test datasets and discusss later the
acceptability.

Attention : due to increased cybersecurity screening I may receive external
emails 30 minutes after their emission.
RESTRICTED

Bertrand RICQUE
Safety and security standardisation Expert
Program Support Manager
Optronics and Defence Division/Customer Support
Safran Electronics & Defense

P +33 (0)1 58 11 96 82   M +33 (0)6 87 47 84 64
bertrand.ricque at safrangroup.com
102 Avenue de Paris
91300 MASSY FRANCE
www.safran-electronics-defense.com

-----Message d'origine-----
De : Les Chambers <les at chambers.com.au>
Envoyé : lundi 26 juin 2023 06:03
� : RICQUE Bertrand (SAFRAN ELECTRONICS & DEFENSE)
<bertrand.ricque at safrangroup.com>; les at chambers.com.au;
koopman.cmu at gmail.com; systemsafety at lists.techfak.uni-bielefeld.de
Objet : RE: [SystemSafety] AI and the virtuous test Oracle

CAUTION:  This message originated from an outside organization. In case of
suspicion, click on "Report to SAFRAN Security" from the Outlook ribbon.

RE your comment: â?oas it impossible, by construction, to identify all
dangerous
situations�

True. But it is possible to identify the subset of highest probability
â?odangerous situationsâ? and deal with them.  I had 10 years of
experience
with this issue in computer control of chemical processing reactors. In the
1970s, we had a safety authority write software to: 1. identify well-
established dangerous states of a reactor; 2. Transition reactor to a safe
state
â?" usually shut down. It was called the â?oabortâ? programming. This
abort code
overrode the control software written by another team. By today's standards
it was primitive but still very effective.
This experience has influenced my thinking on ways and means of dealing
with â?obadâ? or â?ohallucinatingâ? AIs. We need a separate and distinct
â?oabortâ? AI
capable of recognising evil and dealing with it.
Sal Khan (Khan Academy) has implemented another idea in creating his
Khanmigo personal tutor. To improve its accuracy in mathematics, he allows
the AI to have â?opersonal thoughtsâ?. The AI equivalent of mindfulness
(straight out of Marcus Aurelius). He gives the impression that he feeds
back
the AI's response to the AI, so the AI can have a think about what it is
thinking. A bit like briefing a fish on the concept of water.

This supports my pitch for engineers to be taught philosophy. The concepts
of virtue and evil were defined simply 2000 years ago. These definitions
have
stood outside of time and can be implemented with appropriate pattern
matching.
If Marcus Aurelius was reincarnated, he'd be running international seminars
on mindfulness. The foundation ideas have not changed a wit.

Cheers
Les



C2 - Confidential

Answer to the first question. Once the NN is trained, it will always
give the
same output for the same input. It is stupid software logic.

Answer to the second question. The current safety paradigm is based on
perception-interpretation-decision-action loops based on the fact that:
1 Either, all states of the system, leading to all identified losses,
can be
exhaustively defined and their probability of occurrences remains, by
concept and construction of the system, below a tolerable level, enabling
fully autonomous functions (e.g. emergency shutdown functions in process
industries),

2 Or, there is a human in control of the system with the ability of
arbitrating the decisions, meaning the ability to at least perceive and
interpret.

AI deceives point 1 as it impossible, by construction, to identify all
dangerous situations (the training dataset is incomplete and whatever its
size, the operational domain being infinite (or close to infinite from a
practical
point of view), training dataset size divided by infinite = 0.

To deceive the second assumption, just replace a car windscreen by a
video
screen. No need of AI.

AttentionÃ, : due to increased cybersecurity screening I may receive
external
emails 30 minutes after their emission.
RESTRICTED

Bertrand RICQUE
Safety and security standardisation Expert Program Support Manager
Optronics and Defence Division/Customer Support Safran Electronics &
Defense

P +33 (0)1 58 11 96 82   M +33 (0)6 87 47 84 64
bertrand.ricque at safrangroup.com
102 Avenue de Paris
91300 MASSY FRANCE
https://urldefense.com/v3/__http://www.safran-electronics-
defense.com_
_;!!Dl6pPzL6!fVuNoqxkdCCqescc5hd8-9ke7-
o3uVuZEOBtugdgtfrHUBcO3T2a6LyI_
lZBwk7CZBJRmOhVQnZHXCdvIB_h5MY$

-----Message d'origine-----
DeÃ, : Les Chambers <les at chambers.com.au> EnvoyÃf©Ã, : jeudi 22 juin
2023 23:46 Ãfâ,¬Ã, : RICQUE Bertrand (SAFRAN ELECTRONICS & DEFENSE)
<bertrand.ricque at safrangroup.com>; koopman.cmu at gmail.com;
systemsafety at lists.techfak.uni-bielefeld.de
ObjetÃ, : RE: [SystemSafety] AI and the virtuous test Oracle

CAUTION:  This message originated from an outside organization. In
case of suspicion, click on "Report to SAFRAN Security" from the
Outlook
ribbon.

RE your comment, ââ,¬Å"A neural network is software logic.ââ,¬Â

A neural Network is a component of an AI agent. An AI agent is a
suite of general purpose software tools. The agent is configured from a
dataset.

ChatGPT goes on (see dialog below): ââ,¬Å"Synthesizing an AI agent
involves training the machine learning model on a large
dataset.ââ,¬Â

This indicates that the training dataset determines the agentââ,¬â"¢s
behaviour (transfer function - inputs to outputs). In the case of a
Tesla motor
vehicle,
the inputs are camera sensors and the outputs will soon be controls
to the accelerator, brake and steering.

RE your comment, ââ,¬Å"The problem is not AI. It is the exhaustive
deterministic
relationship between the undesired losses and the system states.
Anything breaking this, breaks the current safety paradigm. And AI
is not the single thing to achieve this.ââ,¬Â

Could you please expand on this idea. Possibly using an example.

How would your thought apply here: I am a driver at the wheel of a
Tesla travelling at 110 km/h on the freeway. I have switched the
vehicle to full
self-
driving mode. I am fast asleep. The FSD agent detects a false
positive ââ,¬â?o
a lady
with a pram. It jams on the brakes initiating a rear-end collision
with a following semi.
Q: In this case how is it that, ââ,¬Å"The problem is not AIââ,¬Â

Les

ââ,¬â?Ã¢â,¬â?Ã¢â,¬â?Ã¢â,¬â?Ã¢â,¬â? ChatGPT Dialog
ââ,¬â?Ã¢â,¬â?Ã¢â,¬â?Ã¢â,¬â?Ã¢â,¬â?Ã¢â,¬â?

ChatGPT prompt:
ââ,¬Å"Describe an AI agent from a technical perspective. What are its
components
and
how is it synthesized?ââ,¬Â

Response:
An AI agent, from a technical perspective, typically consists of
several components that work together to perform specific tasks.
Here are the key components and how they are synthesized:

1. Data: The foundation of any AI agent is data. It can be in
various forms such as text, images, audio, or structured data. This
data is used for
training
and providing context to the AI model.

2. Machine Learning Model: The AI agent utilizes a machine learning
model
to
process and generate responses. These models are trained on vast
amounts of data and learn patterns to make predictions or generate
outputs.

3. Natural Language Processing (NLP): NLP is a crucial component for
understanding and processing human language. It involves tasks like
language translation, sentiment analysis, intent recognition, and
entity extraction.
NLP
enables the AI agent to comprehend user inputs and generate relevant
responses.

4. Knowledge Representation: Knowledge representation refers to how
information is stored and organized within the AI agent. It can
employ various
techniques
such as semantic networks, ontologies, or graph databases to store
facts
and
relationships. This allows the agent to access and utilize knowledge
when responding to user queries.

5. Dialogue Management: Dialogue management focuses on handling the
flow of conversation. It involves tracking the dialogue context,
managing user requests and responses, and generating meaningful and
coherent replies.
Reinforcement
learning techniques can be applied to improve dialogue management.

6. Integration and APIs: AI agents can integrate with external
systems and services using APIs (Application Programming
Interfaces). This allows them
to
access relevant data sources, databases, or external functionalities
to provide more comprehensive and accurate responses.

7. User Interface: The AI agent needs a user interface to interact
with
users
effectively. It can be a chat interface, voice-based interface, or a
combination of both. The UI should provide a seamless experience for
users to interact with the agent.

Synthesizing an AI agent involves training the machine learning
model on a large dataset. This process includes steps like
preprocessing the data, designing the model architecture, selecting
appropriate algorithms,
training
the model using optimization techniques, and fine-tuning it for
specific
tasks.
The trained model is then integrated with other components, such as
NLP, knowledge representation, and dialogue management, to create a
functioning AI agent. Regular updates, feedback loops, and
continuous improvement are essential to enhance the agent's
performance over time.

--------END Chat ----------


C2 - Confidential

The problem is not AI. It is the exhaustive deterministic
relationship
between the undesired losses and the system states. Anything
breaking this, breaks the current safety paradigm. And AI is not the
single thing to
achieve
this.

AttentionÃfâ?sÃ, : due to increased cybersecurity screening I may
receive
external
emails 30 minutes after their emission.
RESTRICTED

Bertrand RICQUE
Safety and security standardisation Expert Program Support Manager
Optronics and Defence Division/Customer Support Safran Electronics
& Defense

P +33 (0)1 58 11 96 82   M +33 (0)6 87 47 84 64
bertrand.ricque at safrangroup.com
102 Avenue de Paris
91300 MASSY FRANCE
https://urldefense.com/v3/__http://www.safran-electronics-
defense.com__;!!Dl6pPzL6!bg5nEqH6ID136htdDa-

DgyRz2IZw9arqA9HPKC3p01ZYHoQhzebcyOn5xrgApDNO52A_sLRMk2YgX63j
n5fk7M4$

-----Message d'origine-----
DeÃfâ?sÃ, : systemsafety <systemsafety-bounces at lists.techfak.uni-
bielefeld.de>
De la part de Phil Koopman
EnvoyÃfÆ'Ã,©Ãfâ?sÃ, : jeudi 22 juin 2023 03:32
ÃfÆ'ââ?s¬Ãfâ?sÃ, :
les at chambers.com.au; systemsafety at lists.techfak.uni-
bielefeld.de
ObjetÃfâ?sÃ, : Re: [SystemSafety] AI and the virtuous test Oracle

CAUTION:  This message originated from an outside organization.
In case
of
suspicion, click on "Report to SAFRAN Security" from the Outlook
ribbon.

Les,

Since you welcome riffs, I have something that is not as all-
encompassing,
but might have more immediate application.

I propose that to the degree that "AI" technology is deployed in
a way
that
supplants practical human judgement, the manufacturer of that
system
(in
some cases just the AI part if it is an add-on component) should
be
held
accountable for any action (or inaction) that, if associated
with the
human
that was supplanted, would have constituted negligence.Ãfâ?sÃ,Â
This
should
include
situations in which a human is put in an untenable situation of
supervising
an
AI in a way that puts unreasonable demands upon them, amounting
to a "moral crumple zone"
approach

(https://urldefense.com/v3/__https://papers.ssrn.com/sol3/papers.cfm
?ab


stract_id=2757236__;!!Dl6pPzL6!dV6V79CEWJVLcdXXS5n2wYWdaCGJCzdLlz4
gg9Cz063kcikC8CIr0YMf2lF9o5xNrnA0Av-DS0QOuOFaUivQZX7h$ ).
Liability/negligence if an AI is in substantive control of such
a
situation
should
attach to the manufacturer.

This leads to a more narrow oracle, but perhaps still useful,
than you propose. If a loss event is caused by a lack of
"reasonable" behavior
by an
AI,
the manufacturer is on the hook for negligence, and the
AI/manufacturer owes a duty of care the same as the human who
was supplanted would
have
owed to whoever/whatever might be affected by that negligence.
It has
the
advantage of reusing existing definitions of "reasonable person"
that
have
been hammered out over decades of law. (To be sure that is not
in the
form
of an engineering specification, but case law has a pretty
robust set
of
precedents, such as crashing into something after your properly
functioning
vehicle ran a red light is likely to lead to the driver being
found
negligent.)

This does not require the AI to behave the same as people, and
is not a
full
recipe for "safe" AI. But it puts a floor on things in a way
that is
readily
actionable using existing legal mechanisms and theories. If a
reasonable
person would have avoided a harm, any AI that fails to avoid the
harm
would
be negligent.

I've worked with a lawyer to propose this approach for automated
vehicles,
and it is starting to get some traction. What I write in this
post
(above)
is a
generalization of the concept beyond the narrow automated
vehicle application.
Details here:


https://urldefense.com/v3/__https://safeautonomy.blogspot.com/2023/0
5
/a-liability-approach-for-


automated.html__;!!Dl6pPzL6!dV6V79CEWJVLcdXXS5n2wYWdaCGJCzdLlz4gg
9Cz063kcikC8CIr0YMf2lF9o5xNrnA0Av-DS0QOuOFaUh295b5I$

-- Phil


On 6/21/2023 7:14 PM, Les Chambers wrote:
Hi All

I find myself reflecting on what will become of us.
As systems engineering best practice is overrun by AI.

Practitioners report that neural networks are eating code.
Example 1: The vector field surrounding a Tesla motor vehicle
is an output of a neural network, not the result of software
logic. Soon
the
neural net - not code - will generate controls. The size of
the code base is reducing.  (Elon
Musk)
Example 2: the ChatGPT transformer code base is only 2000 LOC
(Mo Gawdat
https://urldefense.com/v3/__https://youtu.be/bk-
nQ7HF6k4__;!!Dl6pPzL6!



dV6V79CEWJVLcdXXS5n2wYWdaCGJCzdLlz4gg9Cz063kcikC8CIr0YMf2lF9o5xN
rnA0Av
-DS0QOuOFaUpIawiVG$ )

The intelligence resides in terabytes of data, perceptrons and
millions of weighting parameters. All are gathered by
automated
means.
Not subject to human review.

Ergo what will become of our trusty barriers to dangerous
failure:
1. Safety functions - gone
2. Verification - gone
3. Code reviews - gone
4. Validation - How?

On validation, may I suggest the moral AI. A test oracle built
on a virtuous dataset, capable of interrogating the target
system to determine virtue. Test outcomes will morph from
pass/failure to
moral/immoral.

Credible industry players have predicted that soon we will
have AIs orders of magnitude smarter than us. Especially when
they start
talking
to
each other.
The bandwidth will be eye-watering - the increase in
intelligence,
vertical.

New barriers are required. Time to develop an AI that is on
our side
Ãf¢ââ?s‰â,¬Å"
the side of ethics and the moral life. An adult in the room if
you like. We should birth this creature now and raise it as good
parents.

Let us not panic. May I put the proposition: virtue, like
creativity, can be algorithmic.
I have a sense of starting from the beginning - tabula rasa. I
suggest
that high-level thinking on the subject could begin with
ChatGPT
prompts:
1. What is the stoic philosopherÃf¢ââ?s‰â?z¢s concept of
virtue?
2. What are the elements of philosophy relevant to AI?

Let us not forget our engineering mission: Guardians of the
divine Logos, the organizing principle of the universe,
responsible for its creation, maintenance, and order.

Would anyone care to riff on this?

Les

--

Les Chambers

les at chambers.com.au
systemsengineeringblog.com

+61 (0)412 648 992
_______________________________________________
The System Safety Mailing List
systemsafety at TechFak.Uni-Bielefeld.DE
Manage your subscription:
https://urldefense.com/v3/__https://lists.techfak.uni-
bielefeld.de/mai



lman/listinfo/systemsafety__;!!Dl6pPzL6!dV6V79CEWJVLcdXXS5n2wYWdaC
GJCz
dLlz4gg9Cz063kcikC8CIr0YMf2lF9o5xNrnA0Av-
DS0QOuOFaUjXMdlNF$


--
Prof. Phil Koopman   koopman at cmu.edu
(he/him)


https://urldefense.com/v3/__https://users.ece.cmu.edu/*koopman/__;fg!


!Dl6pPzL6!dV6V79CEWJVLcdXXS5n2wYWdaCGJCzdLlz4gg9Cz063kcikC8CIr0YM
f2lF9o5xNrnA0Av-DS0QOuOFaUnsUrevc$

_______________________________________________
The System Safety Mailing List
systemsafety at TechFak.Uni-Bielefeld.DE
Manage your subscription:
https://urldefense.com/v3/__https://lists.techfak.uni-


bielefeld.de/mailman/listinfo/systemsafety__;!!Dl6pPzL6!dV6V79CEWJVL
cd
XXS5n2wYWdaCGJCzdLlz4gg9Cz063kcikC8CIr0YMf2lF9o5xNrnA0Av-
DS0QOuOFaUjXMdlNF$

#
" Ce courriel et les documents qui lui sont joints peuvent
contenir des
informations confidentielles, ÃfÆ'Ã,ªtre soumis aux
rÃfÆ'Ã,¨glementations
relatives au
contrÃfÆ'Ã,´le des exportations ou ayant un caractÃfÆ'Ã,¨re
privÃfÆ'Ã,©.
S'ils ne
vous sont
pas destinÃfÆ'Ã,©s, nous vous signalons qu'il est strictement interdit
de les divulguer, de les reproduire ou d'en utiliser de quelque
maniÃfÆ'Ã,¨re que ce
soit
le contenu. Toute exportation ou rÃfÆ'Ã,©exportation non
autorisÃfÆ'Ã,©e
est interdite Si ce message vous a ÃfÆ'Ã,©tÃfÆ'Ã,© transmis par
erreur,
merci d'en informer l'expÃfÆ'Ã,©diteur et de supprimer
immÃfÆ'Ã,©diatement
de votre systÃfÆ'Ã,¨me informatique ce courriel ainsi que tous les
documents qui y sont attachÃfÆ'Ã,©s."
******
" This e-mail and any attached documents may contain confidential
or
proprietary information and may be subject to export control laws
and regulations. If you are not the intended recipient, you are
notified that
any
dissemination, copying of this e-mail and any attachments thereto or
use of their contents by any means whatsoever is strictly
prohibited. Unauthorized export or re-export is prohibited. If you
have received this e-mail in
error,
please advise the sender immediately and delete this e-mail and all
attached
documents from your computer system."
#



--

Les Chambers

les at chambers.com.au

+61 (0)412 648 992


#
" Ce courriel et les documents qui lui sont joints peuvent contenir
des
informations confidentielles, Ãfªtre soumis aux rÃf¨glementations
relatives au
contrÃf´le des exportations ou ayant un caractÃf¨re privÃf©. S'ils ne
vous sont
pas destinÃf©s, nous vous signalons qu'il est strictement interdit de les
divulguer, de les reproduire ou d'en utiliser de quelque maniÃf¨re que ce
soit
le contenu. Toute exportation ou rÃf©exportation non autorisÃf©e est
interdite Si ce message vous a Ãf©tÃf© transmis par erreur, merci d'en
informer l'expÃf©diteur et de supprimer immÃf©diatement de votre
systÃf¨me informatique ce courriel ainsi que tous les documents qui y sont
attachÃf©s."
******
" This e-mail and any attached documents may contain confidential or
proprietary information and may be subject to export control laws and
regulations. If you are not the intended recipient, you are notified that
any
dissemination, copying of this e-mail and any attachments thereto or use of
their contents by any means whatsoever is strictly prohibited. Unauthorized
export or re-export is prohibited. If you have received this e-mail in
error,
please advise the sender immediately and delete this e-mail and all
attached
documents from your computer system."
#



--

Les Chambers

les at chambers.com.au

+61 (0)412 648 992


#
" Ce courriel et les documents qui lui sont joints peuvent contenir des
informations confidentielles, être soumis aux règlementations relatives au
contrôle des exportations ou ayant un caractère privé. S'ils ne vous sont
pas destinés, nous vous signalons qu'il est strictement interdit de les
divulguer, de les reproduire ou d'en utiliser de quelque manière que ce soit
le contenu. Toute exportation ou réexportation non autorisée est interdite Si
ce message vous a été transmis par erreur, merci d'en informer l'expéditeur
et de supprimer immédiatement de votre système informatique ce courriel ainsi
que tous les documents qui y sont attachés."
******
" This e-mail and any attached documents may contain confidential or
proprietary information and may be subject to export control laws and
regulations. If you are not the intended recipient, you are notified that any
dissemination, copying of this e-mail and any attachments thereto or use of
their contents by any means whatsoever is strictly prohibited. Unauthorized
export or re-export is prohibited. If you have received this e-mail in error,
please advise the sender immediately and delete this e-mail and all attached
documents from your computer system."
#

--

Les Chambers

les at chambers.com.au

+61 (0)412 648 992
_______________________________________________
The System Safety Mailing List
systemsafety at TechFak.Uni-Bielefeld.DE
Manage your subscription: https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/pipermail/systemsafety/attachments/20230627/405b244b/attachment-0001.html>