[SystemSafety] AI and the virtuous test Oracle - action now!

[Les Chambers]

Bernard
Re your comment: 
“I don't see how this makes much sense. There are no international bodies that 
regulate
software-intensive Safety-Critical systems (SISCS for short), except for IMO as 
far as I can tell.
Except for IMO, regulation occurs at the level of nation-states, or the EU 
(whose member states have
delegated certain regulatory activities to the EU in the sense that the EU 
writes directives that
are then taken into national law by the members).”
>>>>>>>
I am confused. Taking the case of IEC 61508, the IEC Mission Statement states:
IEC mission statement 
The mission of the IEC is to achieve worldwide use of IEC International 
Standards and Conformity Assessment systems to ensure the safety, efficiency, 
reliability and interoperability of electrical, electronic and information 
technologies, to enhance international trade, facilitate broad electricity 
access and enable a more sustainable world.

The IEC’s technical committee 65’s strategic business plan states:
SC65A: SYSTEM ASPECTS
To prepare international standards regarding the generic aspects of systems 
used in industrial process measurement, control and manufacturing automation: 
operational conditions (including EMC), methodology for the assessment of 
systems, functional safety, etc. 
SC65A also has a safety pilot function to prepare standards dealing with 
functional safety of electrical/electronic/programmable electronic systems. 

The terms “worldwide use” and “International standards” imply International 
scope. The term “Conformity Assessment” implies regulation.

In one sense you are correct. Non-conformity with standards, promulgated by 
organisations such as the IEC and CENELEC do not trigger a visit from 
compliance police attached to these organisations. You are not handcuffed and 
taken down; you are not sent scary solicitors letters threatening economic 
doom. The consequences are, in fact, more dire. You don't get paid. 
In my experience, the regulation occurs as a second-order effect. Examples: 
Major customers I have dealt with: the Hong Kong Mass Transit Railway (MTR) and 
the Taiwan High Speed Rail Corporation (THSRC) delight in referencing 
standards, such as EN 50128 and IEC 61508 in the compliance section of 
contracts. It's a fair and reasonable strategy, these organisations are tasked 
with keeping the public safe. Having called out a safety standard they feel 
they have done their job. But it's never that simple.

As a bog standard U-Haul systems engineer with dirty fingernails from too many 
years at the coalface (>40), I report aspects of the net result of standards 
compliance as a condition of contract as follows:
1. General confusion amongst contractors (the immature ones that is). There is 
this pathological habit of ticking all the boxes, with gay abandon, in the 
compliance section of a tender response. After all, “We are a competent 
company, how oppressive could these standards be?” One horror story was the 
company that bid $2 million, and won the job only to discover that the “comply” 
tick they had placed against EN 50128 was going to cost them an extra million 
unfunded dollars.
2. Organisational change. A verification and validation group is often added. 
Safety managers, quality managers, and configuration managers appear. More 
processes, more people, more money. 
3. Consultant expense. People such as me are hired to help parse the often 
unintelligible (from the contractor’s point of view) prose in these standards.
4. Life cycle processes are redesigned; procedures are rewritten. 
5. HR issues arise. Your bog-standard U-Haul developer often runs screaming 
from the room when he finds out what he has to do. Some complain endlessly 
about the oppressive nature of these standards. One notable example inspired a 
colleague comment, “Jeez [Fred], it's like we've gotta load your brain with 
this stuff every morning.” Fred replied, “AH but every night I reset!”
6. Political games are played over standards compliance. I've seen one company, 
at 70% complete, announce to the client that they weren't gonna comply.  It was 
high noon with a $13 billion project in jeopardy. The customer caved and the 
contractor did it his own, utterly non-compliant, way. In another case, both 
the customer and the contractor decided to look the other way because an 
immovable delivery date had been promised to the public.
7. People lose their jobs for ticking the wrong box.
8. The standards pain has ended in gain for many of the companies I’ve worked 
for. It's improved their maturity and made them eligible for the painless 
execution of high-value projects. 

ERGO if you announced to any of these players that, “There are no international 
bodies that regulate
software-intensive Safety-Critical systems,” you'd be faced with robust 
disagreement and the odd belly laugh (at least east of Suez - I can't speak for 
the Americans or Europeans). The players faced with standards of compliance 
could care less that the effect is first, second, or Nth order, they feel, and 
are, COMPREHENSIVELY REGULATED!

I sincerely hope that the members of the various standards working groups (for 
example, technical committee 65 working group 10 - IEC 61508), are acutely 
aware of the impact their work has on the international engineering community. 
Every paragraph of these standards has a ripple effect across the planet, 
whenever a client of any stripe (not only government) chooses to attach a 
compliance standard to a contract. The confusion and waste of time over 
compliance standards that I have experienced in my career could be easily 
solved by a simple review technique. Before any standard is released into the 
wild, every normative paragraph should be reviewed as follows:
1. Is it complete?
2. Is it correct?
3. Is it unambiguous? 
4. Is it understandable by a professional engineer with the correct training? 
What training is required? 
5. Is it expressed such that an assessor can easily judge compliance or non-
compliance (just as a test must be designed, such that its result can be easily 
classified as either pass or fail).
6. Informative paragraphs of the standards REGULATING Safety-Critical systems 
development should make it very clear that Safety-Critical systems are 
expensive and take longer to build, require highly skilled professional 
developers and “mature” organisations. You could even include project cost 
models and skill sets required.

Bernard, your lifetime contribution to Safety-Critical standards should be 
acknowledged, along with other working group members who give their time at no 
charge. The profession, in the long run, is better off. I wonder, though, is 
there a need to close the loop on these standards? For example, Sam Altman of 
Open AI is currently touring the world gauging reaction to the impact of 
ChatGPT on the planet Earth. I wonder, has anyone from, say, working group 10 
ever got on a plane and talked to 100 development organisations who have been 
impacted by your work products - with a view to improving the usability of 
same?

Cheers

Les

Informative:

————— Who are MTR and THSRC ————
The Hong Kong Mass Transit Railway (MTR) is operated by the MTR Corporation 
Limited, which is a publicly listed company in Hong Kong. It is responsible for 
the construction and operation of the MTR system, including trains, tracks, and 
stations. While the Hong Kong government has a majority ownership stake in the 
MTR Corporation, it operates as a separate entity with its own management and 
board of directors.

Taiwan High Speed Rail Corporation (THSRC) is a private company responsible for 
operating the Taiwan High Speed Rail (THSR) network. It was established as a 
private entity through a build-operate-transfer (BOT) agreement with the 
government. While the government is a major shareholder in THSRC, 

———— CENELEC ————
CENELEC, based in Brussels, is the European Committee for Electrotechnical 
Standardization, a non-profit organization that develops standards for 
electrical engineering, electronics, and related technologies in Europe1.
EN 50128 is a certification standard issued by CENELEC that specifies the 
process and technical requirements for the development of software for 
programmable electronic systems for use in railway control and protection 
applications213.
The standard applies to all safety-related software used in railway systems, 
such as communication, signalling, and processing systems24. The standard also 
addresses the use of pre-existing software and tools, as well as generic 
software that can be configured by data or algorithms for specific 
applications24.
The standard was first published in 2000 and revised in 2011 and 2020243. The 
standard is aligned with the international standard IEC 622793. The standard 
defines five software safety integrity levels (SSILs) from SSIL0 to SSIL4, 
corresponding to the risk of software failure on the system safety243.

———— ISO IEC ———-
ISO is the International Organization for Standardization, a non-governmental 
organization that develops and publishes international standards for various 
fields and sectors12.
IEC 61508 is an international standard published by the International 
Electrotechnical Commission (IEC), which is a sister organization of ISO that 
focuses on standards for electrical, electronic, and related technologies123.
ISO and IEC have a joint technical committee (JTC 1) that collaborates on 
standards for information technology, including IEC 61508124. Therefore, IEC 
61508 is also published by ISO as ISO/IEC 61508124.
IEC 61508 is a functional safety standard for electrical, electronic, or 
programmable electronic safety-related systems. It covers the entire life cycle 
of the systems, from concept to decommissioning. It defines safety requirements 
and risk reduction measures based on Safety Integrity Levels (SIL).




> Les,
> 
> On 2023-06-27 06:15 , Les Chambers wrote:
> > ..... international bodies
> > that currently regulate software-intensive Safety-Critical systems - who 
cling
> > to regulating processes that have ceased to exist - are likely to be 
overrun
> > and made redundant.
> 
> I don't see how this makes much sense. There are no international bodies that 
regulate 
> software-intensive Safety-Critical systems (SISCS for short), except for IMO 
as far as I can tell. 
> Except for IMO, regulation occurs at the level of nation-states, or the EU 
(whose member states have 
> delegated certain regulatory activities to the EU in the sense that the EU 
writes directives that 
> are then taken into national law by the members).
> 
> And as far as IMO goes, the level of SISCS in large ocean-going vessels seems 
to be of somewhat 
> limited effect on the hazards of shipping (though I am open to 
reconsidering).
> 
> I don't know what "processes that have ceased to exist" you might be 
referring to; can you say?
> 
> Hazard and risk analysis (HRA) is regarded by IEC and ISO as key to standards 
involving safety 
> considerations - that is explicitly what Guide 51 says - and Guide 51 says 
HRA shall be required in 
> such standards, and tells us what it is. The regulation in many states of 
SISCS depends upon 
> adherence to such standards. I don't see that the emergence of ML-based 
subsystems affects a 
> requirement for HRA much at all - but I do see that traditional HRA is put in 
a quandary by how to 
> evaluate systems with ML-based subsystems. The informal development standards 
applied by ML 
> subsystem developers (often called "AI safety") don't work in traditional HRA 
assessments - rather, 
> they do nominally work and rule ML-based subsystems out because reliability 
calculations are not 
> possible.
> 
> However, I do see that there is considerably commercial pressure to approve 
safety-critical software 
> which essentially uses ML-based subsystems for pervasive use, in particular 
in the road-vehicle 
> sector, despite the lack of reliability assessment. But here there are, yes, 
regulatory hurdles. As 
> well as considerable scepticism amongst many engineers. Not helped, of 
course, by revelations such 
> as those by Handelsblatt, which suggests that Tesla knows of way more 
problems with its "Autopilot" 
> SW than have been made public (Handelsblatt got hold of gigabytes of customer 
reports).
> 
> > In favour of organisations such as:
> > 
> > - The Center for Human-Compatible AI at UC Berkeley
> > - The Future of Life Institute
> > - The Center for AI Safety (CAIS)
> > - Stanford Center for AI Safety
> 
> Can you name any reports on the reliability assessment of, say, control 
systems involving ML-based 
> subsystems that any of those institutions have published? (There are quite a 
few such reports 
> around, but those institutions are not where they come from.)
> 
> > .... This is a major
> > inflection point in the evolution of intelligence. Carbon hosts will always 
be
> > limited; silicon is unbounded.
> Well, ChatGPT and its emergent behaviour certainly made the headlines 
recently. It's not new to me. 
> I've been working on two projects since 2017 with language models based on 
word embedding (invented 
> by Google ten years ago: Mikolov, Chen, Corrado and Dean). OpenAI and Google 
and Meta upped the 
> scale and changed the application somewhat in 2021-2022, and then OpenAI puts 
a conversation bot on 
> the open Internet and everybody goes bonkers. Because, rather than just a few 
devoted people (say, 
> at the institutions you name) thinking about issues with chatbots, millions 
of people suddenly are.
> 
> It does seem worth emphasising that Chatbots based on word-embedding 
technology and control systems 
> designed around ML-based environment-interpretation subsystems are two almost 
completely different 
> technologies. What they have in common is ML technology.
> 
> The reason that word-embedding technology made what seems to be a quantum 
leap is the existence of 
> huge corpora. You can train these things, if you wish, on more or less all 
the language that has 
> ever been written down. And OpenAI (and maybe Google and Meta) did. Reported 
to have cost 
> nine-figure sums of money. The CEO of OpenAI has said openly (and I believe 
him) that that is not a 
> sustainable development model. Not necessarily for the cost, for there is 
lots of that kind of money 
> in the world, but for the effort involved and the very special case of the 
entire environment being 
> available (a universal corpus, as it were). Whereas the environment for road 
vehicle operation is 
> not similarly available. It is also vastly more complex, as far as anyone can 
tell. We can't even 
> say what it is. (Whereas conceptualising a corpus is something people have 
been able to do for 
> millenia.) Apple and Google and who knows else have been training their road 
vehicle technology on 
> public roads for well over the decade it took from the invention of word-
embedding technology to the 
> emergence of ChatGPT, and they are nowhere near "prime time" yet.
> 
> Further, I think you're wrong on the silicon level. There are hard physical 
limits to the 
> development of current digital-computational processing units. Moore's Law 
cannot be open-ended. 
> Many HW developers have pointed out we are reaching limits. I would be much 
more inclined to 
> consider an "inflection point" when/if people get quantum computing to work. 
(I won't reiterate that 
> well-rehearsed reasoning here.)
> 
> What does interest me is the political inflection point, if I may term it 
that. FLI put out its 
> Slaughterbot video some years ago, and people such as Stuart Russell tried to 
get everyone to take 
> it very seriously. We can thank our lucky stars that no capable national 
militaries seem to have 
> taken it particularly seriously, for if they had we could well be in a world-
wide political crisis 
> in which no influential politician or national executive in any country could 
ever be seen in the 
> open air ever again. Slaughterbot and similar threats have little to do with 
"intelligence", just 
> with the capabilities of technology developed by people whom society has put 
in category of "AI 
> research". But put a Chatbot on the Internet and all of a sudden the sky is 
falling.
> 
> PBL
> 
> Prof. i.R. Dr. Peter Bernard Ladkin, Bielefeld, Germany
> Tel+msg +49 (0)521 880 7319  www.rvs-bi.de



--

Les Chambers

les at chambers.com.au

+61 (0)412 648 992