[SystemSafety] a discursion stimulated by recent discussions of alleged safety-critical software faults in automobile software

[Les Chambers]

Peter

I made a brief reference to a very long story. The full report can be found
here: http://www.healthpayrollinquiry.qld.gov.au/.

It turns out that, in the interests of getting the problem fixed quickly,
the then Premier of Queensland signed a legal instrument that prevented the
Queensland government from suing the prime contractor in question. I am in
furious agreement that the enquiry would have been justified if there was
some possibility of recovering the hundreds of millions of dollars of wasted
public funds, however that was not to be. Instead the ensuing enquiry was
driven by the political need to placate the anger of the 78,000 employees of
Queensland Health who were adversely affected in some way. 

In general I believe that, in the context of complex systems development,
the punative: "punish them so they don't repeat offend" or the serial killer
approach: "take em off the streets so they can't reoffend" are thirty meter
metaphors - they look good from a distance but when you engage with them
close-up, as I have, they don't work. I have no doubt that these very large
organisations will all reoffend no matter how much you shame them, fine them
or jail them, not because they are evil empires but because they employ
fallible human beings. There are many repeating failure modes. One of the
most common is the sales function getting too much control over a bid
package resulting in a flight of fancy that gets turned into a fixed price
for a complex system that cannot be delivered for the peanuts offered - with
the resulting crashing of schedules, reduction in functionality and cutting
of corners especially in testing ... followed by very expensive failure in
use. The bad actors in these scenarios do not come ashore with battle axes
like Vikings ready to rape and pillage. They while way their incompetent
hours wreaking havoc, making poor decisions or no decisions and when things
turn pear shaped, take their redundancy payouts, get another job and start
again.  This just keeps happening and quite frankly IT WILL NOT DO!

I can offer several solutions that one day will happen because they must
happen. In organisations like NASA they have already happened because of the
obvious link between bad technology and death (astronauts have a marvellous
think-ahead "what can kill me next" attitude). Firstly our profession must
develop the equivalent  of NASA's flight rules. Rules for complex system
development and operation that must NEVER be broken. Graduates must leave
university with these rules ingrained to the point where they would rather
stick a fork in their eye than put 10,000 globals in a real-time
application. I hope you are doing your part here Peter. Secondly, systems
and software engineering management must gain and retain power over
technology implementation. We already have the concept of separation of
concerns in architectural design. Society has had the concept of the
executive, the legislature and the judiciary for centuries. The executive
sets policy and gives leadership, the legislature debates the details and
creates the laws and the judiciary enforces same. Corruption and societal
degradation usually occurs when one personality has power over all three.
This is a frequent feature of failed systems projects. 

Lastly I offer a dangerous idea: that some day, those with the knowledge to
create and maturity to manage complex systems may break free of their chains
of servitude and form a fifth estate. And in so doing save the planet. 

This is my contribution for the year. On Nov 24 I'm scheduled to set sail on
the sloop Northern Child from Las Palmas in the Canary islands bound for
Saint Lucia in the Bahamas. Northern Child's progress can be tracked at:
http://www.performanceyachtcharter.com/

I sincerely hope this technology works.

Cheers

Les

 

From: Peter Bernard Ladkin [mailto:ladkin at rvs.uni-bielefeld.de] 
Sent: Tuesday, November 12, 2013 4:53 PM
To: Les Chambers
Cc: Matthew Squair; Steve Tockey;
systemsafety at lists.techfak.uni-bielefeld.de
Subject: Re: [SystemSafety] a discursion stimulated by recent discussions of
alleged safety-critical software faults in automobile software

 

It seems worth while making again a point I have made before.

 

It is not about blame. Which, by the way, I wouldn't necessarily call an
emotion (Wikipedia, for example, thinks it is an act). It is about
assignment of responsibility for a deleterious event with a view to
dispensing compensation. This is a general principle of human behavior and
lawmaking for thousands of years and occurs in many if not all human
societies. I won't argue here the case for compensating people for harm you
have caused them. I'm glad we adhere to it and that I don't live 1600 years
ago.

 

So, if you are a 1970's hotel owner and a rock group trashes some of your
rooms, you are entitled to a determination of responsibility, and adequate
compensation from those deemed responsible. Since that will often be
disputed (likely not by a 1970's rock group, for which it was a source of
pride), it needs to be decided by the appropriate means, which for us is a
court of law.

 

It used to be the case in GB that hordes of foreigners came ashore from
boats, took what they wanted, trashed the restaurants as if they were
Bullingdon boys, and took women into slavery. They had to be fought off.
When this started being successful, they quit (apart from those who stayed,
which ruined their business model another way). Every three-year old who has
played in a sandbox knows this phenomenon, which manifestly does not stop
when one is older: John Kenneth Galbraith wrote about the power of large
corporations and the consequences for human society between 40 and 55 years
ago. So there is also another  point to this kind of action: resistance
stops other people doing stuff. 

 

Toyota knew they had spaghetti code in this acceleration-control kit. They
wrote so themselves, which you can see in the evidence. They also knew and
know the consequences of such complexity, namely a lack of control over the
behavioral properties of the program. That is also in the evidence. It
didn't stop them using the code again and again (it was still in the 2010
model year, apparently). That won't continue. For example, they have
recently signed a contract with Altran UK to develop examples of useful code
which is free of run-time error.

 

If you don't like principles of fairness and responsibility, and developed
organisations (the courts) with the power to set those principles of
fairness for everyone and every organisation without exception, just try
doing without it..........

 

PBL

Prof. Peter Bernard Ladkin, University of Bielefeld and Causalis Limited


On 12 Nov 2013, at 03:07, "Les Chambers" <les at chambers.com.au> wrote:

What bothers me is the alarming repeat performances we have of these
disasters. And the eye-watering sums of money spent on forensics and
retribution. These events are typically passed over to the legal profession
who proceed to dine out on the assignation of blame.......

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20131113/ff651a70/attachment-0001.html>