[SystemSafety] a public beta phase ???

Mon Jul 18 16:32:08 CEST 2016

Good afternoon Les,

> The argument that 33,000 people are killed in accidents every year, so why
should we care, is also
> drivel. None of these fatalities occurred because a driver trusted a
system that couldn't be trusted.

Do you any evidence to back up that claim? For example can you show that no
one was harmed because a lane keep feature failed to keep a vehicle in lane
or that no one was killed because they became over reliant of features such
as emergency brake assist or any one of the dozen or so driver assist
systems that the current crop of vehicles have (see video below)? I know
it's nearly impossible to prove a negative but I'm also pretty sure that if
we had access to all the relevant data we'd probably find someone, somewhere
who rolled their vehicle because of overconfidence in their ESC system. But
we'd never know because a) the data hasn't been collected on a routine basis
and b) vehicle manufactures don't in general have an mechanism to collect
such data. At least in this regard Tesla seems to be ahead of the game and I
suspect that if Tesla hadn't asked the NHTSA to investigate it's possible we
may never had found out about this either.

The problem here is that absence of evidence isn't the same evidence of
absence. Current systems may have issues we (or anyone) know about and I
know of at least one failure mechanism for ABS that would fool most
implementations from 10 years back (it may not now - I don't know) but ABS
(a driver aid) is mandatory in Europe as is ESC because statistically it's
believed to save lives. Neither provides a guarantee of either safety or
that the system will work all the time; but these days I wouldn't buy a car
without them.

The following video demonstrates some of the benefits of standard systems
(ABS, ESC,TC)
https://www.youtube.com/watch?v=wR1SSxpKitE

And I found this which looks at AEB (autonomous emergency braking) and some
success and some failures.
https://www.youtube.com/watch?v=E_ZNG8cmnlw

Note the driver involved in the tests was British Touring Car champion and
has held an F1 super licence so he actually does know how to drive.

> The media needs to maintain the rage and keep reporting self driving car
fatalities.

Rage is the wrong concept. Rather the media needs to track the issue and
report it in some sort of sane balanced manner, or at least as sane and
balanced as the media seem to be able to muster these days. We should do
likewise.

> Every time you receive a software upgrade in your garage the safety claims
made on your current version
> minus one are null and void. The game starts over again. You drive out the
gate. You roll the dice.
> Thousands of lines of code have been changed, the potential for screwups
is high, exacerbated by the massive
> complexity of these AI fuelled applications.

Only the first and last sentences here are strictly correct, the rest to
some extent is exaggerated.

First for the driver is not necessarily the case the game starts completely
from zero. Tesla have stated that before release they have a lab test
program and they test the software in their own test fleet. I have no idea
what this comprises in detail or how much testing is done but at the level
we have visible information that is reasonably comparable with normal
practice in automotive.

Likewise to state that 1000's lines have been changed is hyperbole - we have
no visibility of what was changed. It might be 1000's of lines, it might be
one (same effect) or it may be zero, for example if it's an update to a
neural network then is could be zero code lines.

> exacerbated by the massive complexity of these AI fuelled applications.

To my mind that is the one point really on the money - how do you cope with
the complexity of AI based systems? You could potentially formally prove the
inference engines and perhaps much of the rest of the system but has been
noted elsewhere the specification of say a neural network or other
stochastically based system is a combination of the size and completeness of
the training AND test sets.

Questions that need to be addressed are
- how big is big enough for either set?
- how do you quantify diversity in either set?
- can you improve on a simple division of training/tests or are there ways
to use one test set in both ways, e.g. develop the test set from the
training set by changing vehicle colours/background etc.

> And furthermore, it takes years for an organisation to develop an
effective safety culture, matter cannot move
> faster than the speed of light nor can Tesla develop a culture that would
rival that of NASA or the aircraft
> industries in the short time they've been in business.

Neither of the two examples has an unblemished record as regards safety,
NASA have overseen the loss of several man carrying spacecraft i.e. Apollo 1
and two shuttles. Recently the aviation industry has given us battery fires
in the 777 and what could only be described as an imaginary safety case on
behalf of the Nimrod. Experience does not necessarily equate to safety it's
something that has to be continually worked on. Whether or not Tesla have it
I'm not sure we can say at this point in time.

Some of the things that do annoy me about reporting and discussion of this
event are:

1. The Tesla in question appears to not have noticed it was involved in a
crash and to have continued down the road without its roof until it ran off
into a field. That doesn't seem to be an appropriate reaction to an accident
and potentially it's worse than the original accident.

2. As is usual in the USA (but not Europe) the truck was not fitted with
side impact bars so a) to the Tesla's radar the road appeared clear (it
apparently classified the side of the vehicle as a road sign) and b) if the
vehicle had run into the bars the impact would possibly have been less bad.
It has been commented elsewhere that that fitting of under-run bars would
prevent 250 deaths a year in the US but after decades of trying it's still
not mandatory.

-----Original Message-----
From: systemsafety
[mailto:systemsafety-bounces at lists.techfak.uni-bielefeld.de] On Behalf Of
Les Chambers
Sent: 18 July 2016 12:15
To: 'Peter Bernard Ladkin'; systemsafety at lists.techfak.uni-bielefeld.de
Subject: Re: [SystemSafety] a public beta phase ???

PBL
John Naughton's article in the Guardian is not sensible, it is uninformed,
illogical and flat out wrong. Obviously written by a person has never had
the experience of writing code that could kill someone; never had to take a
community of untrained operators and put a highly automated system in their
hands from a starting position of total ignorance.
Telling a driver to behave responsibly and keep their hands on the wheel is
a bit like telling a gambler to gamble responsibly. If a car can drive
itself, untrained drivers will take advantage of this feature and put too
much trust in what is currently an untrustworthy system. Drivers put through
focused training will take these warnings seriously. Your average bunny who
can afford a Tesla but has had no training will not. 
The argument that 33,000 people are killed in accidents every year, so why
should we care, is also drivel. None of these fatalities occurred because a
driver trusted a system that couldn't be trusted. 
And lastly RE: Naughton's comment that "... mainstream media will have to
change the way they report self driving cars. Every time a Tesla or a Google
car is involved in a crash, by all means report it. But also report all the
human error crashes that occurred on the same day." Not so. The media needs
to maintain the rage and keep reporting self driving car fatalities. This is
probably the only way we will get the message through to the general public
that if you buy one of these cars you are taking a substantial risk. Every
time you receive a software upgrade in your garage the safety claims made on
your current version minus one are null and void. The game starts over
again. You drive out the gate. You roll the dice. Thousands of lines of code
have been changed, the potential for screwups is high, exacerbated by the
massive complexity of these AI fuelled applications. This is the new normal,
we are now beta testing safety critical systems on the public. PBL you might
as well put a clause in 61508 two okay this behaviour.
And furthermore, it takes years for an organisation to develop an effective
safety culture, matter cannot move faster than the speed of light nor can
Tesla develop a culture that would rival that of NASA or the aircraft
industries in the short time they've been in business. System safety has one
source: motivated people and it takes years to develop that motivation.
They and we will get there eventually but in the meantime the public has a
right to be made aware of the risks they are taking with these vehicles.

Les

-----Original Message-----
From: systemsafety
[mailto:systemsafety-bounces at lists.techfak.uni-bielefeld.de] On Behalf Of
Peter Bernard Ladkin
Sent: Sunday, July 17, 2016 7:29 PM
To: systemsafety at lists.techfak.uni-bielefeld.de
Subject: Re: [SystemSafety] a public beta phase ???

A very sensible comment from John Naughton today in The Observer

https://www.theguardian.com/commentisfree/2016/jul/17/self-driving-car-crash
-proves-nothing-tesla-autopilot

PBL

Prof. Peter Bernard Ladkin, Bielefeld, Germany MoreInCommon Je suis Charlie
Tel+msg +49 (0)521 880 7319  www.rvs-bi.de

_______________________________________________
The System Safety Mailing List
systemsafety at TechFak.Uni-Bielefeld.DE

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus