[SystemSafety] Self-Driving Car Had a Fatal Accident ... (Norman, R 30 61 & many R 30.62)

[Matthew Squair]

Some work that the University of NSW is doing on the liability aspects of
driverless cars.

https://newsroom.unsw.edu.au/news/science-tech/who-blame-when-driverless-cars-have-accident?utm_source=Alumni+Express+Newsletter+%28APRIL+2018%29&utm_campaign=aa1be4651e-EMAIL_CAMPAIGN_2018_03_28&utm_medium=email&utm_term=0_e5cc5bf8d1-aa1be4651e-240644577


On 1 April 2018 at 6:38:43 pm, Peter Bernard Ladkin (ladkin at causalis.com)
wrote:

> People have been arguing in Risks about the Tempe accident, and citing
> numbers (fatalities per
> mile). Don Norman's post is at https://catless.ncl.ac.uk/Risks/30/61#subj1
> and a number of people in
> Risks 30.62 https://catless.ncl.ac.uk/Risks/30/62#subj13 are pointing out
> that he got his numbers
> wrong. Don did issue a correction
> https://catless.ncl.ac.uk/Risks/30/62#subj14 But they are all just
> playing with numbers also.
>
> It is a little odd to see colleagues ignoring the difference between
> statistics and numerology.
> Statistics involves drawing robust conclusions from data to a specified
> level of confidence.
> Numerology is doing arithmetic on numbers.
>
> One incident is not data, it is, at best, a datum. Consider: if the Risks
> discussion on the numbers
> associated with self-driving cars had taken place on March 17th, it would
> have "concluded" that
> self-driving cars were the safest form of road use ever known to mankind.
> On March 19th and since,
> people are "concluding" that it is an order or two of magnitude "worse"
> than human-driven automotive
> vehicles. The term "robust" is not applicable to inferences that flip so
> drastically between one day
> and the next.
>
> Not only that, but the technology of the accident vehicle may well change
> in response to the one
> event, such that that event would not have occurred with the newer
> technology (operating as
> designed). So those gathering numbers can start over again from 0. And
> this can go on and on.
> Indeed, there is an argument (see below) that (a) this is how commercial
> aviation has addressed
> digital automation, and (b) commercial aviation is by far the safest mode
> of travel by a number of
> measures, and (c) continues to improve its safety record for
> high-performance jet transports using
> any measure.
>
> To show the weaknesses of lumping rare events by general category, let me
> consider three points of
> view (please note that I am not advocating these points of view here: I am
> discussing them).
>
> First point of view: there has been one fatal accident with self-driving
> cars.
>
> There is one reasonably robust conclusion to be drawn from a sample of
> one, namely that the event
> was not impossible. I think we knew that anyway.
>
> Second point of view: there have been three fatal accidents with
> self-driving cars.
>
> There was a fatal accident on May 7, 2016 near Williston in Florida, when
> a Tesla Model S on
> "Autopilot" did not avoid a truck crossing the highway.
>
> There was a second one on March 23, 2018 in Mountain View, California.
> Tesla has said that the
> vehicle was travelling on "Autopilot" (see below).
>
> Information about the March 23rd accident is preliminary, but it seems as
> if in both cases, as in
> the Uber accident in Tempe, there was time for an alert human driver to
> register events and take
> avoiding action. There is apparently little or no evidence that either of
> them did.
>
> [The evidence: Tesla is saying that the March 23rd driver had about 5
> seconds and 150m of
> unobstructed view in which to intervene
> https://www.nytimes.com/2018/03/31/business/tesla-crash-autopilot.html
> The NTSB said that both the
> truck driver and the Tesla driver in the May 7th, 2016 collision had up to
> 10.4 seconds to register
> the other vehicle, and the truck's line of sight was 1,132 ft.
> https://www.ntsb.gov/investigations/AccidentReports/Reports/HAR1702.pdf ,
> section 2.2.2, p29.]
>
> So, it does look as though, in both Tesla cases, the car was de facto
> driving itself.
>
> Third view: there have been many fatal accidents with self-driving cars,
> many of them to cyclists
> and even groups of cyclists.
>
> This simple classification in fact applies to many road accidents. Drivers
> being distracted and not
> paying attention to road conditions is an old theme. In this situation,
> the car is self-driving. I
> remember a case of a woman searching for a music tape who hit a group of
> cyclists and killed a
> number of them, I think in Colorado, but I can't seem to find it on the
> WWW (can someone help?).
> Here is a recent one
> http://wishtv.com/2017/06/13/group-of-cyclists-mowed-down-by-driver-in-texas/
> .
> In these cases, there is no question that that the device being operated
> was not equipped to perform
> the required tasks itself.
>
> However, drawing conclusions using this classification is not going to
> tell us anything we didn't
> know about automated road vehicles.
>
> General Discussion.
>
> For robust statistical conclusions, classification of events is crucial.
> There seems a clear
> distinction between automobiles which require continuous human operation,
> and those which, under
> certain road conditions, do not require continuous human operation, and
> those which (are intended
> to) self-drive under all road conditions. The SAE has such a
> classification, with six categories:
> see p24 of
> https://www.ntsb.gov/investigations/AccidentReports/Reports/HAR1702.pdf .
>
> The Tesla model counts as SAE Level 2
> https://www.ntsb.gov/investigations/AccidentReports/Reports/HAR1702.pdf .
> I believe (but have not
> checked) that the Uber vehicle is SAE Level 5.
>
> There is thus a prima facie reason to consider the first view to prevail,
> namely that the SAE
> classification is widely used and we wish to draw conclusions about SAE
> Level 5 vehicles.
>
> However, there are reasons why one might wish to draw conclusions about
> vehicles in multiple
> categories, say Levels 2-5. For these, the second view is appropriate.
> Here are similarities.
> (1) these vehicles were all operating in a "self-driving" mode (if the
> most recent report is correct)
> (2) their minders all appear to have had time to detect and respond (way
> longer than the 1-2
> seconds required for people "in the loop" to respond to an unanticipated
> event).
> If nothing else, these similarities conform with observations about human
> supervisory control which
> have been in the literature for decades.
>
> In their report into the 2016 crash, the NTSB says "until automated
> vehicle systems mature, driver
> engagement remains integral to the automated driving system" (op. cit.
> Section 2.1.1, p27). The
> Board relates what Tesla's manual says of its "Autopilot" functions in
> Section 1.3.4, p13:
>
> [begin quote]
> .... with respect to the Autosteer system, the manual stated, “Warning:
> Autosteer is intended
> for use only on highways and limited-access roads with a fully attentive
> driver” (p. 74). In
> discussing restricted roads, the manual stated that “Autosteer is intended
> for use on freeways and
> highways where access is limited by entry and exit ramps” (p. 75). The
> manual also stated that
> “Autosteer is a hands-on feature. You must keep your hands on the steering
> wheel at all times” (p. 74).
> [end quote]
>
> The accident road in the 2016 crash did not have "access..limited by entry
> and exit ramps". Also,
> "The NTSB concludes that the Tesla’s automated vehicle control system was
> not designed to, and did
> not, identify the truck crossing the car’s path or recognize the impending
> crash....", (op. cit.,
> Section 2.2.4, p30. This is also part of Finding 3, Section 3.1, p41).
>
> In other words, a SAE Level 2 vehicle was being operated with automation
> in circumstances for which
> the automation was not designed or validated, and encountered an
> obstruction situation which it was
> not designed to recognise, and indeed did not recognise. One might add
> that the car was travelling
> (and had been programmed to travel) at 9mph over the posted limit. There
> are a lot of no-no's there.
>
> Indeed, overspeed is known to be a factor in many car accidents over the
> decades. Even to be the
> most common factor in fatal collisions. Is is banal, but nevertheless
> true, to say that, if all road
> regulations were adhered to by all road users, fatal accidents and
> accidents with serious injury
> would be, in comparison with today's figures, rare. Indeed, one main
> safety advantage proposed for
> SAE Level 5 vehicles is that, unlike human operators, they will adhere to
> all road-use regulations.
>
> As far as I know, the question is open (for the public) whether Uber's
> system is designed to detect
> Elaine Herzberg pushing her bicycle across the road in Tempe outside of
> defined crossing points.
> Which may be a different question from whether it is designed to detect
> slow-moving objects on
> conflicting trajectories. The latter is a general question, to which the
> Herzberg event belongs. But
> it may be that there are important characteristics of the Herzberg event
> which Uber's technology is
> not designed to detect. For example, some reports suggest there may be
> "blind spots" in the car's
> environmental sensing
>
> https://www.reuters.com/article/us-uber-selfdriving-sensors-insight/ubers-use-of-fewer-safety-sensors-prompts-questions-after-arizona-crash-idUSKBN1H337Q
> If there are such phenomena, we can presume that Uber will take action to
> mitigate them.
>
> So a SAE Level 5 vehicle will thereby become a different-technology SAE
> Level 5 vehicle to which the
> Herzberg event would not have happened. What happens then to any
> calculation of
> fatality-per-miles-driven? Rather than say "it's still Level 5 so the
> numbers apply", it might be
> more worth-while to subdivide categories: "Level 5 with sensing blind
> spots" versus "Level 5 without
> sensing blind spots". The regulators might get there first and not licence
> any technology for
> public-space trial which has any blind spots greater than such-and-such a
> size. That seems a more
> operative approach than calculating numerical averages over
> constantly-changing categories.
>
> Digitally-automated Commercial Aviation.
>
> Compare with the situation with computer-controlled commercial aircraft.
> The first was the Airbus
> A320, which suffered a fatal accident before it had been taken into
> service, on 1988-06-26. In its
> first five years in service, it had suffered 4. (Actually, five years and
> three months spanned these
> first four accidents, but the first happened technically before regular
> service introduction, on an
> introductory pleasure flight.)
>
> The A318/319/320/321 series has suffered 13 fatal accidents altogether,
> plus two deliberate hostile
> (and successful) actions, from which no commercial aircraft is immune. So,
> from 4 accidents in 5
> years to 13 accidents in nearly 30 years. From a few aircraft in service,
> to a few hundred, and now
> a few thousand. From O(10 exp 4) fleet miles to O(10 exp 8) fleet miles.
> (Note: I imagine the fleet
> miles today are around 100 million - they were around 55 million in 2007,
> as far as I remember.)
>
> Certainly, some statistics can be formulated here, but the point I wish to
> emphasise is that the
> most reliable and useful conclusions have been and are those drawn from
> detailed causal accounts of
> all those accidents (and indeed the two hostile actions), not from
> statistics.
>
> Consider a similar question to that which Norman and co. are attempting to
> answer. Is the A320
> series "safe", or "safer than <something else>"? It seems to me that that
> is too broad a question to
> be answered meaningfully as it stands. There is a lot of inappropriate
> crew handling spread across
> those 15 fatal events. One can ask if the crew is somehow induced by the
> aircraft to indulge in such
> inappropriate behaviour. That is a question which has been raised at every
> one of those 15
> investigations, and answered variously. And a classic study, series of
> studies, by Woods and Sarter
> identified the phenomenon of "mode confusion", which seems to have been
> all but eliminated nowadays
> through combinations of technical change, operating-procedural changes and
> training changes.
>
> The history validates this approach, which is to find out individually
> what might have been wrong,
> and devise and propagate mitigation. The A320 aircraft of today are
> different beasts in many subtle
> and not-so-subtle ways from those of 30 years ago. Statistical inference
> doesn't work very well
> across changing technology. And mitigation may not always address a causal
> factor directly. The
> response to a technological causal factor may well be altered crew
> behaviour (for example,
> concerning high-altitude pitot icing); the response to a human factor
> might well be technological
> (for example, a change to an interface to alleviate potential "mode
> confusion").
>
> PBL
>
> Peter Bernard Ladkin, Bielefeld, Germany
> www.rvs-bi.de
>
>
>
>
>
> ------------------------------
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20180403/e812c3e6/attachment-0001.html>