[SystemSafety] NTSB report on Über fatal accident in Tempe

[Peter Bernard Ladkin]

.. is apparently available, presumably in final-draft, and someone from Ars Technica has seen it.

https://arstechnica.com/cars/2019/11/how-terrible-software-design-decisions-led-to-ubers-deadly-2018-crash/

Looking at the NTSB WWW site (home page), there is a Board meeting on the accident scheduled for
19th November, so the report will not have been publicly released yet.

The ars technica article focuses on two aspects of the system.

1. First, the classification of sensed objects.

a. Something is said about the system not recognising a category of "jaywalking pedestrian". For our
non-US readers, an explanation of the term... In most (all?) US cities, built largely on the
square-block principle, traffic regulations only allow pedestrians to cross a road at a road
intersection. Such intersections are often governed by traffic lights, which include pedestrian
lights. A pedestrian crossing a road elsewhere is said to be "jaywalking". Apparently the Über
system didn't have an object class for "pedestrian crossing the road at a non-pedestrian-crossing".

b. Let T be the time of collision in seconds. After sensing the "object", which was a
pedestrian+bicycle at T-5.2, it was classified as "other", or "vehicle", alternating rapidly, then
"bicycle", then "unknown", then "bicycle". The system computes a trajectory for tracked objects, but
it recomputes the trajectory from zero each time an object is classified. Which has the effect that
if an object is being multiply reclassified in a oscillatory fashion, there is no reliable
trajectory information. There can thus be no rule such as "whatever we think it is, it is moving so
as we're going to hit it".

2. Second, the avoidance algorithms. As is known, Über turned off the Volvo system, because the
radar frequencies in Über's system were the same as or close to those of the Volvo system and the
two could interfere. A plausible reason for turning one of them off, maybe, but a better decision
surely would have been to use different frequencies.

3. Third, the Über system engages in "action suppression". The system doesn't react immediately. A
number of possible reasons are given for this, some good, some poor. But the result in this case is
that emergency braking was only initiated a fifth of a second before impact.

PBL comments:
Ad 1a: Surely a HazardID process would have recognised an inappropriate system of object categories.
It is hard to believe that the classification system didn't have a category for "pedestrian outside
of pedestrian-crossing", since, along with cyclists, pedestrians are a known category of "vulnerable
road user", whose casualty rates in almost all lands have been rising, at the same time as car/truck
casualty rates are reducing, e.g.
http://www.euro.who.int/__data/assets/pdf_file/0004/98779/polbrief_road_injuries.pdf .
Ad 1b: Duuh. Also something which a decent HazID process would have flagged.
Ad 2: There are actually good reasons for turning off one of two duplicate safety systems. If you
let them both run, then you have to analyse all the interactions to see if there are any hazardous
ones, and that can be a monster task, as anyone who has tried to enumerate and analyse the product
of two state machines will know, as well as being hard. One the other hand, if you turn off one of
them, there are surely going to be situations which the system operating does not handle as well as
the system not operating. As in this case. So I don't see point 2 particularly negatively.
Ad 3: Some of the reasons given are poor, such as not wishing to jerk around your CEO too much in a
test ride. But there are surely technical reasons also. For example, if you are
emergency-manoeuvring often in (to humans) spurious situations, you are likely to increase the rate
of collision with other road users. For example, slamming on the brakes in situations in which
humans wouldn't are likely to lead to lots of tail-enders in traffic. Besides, but not
safety-related, you are not going to enamour any riders if you are jerking them around in too many
false-positive emergencies, and there is no point in the system if no one wants to ride it.

PBL

Prof. Peter Bernard Ladkin, Bielefeld, Germany
MoreInCommon
Je suis Charlie
Tel+msg +49 (0)521 880 7319  www.rvs-bi.de





-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20191108/1becfe17/attachment.sig>