[SystemSafety] Autonomous Vehicle Safety

[Strigini, Lorenzo]

> On 17 Dec 2025, at 15:45, Derek M Jones <derek at knosof.co.uk> wrote:
> 
> Peter,
> 
> The authors make some simplistic assumptions, plug some
> numbers in to basic statistical formula, plot some pretty
> graphs, and then conclude that "...  testers cannot drive
> their way to safety."

Well, the authors (Kalra et al from RAND) make a straightforward argument with classical statistical inference: assuming invariance in a system and the way it is used (for a car: environment, pattern of driving etc.), what can you infer about how unlikely accidents are, after the system has operated for a while WITHOUT accidents? Whether they decided their conclusions before "picking numbers" to support it, I do not know. But their conclusion is right: if you use as your only evidence of safety just how much the vehicle has driven without accident, you need to wait a long time before you can claim the level of safety that can be acceptable for vehicles on public roads.

The argument was stated before a bit more in depth by Bev Littlewood and me in 1993 ( Validation of Ultrahigh Dependability for Software-Based Systems. Communications of the ACM, 36(11), pp. 69-80. doi: 10.1145/163359.163373; also https://openaccess.city.ac.uk/id/eprint/1251/ ; a minimal summary is at https://openaccess.city.ac.uk/id/eprint/276/ ). We framed the problem as one of Bayesian inference: how the probabilities that you assign to events (e.g., accidents) after seeing some relevant evidence (operation of the system) should improve over the probabilities you assigned before seeing it. 
We used the example of commercial aviation: to claim high confidence in a bound of 10^-9 probability of accident per hour, the limited amount of pre-certification operation added almost nothing to whatever one claimed before that amount of operation. We also argued that such claims were not plausible, based on the other forms of evidence used to justify them.

As Phil Koopman noted, the assumptions of invariance etc are normally wrong. Yet checking whether a claim would be satisfied at least in the best possible conditions (invariance of everything, so that all evidence collected is certainly relevant) helps one to understand how overoptimistic one's claims are. I think this useful, because we still see claims of having achieved great levels of safety based on seriously inadequate amounts of evidence.

My colleagues and I revisited the problem as stated by Kalra et al in this paper: https://openaccess.city.ac.uk/id/eprint/24779/ , again not in terms of ignoring all about a car except how many miles it has driven without accidents, but in terms of how much all this driving can add to whatever claims you could make before it. We see this as more realistic reasoning than theirs: nobody would develop a vehicle without abundant precautions to support pre-operation confidence in its safety. In later work we have looked at refining the reasoning, e.g. to take into account a change in the car or in its use. Yet strong claims remain hard to prove and confidence after seeing safe operation depends heavily on what confidence you can have before it.

Best regards,

    Lorenzo

> to meet the reliability required to meet the numbers plugged
> in by the authors.
> 
> No attempt to show how the numbers they used connect to
> human accident numbers.
> 
> To my non-automotive-expert eye, it looks like they decided
> on a conclusion and then picked numbers to validate it.
> 
>> https://www.sciencedirect.com/science/article/abs/pii/S0965856416302129
> 
> 2,049 citations on Google Scholar.
> 
>> On the other hand, since the first author is at RAND, maybe there is a report to download at no charge. Indeed so, through
>> 
>> https://www.rand.org/pubs/research_reports/RR1478.html
> 
> 6 citations, and no link to the paper.
> 
> --
> Derek M. Jones           Evidence-based software engineering
> blog:https://shape-of-code.com
> 
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
> Manage your subscription: https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety

__________________
Prof Lorenzo Strigini
Centre for Software Reliability
City St George’s, University of London
Phone: +44 (0)20 7040 8245
www.csr.city.ac.uk