[SystemSafety] Computer Systems and the Law

[Phil Koopman]

The quantitative aspect of software failure rates is not the headline here.

In layperson terms, the issue is that presuming software is defect-free 
in effect puts the burden of proof on the accused to show a software 
defect caused a problem. This is especially problematic if the primary 
or only evidence of bad behavior is the output of that same potentially 
defective software.

Presuming that software can have defects can dramatically shift the 
burden of proof. Now there needs to be proof by whoever is accusing the 
person it was the person and was not the software. Especially in a 
criminal case, just a plausible scenario of software failure might be 
enough to introduce reasonable doubt without needing quantitative 
failure rate data for the specific software under consideration.

I'm not a lawyer, but have dealt with this aspect in many cases as an 
expert. For example, a driver might be accused of injuring another road 
user due to pressing the accelerator rather than the brake, and the data 
recordings were made by the same software which plausibly caused 
acceleration due to a software defect.

--Phil


On 2/5/2025 7:21 PM, Derek M Jones wrote:
> All,
>
>> The current government appears to be thinking differently. As Derek 
>> recently noted, the UK MoJ have just issued a Call for Evidence on 
>> it, indicating they are seriously considering legislating differently.
>
> So the courts finally agree that will software always contain
> coding mistakes that can produce a fault.
>
> Then what?
>
> I suspect that only a few vendors are only to be able
> to reliably produce statements like:
> "We estimate this software will experience 10^(-x) faults
> per hour of operation."
>
> Will vendors be asked to produce the number of reported
> faults every month for the last year, to give an indicator
> of reliability?
>
> Will software evidence always be assumed to be in error?
>

-- 
Phil Koopman    m: 412-260-5955    <phil.koopman at hushmail.com>