[SystemSafety] Computer Systems and the Law

[Prof. Dr. Peter Bernard Ladkin]

On 2025-02-06 00:08 , Phil Koopman wrote:
> The quantitative aspect of software failure rates is not the headline here.
>
> In layperson terms, the issue is that presuming software is defect-free in effect puts the burden 
> of proof on the accused to show a software defect caused a problem. This is especially problematic 
> if the primary or only evidence of bad behavior is the output of that same potentially defective 
> software.
>
> Presuming that software can have defects can dramatically shift the burden of proof.

In my view Phil is quite correct that where the burden of proof should lie is the main issue that 
lawyers and lawmakers are dealing with here.

Some history is worth knowing. I recommend both Ladkin et al "The Law Commission Presumption ...." 
and Marshall et al "Recommendations for the Probity....." of which I have already distributed the 
URLs. As well as James Christie's 2023 paper dealing with the Law Commission's deliberations on this 
matter in the 1990's.

The history is that the Police and Criminal Evidence Act (PACE) 1984 Section 69 said it was up to 
the party introducing computer/computer-generated evidence to demonstrate that the computer was 
working "properly" at the relevant time. This was seen by much of the legal profession as a 
considerable burden, which it arguably is.

On the other hand, it did allow Ross Anderson and his colleagues to get John Munden off, in the mid 
1990's. Munden was an policeman whose ATM card (a magstripe I seem to recall) "suddenly" turned up 
in (I think it was) Northern Ireland and was used to withdraw lots of money. Except it was with him 
at home in Cambridgeshire. He complained to the bank and wanted restitution of the cash. Instead, 
the bank prosecuted him for attempted fraud. He lost the case; he lost his job. Luckily, his 
neighbour was Ross Anderson. With Ross's expert help, the case went to appeal. There are of course 
all kinds of people dealing with ATMs who, back in the old magstripe days, could fiddle with the 
machines, siphon off customer data and use that data to withdraw cash. The bank had claimed that was 
impossible. Munden's team claimed the right to inspection of the software code (following S. 69, the 
bank had to show it was correct), which the appeal court upheld. The bank refused to release the 
code to Munden's team and his conviction was thereby set aside. "Good outcome" except that it cost 
Mr. Munden four years of his life, and his job.

The Law Commission reconsidered S. 69 in reports in 1995 and 1997. Ladkin et al discuss the 1997 
report in detail (although the URL given in the paper is no longer valid. The new one is not hard to 
find). The 1997 report recommended repeal of S. 69. There is a common law presumption, it pointed 
out, that engineered systems are working properly at the relevant time unless there is evidence that 
they weren't. PACE S. 69 was repealed in 1999.

Then came the slew of Post Office prosecutions of subpostmasters for accounting discrepancies as a 
result of using Horizon. There were two common aspects to many/most of these prosecutions. First of 
all, the PO prosecutors said Horizon works at ten+ thousand installations all over the country (and 
Wales and Scotland) without a problem. The accused has discrepancy after discrepancy. So they must 
be stealing/cooking the books/whatever malfeasance. (I recommend reading the transcript of Seema 
Misra's trial on DEESLR to see this line of argument in action. Vol 12, 2015 
https://journals.sas.ac.uk/deeslr/issue/view/328  Kudos to Stephen Mason for the effort he put in to 
obtain the transcript and get permission to publish it. I have another paper in DEESLR Vol 17 2020 
called "Robustness in Software" which looks at this argument in detail.)

It turns out that the Post Office and its expert witnesses knew quite well of all sorts of errors 
that Horizon perpetrated -- there was a Known Error Log -- indeed they also knew that maintenance 
staff at the software company Fujitsu could go in and had gone in to alter sub post office 
bookkeeping figures without knowledge of the subpostmaster. They should have disclosed all this to 
the defence at any of the trials. And didn't. Accused subpostmasters didn't know what to ask for in 
the way of documentation. Attempts to get it were regarded as "fishing expeditions". Seema Misra was 
denied disclosure three times before she was convicted and sent to jail.

Sir Peter Fraser blew this all open with his 2019 judgement Bates v Post Office Limited No. 6: 
Horizon Issues in which he detailed a long list of Horizon bugs that were clearly relevant to what 
had been going on with accused subpostmasters. There is reason this is called a scandal -- before 
and during the trial, POL was trying to deny there was anything such as the Known Error Log for 
Horizon, that Fujitsu had been keeping for, oh, some twenty years.

There are two large issues here. One is multitudinous failures of disclosure. The other is the 
common law presumption (which I like to capitalise as the Presumption). There is disagreement 
amongst lawyers and lawmakers as to which is the "main issue" in the Horizon scandal. Some think it 
was failures of disclosure and the Presumption is OK. Others think the Presumption is inappropriate 
(so do I and my coauthors).

Paul Marshall was asked by the previous UK government in 2020 to recommend what might replace the 
Presumption. He consulted us and that is the provenance of "Recommendations for the probity...." But 
that government decided not to act. Various lawmakers, notably Lord Arbuthnot and Baroness Kidron, 
have since raised the matter in the Lords, with amendments and suchlike. The current government has 
decided to invite evidence on replacing the Presumption with new legislation. Hence the Call for 
Evidence to which Derek drew our attention.


PBL

Prof. Dr. Peter Bernard Ladkin
Causalis Limited/Causalis IngenieurGmbH, Bielefeld, Germany
Tel: +49 (0)521 3 29 31 00