[SystemSafety] Failure Modes in ML
Peter Bernard Ladkin
ladkin at causalis.com
Tue Dec 17 09:24:07 CET 2019
On 2019-12-17 00:31 , Bruce Hunter wrote:
>
>
> Victoria Krakovna also maintains an AI safety resources page which includes database records of AI
> failures. This is at https://vkrakovna.wordpress.com/ai-safety-resources/
> <https://vkrakovna.wordpress.com/ai-safety-resources/>
It is as well to keep in mind that when "AI safety" people talk about safety, they are not talking
about what the IEC means by safety, the freedom from unacceptable risk.
Not that I find the IEC definition satisfactory. It is too indirect. "Risk" seems to me to be a
term which arises out of safety, not the other way around. It doesn't or didn't help that there has
been more than one definition of "risk" hanging around IEC standards for many years. That was
straightened out sometime between 2013 and 2017. But now comes "security risk", a term with a very
concrete meaning to those working in areas where cybersecurity is an issue, but which does not have
a stochastic nature, so "combination of probability and severity of harm" does not do, as there is
no well-defined "probability" that any one has any hope of being able to measure. To indicate its
different nature, I prefer to write it with a hyphen: cybersecurity-risk.
My proposal for cybersecurity-risk is that there is a progression of five well-defined stages
leading to a successful exploitation (or cyber attack if you prefer this term). At each stage there
is at least a qualitative likelihood that can be evaluated. It might be unknown at what stage we are
in at any given moment in time; so in this case a statement of risk would consist in (a) a selection
of possible stages, along with (b) per stage, the conditional probability that in fact we are in the
next stage. So risk is not a number (or a "combination of" two numbers, as the IEC currently has
it)but an array of tuples.
I am also very tempted to assimilate "exploitation" to "loss of system integrity" through direct or
indirect intentional manipulation; "subversion of system integrity" if you care to define what
"subversion" is. But then that clashes with the rather odd notion of "integrity" used in IEC 61508.
I think there are some obvious ways to sort that out, which I wrote about in 2017 and 2018, but
there seems to be zero interest in getting it sorted in the 61508 MTs.
Founding a straightforward concept such as "safety" on such a complex intellectual structure as
"risk" seems to me misplaced. "The (analysis and) avoidance of harm" might be a more encompassing,
straightforward definition. The AI people seem to mix this up with reliability at times.
> I think it is a bit too early to dismiss ML failures as just software or systematic failures. True
> ML failures like systematic failures (IEC 61508-4 3.6.6) are "related in a deterministic way to a
> certain cause "; in this case the "learning" process; environment; and data.
Are you so sure it is deterministic? FP numbers are calculated by the trillion. I doubt we can
determine the effect of rounding errors.
> It also depends, somewhat, on whether
> learning is part of the design process and then when validated, locked baselined or whether the
> system continues to learn and thus acquires more "systematic faults".
So, static or dynamic nets. As of ten years ago, NASA seemed to think there was some (at that time
largely unrealised) hope that static DLNNs could be analysed via the associated Liapunov functions
but that there was no such hope for dynamic DLNNs (Schumann and Liu
https://www.springer.com/de/book/9783642106897 ). Nevertheless, the DLNNs which "worked" most
effectively were dynamic, for example on the Propulsion Controlled Aircraft projects.
> ... governments ... are wowed by the buzz but not willing to
> contemplate the risks.
That may be true of some governments, but it doesn't seem to be true of, for example, California's.
The DMV has rules for allowing autonomous vehicles on the roads, and when Uber decided to ignore
those rules DMV got them off the streets pretty quickly (and they went to Arizona). There has not
been serious harm caused by a road AV accident in California, as far as I know, in which the
operator was following the procedures (the Tesla accidents have all involved operator hubris,
although the NHTSA has pointed out how the automated systems contribute to that hubris. The Waymo
operations appear to be well-conducted. Uber famously was filmed nearly hitting a cyclist on its
first day out in SF, which turned out to be a good thing as it alerted the DMV to the fact that Uber
wasn't conforming with its requirements for permits).
I think we could conclude that that is in any case going reasonably well. There has been one fatal
accident, thoroughly investigated by the NHTSA with a public report, and Uber has necessarily
reacted to those findings, just as Tesla has reacted to the NHTSA findings on its accidents.
Compare those thorough investigations-with-consequences to what happens with human-controlled
vehicles on the roads daily. Serious local traffic accidents are all dutifully reported in my local
newspaper. People losing control on icy roads at the first frost of the year. Or, in this year,
after the first rain for some weeks has slicked down the roads. Truck drivers on the very full
Autobahn not noticing that the traffic ahead of them has slowed or stopped and, if they live, often
found to be using drugs to keep awake on longer-than-permitted schedules. It could be controlled, of
course, but isn't, partly because most taxpayers don't want to pay for it and partly because
encountering a serious accident is rare, unless it's your job. EU states are non-uniform in their
approaches to traffic safety. Some have up to four times the accident rates of others.
We have a 30kph zone in my village, whose through-road has a long, mostly blind, sloping, 135° curve
with 100m radius. Which people customarily traverse at up to twice the limit. The city has purchased
two portable radar devices and one has been installed at the beginning of the zone for a few days
now. It is remarkable how traffic has calmed, and people now cross the road easily without having to
shuffle-run or keep tight hold of their kids. I'd rather like Waymo to chose our village as a hot
spot for their tests. Nobody would be able to travel over 30kph because the Waymo vehicles wouldn't.
We residents would all be a lot happier because, whether you think the Waymo car will slow for you
or not see you, all traffic negotiations would take place a lot more slowly, calmly and carefully,
so our quality of life would improve.
It is not just avoiding harm, of course. Stuart Russell, for example, has been publicising the ways
in which current AI could be used to promulgate harm, including being involved with the Slaughterbot
video. I think the US DoD is well aware of issues with automated or semi-automated warfare. There
have been books on it for years, some of them very good, such as Chamayou's Drone Theory and
Gusterson's Drone.
PBL
Prof. Peter Bernard Ladkin, Bielefeld, Germany
MoreInCommon
Je suis Charlie
Tel+msg +49 (0)521 880 7319 www.rvs-bi.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.techfak.uni-bielefeld.de/pipermail/systemsafety/attachments/20191217/080e639c/attachment-0001.sig>
More information about the systemsafety
mailing list