[SystemSafety] Failure Modes in ML

[Bruce Hunter]

 On 2019-12-17 00:31 , Peter Bernard Ladkin wrote:

>> ... governments ... are wowed by the buzz but not willing to
>> contemplate the risks.

>That may be true of some governments, but it doesn't seem to be true of,
for example, California's.

Yes, I did generalised too much with my comments. There are a many good
initiatives by government agencies to address the risk of ML and autonomous
vehicles. California seems to have the right balance in setting and
enforcing rules to keep other road users safe.

Australia's National Transport Commission and Ausroads have, however, only
published "*Guidelines* for trials of automated vehicles in Australia" in
response to "Australia’s transport ministers have endorsed an ambitious
roadmap of reform that will facilitate the introduction of more automated
vehicles on our roads".

It's a tricky thing for politicians to correctly balance risk and benefit.

Thank you for the other references and thoughts on AI and ML risks related
to safety (freedom from harm). I'll research them with interest.

I'm a bit concerned when IIoT, Edge Technology and Could-based AI will
start being used for safety-related systems. This may have some great
benefits but some possible big risks with dependability. There is already
some that push for this to be used in alarm monitoring.

Regards,
Bruce


On Tue, 17 Dec 2019 at 19:24, Peter Bernard Ladkin <ladkin at causalis.com>
wrote:

>
>
> On 2019-12-17 00:31 , Bruce Hunter wrote:
> >
> >
> > Victoria Krakovna also maintains an AI safety resources page which
> includes database records of AI
> > failures. This is at
> https://vkrakovna.wordpress.com/ai-safety-resources/
> > <https://vkrakovna.wordpress.com/ai-safety-resources/>
>
> It is as well to keep in mind that when "AI safety" people talk about
> safety, they are not talking
> about what the IEC means by safety, the freedom from unacceptable risk.
>
> Not that I find the IEC definition satisfactory.  It is too indirect.
> "Risk" seems to me to be a
> term which arises out of safety, not the other way around. It doesn't or
> didn't help that there has
> been more than one definition of "risk" hanging around IEC standards for
> many years. That was
> straightened out sometime between 2013 and 2017. But now comes "security
> risk", a term with a very
> concrete meaning to those working in areas where cybersecurity is an
> issue, but which does not have
> a stochastic nature, so "combination of probability and severity of harm"
> does not do, as there is
> no well-defined "probability" that any one has any hope of being able to
> measure. To indicate its
> different nature, I prefer to write it with a hyphen: cybersecurity-risk.
>
> My proposal for cybersecurity-risk is that there is a progression of five
> well-defined stages
> leading to a successful exploitation (or cyber attack if you prefer this
> term). At each stage there
> is at least a qualitative likelihood that can be evaluated. It might be
> unknown at what stage we are
> in at any given moment in time; so in this case a statement of risk would
> consist in (a) a selection
> of possible stages, along with (b) per stage, the conditional probability
> that in fact we are in the
> next stage. So risk is not a number (or a "combination of" two numbers, as
> the IEC currently has
> it)but an array of tuples.
>
> I am also very tempted to assimilate "exploitation" to "loss of system
> integrity" through direct or
> indirect intentional manipulation; "subversion of system integrity" if you
> care to define what
> "subversion" is. But then that clashes with the rather odd notion of
> "integrity" used in IEC 61508.
> I think there are some obvious ways to sort that out, which I wrote about
> in 2017 and 2018, but
> there seems to be zero interest in getting it sorted in the 61508 MTs.
>
> Founding a straightforward concept such as "safety" on such a complex
> intellectual structure as
> "risk" seems to me misplaced. "The (analysis and) avoidance of harm" might
> be a more encompassing,
> straightforward definition. The AI people seem to mix this up with
> reliability at times.
>
> > I think it is a bit too early to dismiss ML failures as just software or
> systematic failures. True
> > ML failures like systematic failures  (IEC 61508-4 3.6.6) are "related
> in a deterministic way to a
> > certain cause "; in this case the "learning" process; environment; and
> data.
>
> Are you so sure it is deterministic? FP numbers are calculated by the
> trillion. I doubt we can
> determine the effect of rounding errors.
>
> > It also depends, somewhat, on whether
> > learning is part of the design process and then when validated, locked
> baselined or whether the
> > system continues to learn and thus acquires more "systematic faults".
>
> So, static or dynamic nets. As of ten years ago, NASA seemed to think
> there was some (at that time
> largely unrealised) hope that static DLNNs could be analysed via the
> associated Liapunov functions
> but that there was no such hope for dynamic DLNNs (Schumann and Liu
> https://www.springer.com/de/book/9783642106897 ). Nevertheless, the DLNNs
> which "worked" most
> effectively were dynamic, for example on the Propulsion Controlled
> Aircraft projects.
>
> > ... governments ... are wowed by the buzz but not willing to
> > contemplate the risks.
>
> That may be true of some governments, but it doesn't seem to be true of,
> for example, California's.
> The DMV has rules for allowing autonomous vehicles on the roads, and when
> Uber decided to ignore
> those rules DMV got them off the streets pretty quickly (and they went to
> Arizona). There has not
> been serious harm caused by a road AV accident in California, as far as I
> know, in which the
> operator was following the procedures (the Tesla accidents have all
> involved operator hubris,
> although the NHTSA has pointed out how the automated systems contribute to
> that hubris. The Waymo
> operations appear to be well-conducted. Uber famously was filmed nearly
> hitting a cyclist on its
> first day out in SF, which turned out to be a good thing as it alerted the
> DMV to the fact that Uber
> wasn't conforming with its requirements for permits).
>
> I think we could conclude that that is in any case going reasonably well.
> There has been one fatal
> accident, thoroughly investigated by the NHTSA with a public report, and
> Uber has necessarily
> reacted to those findings, just as Tesla has reacted to the NHTSA findings
> on its accidents.
>
> Compare those thorough investigations-with-consequences to what happens
> with human-controlled
> vehicles on the roads daily. Serious local traffic accidents are all
> dutifully reported in my local
> newspaper. People losing control on icy roads at the first frost of the
> year. Or, in this year,
> after the first rain for some weeks has slicked down the roads. Truck
> drivers on the very full
> Autobahn not noticing that the traffic ahead of them has slowed or stopped
> and, if they live, often
> found to be using drugs to keep awake on longer-than-permitted schedules.
> It could be controlled, of
> course, but isn't, partly because most taxpayers don't want to pay for it
> and partly because
> encountering a serious accident is rare, unless it's your job. EU states
> are non-uniform in their
> approaches to traffic safety. Some have up to four times the accident
> rates of others.
>
> We have a 30kph zone in my village, whose through-road has a long, mostly
> blind, sloping, 135° curve
> with 100m radius. Which people customarily traverse at up to twice the
> limit. The city has purchased
> two portable radar devices and one has been installed at the beginning of
> the zone for a few days
> now. It is remarkable how traffic has calmed, and people now cross the
> road easily without having to
> shuffle-run or keep tight hold of their kids. I'd rather like Waymo to
> chose our village as a hot
> spot for their tests. Nobody would be able to travel over 30kph because
> the Waymo vehicles wouldn't.
> We residents would all be a lot happier because, whether you think the
> Waymo car will slow for you
> or not see you, all traffic negotiations would take place a lot more
> slowly, calmly and carefully,
> so our quality of life would improve.
>
> It is not just avoiding harm, of course. Stuart Russell, for example, has
> been publicising the ways
> in which current AI could be used to promulgate harm, including being
> involved with the Slaughterbot
> video. I think the US DoD is well aware of issues with automated or
> semi-automated warfare. There
> have been books on it for years, some of them very good, such as
> Chamayou's Drone Theory and
> Gusterson's Drone.
>
> PBL
>
> Prof. Peter Bernard Ladkin, Bielefeld, Germany
> MoreInCommon
> Je suis Charlie
> Tel+msg +49 (0)521 880 7319  www.rvs-bi.de
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/pipermail/systemsafety/attachments/20191220/379adf41/attachment.html>