[SystemSafety] Machine learning failure modes and failure rates

[Ganesh J. Pai]

Dear System Safety List members,

The SAE G-34 Committee for Artificial Intelligence in Aviation and the
EUROCAE WG-114 for AI  have been working jointly on ARP 6983
<https://www.sae.org/standards/content/arp6983/> (ED number yet to be
assigned). The purpose of this document is to give process assurance
guidance for aeronautical products implementing AI (the first issue of this
document specifically targets supervised, offline, machine learning) with a
view to being endorsed by the respective civil aviation regulators as an
acceptable means of compliance to regulations in the certification/approval
processes.

The document is, to some extent, aiming to align itself with the objectives
and suggested means of compliance formulated by EASA's Concept Paper
Proposed Issue 02 on Levels 1 & 2 Machine learning applications
<https://www.easa.europa.eu/en/downloads/137631/en> (See the linked
document for what those levels mean). I won't comment on what "alignment"
means here, but suffice it to say that EASA is an active participant in the

The EASA document puts forth a concept of "ML Constituent (MLC)" (not my
first choice of terminology but that's the accepted term now), which is
equivalent---for all intents and purposes---to an ARP 4754 / ARP 4761 *item*,
but including one or more ML models (MLMs).

With this background in place, one of the topics of discussion that has
emerged is whether or not an MLM (and, in turn, an MLC) has failure modes
and failure rates. As you can probably imagine, the topic is already
controversial and there is the same dichotomy of positions/opinions as when
that question were posed of conventional software

The EASA document linked above does admit failure modes and failure rates
as valid concepts for ML models, which I think contradicts their position
that software does not have failure modes or a failure rate.

I would very much like to hear your opinions and arguments on the topic
(especially if you're involved in the aviation domain).

Thanks,
Ganesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/pipermail/systemsafety/attachments/20230815/e366db14/attachment.html>