[SystemSafety] AI and the virtuous test Oracle - intelligence
Prof. Dr. Peter Bernard Ladkin
ladkin at techfak.de
Sat Jul 1 11:02:53 CEST 2023
On 2023-06-30 20:34 , Steve Tockey wrote:
> I attended a conference earlier this year in Mysuru, India where an invited speaker, Dr. Seema Chopra from Boeing talked about adding in elements into AI systems whereby that system can be asked to “explain” why it gave the result that it did. Seemed like an interesting area of research that could maybe help in this area.
Interest in so-called "explainable AI" is around thirty years old. It can be thought to have started
with the use of expert systems in medical-clinical diagnoses; physicians needed to have a rationale
for the decisions they made aided by the systems. A short, though not necessarily completely
accurate, history is at https://en.wikipedia.org/wiki/Explainable_artificial_intelligence
But for DLNNs and related technologies it's a wish, not a developed technology.
Interest in it has expanded because of the need in some circumstances to explain why we should trust
the results of DLNNs and most other ML engines. We should be wary of assuming that because there is
already a name for the task, that there is anything approaching a solution. (You can apply similar
wariness to the term "AI" itself.)
It is possible to concoct rationales from the operation of expert systems or theorem provers - you
look at how they used the rules to come up with their output. However, it is still really difficult
to present those in human-surveyable form, as anyone who has used a theorem-prover in earnest will
attest. Indeed, that is why there is a name for it and ongoing research. Note that the problem with
trying similar for DLNNs is the complete lack of any kind of technical connection between the way
the engine operates (its algorithm/s) and what would count as justification for a human assessor.
Note that there are significant difficulties, emanating from such phenomena as that noted by Ross
Anderson and colleagues, that you can alter the outputs of a DLNN by permuting the training data.
Intuitively, any justification for trusting DLNN output as a solution to whatever real-world problem
you are addressing should be invariant under permutation of training data.
Note that when humans come up with concocted explanations for things (say, for things they did) it
is called rationalisation and usually deprecated.
There are also domains in which it is hard to envisage how any "explainable AI" might work, for
example, real-time control systems. For a pilot, for example, any such additional input is more
likely to be a distraction than a help. If the airplane control system isn't doing what you want,
then the immediate goal is to stop that behaviour and have it do what you want, not necessarily to
have it explain itself.
--
PBL
Prof. i.R. Dr. Peter Bernard Ladkin, Bielefeld, Germany
Tel+msg +49 (0)521 880 7319 www.rvs-bi.de
More information about the systemsafety
mailing list