[SystemSafety] NYTimes: The Next Accident Awaits

[Matthew Squair]

>On Mon, Feb 3, 2014 at 7:07 PM Patrick Graydon <patrick.graydon at gmail.com>
wrote


> I don't see how those experiments (either the original or the follow-up
work) are particularly relevant.

Apologies for the late response, I blame the time difference and pressures
of work.

Before I start, my definition of a safety case is 'a documented body of
evidence that provides a convincing and valid argument that a system is
adequately safe for a given application in a given environment'. Usually
(but not always) it includes graphical representations.

The original Fischoff, Slovic and Lichtenstein 1978 study (on DTIC) was
funded by DARPA and looked at fault trees because of the WASH 1400 reactor
safety study. As the authors pointed out fault trees were the primary
methodological tools for that study, hence their interest in whether the
technique was sound. As graphical techniques such as fault trees, event
trees, master logic diagrams and goal structuring notation etc etc are the
primary tools we use to analyse and express the 'safety' of systems I think
it's still relevant.

The out of sight, out of mind affect they found is not I suggest a small
issue that can be easily resolved, it's actual a persistent and significant
problem even for experts, as Fischoff et al found in the original study
when they asked experts to review the provided tree. To quote "The most
dramatic result of these studies was subjects' inability to appreciate how
much had been omitted from the pruned fault trees". So the effect is
evident in experts review of evidence presented in such a fashion and is
still pertinent to independent assessment by any expert.

Greenwell, Knight, Holloway and Pease (2006) have reviewed a number of
safety cases for logical fallacies, they found omission of evidence in all
three of the safety cases studied, the Opalinus Clay repository, EUR Whole
Airspace (preliminary study) and EUR RVSM  (pre implementation). As a
specific example of what they found in, in the case of the EUR Whole
airspace argument they found it failed to consider possible interactions
between geographical areas, while for the Opalinus clay study they found
that the selection of uncertainty scenarios was based on expert judgement
with no evidence as to which were rejected. Peter Ladkin has also
separately published an analysis of the RVSM safety case, which looked at
how that case failed to address the safety of the system as it was actually
going to be operated. These are not to my mind nickel and dime type issues.

All of the cases examined had been published, with I presume the review and
blessing of the associated regulator, yet these omissions escaped?

Maybe we need a meta analysis of all the work done.

>...whether a safety case regime systematically accepts more shoddy systems
after regulator/ISA review than a so-called 'prescriptive' system would...

Rather what is the likelihood that we don't know that we have a problem in
either regime, and why?

>As to arguments that a system is unsafe, could you explain how that would
work?

Well like Popper said, science advances on the basis of disconfirming
evidence. So at the organisational level apply the approach used in
security where a red team is constituted to specifically find a weakness in
the system, which would offset the,"all that we've covered is all that
there is" syndrome. Or at the methodology level the use of a technique such
as TRIZ during hazard identification to reverses how we look at a system.
Or during testing to emphasise tests that will break the system, regardless
of whether the test is reasonable, then looking at what that tells us about
predicted behaviour.

Regards,


On Mon, Feb 3, 2014 at 7:07 PM, Patrick Graydon
<patrick.graydon at gmail.com>wrote:

>
> On 3 Feb 2014, at 02:36, Matthew Squair <mattsquair at gmail.com> wrote:
>
> > There is for example experimental evidence going back to Slovic and
> Fischoffs work in the 70s and Silveras follow up work in the 00s on how the
> structuring of fault trees can lead to an effect known as omission neglect,
> see here (http://wp.me/px0Kp-1YN) for further discussion of the effect. I
> see no reason why such graphical techniques as GSN should be immune to the
> same problem, or safety cases in the broader sense.
>
> I don't see how those experiments (either the original or the follow-up
> work) are particularly relevant.  In all of them, the subjects were given
> the fault trees and told to use them as an aid to a subsequent task; the
> experimenters were measuring how presentation in them biased their
> performance in that task.  But in none of them was anyone explicitly tasked
> with checking the given fault trees, as an ISA or a regulator would a
> safety case.  Because no-one took on the role of a skeptical critic, I
> don't see the experimental context as particularly analogous to safety-case
> regulatory regimes.
>
> Moreover, if this was really to weigh in on the question of whether a
> safety case regime systematically accepts more shoddy systems after
> regulator/ISA review than a so-called 'prescriptive' system would, the
> experimental context would have to clearly be more analogous to the context
> of one of those than the other.  But in *both* we have people presenting
> information (that might be framed one way or another) to
> regulators/assessors.
>
> Don't get me wrong, I am not claiming to have the answer here.  But I find
> the evidence that has been offered to date so weak as to be useless.  I
> second Drew's call for serious, systematic study of this.
>
> As to arguments that a system is unsafe, could you explain how that would
> work?  Trying to discover all of the ways that a system is dangerous is a
> good way to find them, as trying to discover all of the ways that an
> argument is flawed is how we find flaws in arguments (safety and
> otherwise).  But what are the criteria on which we decide whether something
> is good enough?
>
> This approach seems to be a case of demonstrating a negative.  In an
> inductive argument, you do this by showing how many possibilities you have
> examined and discarded.  E.g., if I wanted to claim that there are no
> Ferraris in my bedroom, I could back that up by claiming that I have looked
> into every space in that room big enough to hold one in such a way that I
> would expect to see one if it was there and that my search revealed
> nothing.  In the case of safety, wouldn't you have to argue over how you'd
> gone about looking for hazards (and dealt with all you'd found), how you'd
> gone about looking for causes to those (and dealt with all of those), how
> you'd gone about verifying that your system as deployed did what your
> analysis (and the resulting safety requirements) required, etc.  This
> sounds an awful lot to me like the standard guidance for safety case
> structure.  Or do you have something else in mind?
>
> -- Patrick
>
>


-- 
*Matthew Squair*
MIEAust CPEng

Mob: +61 488770655
Email: MattSquair at gmail.com
Website: www.criticaluncertainties.com <http://criticaluncertainties.com/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20140204/77a0fa3a/attachment-0001.html>