[SystemSafety] Bossavit's Leprechauns book

[Peter Bernard Ladkin]

A couple of people have recommended Bossavit's book The Leprechauns of Software Engineering.: How
Folklore Turns Into Fact And What To Do About It (self-published).

It is about 180pp long (in the printed version). Thanks to Tim Schürmann for lending me his copy. I
read it in a couple of hours on Saturday night.

There are four themes.

1. Don't believe everything you read/take everything you read at face value, even in "respectable"
publications.

2. The claimed factor-of-ten difference in productivity amongst professional programmers.

3. Where the Waterfall model came from.

4. The rising cost of discovering defects later in the "Waterfall" chain.

Point 1 is taking coals to Newcastle for many of us who have been involved in the
scientific-publication process. It could be a welcome warning for those who tend to take what is
published in respectable journals as established fact. It is by no means unique to SWE.

Points 2, 3, and 4 are well-known memes, or tropes, in SWE. There is some useful material in
Bossavit's attempt to trace them back.

The material could be assembled into a short paper (6pp?) for, say, IEEE Software. I think it would
be worth so doing. Bossavit says he is going to keep at it, with other memes. Let us hope he does,
for tracing memes is a worthwhile social enterprise for engineering. I could suggest some more --
going through a list of recent themes on this list:
* "Formal methods don't work"
* "Formal methods take up resources in development and don't pay them back in increased SW quality"
* "You cannot evaluate software quality statistically"
* "C is as good as any other procedural language for writing critical software"
* "You don't need a language to enforce strong data typing because your static analysis tools can
check if there could/would be type violations"
* "We can write better standards for critical SWE than what is there already"
* "MISRA C is crap"
* "MISRA C is brilliant"
* "Coding standards are not helpful"
* "All critical SW development must be performed according to a coding standard"

An example of Point 1 from another field. Back in the 1990's, the researchers at Boeing's then
Radiation Lab (it has been disbanded) had a sizeable literature pointing to SEEs at altitude
primarily due to neutrons. I worked in close proximity to a well-known team of particle physicists,
so I asked one: "Why do they think it is neutrons? Why can't it be electrons?" He pointed me to the
Particle Physics Booklet, published regularly by the Review of Particle Physics, which is C15 of the
European Physical Journal. I learnt that actually not all that much is known about cosmic rays (the
results of atmospheric interactions initiated by near-earth space radiation, which largely consists
of alpha particles). Spacecraft people have to defend largely against alpha particles. I traced the
citation chain back to an article in the IEEE Transactions on Electron Devices, which reported the
results of some experiments with DRAMs in the neutron accelerator in New Mexico. They put half the
DRAMs in the neutron beam and half out. Those in the beam suffered significantly more SEEs than
those out. Established? Not quite -- the asymmetry of the flips was also significant. The obvious
possible explanation was that it was mostly charged particles causing the flips. But the researchers
just left the phenomenon unexplained. I went to one of the experimental particle physicists and said
"This just can't be right. The main effect noted in this paper is an asymmetry of flips which
suggests charged particles. But the authors say it's neutrons and ignore the effect." He said, "That
would be typical. Ninety per cent of all such empirical work is crap. One has to learn how to pick
out the stuff that's not."

Bossavit also mentions Bruno Latour's work on science as it is actually practiced. I have had a hard
time reading Latour, as I do with many STS authors, but some of the work is masterful, especially
that of Harry Collins and Donald MacKenzie.

To Points 2-4. Bossavit is right to claim they haven't been proven. But the question is less whether
there is a scientific fact about, for example, professional-programmer productivity -- of course
there are, indeed there are probably many thousands to millions of them, very few of which are in
any way useful. The practical question is, given the almost insuperable plethora of confounding
factors in any experiment you might care to try, whether there is indeed such a substantial
difference and what its magnitude might be.

I have met programmers whom I have judged to be at least ten times as productive as I was (they
turned something out in 60 hours, which I used, and had judged it would have taken me somewhat of
the order of four months to produce). So I know there is a substantial difference. Just as I know
that there are people who can solve Sudoku puzzles faster than I can, and also people who can't
solve all the Sudoku puzzles that I can. But how big is it reasonable to take that difference? Give
me a number. If it is not 10, is it 2? (Hardly.) Is it 2000? (Hardly. Nobody can type that fast.)
There could be different numbers for different situations: you need to hire a programmer to get a
job done. Do you hire someone straight out of college whose teachers thought heshe was pretty good,
or do you engage a consultant with a very high reputation and a substantial list of satisfied
clients? The answer is usually: it depends. I suggest 10 is a pretty good number. But I wouldn't be
surprised if 5 would turn out to be more helpful. I would be surprised if it were 50.

Similarly with the claim about the cost of discovery of defects. People like to claim it is a
geometric progression in the phases of the waterfall. What would be the basis for this? I suggest
the following. One change in requirements is likely to cause X changes in design when fixed. One
change in design is likely to cause Y changes in code. One change in code is likely to cause Z
changes in tests to be performed. Can you put any numbers on X, Y, Z? Of course you can't. It would
take a lot of work even to turn these claims into something that could be meaningfully tested
statistically. But as a rule of thumb, does it make sense to consider them all the same to a first
approximation? Well, why not? And suppose you do consider them all the same, what would that number
be? Boehm looked at a bunch of things and said "4.6". He looked pretty carefully, but equally he was
careful to circumscribe what he was in fact looking at (and, as Bossavit points out, it is
particular enough not to be necessarily useful widely). Others have said "10". It is unlikely to be
1, and it is unlikely to be 100,000. What is the best number to use? This is not the kind of
question one can answer to a 6-sigma confidence judgement. But it is better to have a half-way
indicative number than to have none.

This is the kind of thing people do in business on a daily basis. You simplify the question until
you can guess at some kind of answer, and use it as a guide for decisions. Somebody comes to you
with a SW project. You have to look at it and figure out how much it will cost you in resources, in
order to make a bid. There are people who consistently get that kind of thing right; there are
people who often get it right; and then there are people who used to be in the business......

As for Bossavit's investigations into the Waterfall model, I think he gets it wrong. He wants to
suggest 1987 as the take-off point, and relates it to ICSE 1987. I was there (indeed, I had a paper
in it). The models were well-established before then. When I started getting into this in 1985,
Royce's Waterfall and Boehm's Spiral were already well-known and talked about in those terms, indeed
those were amongst the first two models I learned. It was evident to those who looked at Win's paper
or talked to him that retracing/feedback was involved. I also imagine he would have said he was just
the guy who wrote down some useful separation of concerns because he was that kind of person (maybe
he did say that and I forgot). Waterfall is still a very useful separation of concerns, in
particular as people have been observing here that people are still often missing the source
(written requirements. Lockheed built largely military systems; they didn't and don't miss
requirements).

I think Bossavit is wrong in suggesting the term "software crisis" was barely used at the Garmisch
1968 conference, and that it was really due to Edsger Dijkstra a couple of years later. For example,
in the report Section 7.1.2 Problem Areas, one can read the intro "There was a considerable amount
of debate on what some members chose to call the ‘software crisis’ or the ‘software gap’. As will be
seen from the quotations below, the conference members had widely differing views on the
seriousness, or otherwise, of the situation, and on the extent of the problem areas." The word
"crisis" occurs a dozen times in the manuscript.

PBL

Prof. Peter Bernard Ladkin, Bielefeld, Germany
MoreInCommon
Je suis Charlie
Tel+msg +49 (0)521 880 7319  www.rvs-bi.de





-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20181203/e034caf3/attachment.sig>