<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Derek</p>
<p>We seem to be misunderstanding each other.</p>
<p>I'm interested in whether the CMU data shows how many defects
were introduced during development and later found during later
stages of development – and, where it does, whether that has been
further analysed to show any relationships with component size,
for example.</p>
<p>On the reasonable assumption that "usage data" means usage after
development has ceased and the product has been released into use,
"usage data" is irrelevant to the question I asked about the CMU
datasets. Of ourse, many defects are only found in use, or never
found, so the defect injection data doesn't tell us the defect
density in the <b>delivered</b> code.<br>
</p>
<p><br>
</p>
<p>There is a <b>separate</b> issue about how frequently defects in
software manifest themselves as evident errors in use. The only
alalysis I have ever seen on this was the paper by Adams, who
analysed huge datasets collected by IBM from the logs of mainframe
users (I think in the 1980s). That showed that many latent defects
only become manifest very infrequently (tens of thousands of hours
of user connect time, as I recall). I haven't the reference to
hand but I'll find it if you want it.</p>
<p><br>
</p>
<p>Martyn<br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 09/06/2021 12:16, Derek M Jones
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:1a7de9ad-1aa5-b635-d479-e1fd72a5aa9c@knosof.co.uk">Martyn,
<br>
<br>
The 2106.03679 paper data derives from the Watts Humphrey's work
<br>
at CMU. There is lots more to be found in this data, and the
<br>
paper is a start.
<br>
<br>
<blockquote type="cite">
<blockquote type="cite">Defect per KLOC is meaningless unless it
is connected with usage
<br>
data, e.g., there can be zero defects per KLOC (because the
software
<br>
has no users), or lots per KLOC because it has millions of
users.
<br>
</blockquote>
<br>
The datasets from <a class="moz-txt-link-freetext" href="http://arxiv.org/abs/2106.03679">http://arxiv.org/abs/2106.03679</a> that you
analysed contain defects injected and defects found later in
development and repaired. have you analysed those?
<br>
</blockquote>
<br>
This is the data behind figure 6.42.
<br>
<br>
But there is no usage data.
<br>
<br>
<blockquote type="cite">
<blockquote type="cite">I've never seen a breakdown by
individual. It's possible to do, when
<br>
mining github (actually this is by user id, and there are
cases of
<br>
the same person having multiple ids), but again usage needs to
be
<br>
taken into account.
<br>
</blockquote>
<br>
Again, the <a class="moz-txt-link-freetext" href="http://arxiv.org/abs/2106.03679">http://arxiv.org/abs/2106.03679</a> data seems to show
individuals. The Watts Humphrey study below does that too.
<br>
</blockquote>
<br>
Yes, tasks are associated with individuals.
<br>
But again, no usage data.
<br>
<br>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">There was data of this sort from the
SEI 30 years ago and some from UK MoD, and some reports by
the CHAOS group twenty years ago but nothing I know of
recently.
<br>
</blockquote>
<br>
</blockquote>
The SEI data I referred to was from a study carried out by Watts
Humphrey, of the Software Engineering Institute at
Carnegie-Mellon University, analysed the fault density of more
than 8000 programs written by 810 industrial software
developers.
resources.sei.cmu.edu/asset_files/SpecialReport/2009_003_001_15035.pdf
p132
<br>
</blockquote>
<br>
Thanks for the link. I had not seen this collection of Watts
Humphrey
<br>
columns before.
<br>
<br>
The column name of table 25-1 should read:
<br>
"Average detected defects per KLOC".
<br>
The question is then: How much effort was put into
<br>
detecting defects?
<br>
The metric Defects_per_KLOC only makes sense when effort
<br>
to detect the defacts is taken into account.
<br>
I can create programs with zero Defects_per_KLOC, simply by
<br>
putting zero effort into detecting defects.
<br>
<br>
<blockquote type="cite">
<blockquote type="cite">UK MoD? This does not ring any bells
for me. Do you have a reference,
<br>
<br>
</blockquote>
My reference was to the analysis of Boeing flight control
software published in Crosstalk
<br>
German, A.: Software static code analysis lessons learned.
Crosstalk
<br>
16(11) (2003)
<br>
</blockquote>
<br>
Thanks for the reference.
<br>
Table 1 lists Anomalies per lines of code.
<br>
But again, no indication of the effort involved in detecting
<br>
those anomalies.
<br>
<br>
<blockquote type="cite">and to the review of the Full Authority
Digital Engine Controller that was installed in Chinook
helicopters; which is described in a House of Commons report
into the Mull of Kintyre Chinook accident on 2 June 1994 . This
said:/In the </blockquote>
<br>
I will have a look at this, but I suspect that effort to detect
data is
<br>
not included.
<br>
<br>
<blockquote type="cite">summer of 1993 an independent defence IT
contractor, EDS-SCICON, was instructed to review the FADEC
software; after examining only 18 per cent of the code they
found 486 anomalies and stopped the review/.
<br>
</blockquote>
<br>
Did they record the effort (I imagine their time), needed
<br>
to detect each anomaly? This kind of data is rare.
<br>
<br>
<br>
</blockquote>
</body>
</html>