[SystemSafety] Analysis of some Work Breakdown Structure projects

Derek M Jones derek at knosof.co.uk
Wed Jun 9 13:16:15 CEST 2021


Martyn,

The 2106.03679 paper data derives from the Watts Humphrey's work
at CMU.  There is lots more to be found in this data, and the
paper is a start.

>> Defect per KLOC is meaningless unless it is connected with usage
>> data, e.g., there can be zero defects per KLOC (because the software
>> has no users), or lots per KLOC because it has millions of users.
> 
> The datasets from http://arxiv.org/abs/2106.03679 that you analysed contain defects injected and defects found later in 
> development and repaired. have you analysed those?

This is the data behind figure 6.42.

But there is no usage data.

>> I've never seen a breakdown by individual.  It's possible to do, when
>> mining github (actually this is by user id, and there are cases of
>> the same person having multiple ids), but again usage needs to be
>> taken into account.
> 
> Again, the http://arxiv.org/abs/2106.03679 data seems to show individuals. The Watts Humphrey study below does that too.

Yes, tasks are associated with individuals.
But again, no usage data.

>>> There was data of this sort from the SEI 30 years ago and some from UK MoD, and some reports by the CHAOS group 
>>> twenty years ago but nothing I know of recently.
>>
> The SEI data I referred to was from a study carried out by Watts Humphrey, of the Software Engineering Institute at 
> Carnegie-Mellon University, analysed the fault density of more than 8000 programs written by 810 industrial software 
> developers. resources.sei.cmu.edu/asset_files/SpecialReport/2009_003_001_15035.pdf p132

Thanks for the link.  I had not seen this collection of Watts Humphrey
columns before.

The column name of table 25-1 should read:
"Average detected defects per KLOC".
The question is then: How much effort was put into
detecting defects?
The metric Defects_per_KLOC only makes sense when effort
to detect the defacts is taken into account.
I can create programs with zero Defects_per_KLOC, simply by
putting zero effort into detecting defects.

>> UK MoD?  This does not ring any bells for me.  Do you have a reference,
>>
> My reference was to the analysis of Boeing flight control software published in Crosstalk
>     German, A.: Software static code analysis lessons learned. Crosstalk
>     16(11) (2003)

Thanks for the reference.
Table 1 lists Anomalies per lines of code.
But again, no indication of the effort involved in detecting
those anomalies.

> and to the review of the Full Authority Digital Engine Controller that was installed in Chinook helicopters; which is 
> described in a House of Commons report into the Mull of Kintyre Chinook accident on 2 June 1994 . This said:/In the 

I will have a look at this, but I suspect that effort to detect data is
not included.

> summer of 1993 an independent defence IT contractor, EDS-SCICON, was instructed to review the FADEC software; after 
> examining only 18 per cent of the code they found 486 anomalies and stopped the review/.

Did they record the effort (I imagine their time), needed
to detect each anomaly?  This kind of data is rare.


-- 
Derek M. Jones           Evidence-based software engineering
tel: +44 (0)1252 520667  blog:shape-of-code.coding-guidelines.com


More information about the systemsafety mailing list