[SystemSafety] Analysis of some Work Breakdown Structure projects

[Derek M Jones]

Steve,

Detailed discussion of the internals of the CESAW data
is probably outside the scope of this list.

There is now a CESAW channel on the Evidence-based
software engineering Slack channel:
https://evidencebased-uef5553.slack.com

> You don’t need that for an R% calculation. You only need total time spent on rework divided by total time spent on the project. I looked at CESAW_defect_facts.csv and saw that each row has a project_key column and a defect_fix_time_minutes column.

True.  If you connected this with the task-id in the task-facts
file, you would get the WBS-element-id, and from that you can find
the inspection tasks, giving you access to all the work associated
with the fix.

> CESAW_project_summary.csv has the overall effort data for each project in column “Effort [hours]”.
> 
> Overall R% for the entire data set should be calculate-able by adding up all defect_fix_time_minutes in CESAW_defect_facts.csv and dividing by 60 to convert to hours, then dividing that by the total of the Effort [Hours] column in CESAW_project_summary.csv.
> 
> R% on a project-by-project basis should be calculate-able by adding up all defect_fix_time_minutes in CESAW_defect_facts.csv that have the same project_key and dividing by 60 to convert to hours, then dividing that by the Effort [Hours] value for the row with the same project_key in CESAW_project_summary.csv.
> 
> 
> 
> Martyn,
> 
> “I'd be interested in data on the defects injected and fixed.  How many per KLOC, how variable between individuals, what relationship to component size (LOC), what percentage injected in each phase, where they were found, how long a defect typically remains before being found and fixed ...”
> 
> Table CESAW_defect_facts.csv has columns including:
> 
> *) wbs_element_key
> *) person_key
> *) injected_phase_name
> *) injected_phase_type
> *) defect_found_date
> *) defect_removed_phase_key
> *) removed_phase_name
> *) removed_phase_type
> 
> It seems like you could at least get some of the data you’re looking for, e.g., % injected by phase, % detected by phase, how they were found (failure vs. construction vs. appraisal). You should be able to match the WBS element key in CESAW_defect_facts.csv to the corresponding task in CESAW_task_fact.csv to get calendar dates for when the original work on the deliverable was done.
> 
> 
> — steve
> 
> 
> 
> 
> 
> -----Original Message-----
> From: Derek M Jones <derek at knosof.co.uk<mailto:derek at knosof.co.uk>>
> Organization: Knowledge Software, Ltd
> Date: Wednesday, June 9, 2021 at 7:13 PM
> To: Steve Tockey <Steve.Tockey at construx.com<mailto:Steve.Tockey at construx.com>>, "systemsafety at lists.techfak.uni-bielefeld.de<mailto:systemsafety at lists.techfak.uni-bielefeld.de>" <systemsafety at lists.techfak.uni-bielefeld.de<mailto:systemsafety at lists.techfak.uni-bielefeld.de>>
> Subject: Re: [SystemSafety] Analysis of some Work Breakdown Structure projects
> 
> Steve,
> 
> Use of a Work Breakdown Structure (WBS) is simply a good project management practice. It’s orthogonal to Waterfall (or other “plan-based”) lifecycles vs. Agile ones.
> 
> I was surprised to see how many non-software projects use it.
> 
> The WBS is no more and no less than simply a catalog of the work (to be) done on that project...
> A running battle I have with the agilista crowd is their insistence that, “our product backlog is our requirements”. No it’s not. Your product backlog is the Agile manifestation of your WBS. The requirements live elsewhere, but that’s another story for another time.
> 
> Everybody has their own best practices.  What they all
> share is a lack of evidence showing almost anything other
> than the fact that they are used.
> 
> “I'm not sure of the kind of questions that those involved might ask about such data.  Suggestions welcome.”
> 
> There is an implicit "Assuming the data has the information needed to
> answer the question."
> 
> To me, an extremely useful thing to look at would be “Rework Percentage” (R%). Add up total labor spent repairing defects (it’s a
> The WBS structure in the data is essentially parent links,
> which enables things like figure 2 to be created.
> 
> The rework task information is the same as all the other task
> information, e.g., date, time, person-id, etc.  But there is
> no field specifying what previous tasks have been reworked.
> 
> I was hoping to be able to spot the iterations, but could not
> see a simple way to do it.  Somebody familiar with using WBS
> may be able to see some tell-tale patterns that indicate iterations.
> 
> defect if it is a shortcoming found after the creator of that work claimed that that work was complete, e.g., going into a review or inspection of some sort) and divide that by the total labor on that project.
> 
> The most common task in the data are inspections of one sort or
> another.
> 
> Is reworking a short coming, or an intrinsic part of the process
> of building something new?
> 
> Writing software is a process of discovery, and accepting reworking
> may be the most cost effect way of progressing.
> 
> The limited but reliable data I have from 5 organizations of between 50 and 350 developers shows a weighted average of 61.2%. In other words, 61.2 out of every 100 labor hours spent in those organizations goes to fixing defects discovered after the worker claimed their work was complete. This leads directly to the obnoxious but unavoidable conclusion that,
> “Rework is not only the single largest driver of cost and schedule on a typical software project; it is bigger than all other drivers combined!”
> 
> And what approach would be less costly?  Those who sing the benefits
> of complete upfront design are guilty of hand-waving, just as
> much as everybody else.
> 
> Having said that, R% is highly manageable. I don’t have published, reference-able data but I do have experience on software projects that makes me highly confident R% can be well under 10%. I also have anecdotal data that suggests “Acceptance Test Driven Development” can cut R% in half. R% is the net effect of what Martin Thomas is looking for in timing differences between defect injection and detection. The bigger the difference, the higher R%.
> 
> Why is 10% better than 90%?  It depends on the nature of
> the problem.
> 
> I saw in your paper that the CESAW data set does include rework data. So I think looking at R% both overall and on a project-by-project basis would be interesting and useful.
> 
> There is lots of interesting data in there.
> 'All' that is are good ideas to connect it all
> together :-)
> 
> — steve
> -----Original Message-----
> From: systemsafety <systemsafety-bounces at lists.techfak.uni-bielefeld.de<mailto:systemsafety-bounces at lists.techfak.uni-bielefeld.de><mailto:systemsafety-bounces at lists.techfak.uni-bielefeld.de>> on behalf of Derek M Jones <derek at knosof.co.uk<mailto:derek at knosof.co.uk><mailto:derek at knosof.co.uk>>
> Organization: Knowledge Software, Ltd
> Date: Tuesday, June 8, 2021 at 3:07 AM
> To: "systemsafety at lists.techfak.uni-bielefeld.de<mailto:systemsafety at lists.techfak.uni-bielefeld.de><mailto:systemsafety at lists.techfak.uni-bielefeld.de>" <systemsafety at lists.techfak.uni-bielefeld.de<mailto:systemsafety at lists.techfak.uni-bielefeld.de><mailto:systemsafety at lists.techfak.uni-bielefeld.de>>
> Subject: [SystemSafety] Analysis of some Work Breakdown Structure projects
> All,
> Until recently data on the use of Agile in safety critical software
> was almost non-existent.
> I'm not sure if the use of work breakdown structure counts as
> Agile, but here is an analysis of lots of data (project 615
> was one of the safety critical projects):
> http://arxiv.org/abs/2106.03679
> Never having used WBS, I'm not sure of the kind of questions
> that those involved might ask about such data.  Suggestions
> welcome.
> --
> Derek M. Jones           Evidence-based software engineering
> tel: +44 (0)1252 520667  blog:shape-of-code.coding-guidelines.com
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE<mailto:systemsafety at TechFak.Uni-Bielefeld.DE><mailto:systemsafety at TechFak.Uni-Bielefeld.DE>
> Manage your subscription: https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety
> 
> --
> Derek M. Jones           Evidence-based software engineering
> tel: +44 (0)1252 520667  blog:shape-of-code.coding-guidelines.com
> 
> 

-- 
Derek M. Jones           Evidence-based software engineering
tel: +44 (0)1252 520667  blog:shape-of-code.coding-guidelines.com