[SystemSafety] Qualifying SW as "proven in use" [Measuring Software] & [THE DARK NIGHT OF THE SOUL]

[Les Chambers]

Steve
Can you cite your source for the "study of 18 production applications"? That
would be interesting.
Back in the day I provided a McCabe complexity analysis service. The worst
offenders were real-time operating systems. Today the problems caused by
module level complexity are exacerbated by complex and fragmented
architectures. If you take the bog standard dynamic web application as an
example we have the following:
Assuming we implement an allegedly simple model, view, controller
architecture there is:
- PHP controller code for each web page. 
- Various PHP class libraries and configuration files (dynamically included
on the server at run time) .
- PHP view code that dynamically modifies the HTML
- HTML itself
- JavaScript view code written au naturale (dynamically included on the
client at run time)
- JavaScript libraries such as Ajax and jQuery that impact web page
behaviour and communicate with back-end databases (dynamically included on
the client at run time)
- Miscellaneous other JavaScript libraries that perform UI functions such as
nifty drop-down menus (dynamically included on the client at run time)
- Cascading style sheet (CSS) files that impact the format and behaviour of
web pages (for example they can make web page elements disappear). Note that
it is common practice for JavaScript to interact with CSS and change its
behaviour on the fly. Further, most JavaScript libraries can be directly
sourced by page links to external library repositories that are subject to
change not in the control of the developer. There are training organisations
out there that suggest programmers do this to stay current with the new and
the grovy!!! 
- Next we have the PHP engine with behaviour impacted by a configuration
file. And various releases that have slightly different behaviour in
interpreting PHP code.
- The web server itself with its attendant configuration file.
- The database server with its configuration parameters and multiple and
sundry variations on SQL implementation (MySQL, SQL Server, Oracle).
- Then if you are into an ultramodern nontrivial application you are looking
at a cloud environment such as Amazon Web Services with multiple virtual
machines running any number of UNIX variants with attendant transaction
switching, load balancing and database replication facilities not to mention
firewalls with their associated configuration.
- Lastly we have the user's browser (Internet Explorer, Firefox, Chrome,
Safari, Opera) all with slightly different implementations of HTML and
JavaScript. Internet Explorer is the worst offender with implementations
changing by release level.(I have grown quite attached to Chrome).
On the strength of all this, some time ago, I gave up hope of ever providing
value to a client with a McCabe complexity analysis. I'd be handing out
speeding tickets at the Indi 500. There are bigger fish to fry  ...  for
example evasive action on the side effects of the non-declarative weakly
typed PHP language.  You are forced into using it by the massive support in
web environments not to mention its free and has highly active forums that
will get you out of any problem you will ever get into. Legend has it that
it has an error message in Hebrew. 
But I know I'm raving here. And you might wonder what's this got to do with
safety critical software. The short answer is money and lives. We all do our
banking with this stuff and sooner or later, if it hasn't happened already,
this software will be despatching ambulances, police cars and strike jets.
Electricity grids are probably already involved. I'm personally aware of one
gentleman who can tune a gas treatment plant in New Guinea from his living
room in Brisbane.
I have a cycling buddy who is a professor of computer science (emeritus) -
expert in language compiler development. He looks on with some sadness at
the poor state of language development and its suicidal march into
fragmented architectures that can only be described as side-effect city,
"... did not the silly buggers listen in class!!!  AT ALL!!!!". 
Wasn't it Dostoyevsky who said that, "man is a creature who can get used to
anything". I guess that's our fate.

Cheers
Les

-----Original Message-----
From: systemsafety-bounces at techfak.uni-bielefeld.de
[mailto:systemsafety-bounces at techfak.uni-bielefeld.de] On Behalf Of Steve
Tockey
Sent: Tuesday, June 25, 2013 4:21 AM
To: Peter Bishop; systemsafety at techfak.uni-bielefeld.de
Subject: Re: [SystemSafety] Qualifying SW as "proven in use" [Measuring
Software]


Actually, getting the evidence isn't that tricky, it's just a lot of work.
Essentially all one needs to do is to run a correlation analysis
(correlation coefficient) between the proposed quality measure on the one
hand, and defect tracking data on the other hand.

For example, the code quality measure "Cyclomatic Complexity" (reference:
Tom McCabe, ³A Complexity Measure², IEEE Transactions on Software
Engineering, December, 1976) was validated many years ago by simply
finding a strong positive correlation between the cyclomatic complexity of
functions and the number of defects that were logged against those same
functions (I.e., code in that function needed to be changed in order to
repair that defect).

According to one study of 18 production applications, code in functions
with cyclomatic complexity <=5 was about 45% of the total code base but
this code was responsible for only 12% of the defects logged against the
total code base. On the other hand, code in functions with cyclomatic
complexity of >=15 was only 11% of the code base but this same code was
responsible for 43% of the total defects. On a per-line-of-code basis,
functions with cyclomatic complexity >=15 have more than an order of
magnitude increase in defect density over functions measuring <=5.

What I find interesting, personally, is that complexity metrics for
object-oriented software have been around for about 20 years and yet
nobody (to my knowledge) has done any correlation analysis at all (or, at
a minimum they have not published their results).

The other thing to remember is that such measures consider only the
"syntax" (structure) of the code. I consider this to be *necessary* for
code quality, but far from *sufficient*. One also needs to consider the
"semantics" (meaning) of that same code. For example, to what extent is
the code based on reasonable abstractions? To what extent does the code
exhibit good encapsulation? What are the cohesion and coupling of the
code? Has the code used "design-to-invariants / design-forchange"? One can
have code that's perfectly structured in a syntactic sense and yet it's
garbage from the semantic perspective. Unfortunately, there isn't a way
(that I'm aware of, anyway) to do the necessary semantic analysis in an
automated fashion. Some other competent software professionals need to
look at the code and assess it from the semantic perspective.

So while I applaud efforts like SQALE and others like it, one needs to be
careful that it's only a part of the whole story. More work--a lot
more--needs to be done before someone can reasonably say that some
particular code is "high quality".


Regards,

-- steve





-----Original Message-----
From: Peter Bishop <pgb at adelard.com>
Date: Friday, June 21, 2013 6:04 AM
To: "systemsafety at techfak.uni-bielefeld.de"
<systemsafety at techfak.uni-bielefeld.de>
Subject: Re: [SystemSafety] Qualifying SW as "proven
in	use"	[Measuring	Software]

I agree with Derek

Code quality is only a means to an end
We need evidence to to show  the means actually helps to achieve the ends.

Getting this evidence is pretty tricky, as parallel developments for the
same project won't happen.
But you might be able to infer something on average over multiple projects.

Derek M Jones wrote:
> Thierry,
> 
>> To answer your questions:
>> 1°) Yes, there is some objective evidence that there is a correlation
>> between a low SQALE index and quality code.
> 
> How is the quality of code measured?
> 
> Below you say that SQALE DEFINES what is "good quality" code.
> In this case it is to be expected that a strong correlation will exist
> between a low SQALE index and its own definition of quality.
> 
>> For example ITRIS has conducted a study where the "good quality" code
>> is statistically linked to a lower SQALE index, for industrial
>> software actually used in operations.
> 
> Again how is quality measured?
> 
>> No, there is not enough evidence, we wish there would be more people
>> working on getting the evidence.
> 
> Is there any evidence apart from SQALE correlating with its own
> measures?
> 
> This is a general problem, lots of researchers create their own
> definition of quality and don't show a causal connection to external
> attributes such as faults or subsequent costs.
> 
> Without running parallel development efforts that
> follow/don't follow the guidelines it is difficult to see how
> reliable data can be obtained.
> 

-- 

Peter Bishop
Chief Scientist
Adelard LLP
Exmouth House, 3-11 Pine Street, London,EC1R 0JH
http://www.adelard.com
Recep:  +44-(0)20-7832 5850
Direct: +44-(0)20-7832 5855
_______________________________________________
The System Safety Mailing List
systemsafety at TechFak.Uni-Bielefeld.DE

_______________________________________________
The System Safety Mailing List
systemsafety at TechFak.Uni-Bielefeld.DE