[SystemSafety] Logic

[Steve Tockey]

This may very well always be a challenge. People generally have a way too
short term outlook, particularly mid-level managers in big corporations. I
would like to (optimistically) extend Heath Raftery's example as follows
(by the way, I refuse to refer to person B as "Engineer B" because they
clearly aren't one):

Possibility A) Person B's implementation of doodad D is still little more
than just flashing LEDs and clicking relays when Engineer A's solution is
ready to go into production. Engineer A's production version works
essentially flawlessly.

Possibility B) Person B does provide a "production version" of doodad D
however that production system gives defective output on every 32nd use
and crashes--requiring a complete OS reboot--on every 64th use. Engineer
A's production version works essentially flawlessly.

But will the decision makers ever even notice??? Sadly, probably not.



I have a real-world example of a variant on this theme. Details are
changed to keep PBL and his organization out of trouble.

I worked for about 8 years at a company that makes very high-tech
transportation devices. I'll use cars as an analogy, but their devices are
at least an order of magnitude more complex than cars.

We start off with Car Product Line 1, which the company has been building
with, say, gasoline engines for almost 20 years. There's a comprehensive
suite of "automated test equipment" (ATE) software--embedded in a hardware
platform--for testing Product Line 1 cars as they are being manufactured.
All existing test programs were traditionally-built C code that ran on
HP/UX 9. Along comes the need to produce Product Line 1 cars, but with
diesel engines. The "engine simulator ATE" application for gasoline
engines is 25K SLOC. The most reasonable estimate is that engine sim ATE
for diesel engines will also be about 25K SLOC however code reuse is not
possible for reasons that can't be elaborated here. Nonetheless, at
typical programmer productivity rates and the project's allocated staffing
level, that's more than a year of development. The problem is that we're
already in July and the first diesel engine car will be coming through the
factory the following March. We only have 8 months, not 12. They simply
couldn't wait until the following July (or, realistically, much later
given typical software project schedule over-runs) for the diesel version
of the engine sim ATE software.

The project manager, Mike (his real name), had worked with me before on
some non-related projects and was aware of my involvement in model-based
development so he invited me to give the team of four a presentation on
the topic. The team was intrigued with the idea that we could
significantly accelerate delivery because that's exactly what was needed.
Everyone agreed to take the model-based development route. Estimates
derived from early modeling predicted a mid-January completion date for
the model-based project. We could get it done in about 6 months, well
ahead of the March need date.

A mid-level manager (to remain nameless), having experienced horrible
software project delays--due to necessary debugging--in the past, insisted
on having the initial code written by the end of November. This was
intended to allow adequate debugging time before the need date for the
first diesel car. Long story short, the requirements modeling took until
the middle of October. The design modeling took until early December. When
that mid-level manager visited the project in early December, he almost
went into orbit when he realized that the team had not yet written even
one line of production code and the project was already past the point
that he had mandated for "code complete". Mike almost lost his job right
then and there.


There was a slight underestimate in the project, code complete (13K SLOC)
and hardware integration was completed around January 21, not January 14
as predicted back in late July. We had done all of the testing that could
be done without an actual car and everything worked as expected. The
engine sim ATE system then sat there until mid March, waiting for the
first diesel car. When that first diesel car was ready to be tested, both
it and the ATE performed flawlessly.

A little more than a year earlier, the corporate executive management team
approved the engineering & development of Car Product Line 2. The schedule
from approval to Product Line 2 car #1 rolling off the assembly line was
2.5 years. The entire ATE software suite for Car Product Line 2 was
included in the schedule and needed all of the 2.5 years for development.
Unfortunately, that project's original software team wasted the first 1.25
years. When the executive management did a check of the Car Product Line 2
engineering & development critical path, they realized that the ATE
software team was still sitting back at the starting line. The team
members had authored a few interesting technical papers and played a lot
of computer games but had made zero progress on actually producing ATE
software. Most of that original team got reassigned to other projects and
a new team was brought in. This new team noticed that Car Product Line 1's
engine sim ATE was completed in about half the time that had been
predicted, and that's pretty much what they had: half the time. So I and
three of the four from Car Product Line 1 engine sim ATE were brought over
to get Car Product Line 2 ATE going.

There was a management mandate to "reuse as much of the Car Product Line 1
code as possible". Unfortunately, code re-use was simply not an option
because for some reason Car Product Line 2 had chosen C++ on HP/UX 10 for
implementation. We did reuse a little code, but only 83 SLOC. Long story
short, the entire ATE suite for Car Product Line 2 was delivered 3 days
ahead of the need date for car #1. Keep in mind that the full ATE suite
was a far bigger job, we had 30 developers and delivered 113K SLOC. 6-8
weeks after going live on the factory floor, we met with the ATE operators
to see how they liked it. Simply, they were amazed at how such a complex
piece of software could work so flawlessly from the very beginning. They
had set up a contest to see which operator could crash ATE and nobody had
been able to.

With such fully documented, high quality code the middle managers decided
they didn't need nearly as many software weenies to maintain the Product
Line 2 ATE code base. In their infinite wisdom they completely ignored the
fact that we had built a team that took a project in seriously deep doo
doo and made it successful. Rather than find another project that was in
deep doo doo and turn the extra people loose on that, the excess staff got
laid off (made redundant). The team's reward for doing a great job was
that most of them lost their job. Sigh...

Now, wind the clock forward about 12-15 years later. I'm no longer working
at the manufacturing company. By this time they were about half way into
Car Product Line 3 engineering and development. Deja vu all over again as
they realized that the original ATE software team had wasted the first
half of the project schedule. Again, those people got re-assigned to other
projects and I got a panic call from the new ATE software team. "Aren't
you the guy who bailed out the Product Line 2 ATE software project?".
"Yes, why?" "We desperately need your help..."

But again, code reuse was simply not an option because the Product Line 3
ATE project had already committed to C#/.net. Nonetheless, ATE software
was ready well before Product Line 3 Car #1 was in a position to be
tested. And when tested, both car #1 and ATE software performed flawlessly.

One very important lesson that this company never learned was that one
major reason each of these projects were able to finish on time/early was
because we reduced the amount of rework to negligible levels. Software
projects at that company, like traditional software projects everywhere,
suffered from 50-60% rework ("debugging"). All of the model-based ATE
software projects featured peer review of the models that revealed and
allowed removal of the majority of the defects before a single line of
code was ever written. Rework on these projects was well under 10%,
probably closer to 5%.

The other very important lesson that the company never learned was that
the other major reason for finishing on time/early was because of
requirements model reuse. If you laid the three requirements models
side-by-side you would notice that 80-90% of the content was identical.
>From a "what does it mean to test a car?" perspective, each version of ATE
was largely just a minor modification of the earlier version, thus saving
huge amounts of requirements development time.

In the end, my point is that the data is there. Projects have been done
this way and those projects have been successful. But the business has to
take the blinders off and understand what was done differently and why it
made a difference. They seem to be totally incapable of this. Insert
another sigh here...

I should add that what was done on these projects was not strictly "formal
methods" in the sense that's being hotly debated here. We didn't use Z,
VDM, or any of those formal languages. We didn't use theorem provers
either. We used UML (and pre-UML because of project timing) class diagrams
and state charts mostly, but we had a carefully defined and enforced (via
the model inspections) single interpretation of that modeling language.
Much like I mentioned in the Jeannette Wing "A Specifier's Introduction to
Formal Methods" paper earlier, the modelers were using a comfortable
surface syntax (UML) but there was a rigorous (albeit not exactly
formally-defined) semantic to those models.

I can only speculate on the scalability of formal methods based on my
experience. I suspect that they will scale just fine, provided that the
people doing the majority of the "methods" work can work in comfortable
surface syntaxes like UML and keep the Z, VDM, Larch, etc. stuff hidden
under the covers. If anyone wants to do theorem proof of some interesting
property, they are free to do so. Simply take the existing UML model and
translate it into the underlying formal language equivalent and run the
analysis on that. Every property proven about the formal representation
has to apply to the UML version because they are equivalent
semantics--they are just in a different syntax.

I don't have to speculate on the scalability of the ("semi-formal"?)
model-based development process. I've personally been involved on projects
that had more than 250 programmers working for about 5 years on code bases
up to about 5-10 million SLOC. The projects delivered on time (or early)
and the users were amazed by how few defects they encountered in actual
use. We just have to find a way to get the corporate decision makers to
notice...



-- steve



-----Original Message-----
From: Heath Raftery <hraftery at restech.net.au>
Organization: ResTech Pty Ltd
Date: Wednesday, February 19, 2014 3:35 PM
To: "systemsafety at lists.techfak.uni-bielefeld.de"
<systemsafety at lists.techfak.uni-bielefeld.de>
Subject: Re: [SystemSafety] Logic

On 19/02/2014 11:28 PM, Michael J. Pont wrote:
> It may - of course - be that the organisations I have closest contact
>with
> are atypical: they are, after all, a self-selecting group.  However,
>while
> I'm sure that there are many organisations that have mature processes in
> place for the development of real-time embedded systems, I'm equally sure
> that this isn't the norm.
>
> If we assume - for the moment - that my model is correct, how do we
>ensure
> that the situation is different in 10 years time?

Great points. I'd suggest that changes to education focus, while very
important, wont be the necessary trigger. There needs to be a market
force. The scenario that plays out in my world goes like this:

1. Customer C requests doodad D to solve problem P.
2. Engineer A says right, no problem, we just need to articulate the
requirements and capture them in an unambiguous way. Formal methods can
help, I'll show you the way.
3. Engineer B says, no problem, in fact here's a prototype I whipped up.
We're almost there.

Engineer A studied embedded development at an excellent facility and has
sound knowledge of formal methods.

Engineer B taught herself programming and has been writing code since
before she could drive.

4. A's manager asks how D is coming along and A says fine, we're working
through the requirements.
5. B's manager asks how D is coming along and B says fine, look I've got
the LEDs flashing and the relays clicking.

Guess which engineer gets rewarded?

_______________________________________________
The System Safety Mailing List
systemsafety at TechFak.Uni-Bielefeld.DE