waterfall (was Re: REPOST: Re: Book "python programming patterns". anybody read this??)

[Alex Martelli]

"Peter Milliken" <peter.milliken at gtech.com> wrote in message
news:a0vvrj$e862 at news1.gtech.com...
    ...
> > How do you KNOW that "the customer's requirements are definitive"?
>
> How can you BUILD something if you don't know what it is? If you are a

By *finding out* what it is.  The process of "building", in the widest 
sense of the word, is the process of finding out what is being built.

> software house then you have to get an agreed contractual basis as to what
> you deliver - otherwise you are always changing what you deliver - which
is

Remember that well over 90% of software development is in-house, so that the
specific business models you may have in mind are quite marginal in the
world of software development as a whole.  Further, a good slice of what is
NOT "in house" is development of software to be sold (or rented, subscribed
to, included as novelty gift in potato chip packs, etc) "off the shelf", so
there is absolutely no "agreed contractual basis" with your customers until
AFTER you have the software ready for sale (rent, subscription, etc).


> in software otherwise you go out of business backwards. Of course, if you
> have the "luxury" of developing for an inhouse part of the company then
you
> (the supplier) and the customer can screw around to your hearts content

Except that this is not the customer's interest: they want SOLUTIONS to
their problems, the latter often being rather ill-defined ones.  So, it's 
in their best interest to work with you to help find out exactly what they 
need, and with what priority in terms of what needs to work first, what can 
come later, what would "be nice to have" but not vital, and so on.

Further, YOUR interests are also to similarly develop successful, useful, 
usable, software systems -- systems that get used and deliver results.  
That's assuming you're an engineer at heart, rather than a bureaucrat whose 
key goals are following established procedure and always having somebody 
else to blame when things go wrong, of course.

> So nailing down the requirements is always the first step - agreement
means

It almost never is.  Use velcro or masking tape, not nails.

The requirements are going to be refined, prioritized, changed in scope, 
refactored, reprioritized, etc, throughout the life of a successful 
software system.  Those nails would get bent out of shape and rust far too 
soon to be any real use.

> failures that I have seen and read about have one underlying common
thread -
> the requirements were still being changed at the time the project was
> cancelled.

I.e., the customer was in the process of finding out what they actually
needed -- a perfectly normal interactive discovery process -- and somewhere
in some decision-making positions were people who did NOT realize this and
were not prepared for it.  Most likely because they harbored [expletive
deleted] notions about "nailing down the requirements is always the first
step".

BTW, you may not read about them as failures, but examples also abound
of software that WAS painstakingly specified in minute detail before any
step towards building it was taken, then later delivered -- and never
used, being totally irrelevant to the business needs.  In my (indirect)
experience, this tends to happen in public procurement, where laws and
regulations may indeed mandate such an absurd process.  It also happens
in development for "off-the-shelf" sale (etc), in which case it tends to
be more visible since the end product, despite fulfilling its "nailed down"
specs perfectly, doesn't sell.

It may not be something you will read about as a failure, but, both as
a taxpayer AND as an engineer, I consider these disasters, which are far
from unheard of, every bit as bad as other development disasters.


> > Let's say: use waterfall when you are ABSOLUTELY certain that the
> > customer's requirements are definitive and you have understood them
> > correctly and completely at once.  To make "absolutely certain" more
> > concrete, let's say that you will undertake to disembowel yourself
> > ritually if it turns out you were wrong -- that the requirements were
> > NOT quite as clear, well-communicated, well-understood, definitive
and/or
> > nailed-down as you thought.  If you're not willing to commit to this,
> > then you're not ABSOLUTELY certain, and you shouldn't use waterfall.
>
> I think you are getting too rigid here - all software models are based
upon
> the fundamental steps of requirements analyise, design, code and test.
Even

You may call then "steps" if you wish, and you have missed quite a few
crucial ones (such as "deploy", "document and/or train", "tune", ...),
but that doesn't make them anywhere as separate and/or sequential as all
that.

> models based upon rapid protyping follow the same basic steps (the
> requirements are very loose obviously, the aim of protyping is to refine
> them) - the protype is built to a set of understood requirements, the
> protype has some form of design, there is coding and there is testing.

"There is" understanding, design, coding, testing, and other things yet
"there are".  But the optimal sequencing, interleaving, merging and
splitting of the various activities are not optimally rigid -- YOU tell
ME "I am getting too rigid", I, on the other hand, claim Agile Development
processes are the very antithesis of rigidity and Waterfall IS rigidity in 
software development -- an inappropriate, inferior and unsuitable 
methodology.


> So I am getting confused by your responses - all you seem to do is blast
> waterfall - I haven't seen any attempt at some constructive response here,
a

All I want to do in this thread is make sure waterfall is and stay blasted,
since benighted discussors are trying to revive the zombie.  Excellent
textbooks, articles, courses etc, abound, on many superior alternatives,
such as Rational Unified Process, Extreme Programming, and so forth.

> proposal for some other model because it fits better into your concepts of
> software development or personal situations.

Read anything published in the field in the last few years and you'll
find nothing but.  Meanwhile, the monster of Waterfall needs to be
spiked through the heart each and every time it threatens to revive, which 
is basicall what I'm doing here.

> All I have said is that waterfall can and has worked in the past and

It can't and hasn't, in any relevant case (see below).

> shouldn't be dismissed upon the argument that "it hasn't worked in the
past,
> so therefore lets not consider it" (my interpretation of your responses).
> This seems a nonsense argument at best.

It has never worked, it can never work, it's an absurdity based on a
total misconception of the nature of programming, of engineering, of
the human mind, of human society, and of the universe.  There, good
enough for you?

You're of course driving me to rather extreme assertions, just as I
would be if (e.g.) debating some other similar absurdity which every
historical experience and common sense show can't work, yet upon
which altar blood (true or metaphorical) has been shed aplenty in
the past and threatens to be shed again until and unless the nightmare
is driven forever from the sublunar world.

There ARE trivial 'projects' (not worth the name of 'projects') which
basically consist in doing for the hundredth time much the same thing
as you've done the last 99 times.  Boring, but true; in this case the
amount of "interleaving" needed may diminish, potentially down to a
lower bound of zero.  This is much like saying that "communism" can
work within a small, close-knit family where everybody trust and love
each other: in such a scenario there is no need to account for "whose
money it is", who's earning and who's spending, and so on.  But it
*JUST DOESN'T SCALE* -- it's no basis for economics, which has to deal
with the vast majority of interesting cases, where people, far from
loving each other, may barely _stand_ each other.  Idealizing and
extrapolating the tiny "loving family" scenario leads to horrors that
range from endless queues outside of empty shops all the way to rivers
of blood down the streets.  Wrong-headed software development approaches
so far have less potential for utter disaster -- but, deaths HAVE resulted
from "waterfall" development (specifically its hubris-like lack of
check-backs and continuous verification), as any reader of the Risks
column will know.

Waterfall proponents traditionally wanted to dismiss such occurrences
as (paraphrasing) "operator error".  That's a good symptom of a system
that's not fit for use by human beings, and the rationalization attempts
for the system.  Sure, there's nothing wrong with communism per se, only
when you apply it to human beings do problems appear (for all I know,
Martians may be quite happy with it).  Perfect beings might be able to
apply Waterfall -- I don't really care about them (being perfect, they'll
surely be good at picking their methodologies anyway): "Know then thyself,
presume not God to scan; The proper study of Mankind is Man".

Man is fallible, and has evolved in an environment of trial-and-error, 
continuous feedback, continuous adjustment.  The worst failures come when 
one's pride rears up and proclaims that one HAS fully grasped the issues, 
so no more tentative, feedback, adjustments, etc, are possible -- when one 
denies the fallibility, or fails to accept the indispensable "checks and 
balances" that compensate for it and make it bearable.  "Pride comes before 
a fall".

On the subject of "operator error", see Perrow, "Normal Accidents"; and to 
some extent Norman, "Design of Everyday Things".  On inflexibility 
(resulting from excessive pride from past successes) as the key aspect in 
the (economic) fall of every Empire, Carlo Cipolla has an excellent short 
article in his delightful, highly mixed book "Le Tre Rivoluzioni" (there's 
probably an English version somewhere too, given that Cipolla has written 
and taught in English even more than in Italian -- I can look for it, if 
you want to try and read it).  Kennedy's "Rise and Fall of the Great 
Powers" is, to some extent, a look at the same issue with wider scope.

Waterfall is highly reminiscent of each of these general issues.


> any superior state. It appears more to me that the new genes entering the
> pool from colleges every year think they are the next Leonardo Da Vinci of

A failure, as an engineer, of course -- he applied "waterfall", rather than
iteratively testing and checking that he knew what he was doing.  So, none
of his machines ever really worked well, and some of his incredible works of
art (artists can afford far more pride than engineers can) were not as 
lasting as those of his contemporaries (who, more humbly, tried things in 
small increments and kept double-checking with customers...).

> software development, that they can ignore the lessons of the past and

I.e., human fallibility?  If new graduates are so full of misplaced pride 
that they think they can use Waterfall, rather than solid, interactive 
mingled-steps development, I entirely blame their teachers.

> stride out there with supreme confidence that they know what is right -
> hence people like me (who have been here for several decades now) see the
> same mistakes being made time and time again. I am sure that you are just

Yes, I do see Waterfall being praised -- but more likely by people my
contemporaries, such as you, rather than by young brilliant graduates (who 
are more likely, at least, to have read recent relevant literature).


> confident as I am that you "know what is right". I have seen what works

It's exactly because *I know that I don't know* (and that others don't know 
either), that I consider it a horror to praise and defend a methodology 
that could only work for perfect (and mind-reading) developers who are 
developing for idealized customers who DO know exactly what they need.

> what doesn't, I don't pretend that I know it all, but what I do know is
> correct for the circumstances I have experienced :-). So waterfall can and
> does work in the right context. So do the other models - there is no
single
> model that will work best for all circumstances.

No, but, as I don't care about what works or doesn't for hypothetical 
Martians, I do know there's a model that will work horribly in all 
interesting circumstances: it's called Waterfall.


> So, unless you are "absolutely certain" then I hope you are on a time and
> materials contract, otherwise there will be a win-win situation and the

That I am absolutely certain that Waterfall doesn't work for human beings 
doesn't mean I already know what does work best for any given situation, 
since there are so many other possibilities; it means I, and the rest of 
the team, _including the customer_, can *find out*, and interactively as 
well as iteratively develop and refine solutions and methods that "work 
well enough" to deliver significant business value, reliably and repeatably.


Alex