Is there a "Large Scale Python Software Design" ?

[Peter Hansen]

Jonathan Ellis wrote:
> Peter Hansen wrote:
>>I'm getting the impression you also haven't tried any significant
>>test-driven development.  
> 
> I think I responded to this already --
> 
>>>Testing is good; preventing entire classes of errors from ever
>>>happening at all is better, particularly when you get large.
> 
> Oh yes; so I did. :)  (See my reply to another subthread for one
> example of when static type checking saved me a LOT of work.)

And you've reemphasized my point.  "Testing" is not test-driven
development.  In fact, test-driven development is about *design*,
not just about testing.  The two are related, but definitely not
the same thing, and eliminating TDD with a wave of a hand intended
to poo-poo mere testing is to miss the point.  Once someone has
tried TDD, they are unlikely to lump it in with simple "unit testing"
as it has other properties that aren't obvious on the surface.

> What is the biggest system you have built with python personally?  I'm
> happy to be proven wrong, but honestly, the most enthusiastic "testing
> solves all my problem" people I have seen haven't worked on anything
> "large" -- and my definition of large agrees with Alex's; over 100
> kloc, more than a handful of developers.

The topic of the thread was large projects with _large teams_,
I thought, so I won't focus on my personal work.  The team I
was leading worked on code that, if I recall, was somewhat over
100,000 lines of Python code including tests.  I don't recall
whether that number was the largest piece, or combining several
separate applications which ran together but in a distributed
system...  I think there were close to 20 man years in the main
bit.

(And remembering that 1 line of Python code corresponds to
some larger number, maybe five or ten, of C code, that should
qualify it as a large project by many definitions.)

> So people don't get me wrong: I love python.  Most of my programming
> friends call me "the python zealot" behind my back.  I just don't think
> it's the right tool for every problem.

Neither do I.  The above project also involved some C and
some assembly, plus some Javascript and possibly something else
I've forgotten by now.  We just made efforts to use Python *as
much as possible* and it paid off.

> Specifically, in my experience, statically-typed languages make it much
> easier to say "okay, I'm fixing a bug in Class.Foo; here's all the
> places where it's used."  This lets me see how Foo is actually used --
> in a perfect world, Foo's documentation is precise and up to date, but
> I haven't worked anywhere that this was always the case -- which lets
> me make my fix with a reasonable chance of not breaking anything.
> Compile-time type checking increases those chances.  Unit tests
> increase that further, but relying on unit tests as your first and only
> line of defense is suboptimal when there are better options.

But what if you already had tests which allowed you to do exactly
the thing you describe?  Is there a need for "better options"
at that point?  Are they really better?  When I do TDD, I can
*trivially* catch all the cases where Class.Foo is used
because they are all exercised by the tests.  Furthermore, I
can catch real bugs, not just typos and simple things involving
using the wrong type.  A superset of the bugs your statically
typed language tools are letting you catch.  But obviously
I'm rehashing the argument, and one which has been discussed
here many times, so I should let it go.

>>Having experience with both approaches, and choosing one over
>>the other, gives one greater credibility than having experience
>>with just one approach, yet clinging to it...
> 
> You are incorrect if you assume I am unfamiliar with python.  

I assumed no such thing, just that you were unfamiliar with
large projects in Python and yet were advising the OP on its
suitability in that realm.  You're bright and experienced, and
your comments have substance, but until you've actually
participated in a large project with Python and seen it fail
gloriously *because it was not statically typed*, I wouldn't
put much weight on your comments in this area if I were the
OP.  That's all I was saying...

-Peter