[Python-Dev] Re: Stability and change

Alex Martelli aleax@aleax.it
Mon, 8 Apr 2002 10:20:04 +0200

On Monday 08 April 2002 00:08, Guido van Rossum wrote:
> Alex, please try brevity. :-)
> > It would help to some degree if the "stable" releases could be shown
> > to have SOME kind of enhancement.  Not as much as if language
> > changes which did not break existing correct code could also be
> > there, but some.  Modules such as email are a big part of the draw
> > of 2.2, for example (not quite as big as language-level enhancements,
> > but big).
> Now there's a good proposal.  But it costs time and effort that takes
> away from our main mission (and PythonLabs is already under a lot of
> pressure).

You know PythonLabs's missions and goals better than I do.  If such
goals weigh "making a new and better Python" far more highly than
helping spread and preserve the use of Python, then you may well be
right that the extra effort for (e.g.) making email part of the standard
distribution vs the current "maintaining a separate distro" would be a
bad allocation of scarce resources.

>  I wonder if there's a commercial market for this?  Maybe
> ActiveState could back-port important packages or language changes and
> insert them into ActivePython?

If I understand correctly, ActiveState has backed off (because of not
sufficient revenue returns) from "owning" Hammond's win32all
extensions, which are truly indispensable to firms using Python for
Windows-only development.  If win32all did not "have a commercial
market" (sufficient to justify ActiveState's investments), then I don't see 
what other packages could.

> [proposal to semi-backport bool snipped]
> > Thus it would help us send the message I think prospective users
> > want to hear -- that Python stands for stability AND change, that
> > there is not necessarily a contradiction between the two goals.  In
> > a very minor way, to be sure.
> Given the requirements you give elsewhere in your message, I'm not
> sure how to satisfy all these constraints.  Let's be concrete.  Should
> we backport stuff to 2.1 (the earliest release I'm willing to
> consider)?  If so, what?  Look at the "What's New?" section in
> http://www.python.org/2.2/.  The new Windows installer has already
> been backported to 2.1.  Type-class unification is obviously out.  So
> are interators and generators (too much effort).  So is // division (I
> think).  Returning longs instead of raising OverflowError on int
> operations would probably break too much code, so it's better not to

Something requiring much effort is not necessarily "obviously out": it
depends on how important you judge the results of those effort versus
the results you could have by spending the effort elsewhere.  In terms
of "how it looks to actual and prospective users of Python" rather than
"how much effort would it be to implement", iterators (not generators)
and // division look good -- enabling "for x in dict" and "if x in dict" are
two features of instant appeal -- language enhancements that do not 
break previously correct code.

It's of course possible to construct a program that would break due to
any such change, because it expects and relies upon a specific
exception being raised when such usage is attempted -- when the
usage becomes legal and meaningful, crack.  But in most cases that
would be a "sea-lawyer" level issue.  You're probably right that, alas,
overflow on int operations wouldn't be feasible to remove (I don't
know).  Syntax changes that make some source code legal where
it didn't use to be, e.g. // , are probably at the other extreme -- in
practice one would not rely on e.g. compile('a//b') throwing a
SyntaxException and thus break if it doesn't.

To "be concrete" as you ask, I think that targeting 2.1 as the stable
release would not be the best allocation of effort, even though I see
others disagree.  Basically, the argument against that would be that
2.2 was _announced_ as the new stable release, not as a bleeding
edge experimental one, and making it "retroactively experimental"
would send exactly the wrong message.  So, I think effort would be
better spent ensuring that enhancements for 2.3 are well divided
between [a] those which are safe to have in some 2.2.* as well, vs
[b] those which ARE experimental (break previously correct code) 
and thus should only go into 2.3.

> backport.  That leaves the email package and xmlrpclib (both available
> as separate distros!), and large file support -- which I believe *did*
> get ported to 2.1.2.

The fact that email and xmlrpclib are already installable as separate
distros on 2.1.* suggests the amount of effort to put them into the
standard library of 2.1.something might be modest; therefore, even if
the advantage is also modest, it might still be worth doing.

> > More relevantly, I'm not sure I had managed to explain exactly why I
> > think "stability and change" IS important to Python adoption in a
> > certain category of software shops.
> Unfortunately, it's not clear what adaptation in that (apparently very
> change-resistant) category buys *Python*.  IOW why should we, mostly
> volunteers working for the community, care?

Assuming one cares about Python being widely used, "capturing" a new
"customer" that is highly change-resistant has better returns than
getting one who's very change-prone.  Exactly because of that user's
inertia, they'll stick with Python once they've decided for it -- some
other shop who'll gladly have a carefree fling with Python is just as
liable to leave it for Ruby, or whatever, tomorrow.

Apparently we DO want to promote the spread of Python (otherwise
the purpose of the python marketing list and the PSF becomes murky
to me).  We believe the world will be a better place if more development
uses Python rather than different languages.  That goal is well served
by helping Python acceptance by development shops that do care about
stability, because such shops are a substantial fraction of our industry.

Over the decades, the people who are now the technical managers
and senior developers in such shops have seen too much of their work
laid waste by incompatible changes in platforms, and have thus become
quite sensitive to that specific technological risk; they've seen some
other fraction of their efforts similarly wasted by having been spent on
platforms who became "really stable" (i.e., dead) and are thus also
sensitized to that antisymmetric risk.  These are survival traits in the
software jungle.  Our industry suffers from so many "bandwagons" (in
platforms, methodologies, tools, etc) that learning when NOT to jump
on one -- developing a nose for the symptoms of the two opposed
risks, excessive/churning rate of change and the "sunset boulevard"
syndrome of a slowly dying technology -- make a shop's survival more
likely.  So, I think we need to avoid triggering either reflex (in sensible
cases: we can't really make Logajan happy, I think).

That is, IF we want to promote Python's spread -- if we do believe
we'll be happier if Python is widely accepted rather than less used
than it deserves to be.  I'm reasonably sure of that, myself.  The more
Python is widespread, the more of a "Python market" commercial
ventures perceive as existing, the better my chances to find a good
Python port to some weird new platform, the better some other
enthusiast's chances to find jobs based on Python (or mine, should
my current arrangements fail or terminate), the better tools I'm likely
to have my pick amongst, and so forth.  I didn't think this needed to
be re-examined on this thread, but I'm quite game if it does.

> > What numbering or naming scheme we adopt has importance in
> > determining what we communicate, but before that, deciding what we
> > DO want to communicate is probably more important.
> Indeed.  But I'd love it if all we need to do is change our
> communication patterns, because that takes much less effort than fully
> maintaining two separate release tracks.

I understand that doing nothing except changing communication patterns
entails less effort than other changes.  Exactly because of that, the
other changes send a stronger message: they bespeak real commitment
to "stability AND change" _because_ they don't come for free.  Economy
Nobel prizes last year all went to economists whose major work has been
on this subject: asymmetric-information situations and message-sending.

For example, that's how advertising, sponsoring sports, etc, mostly work:
a firm sends a message "we really believe in this product" by splurging a
lot of money in promoting it, _because_ it's substantial effort -- effort it
would make no sense for them to spend if they (who have more info on
the product than the prospective buyer) really knew it was a dud.  I'm
summarizing and oversimplifying, but that's the gist.

> > I think such a "retroactive" rebranding of 2.2 as "bleeding-edge" would
> > be a serious blow to those of us who have accepted it as stable
> > enough as our base over the last 3 months -- in books, magazine
> > articles, advocacy within our firms and outside of them, etc.  For
> > example, AB Strakt has decided 2.2 was stable enough for us (and
> > it was being presented as such) -- our new code now uses
> > "for x in somedict:", "if x in somedict:", etc, for example -- now
> > hearing that it wasn't, and 2.2.X micro-releases might introduce
> > backwards-incompatible changes, would be a real problem indeed.
> I'm happy to hear that 2.2 is stable for you -- exactly as I
> intended and expected.  But what confuses me is that while you're
> happy with 2.2's stability, you don't want to testify of this fact to
> the more conservative shops.  Rather than telling them "honestly, I
> cannot recommend 2.2", why don't you tell them "I've used 2.2 for
> three months and it's as solid as a rock"?

I _do_ testify that all new code I've written or ported to 2.2 has run
without a hitch so far.  That's quite different from saying that existing
code written for 2.1 ran unchanged with 2.2, because it didn't -- nor
do I think it reasonably COULD have, mind you.  E.g., any time I
introspected an instance via dir() I only used to get per-instance
attributes -- now I get attributes it gets from its class too; it's an
enhancement, but it does fool any such introspection that I had.

Unit-tests might help, but unfortunately I relied on doctest for those,
so I get a zillion "breakage" reports that aren't all that helpful.  Guess
I misused doctest (but it's SO much handier than unittest that it's an
"attractive nuisance", consarn it!-).

Once you do manage to identify the real breakage points, it's not
(generally) all that hard to turn 2.1 code into 2.2 code.  People who
maintain both 2.1 and 2.2 versions of their packages are no doubt
better placed to evaluate that.  But the message that "only little
broke, and fixing it was not too terrible" is not strong enough for
the shop looking for stability -- which I guess help explains 2.1's
relative longevity, no?

> > I perceive Booleans as small (albeit negatively), but both our CTO and
> > principal investor have vehemently posted against them, so it would
> > be hard to 'sell' them even if I were quite favorable.
> If I didn't suspect that you, the CTI and the P.I. together are 75% of
> the company, I would say that they are micromanaging you. :-)

No: there's 12 of us, but we also have a strong commitment to *no
"code ownership"*.  We need to agree on coding standards that are
strong enough that we _can't tell_ who wrote what piece -- at
least, that's the ideal (and part of XP).  And the standards must get
consensus, which is no picnic (but the alternatives are worse).

> > Still, if that was the one hurdle to jump to get to a "stable and
> > changing" release, who knows.  But if jumping that small hurdle only
> > gets to more of the same, either unstability or feature-frozen
> > versions, the carrot may be too small.
> I suspect that the people who complain that Python's rate of change is
> too fast are misguided.  There's a lot of fuss on the newsgroup about
> potentially breaking code and undesired changes, and before you know
> it, everybody just *believes* that Python is unstable and changes too
> fast.  But if you try it, you find very little incompatibility, and
> what's there is easily dealt with.  The people on the newsgroup who
> complain that the introduction of bool is such a major change that it
> would only be acceptable in a major release (i.e. 3.0) vastly
> overestimate its impact, which is minuscule (zero in most cases).

The impact is small (that's my evaluation) in terms of maintenance of
existing code (not necessarily all that small in teaching the language
or writing about it, but that's another issue).  Total breakage of
doctests is probably the worst part (maybe it's just what we needed
to stop using doctests and go for other approaches instead).  Still,
"zero in most cases" is NOT the situation with doctest usage today: I
do believe I was not the only one to fall into its attractive-nuisance

> Logajan's main gripe (and also McMillan's biggest effort in keeping
> his codebase cross-version portable) was about things that changed
> between 1.5.2 and 2.0 -- that *was* a major release.  And at the time
> we didn't have PEPs 4-6, there was no warning module, and so on.
> So maybe there's only (or mostly) a problem of *perception*?

Mostly, yes, and I do agree that Logajan's impossible to satisfy -- I
see no amount of effort that could have kept him happy, given his
expressed complaints.

But if you're implying that said perception (by prospective users) is
unfounded, I disagree.  It has a solid basis in reality.  Some previously
correct code does break at each minor release (doctests first -- as
apparently is their "canary in a coal mine" intended role -- but that
is quite visible and contributes highly to perception).  To some extent
("sea-lawyering" level) that may be inevitable.  Moreover, the "status"
perceived for releases such as 2.1.2, etc, seems correct to me: you
do NOT want to invest substantial effort on them -- isn't this what
you're saying throughout this?  So, the perception "not quite dead but
close, BDFL doesn't care much at all about them" is correct, no?

It may be possible to change perception without changing reality, but
I have some doubt.  "You can fool some of the people all of the time",
etc, etc:-).