[Python-Dev] Proposal to revert r54204 (splitext change)

[glyph at divmod.com]

On 08:43 pm, jeremy at alum.mit.edu wrote:
>On 3/15/07, glyph at divmod.com <glyph at divmod.com> wrote:
>>On 05:51 pm, pje at telecommunity.com wrote:
>> >At 07:45 AM 3/15/2007 +0100, Martin v. Löwis wrote:
>> >>I apparently took the same position that you now take back then,
>> >>whereas I'm now leaning towards (or going beyond) the position
>> >>Tim had back then, who wrote "BTW, if it *weren't* for the code 
>>breakage,
>> >>I'd be in favor of doing this."
>> >
>> >If it weren't for the code breakage, I'd be in favor too.  That's not 
>>the
>> >point.
>> >
>> >The point is that how can Python be stable as a language if 
>>precedents can
>> >be reversed without a migration plan, just because somebody changes 
>>their
>> >mind?  In another five years, will you change your mind again, and 
>>decide
>> >to put this back the way it was?
>>
>>Hear, hear.  Python is _not_ stable as a language.  I have Java 
>>programs
>>that I wrote almost ten years ago which still run perfectly on the 
>>latest
>>runtime.  There is python software I wrote two years ago which doesn't 
>>work
>>right on 2.5, and some of the Python stuff contemporary with that Java 
>>code
>>won't even import.
>
>I think the problem has less to do with bug fixing than with lack of
>any clear specifications or documentation about what developers can
>depend on.    You could probably make a case that any change that
>doesn't fix a crash bug is likely to cause some particular program to
>behave differently.

Absolutely.  One of the reasons I'm working on documenting this process 
is that, of course, *everything* can't be made compatible.  The mere act 
of adding a function or module adds a detectable change in behavior that 
some program *might* insanely depend on.

A clear understanding of what is meant by "backwards compatibility" is 
equally important to developers trying to future-proof their code as it 
is to those trying to make sure they don't break code which has been 
future-proofed.  This is a form of social contract, and both sides need 
to know about it.
>Take bug 1504333 which lead to a change in sgmllib behavior for angle
>brackets in quoted attribute values.  Did the sgmllib documentation
>explain that the fixed behavior was incorrect?  Might a programmer
>working with sgmllib have written code that depended on this bug?  Do
>you object to this bug fix?

I don't know enough about the specification to say for sure, but I 
suspect that it is a legitimate bug fix, because sgmllib is implementing 
an externally-determined spec.  In cases where the spec is flagrantly 
violated, it seems like it should be fixed to adhere to it.
>For many of these bugs, some people will have written code against the
>documentation and some people against the implementation or behavior.
>(In this case, the documentation is vague or conflicting.)  I don't
>think I know how to balance the important of these two classes of
>users.  Some code is going to break the first time they run into the
>under-specific edge case, some code is going to break when the
>specification and implementation are clarified.  You have to weigh
>which you think is more likely and which will benefit users the most.

If the documentation is vague and conflicting, then it seems likely that 
a parsing option could be added.  I am not advocating perfect, 100% 
backwards compatibility, merely some standards for what happens when a 
(potentially) incompatible change is made.  For example, you could add a 
flag to the parser which tweaks the treatment of quoted angle brackets, 
and warn if the argument is not passed that the default will change in 
the future (or, better yet, that the argument will be required in the 
future).  Or, you could provide a separate name for invoking the 
different behavior.
>I think everyone wants to do the "right thing" by Python's users, but
>it's not clear what that right thing is.

I really think that starting with the "golden rule" would be a good 
idea.  Would Python core developers mind if something analogous in the C 
runtime changed?  How would they cope with it?  What kind of feedback 
would you expect the C compiler or runtime to provide in such a case? 
Python should do unto others, etc.
>Could you point out a few such programs that people on python-dev can
>look at?  I think it would be useful to gather some data about the
>kind of migration pains real users are having.  I believe Martin and
>others are trying to do the right thing.  Real data is more likely to
>convince them than passionate arguments on python-dev.

(I assume you're responding to my other comment about my programs not 
running, even though that's not what you quoted.)

I don't think these programs would contribute much to the discussion. 
I've probably got them archived somewhere, but they were broken circa 
2.3 and I don't think I've run them since.  I doubt they would make any 
sense to anyone here, and we would all get into a heated debate as to 
whether my usage of Python was valid or not (hint: it was *REALLY* 
gross).

In fact, let's back up a step.  These programs were never released as 
open-source.  Pretend for a moment that it is beyond my power to reveal 
the highly proprietary for loops and if statements contained therein to 
the general public.  Why should python-dev get to see my source code? 
Microsoft doesn't have access to millions of enterprises' code all over 
the world, and yet, they manage to provide a (relatively) stable 
platform for developing such code.  The Mozilla foundation doesn't 
receive sources from Adobe, but the flash plugin manages to run on their 
browser for years at a time with no change.  Sun does not require IBM 
reveal their source code to keep WebSphere working with new releases of 
Java.  When was the last time you needed to approach the Linux kernel 
mailing list with some of your code to discuss their rapid evolution 
which was breaking all your applications?

I am sympathetic to the Python team's desire to evolve and collaborate 
with the community, so I am overstating the case a bit here.  I think 
that this experiment in a platform that actually has meaningful changes 
and deprecations going on that require developer participation is 
interesting, and probably the way that all software will eventually be 
moving.

My point is that Python is going to be operating at a massive 
disadvantage to other languages and environments by being a living, 
dynamic, evolving environment in the first place.  To combat that, it 
needs to be as conservative as possible about the types of change that 
are accepted.  The process of evolution needs to be tightly managed, and 
very good, highly explicit channels of communication are necessary in 
the code, on the website, in the releases, on the mailing lists, and so 
on.

I keep hearing the "NEWS" file bandied about as the solution here, and 
that isn't going to cut it.  The change that caused the most angst to 
fix in Twisted for Python 2.5 was buried on line 751 of the NEWS file, 
in the "what's new" for one of the betas.  We discovered it because of 
our test coverage, we would never have found it in the NEWS file.

There were HUNDREDS of other issues in that file, many of which have 
absolutely no observable effect on the semantics of _any_ working python 
program, like minor fixes for obvious bugs like memory-management 
issues.  These are mixed in with no prioritization amongst and zingers 
like new-style exceptions and id()'s return value changing.

Of course, there is *some* code I can disclose.  Have a look here:
    http://twistedmatrix.com/trac/ticket/1842#comment:5
Migrating Twisted from Python *2.4* to *2.5* took over a week, and 
several people.  There were 10 separate issues that needed to be fixed. 
This is very recent, very active code, that had already had some 
preparation from a few people following beta releases, not some gnarly 
old legacy application.
>I think you missed the point here.  The hypothetical question was not
>about any particular majority, but rather that regardless of which
>group you poll, the majority decision may not be the right one.  Even
>a majority of Twised users :-).

Erm... perhaps I was insufficiently clear.  My comments there were that 
this is yet another example of letting muddy thinking and vague feelings 
drive the development process.  If the development process were 
democratic (and let me be crystal clear, I think that democratic 
development is a _VERY BAD_ idea) then at least it would be an explicit 
process, not just one guy thinking that maybe a lot of people like a 
particular change.
>>According to that highly scientific study, at this point in time, 
>>"Nobody
>>disagrees" :).  (One in favor, zero against.)

(And *this* was just a joke...)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070315/0e5abe6d/attachment-0001.htm