Tim Peters writes:

I checked in a surprisingly large patch to change the way we collect a generation.

Do you think this should be backported to 2.2.2 as well? Regards, Martin

[MvL]

Do you think this should be backported to 2.2.2 as well?

I already backported the part that's arguably "a bugfix" (the part that would have solved Kevin's severe speed problem, had 2.2.1 not had a different bug that prevented the dramatic slowdown he saw in CVS Python). All that remains is new optimizations, and those can't be sold as a bugfix. Large (in the sense of lines of code) changes to gc really need to go thru lots of testing too. The optimizations involve some new algorithms, not just tweaking the former ones. So, no. If it were-- or evolves into --a dramatic speedup, maybe.

[martin@v.loewis.de]

I understand that it is not a requirement anymore that changes to Python 2.2 are "pure bugfixes". Instead, people expect that Python 2.2 evolves and continues to grow new features, as long as they are "strictly backwards compatible".

Alex made a case here for "new features", but the Python Business Forum hasn't shown interest in that. Like most businessfolk, I expect they'll ignore such issues until someone discovers that the lack of a new feature is putting them out of business <0.8 wink>.

For any user-visible feature, it is normally debatable whether it is "strictly backwards compatible", since it is, by nature, a change in observable behaviour.

This specific case is not in that category (i.e. has no user-observable behaviour change), so I think it qualifies for 2.2 - provided there is enough trust in its correctness.

The "bugfix part" of these changes certainly had user-visible aspects, in that before it was possible for objects in older generations to get yanked back into younger generations. This can affect when objects get collected, and so throw off over-tuned programs slinging gc.enable() and disable() "at exactly the best time(s)".

... I'm concerned that backporting more changes to Python 2.2 will become difficult in that area, if the GC implementations vary significantly.

Maintaining multiple branches is always a PITA.

Maybe this can be reconsidered when there actually is another change to backport.

Anyone who is so inclined is welcome to reconsider it non-stop <wink>.

[MaL, replying to me, but presumably bonding with Martin again <wink>]

Patch level releases should *never* include new features (unless these are essential to fix a serious bug or a simple byproduct of a fix). I don't know where you got the impression that Python should move back to the 1.5 branch development process where patch levels added new features.

The pre-PBF Patch Czars generally took a hard "no new features!" stance, but it seems to be up in the air now.

W/r to the PBF: at EuroPython we did a poll to see which version to base the PBF's activities on. The result was that a majority voted for Python 2.2 as first target.

Cool! Good choice.

Patch levels are there to stabilize a release, not make it more powerful.

This is one popular view, although there's plenty of wiggle room in what "stabilize" means (e.g., is it "stabilizing" to port Python to a new platform? to speed a bottleneck? to add a new encoding? etc).

Martin v. Loewis wrote:

"M.-A. Lemburg" writes:

...

Patch level releases should *never* include new features (unless these are essential to fix a serious bug or a simple byproduct of a fix). I don't know where you got the impression that Python should move back to the 1.5 branch development process where patch levels added new features.

...

From discussions on python-dev...

...

Patch levels are there to stabilize a release, not make it more powerful.

What precisely does that mean?

Mainly that only bugs should be fixed. Adding new features doesn't help in fixing bugs since you can't expect that existing code for a particular Python branch will get changed to make use of it. Stabilizing means that code using the existing features in a branch runs more stable, i.e. there are fewer situations where a program can trigger a bug hiding in the Python release.

Specific case in question: xml.dom.minidom.toxml does not support the specification of an encoding of the resulting XML document. Instead, if there are non-ASCII characters in the output document, it returns a Unicode object that starts with u"<?xml version='1.0' ?>". People cannot write this to a file as-is, and they cannot encode it in anything but UTF-8 (because the document would then be incorrect).

So I added an optional encoding= argument to .toxml, for 2.3. The question now is: should that argument also be made available for 2.2.2?

Adding the argument would only help applications which would make use of it. An application written for Python 2.2 couldn't do this since the optional argument wouldn't be available. BTW, the above is trying to fix an application bug rather than a Python one: if the application cannot deal with Unicode, it is not non-ASCII compatible. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

...

...

...

...

"MAL" == M writes:

MAL> Adding the argument would only help applications which would MAL> make use of it. An application written for Python 2.2 MAL> couldn't do this since the optional argument wouldn't be MAL> available. Ok, here's another question. When I updated the email package in Python 2.3, Guido wanted me to backport it to Python 2.2.x. I did that once, but there's been a lot of changes since then, both bug fixes, API "fixes", and new functionality. The email package can be installed separately as a distutils package, and it is compatible all the way back to Python 2.1.x. Which means someone /could/ install the latest version in their site-packages and have the new functionality in any of the last 3 versions of Python, although it would be tricky for Python 2.2.1. So does it make sense to backport the latest email package to Python 2.2.2? That's what Guido wanted, and I could argue that doing so improves stability of that branch, because while it adds a lot of new stuff, the old stuff was fairly well broken. E.g. you can't properly encode RFC 2047 headers in Python 2.2.1's email package. Backporting allows application writers to fix their code so that it works compatibly and correctly across more versions of Python than if we didn't backport. It also makes no sense to maintain two different code bases (especially now that that's been reduced from 3! :). OTOH, it definitely adds new features. Maybe email is special because it was so new in Python 2.2, and so I took a more naive approach to some issues that a wider use uncovered. it-ain't-always-simple-ly y'rs, -Barry

Martin v. Loewis wrote:

"M.-A. Lemburg" writes:

...

I don't think that porting to a new platform falls under this definition, a new encoding might (but then only if the encoding is so popular that people consider its absence a bug)

Here we go. If "many people consider absence of foo a bug" is enough to allow for a change, I can backport any change if I only find enough people to testify that absence of that change is a bug...

No, I was not talking about a missing foo; the comment was specifically about an encoding. Adding a new encoding would not need applications to be changed since the encoding information is part of the processed data. Anyway, if this confuses too much, simply go for the more restrictive: no new features at all paradigm. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

Barry A. Warsaw wrote:

...

...

...

...

...

"MAL" == M writes:

MAL> Adding the argument would only help applications which would MAL> make use of it. An application written for Python 2.2 MAL> couldn't do this since the optional argument wouldn't be MAL> available.

Ok, here's another question. When I updated the email package in Python 2.3, Guido wanted me to backport it to Python 2.2.x. I did that once, but there's been a lot of changes since then, both bug fixes, API "fixes", and new functionality.

The email package can be installed separately as a distutils package, and it is compatible all the way back to Python 2.1.x. Which means someone /could/ install the latest version in their site-packages and have the new functionality in any of the last 3 versions of Python, although it would be tricky for Python 2.2.1.

So does it make sense to backport the latest email package to Python 2.2.2? That's what Guido wanted, and I could argue that doing so improves stability of that branch, because while it adds a lot of new stuff, the old stuff was fairly well broken. E.g. you can't properly encode RFC 2047 headers in Python 2.2.1's email package. Backporting allows application writers to fix their code so that it works compatibly and correctly across more versions of Python than if we didn't backport. It also makes no sense to maintain two different code bases (especially now that that's been reduced from 3! :).

OTOH, it definitely adds new features. Maybe email is special because it was so new in Python 2.2, and so I took a more naive approach to some issues that a wider use uncovered.

it-ain't-always-simple-ly y'rs,

Never said it was... :-) For cases like the email package or distutils, I think it's perfectly OK to only provide the updates for older Python releases as separate download. Both have their own way of life, so IMHO this is acceptable. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

On Tuesday 02 July 2002 21:06, Tim Peters wrote:

[martin@v.loewis.de]

...

I understand that it is not a requirement anymore that changes to Python 2.2 are "pure bugfixes". Instead, people expect that Python 2.2 evolves and continues to grow new features, as long as they are "strictly backwards compatible".

Alex made a case here for "new features", but the Python Business Forum hasn't shown interest in that.

As a Python Business Forum member and board member, I think I can state that if a (business) case is indeed made, the PBF interest is there.

Like most businessfolk, I expect they'll ignore such issues until someone discovers that the lack of a new feature is putting them out of business <0.8 wink>.

I suspect instead that a businessperson clever enough to pick Python rather than heavily-hyped Java or widely-popular Perl or PHP is most likely to be an unusually clever businessperson, with some level of perception of what programming productivity is worth and how to get it.

...

For any user-visible feature, it is normally debatable whether it is "strictly backwards compatible", since it is, by nature, a change in observable behaviour.

This specific case is not in that category (i.e. has no user-observable behaviour change), so I think it qualifies for 2.2 - provided there is enough trust in its correctness.

The "bugfix part" of these changes certainly had user-visible aspects, in that before it was possible for objects in older generations to get yanked back into younger generations. This can affect when objects get collected, and so throw off over-tuned programs slinging gc.enable() and disable() "at exactly the best time(s)".

Performance change is not quite the same thing as behavior change. I agree with Martin that, assuming a performance-oriented change is 'known' to be correct (no change in the inputs-to-outputs behavior of programs), the criterion should be one of overall benefit rather han one of Pareto optima.

...

I'm concerned that backporting more changes to Python 2.2 will become difficult in that area, if the GC implementations vary significantly.

Maintaining multiple branches is always a PITA.

Yes, but the degree of pain varies with the branches' separation.

...

Maybe this can be reconsidered when there actually is another change to backport.

Anyone who is so inclined is welcome to reconsider it non-stop <wink>.

I suspect we'll indeed reconsider it. Whether we do something about it after the reconsideration will depend on cost-benefit analysis... Alex

Martin v. Loewis wrote:

"M.-A. Lemburg" writes:

...

For cases like the email package or distutils, I think it's perfectly OK to only provide the updates for older Python releases as separate download. Both have their own way of life, so IMHO this is acceptable.

In neither case, this is really possible: Once you have the package in the Python core, a separate installation in site-packages cannot override the core implementation.

True, but it is easily possible to install those packages in a directory which is scanned before the standard lib, thus overriding the distribution versions: python setup.py install install-lib=~/lib

I believe that was the motivation for Barry to consider backporting large amounts of changes. The same holds for distutils, except that there aren't that many major changes.

If that's the case, then we probably ought to make it easier for user installed Python add-ons to override builtin packages. This would help to get rid off the hacks which the PyXML distribution has to use in order to achieve the same. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

"M.-A. Lemburg" writes:

True, but it is easily possible to install those packages in a directory which is scanned before the standard lib, thus overriding the distribution versions:

python setup.py install install-lib=~/lib

Why is ~/lib scanned before the standard lib? Regards, Martin

Martin v. Loewis wrote:

Gerhard Häring writes:

...

This might sound clueless, but wouldn't it be a good idea to change that? So that site-packages comes before Lib/ in sys.path?

No, this is by design, to prevent people from overriding the standard library. Essentially, all module names in the standard library are reserved; this procedure enforces that (somewhat, you can always insert things in the beginning of sys.path).

Uhm, just for the record: all paths defined in PYTHONPATH are inserted before the std lib dirs in sys.path on startup, so the "restriction" is not really all that restrictive. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

...

...

...

...

"MAL" == M writes:

MAL> True, but it is easily possible to install those packages in MAL> a directory which is scanned before the standard lib, thus MAL> overriding the distribution versions: MAL> python setup.py install install-lib=~/lib A general solution that requires uses to set environment variables isn't acceptable IMO. >> I believe that was the motivation for Barry to consider >> backporting large amounts of changes. The same holds for >> distutils, except that there aren't that many major changes. MAL> If that's the case, then we probably ought to make it easier MAL> for user installed Python add-ons to override builtin MAL> packages. +1 MAL> This would help to get rid off the hacks which the PyXML MAL> distribution has to use in order to achieve the same. Yup, see my previous response. -Barry

"M.-A. Lemburg" writes:

Perhaps we could have some kind of category for distutils packages which marks them as system add-ons vs. site add-ons.

One approach would be for distutils to have a list of system packages built-in, depending on the Python release. That list would cover PyXML and email; perhaps others. Of couse, taking the _xmlplus hack out of PyXML will cause backwards compatibility problems (regardless what the alternative hook is). Regards, Martin

"Gordon McMillan" writes:

...

Of couse, taking the _xmlplus hack out of PyXML will cause backwards compatibility problems (regardless what the alternative hook is).

How? As long as "import xml" gets them _xmlplus, I can't see how it would break anything.

Of course, once the hack that is taken out of PyXML, there won't be any _xmlplus anymore. I was thinking about applications that package Python applications, like freeze or Installer. People might have taken into account that they have to look inside _xmlplus as well. If the hack changes, they have to take into account that they need to look somewhere else, instead. Regards, Martin

...

...

...

...

"MAL" == M writes:

>> My suggestion would be to prepend a new directory on the >> standard search path, let's call it site-upgrade for now. A >> normal "python setup.py install" would still install to >> site-packages, but we'd add a "python setup.py upgrade" command >> that would install to site-upgrade. MAL> +1 (maybe with s/site-upgrade/system-packages) I like that: system-packages. MAL> Not sure whether it's already possible or not, but I'd prefer MAL> to keep the install command and have the package provide this MAL> information (site-packages vs. system-packages) as part of MAL> the setup.py or setup.cfg file. Ok, yeah. I think it would be a good idea for the package to somehow register itself as an upgrade to an existing system package. I still want the install command to install to site-packages, but whether the upgrade happens as an upgrade command or "python setup.py install -U" or some other mechanism is up for grabs. -Barry

On 08 July 2002, M.-A. Lemburg said:

Not sure whether it's already possible or not, but I'd prefer to keep the install command and have the package provide this information (site-packages vs. system-packages) as part of the setup.py or setup.cfg file.

Perhaps we could have some kind of category for distutils packages which marks them as system add-ons vs. site add-ons.

+1 -- this should definitely be up to the package author/packager, not the local admin. I once tried to convince Guido that the ability to occasionally upgrade standard library modules/packages would be a good thing, but he wasn't having it. Any change of heart, O Mighty BDFL? Greg -- Greg Ward - Python bigot gward@python.net http://starship.python.net/~gward/ What the hell, go ahead and put all your eggs in one basket.

barry wrote:

MAL> The user would in both cases type 'python setup.py install' MAL> but the install command would automatically choose the MAL> right target subdir (site-packages/ or system-packages/).

Except you can't always tell if its a system package or an add-on. email for example is an add-on for Python 2.1, but a system package for Python 2.2.

assuming that the package maintainer is informed when a package is added to the standard library, that packages won't move in and out too much, and/or that most users probably don't want to down- grade to an older package version, you could of course write: if sys.version_info >= (2, 2): pkgtype = "system-package" else: pkgtype = "site-package" setup(... pkgtype=pkgtype ...) in your setup.py file, once your package has been added. </F>

...

...

...

...

"MAL" == M writes:

MAL> Anyway, regardless of how we do it, we need to add the MAL> 'system-packages' dir to just before the '.../lib/pythonX.X' MAL> entry in sys.path. If there's consent about this, I'd suggest MAL> to move ahead in this direction as first step. +1 Perhaps also backport to 2.2 and (maybe? maybe not?) 2.1. -Barry

...

Perhaps we could have some kind of category for distutils packages which marks them as system add-ons vs. site add-ons.

+1 -- this should definitely be up to the package author/packager, not the local admin. I once tried to convince Guido that the ability to occasionally upgrade standard library modules/packages would be a good thing, but he wasn't having it. Any change of heart, O Mighty BDFL?

Before I answer that, here's a question. Why do we think it's a good idea to distribute upgrades as separate add-ons while we don't think it's okay to distribute such upgrades with bugfix releases? Doesn't this just increase the variability of site configurations, and hence version interaction hell? --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:

...

...

...

Perhaps we could have some kind of category for distutils packages which marks them as system add-ons vs. site add-ons.

+1 -- this should definitely be up to the package author/packager, not the local admin. I once tried to convince Guido that the ability to occasionally upgrade standard library modules/packages would be a good thing, but he wasn't having it. Any change of heart, O Mighty BDFL?

Before I answer that, here's a question. Why do we think it's a good idea to distribute upgrades as separate add-ons while we don't think it's okay to distribute such upgrades with bugfix releases?

[MAL]

The idea is to provide bugfixes for Python versions which are no longer being maintained. Of course, the effect would only show a few years ahead.

Hm, if you really are fixing bugs in old versions, why not patch the Python installation in-place rather than trying to play nice?

...

Doesn't this just increase the variability of site configurations, and hence version interaction hell?

I don't think that core packages are any different than other third party packages: they are usually independent enough from the rest of the code that upgrades don't affect the workings of the other code using it. The internals are free to change, though, e.g. to accomodate bug fixes, etc.

Well, I don't expect that we'll do independent upgrades for core packages, so I propose to end this thread. --Guido van Rossum (home page: http://www.python.org/~guido/)