I checked in a surprisingly large patch to change the way we collect a generation. Before: Things directly reachable from outside the young generation were moved into a 'reachable' set in one pass. Things indirectly reachable from outside the young generation were moved into 'reachable' in a second pass. The 'young' list then contained the unreachable objects. After: Things unreachable from outside the young generation are moved into an 'unreachable' set in one pass. The 'young' list contains the reachable objects when that's done. The point was that almost everything is reachable in the end, and moving an object between lists costs six pointer stores (updating prev and next pointers in the object, and in each of the two lists). So if most stuff is doomed to be reachable in the end, better to move the unreachable stuff than to move the reachable stuff. This seems to be a nice little win. If you want to know more, read the comments. An instrumented version showed this over a run of the Python test suite: scanned 7437363 moved 36854 movedback 34389 where scanned # of times the loop in move_unreachable() went around == # of objects moved # of times "if (gc->gc.gc_refs == 0)" in move_unreachable() triggered == # of objects moved into an unreachable set movedback # of times "if (gc_refs == GC_TENTATIVELY_UNREACHABLE)" in visit_reachable() triggered == the number of times move_unreachable() guessed wrong and an object had to be moved back into a reachable set So the change saved about 7e6 object moves here, for 6x as many pointer stores, and gc is finding very little that's unreachable in the end. Surprisingly, the worst (for some technical meaning of "worst" I'll leave to your imagination) stats I've seen came out of running Zope3's test suite: scanned 649444 moved 56124 movedback 43576 It's surprising for lots of reasons, including how relatively little work gc is doing in total, and how relatively much was found to be unreachable (about 12 thousand objects). The latter is surprising because Zope code has traditionally tried like heck not to create cycles. The Python test suite *almost* gave "the best" (most favorable to the change) stats I've seen. Only a variant of Kevin Jacobs's little test case looked better so far: scanned 12322200 moved 244 movedback 244 It would be nicer if we could drive scanned there down to 0 <wink>.
[MvL]
Do you think this should be backported to 2.2.2 as well?
I already backported the part that's arguably "a bugfix" (the part that would have solved Kevin's severe speed problem, had 2.2.1 not had a different bug that prevented the dramatic slowdown he saw in CVS Python). All that remains is new optimizations, and those can't be sold as a bugfix. Large (in the sense of lines of code) changes to gc really need to go thru lots of testing too. The optimizations involve some new algorithms, not just tweaking the former ones. So, no. If it were-- or evolves into --a dramatic speedup, maybe.
Tim Peters <tim.one@comcast.net> writes:
All that remains is new optimizations, and those can't be sold as a bugfix.
I understand that it is not a requirement anymore that changes to Python 2.2 are "pure bugfixes". Instead, people expect that Python 2.2 evolves and continues to grow new features, as long as they are "strictly backwards compatible". For any user-visible feature, it is normally debatable whether it is "strictly backwards compatible", since it is, by nature, a change in observable behaviour. This specific case is not in that category (i.e. has no user-observable behaviour change), so I think it qualifies for 2.2 - provided there is enough trust in its correctness.
Large (in the sense of lines of code) changes to gc really need to go thru lots of testing too. The optimizations involve some new algorithms, not just tweaking the former ones.
I'm concerned that backporting more changes to Python 2.2 will become difficult in that area, if the GC implementations vary significantly. Maybe this can be reconsidered when there actually is another change to backport. Regards, Martin
[martin@v.loewis.de]
I understand that it is not a requirement anymore that changes to Python 2.2 are "pure bugfixes". Instead, people expect that Python 2.2 evolves and continues to grow new features, as long as they are "strictly backwards compatible".
Alex made a case here for "new features", but the Python Business Forum hasn't shown interest in that. Like most businessfolk, I expect they'll ignore such issues until someone discovers that the lack of a new feature is putting them out of business <0.8 wink>.
For any user-visible feature, it is normally debatable whether it is "strictly backwards compatible", since it is, by nature, a change in observable behaviour.
This specific case is not in that category (i.e. has no user-observable behaviour change), so I think it qualifies for 2.2 - provided there is enough trust in its correctness.
The "bugfix part" of these changes certainly had user-visible aspects, in that before it was possible for objects in older generations to get yanked back into younger generations. This can affect when objects get collected, and so throw off over-tuned programs slinging gc.enable() and disable() "at exactly the best time(s)".
... I'm concerned that backporting more changes to Python 2.2 will become difficult in that area, if the GC implementations vary significantly.
Maintaining multiple branches is always a PITA.
Maybe this can be reconsidered when there actually is another change to backport.
Anyone who is so inclined is welcome to reconsider it non-stop <wink>.
Tim Peters wrote:
[martin@v.loewis.de]
I understand that it is not a requirement anymore that changes to Python 2.2 are "pure bugfixes". Instead, people expect that Python 2.2 evolves and continues to grow new features, as long as they are "strictly backwards compatible".
Alex made a case here for "new features", but the Python Business Forum hasn't shown interest in that. Like most businessfolk, I expect they'll ignore such issues until someone discovers that the lack of a new feature is putting them out of business <0.8 wink>.
Patch level releases should *never* include new features (unless these are essential to fix a serious bug or a simple byproduct of a fix). I don't know where you got the impression that Python should move back to the 1.5 branch development process where patch levels added new features. W/r to the PBF: at EuroPython we did a poll to see which version to base the PBF's activities on. The result was that a majority voted for Python 2.2 as first target. Patch levels are there to stabilize a release, not make it more powerful. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
[MaL, replying to me, but presumably bonding with Martin again <wink>]
Patch level releases should *never* include new features (unless these are essential to fix a serious bug or a simple byproduct of a fix). I don't know where you got the impression that Python should move back to the 1.5 branch development process where patch levels added new features.
The pre-PBF Patch Czars generally took a hard "no new features!" stance, but it seems to be up in the air now.
W/r to the PBF: at EuroPython we did a poll to see which version to base the PBF's activities on. The result was that a majority voted for Python 2.2 as first target.
Cool! Good choice.
Patch levels are there to stabilize a release, not make it more powerful.
This is one popular view, although there's plenty of wiggle room in what "stabilize" means (e.g., is it "stabilizing" to port Python to a new platform? to speed a bottleneck? to add a new encoding? etc).
Tim Peters wrote:
[MaL, replying to me, but presumably bonding with Martin again <wink>]
Patch level releases should *never* include new features (unless these are essential to fix a serious bug or a simple byproduct of a fix). I don't know where you got the impression that Python should move back to the 1.5 branch development process where patch levels added new features.
The pre-PBF Patch Czars generally took a hard "no new features!" stance, but it seems to be up in the air now.
I wonder why... just because Fossetts can't get back to solid ground doesn't mean we have to follow him ;-)
W/r to the PBF: at EuroPython we did a poll to see which version to base the PBF's activities on. The result was that a majority voted for Python 2.2 as first target.
Cool! Good choice.
Patch levels are there to stabilize a release, not make it more powerful.
This is one popular view, although there's plenty of wiggle room in what "stabilize" means (e.g., is it "stabilizing" to port Python to a new platform? to speed a bottleneck? to add a new encoding? etc).
"Stabilize" should mean to make triggering bugs in a Python release less likely. I don't think that porting to a new platform falls under this definition, a new encoding might (but then only if the encoding is so popular that people consider its absence a bug), performance tweaks are probably within range if they are in the micro-optimization area and hidden within the interpreter. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
"M.-A. Lemburg" <mal@lemburg.com> writes:
I don't think that porting to a new platform falls under this definition, a new encoding might (but then only if the encoding is so popular that people consider its absence a bug)
Here we go. If "many people consider absence of foo a bug" is enough to allow for a change, I can backport any change if I only find enough people to testify that absence of that change is a bug... Regards, Martin
Martin v. Loewis wrote:
"M.-A. Lemburg" <mal@lemburg.com> writes:
I don't think that porting to a new platform falls under this definition, a new encoding might (but then only if the encoding is so popular that people consider its absence a bug)
Here we go. If "many people consider absence of foo a bug" is enough to allow for a change, I can backport any change if I only find enough people to testify that absence of that change is a bug...
No, I was not talking about a missing foo; the comment was specifically about an encoding. Adding a new encoding would not need applications to be changed since the encoding information is part of the processed data. Anyway, if this confuses too much, simply go for the more restrictive: no new features at all paradigm. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
"M.-A. Lemburg" <mal@lemburg.com> writes:
Patch level releases should *never* include new features (unless these are essential to fix a serious bug or a simple byproduct of a fix). I don't know where you got the impression that Python should move back to the 1.5 branch development process where patch levels added new features.
Martin v. Loewis wrote:
"M.-A. Lemburg" <mal@lemburg.com> writes:
Patch level releases should *never* include new features (unless these are essential to fix a serious bug or a simple byproduct of a fix). I don't know where you got the impression that Python should move back to the 1.5 branch development process where patch levels added new features.
From discussions on python-dev...
Patch levels are there to stabilize a release, not make it more powerful.
What precisely does that mean?
Mainly that only bugs should be fixed. Adding new features doesn't help in fixing bugs since you can't expect that existing code for a particular Python branch will get changed to make use of it. Stabilizing means that code using the existing features in a branch runs more stable, i.e. there are fewer situations where a program can trigger a bug hiding in the Python release.
Specific case in question: xml.dom.minidom.toxml does not support the specification of an encoding of the resulting XML document. Instead, if there are non-ASCII characters in the output document, it returns a Unicode object that starts with u"<?xml version='1.0' ?>". People cannot write this to a file as-is, and they cannot encode it in anything but UTF-8 (because the document would then be incorrect).
So I added an optional encoding= argument to .toxml, for 2.3. The question now is: should that argument also be made available for 2.2.2?
Adding the argument would only help applications which would make use of it. An application written for Python 2.2 couldn't do this since the optional argument wouldn't be available. BTW, the above is trying to fix an application bug rather than a Python one: if the application cannot deal with Unicode, it is not non-ASCII compatible. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
"MAL" == M <mal@lemburg.com> writes:
MAL> Adding the argument would only help applications which would MAL> make use of it. An application written for Python 2.2 MAL> couldn't do this since the optional argument wouldn't be MAL> available. Ok, here's another question. When I updated the email package in Python 2.3, Guido wanted me to backport it to Python 2.2.x. I did that once, but there's been a lot of changes since then, both bug fixes, API "fixes", and new functionality. The email package can be installed separately as a distutils package, and it is compatible all the way back to Python 2.1.x. Which means someone /could/ install the latest version in their site-packages and have the new functionality in any of the last 3 versions of Python, although it would be tricky for Python 2.2.1. So does it make sense to backport the latest email package to Python 2.2.2? That's what Guido wanted, and I could argue that doing so improves stability of that branch, because while it adds a lot of new stuff, the old stuff was fairly well broken. E.g. you can't properly encode RFC 2047 headers in Python 2.2.1's email package. Backporting allows application writers to fix their code so that it works compatibly and correctly across more versions of Python than if we didn't backport. It also makes no sense to maintain two different code bases (especially now that that's been reduced from 3! :). OTOH, it definitely adds new features. Maybe email is special because it was so new in Python 2.2, and so I took a more naive approach to some issues that a wider use uncovered. it-ain't-always-simple-ly y'rs, -Barry
[people ask assorted hypothetical questions about backporting] Note that if the PBF is a success (and I sure hope that it is!), backporting stuff to the py-in-a-tie release line is supposed to become its job, not Python-Dev's. They'll backport whatever they see fit, and it won't matter what even Guido thinks then. In the meantime, I suggest *we* stick to backporting unarguable bugfixes. How can you tell whether something is unarguable? If in doubt, backport it, and if someone complains, tell them to revert it <0.8 wink>.
Barry A. Warsaw wrote:
"MAL" == M <mal@lemburg.com> writes:
MAL> Adding the argument would only help applications which would MAL> make use of it. An application written for Python 2.2 MAL> couldn't do this since the optional argument wouldn't be MAL> available.
Ok, here's another question. When I updated the email package in Python 2.3, Guido wanted me to backport it to Python 2.2.x. I did that once, but there's been a lot of changes since then, both bug fixes, API "fixes", and new functionality.
The email package can be installed separately as a distutils package, and it is compatible all the way back to Python 2.1.x. Which means someone /could/ install the latest version in their site-packages and have the new functionality in any of the last 3 versions of Python, although it would be tricky for Python 2.2.1.
So does it make sense to backport the latest email package to Python 2.2.2? That's what Guido wanted, and I could argue that doing so improves stability of that branch, because while it adds a lot of new stuff, the old stuff was fairly well broken. E.g. you can't properly encode RFC 2047 headers in Python 2.2.1's email package. Backporting allows application writers to fix their code so that it works compatibly and correctly across more versions of Python than if we didn't backport. It also makes no sense to maintain two different code bases (especially now that that's been reduced from 3! :).
OTOH, it definitely adds new features. Maybe email is special because it was so new in Python 2.2, and so I took a more naive approach to some issues that a wider use uncovered.
it-ain't-always-simple-ly y'rs,
Never said it was... :-) For cases like the email package or distutils, I think it's perfectly OK to only provide the updates for older Python releases as separate download. Both have their own way of life, so IMHO this is acceptable. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
"M.-A. Lemburg" <mal@lemburg.com> writes:
For cases like the email package or distutils, I think it's perfectly OK to only provide the updates for older Python releases as separate download. Both have their own way of life, so IMHO this is acceptable.
In neither case, this is really possible: Once you have the package in the Python core, a separate installation in site-packages cannot override the core implementation. I believe that was the motivation for Barry to consider backporting large amounts of changes. The same holds for distutils, except that there aren't that many major changes. Regards, Martin
"Martin v. Loewis" <martin@v.loewis.de> wrote:
"M.-A. Lemburg" <mal@lemburg.com> writes:
For cases like the email package or distutils, I think it's perfectly OK to only provide the updates for older Python releases as separate download. Both have their own way of life, so IMHO this is acceptable.
In neither case, this is really possible: Once you have the package in the Python core, a separate installation in site-packages cannot override the core implementation.
This might sound clueless, but wouldn't it be a good idea to change that? So that site-packages comes before Lib/ in sys.path? Gerhard -- mail: gerhard.haering@gmx.de registered Linux user #64239 web: http://www.cs.fhm.edu/~ifw00065/ OpenPGP public key id 86AB43C0 public key fingerprint: DEC1 1D02 5743 1159 CD20 A4B6 7B22 6575 86AB 43C0 reduce(lambda x,y:x+y,map(lambda x:chr(ord(x)^42),tuple('zS^BED\nX_FOY\x0b')))
Gerhard Häring <gerhard.haering@gmx.de> writes:
This might sound clueless, but wouldn't it be a good idea to change that? So that site-packages comes before Lib/ in sys.path?
No, this is by design, to prevent people from overriding the standard library. Essentially, all module names in the standard library are reserved; this procedure enforces that (somewhat, you can always insert things in the beginning of sys.path). Regards, Martin
Martin v. Loewis wrote:
Gerhard Häring <gerhard.haering@gmx.de> writes:
This might sound clueless, but wouldn't it be a good idea to change that? So that site-packages comes before Lib/ in sys.path?
No, this is by design, to prevent people from overriding the standard library. Essentially, all module names in the standard library are reserved; this procedure enforces that (somewhat, you can always insert things in the beginning of sys.path).
Uhm, just for the record: all paths defined in PYTHONPATH are inserted before the std lib dirs in sys.path on startup, so the "restriction" is not really all that restrictive. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
Martin v. Loewis wrote:
"M.-A. Lemburg" <mal@lemburg.com> writes:
For cases like the email package or distutils, I think it's perfectly OK to only provide the updates for older Python releases as separate download. Both have their own way of life, so IMHO this is acceptable.
In neither case, this is really possible: Once you have the package in the Python core, a separate installation in site-packages cannot override the core implementation.
True, but it is easily possible to install those packages in a directory which is scanned before the standard lib, thus overriding the distribution versions: python setup.py install install-lib=~/lib
I believe that was the motivation for Barry to consider backporting large amounts of changes. The same holds for distutils, except that there aren't that many major changes.
If that's the case, then we probably ought to make it easier for user installed Python add-ons to override builtin packages. This would help to get rid off the hacks which the PyXML distribution has to use in order to achieve the same. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
"M.-A. Lemburg" <mal@lemburg.com> writes:
True, but it is easily possible to install those packages in a directory which is scanned before the standard lib, thus overriding the distribution versions:
python setup.py install install-lib=~/lib
Why is ~/lib scanned before the standard lib? Regards, Martin
Martin v. Loewis wrote:
"M.-A. Lemburg" <mal@lemburg.com> writes:
True, but it is easily possible to install those packages in a directory which is scanned before the standard lib, thus overriding the distribution versions:
python setup.py install install-lib=~/lib
Why is ~/lib scanned before the standard lib?
Because I have it defined in PYTHONPATH :-) As I said, perhaps we need to make it easier to override std lib packages... -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
"MAL" == M <mal@lemburg.com> writes:
MAL> True, but it is easily possible to install those packages in MAL> a directory which is scanned before the standard lib, thus MAL> overriding the distribution versions: MAL> python setup.py install install-lib=~/lib A general solution that requires uses to set environment variables isn't acceptable IMO. >> I believe that was the motivation for Barry to consider >> backporting large amounts of changes. The same holds for >> distutils, except that there aren't that many major changes. MAL> If that's the case, then we probably ought to make it easier MAL> for user installed Python add-ons to override builtin MAL> packages. +1 MAL> This would help to get rid off the hacks which the PyXML MAL> distribution has to use in order to achieve the same. Yup, see my previous response. -Barry
"MvL" == Martin v Loewis <martin@v.loewis.de> writes:
>> For cases like the email package or distutils, I think it's >> perfectly OK to only provide the updates for older Python >> releases as separate download. Both have their own way of life, >> so IMHO this is acceptable. MvL> In neither case, this is really possible: Once you have the MvL> package in the Python core, a separate installation in MvL> site-packages cannot override the core implementation. MvL> I believe that was the motivation for Barry to consider MvL> backporting large amounts of changes. The same holds for MvL> distutils, except that there aren't that many major changes. Exactly. For my own purposes (e.g. Mailman) it's no problem; I provide my own email package and arrange for MM to use it before the Python standard one. I actually think it's the right thing for a normal distutils install to not override the standard version. But I also think there is a use case for allowing a standard package to be separately upgraded for a particular Python installation. As more and more standard Python libraries are packagized, they will probably have life-cycles separate from the Python core themselves (this will only be more true once we evolve toward a CPAN-like arrangement). So I think we will eventually need a way to upgrade (not override :) a standard library package. My suggestion would be to prepend a new directory on the standard search path, let's call it site-upgrade for now. A normal "python setup.py install" would still install to site-packages, but we'd add a "python setup.py upgrade" command that would install to site-upgrade. -Barry
Barry A. Warsaw wrote:
But I also think there is a use case for allowing a standard package to be separately upgraded for a particular Python installation. As more and more standard Python libraries are packagized, they will probably have life-cycles separate from the Python core themselves (this will only be more true once we evolve toward a CPAN-like arrangement). So I think we will eventually need a way to upgrade (not override :) a standard library package.
+1
My suggestion would be to prepend a new directory on the standard search path, let's call it site-upgrade for now. A normal "python setup.py install" would still install to site-packages, but we'd add a "python setup.py upgrade" command that would install to site-upgrade.
+1 (maybe with s/site-upgrade/system-packages) Not sure whether it's already possible or not, but I'd prefer to keep the install command and have the package provide this information (site-packages vs. system-packages) as part of the setup.py or setup.cfg file. Perhaps we could have some kind of category for distutils packages which marks them as system add-ons vs. site add-ons. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
"M.-A. Lemburg" <mal@lemburg.com> writes:
Perhaps we could have some kind of category for distutils packages which marks them as system add-ons vs. site add-ons.
One approach would be for distutils to have a list of system packages built-in, depending on the Python release. That list would cover PyXML and email; perhaps others. Of couse, taking the _xmlplus hack out of PyXML will cause backwards compatibility problems (regardless what the alternative hook is). Regards, Martin
On 8 Jul 2002 at 18:24, Martin v. Loewis wrote:
Of couse, taking the _xmlplus hack out of PyXML will cause backwards compatibility problems (regardless what the alternative hook is).
How? As long as "import xml" gets them _xmlplus, I can't see how it would break anything. I'd say it's broken already, since code written for _xmlplus assumes a different contract, and that is completely implicit. -- Gordon http://www.mcmillan-inc.com/
"Gordon McMillan" <gmcm@hypernet.com> writes:
Of couse, taking the _xmlplus hack out of PyXML will cause backwards compatibility problems (regardless what the alternative hook is).
How? As long as "import xml" gets them _xmlplus, I can't see how it would break anything.
Of course, once the hack that is taken out of PyXML, there won't be any _xmlplus anymore. I was thinking about applications that package Python applications, like freeze or Installer. People might have taken into account that they have to look inside _xmlplus as well. If the hack changes, they have to take into account that they need to look somewhere else, instead. Regards, Martin
On 8 Jul 2002 at 19:03, Martin v. Loewis wrote:
"Gordon McMillan" <gmcm@hypernet.com> writes:
Of couse, taking the _xmlplus hack out of PyXML will cause backwards compatibility problems (regardless what the alternative hook is).
How? As long as "import xml" gets them _xmlplus, I can't see how it would break anything.
Of course, once the hack that is taken out of PyXML, there won't be any _xmlplus anymore.
I was thinking about applications that package Python applications, like freeze or Installer. People might have taken into account that they have to look inside _xmlplus as well. If the hack changes, they have to take into account that they need to look somewhere else, instead.
py2exe doesn't do _xmlplus (unless that's changed recently) - Thomas has people overlay xml with _xmlplus. Installer does do it, but it's a horrid hack (one bad hack deserves another) and I'd be delighted to remove it. -- Gordon http://www.mcmillan-inc.com/
"MAL" == M <mal@lemburg.com> writes:
>> My suggestion would be to prepend a new directory on the >> standard search path, let's call it site-upgrade for now. A >> normal "python setup.py install" would still install to >> site-packages, but we'd add a "python setup.py upgrade" command >> that would install to site-upgrade. MAL> +1 (maybe with s/site-upgrade/system-packages) I like that: system-packages. MAL> Not sure whether it's already possible or not, but I'd prefer MAL> to keep the install command and have the package provide this MAL> information (site-packages vs. system-packages) as part of MAL> the setup.py or setup.cfg file. Ok, yeah. I think it would be a good idea for the package to somehow register itself as an upgrade to an existing system package. I still want the install command to install to site-packages, but whether the upgrade happens as an upgrade command or "python setup.py install -U" or some other mechanism is up for grabs. -Barry
Barry A. Warsaw wrote:
"MAL" == M <mal@lemburg.com> writes:
>> My suggestion would be to prepend a new directory on the >> standard search path, let's call it site-upgrade for now. A >> normal "python setup.py install" would still install to >> site-packages, but we'd add a "python setup.py upgrade" command >> that would install to site-upgrade.
MAL> +1 (maybe with s/site-upgrade/system-packages)
I like that: system-packages.
MAL> Not sure whether it's already possible or not, but I'd prefer MAL> to keep the install command and have the package provide this MAL> information (site-packages vs. system-packages) as part of MAL> the setup.py or setup.cfg file.
Ok, yeah. I think it would be a good idea for the package to somehow register itself as an upgrade to an existing system package. I still want the install command to install to site-packages, but whether the upgrade happens as an upgrade command or "python setup.py install -U" or some other mechanism is up for grabs.
Hmm, maybe I wasn't clear enough: I think that a distutils package should have a flag in its setup.py which lets distutils tell whether it's a site package or a system package, e.g. setup(... pkgtype='site-package' ...) vs. setup(... pkgtype='system-package' ...) (with pkgtype='site-package' as default value if not given) The user would in both cases type 'python setup.py install' but the install command would automatically choose the right target subdir (site-packages/ or system-packages/). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
"MAL" == M <mal@lemburg.com> writes:
MAL> Hmm, maybe I wasn't clear enough: I think that a distutils MAL> package should have a flag in its setup.py which lets MAL> distutils tell whether it's a site package or a system MAL> package, e.g. | setup(... pkgtype='site-package' ...) | vs. | setup(... pkgtype='system-package' ...) MAL> (with pkgtype='site-package' as default value if not given) MAL> The user would in both cases type 'python setup.py install' MAL> but the install command would automatically choose the MAL> right target subdir (site-packages/ or system-packages/). Except you can't always tell if its a system package or an add-on. email for example is an add-on for Python 2.1, but a system package for Python 2.2. Ignoring this specific example for now (since none of this will exist until Py2.3 anyway), it seems to me that there will be future packages for which this is true too. In that case, hardwiring site vs. system in the package's setup.py isn't the right approach. -Barry
barry wrote:
MAL> The user would in both cases type 'python setup.py install' MAL> but the install command would automatically choose the MAL> right target subdir (site-packages/ or system-packages/).
Except you can't always tell if its a system package or an add-on. email for example is an add-on for Python 2.1, but a system package for Python 2.2.
assuming that the package maintainer is informed when a package is added to the standard library, that packages won't move in and out too much, and/or that most users probably don't want to down- grade to an older package version, you could of course write: if sys.version_info >= (2, 2): pkgtype = "system-package" else: pkgtype = "site-package" setup(... pkgtype=pkgtype ...) in your setup.py file, once your package has been added. </F>
Fredrik Lundh wrote:
barry wrote:
MAL> The user would in both cases type 'python setup.py install' MAL> but the install command would automatically choose the MAL> right target subdir (site-packages/ or system-packages/).
Except you can't always tell if its a system package or an add-on. email for example is an add-on for Python 2.1, but a system package for Python 2.2.
assuming that the package maintainer is informed when a package is added to the standard library, that packages won't move in and out too much, and/or that most users probably don't want to down- grade to an older package version, you could of course write:
if sys.version_info >= (2, 2): pkgtype = "system-package" else: pkgtype = "site-package"
setup(... pkgtype=pkgtype ...)
in your setup.py file, once your package has been added.
Right. A package author whose package moves into the core would have to do this anyway, if s/he wants to maintain backwards compatibility with older Python versions, since the distutils package in those versions would not accept the new keyword. Anyway, regardless of how we do it, we need to add the 'system-packages' dir to just before the '.../lib/pythonX.X' entry in sys.path. If there's consent about this, I'd suggest to move ahead in this direction as first step. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
"MAL" == M <mal@lemburg.com> writes:
MAL> Anyway, regardless of how we do it, we need to add the MAL> 'system-packages' dir to just before the '.../lib/pythonX.X' MAL> entry in sys.path. If there's consent about this, I'd suggest MAL> to move ahead in this direction as first step. +1 Perhaps also backport to 2.2 and (maybe? maybe not?) 2.1. -Barry
"MAL" == M <mal@lemburg.com> writes:
MAL> Anyway, regardless of how we do it, we need to add the MAL> 'system-packages' dir to just before the '.../lib/pythonX.X' MAL> entry in sys.path. If there's consent about this, I'd suggest MAL> to move ahead in this direction as first step.
+1
Perhaps also backport to 2.2 and (maybe? maybe not?) 2.1.
Smells like a new feature to me, so -1 on a 2.2 backport. I haven't seen enough of this thread to comment on this for 2.3. --Guido van Rossum (home page: http://www.python.org/~guido/)
On 08 July 2002, M.-A. Lemburg said:
Not sure whether it's already possible or not, but I'd prefer to keep the install command and have the package provide this information (site-packages vs. system-packages) as part of the setup.py or setup.cfg file.
Perhaps we could have some kind of category for distutils packages which marks them as system add-ons vs. site add-ons.
+1 -- this should definitely be up to the package author/packager, not the local admin. I once tried to convince Guido that the ability to occasionally upgrade standard library modules/packages would be a good thing, but he wasn't having it. Any change of heart, O Mighty BDFL? Greg -- Greg Ward - Python bigot gward@python.net http://starship.python.net/~gward/ What the hell, go ahead and put all your eggs in one basket.
Perhaps we could have some kind of category for distutils packages which marks them as system add-ons vs. site add-ons.
+1 -- this should definitely be up to the package author/packager, not the local admin. I once tried to convince Guido that the ability to occasionally upgrade standard library modules/packages would be a good thing, but he wasn't having it. Any change of heart, O Mighty BDFL?
Before I answer that, here's a question. Why do we think it's a good idea to distribute upgrades as separate add-ons while we don't think it's okay to distribute such upgrades with bugfix releases? Doesn't this just increase the variability of site configurations, and hence version interaction hell? --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
Perhaps we could have some kind of category for distutils packages which marks them as system add-ons vs. site add-ons.
+1 -- this should definitely be up to the package author/packager, not the local admin. I once tried to convince Guido that the ability to occasionally upgrade standard library modules/packages would be a good thing, but he wasn't having it. Any change of heart, O Mighty BDFL?
Before I answer that, here's a question. Why do we think it's a good idea to distribute upgrades as separate add-ons while we don't think it's okay to distribute such upgrades with bugfix releases?
The idea is to provide bugfixes for Python versions which are no longer being maintained. Of course, the effect would only show a few years ahead.
Doesn't this just increase the variability of site configurations, and hence version interaction hell?
I don't think that core packages are any different than other third party packages: they are usually independent enough from the rest of the code that upgrades don't affect the workings of the other code using it. The internals are free to change, though, e.g. to accomodate bug fixes, etc. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
Guido van Rossum wrote:
Perhaps we could have some kind of category for distutils packages which marks them as system add-ons vs. site add-ons.
+1 -- this should definitely be up to the package author/packager, not the local admin. I once tried to convince Guido that the ability to occasionally upgrade standard library modules/packages would be a good thing, but he wasn't having it. Any change of heart, O Mighty BDFL?
Before I answer that, here's a question. Why do we think it's a good idea to distribute upgrades as separate add-ons while we don't think it's okay to distribute such upgrades with bugfix releases?
[MAL]
The idea is to provide bugfixes for Python versions which are no longer being maintained. Of course, the effect would only show a few years ahead.
Hm, if you really are fixing bugs in old versions, why not patch the Python installation in-place rather than trying to play nice?
Doesn't this just increase the variability of site configurations, and hence version interaction hell?
I don't think that core packages are any different than other third party packages: they are usually independent enough from the rest of the code that upgrades don't affect the workings of the other code using it. The internals are free to change, though, e.g. to accomodate bug fixes, etc.
Well, I don't expect that we'll do independent upgrades for core packages, so I propose to end this thread. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
Guido van Rossum wrote:
Perhaps we could have some kind of category for distutils packages which marks them as system add-ons vs. site add-ons.
+1 -- this should definitely be up to the package author/packager, not the local admin. I once tried to convince Guido that the ability to occasionally upgrade standard library modules/packages would be a good thing, but he wasn't having it. Any change of heart, O Mighty BDFL?
Before I answer that, here's a question. Why do we think it's a good idea to distribute upgrades as separate add-ons while we don't think it's okay to distribute such upgrades with bugfix releases?
[MAL]
The idea is to provide bugfixes for Python versions which are no longer being maintained. Of course, the effect would only show a few years ahead.
Hm, if you really are fixing bugs in old versions, why not patch the Python installation in-place rather than trying to play nice?
We don't have an easy way of doing this, unless of course we trick python setup.py install to install directly into .../lib/pythonX.X rather than a sub directory on the path.
Doesn't this just increase the variability of site configurations, and hence version interaction hell?
I don't think that core packages are any different than other third party packages: they are usually independent enough from the rest of the code that upgrades don't affect the workings of the other code using it. The internals are free to change, though, e.g. to accomodate bug fixes, etc.
Well, I don't expect that we'll do independent upgrades for core packages, so I propose to end this thread.
Barry is already doing this with the email package and I would expect more such packages to make their way into the core. The PyXML package also has a life of its own outside the core distribution and could benefit from this. I think it's too early to end the thread. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
On Tuesday 02 July 2002 21:06, Tim Peters wrote:
[martin@v.loewis.de]
I understand that it is not a requirement anymore that changes to Python 2.2 are "pure bugfixes". Instead, people expect that Python 2.2 evolves and continues to grow new features, as long as they are "strictly backwards compatible".
Alex made a case here for "new features", but the Python Business Forum hasn't shown interest in that.
As a Python Business Forum member and board member, I think I can state that if a (business) case is indeed made, the PBF interest is there.
Like most businessfolk, I expect they'll ignore such issues until someone discovers that the lack of a new feature is putting them out of business <0.8 wink>.
I suspect instead that a businessperson clever enough to pick Python rather than heavily-hyped Java or widely-popular Perl or PHP is most likely to be an unusually clever businessperson, with some level of perception of what programming productivity is worth and how to get it.
For any user-visible feature, it is normally debatable whether it is "strictly backwards compatible", since it is, by nature, a change in observable behaviour.
This specific case is not in that category (i.e. has no user-observable behaviour change), so I think it qualifies for 2.2 - provided there is enough trust in its correctness.
The "bugfix part" of these changes certainly had user-visible aspects, in that before it was possible for objects in older generations to get yanked back into younger generations. This can affect when objects get collected, and so throw off over-tuned programs slinging gc.enable() and disable() "at exactly the best time(s)".
Performance change is not quite the same thing as behavior change. I agree with Martin that, assuming a performance-oriented change is 'known' to be correct (no change in the inputs-to-outputs behavior of programs), the criterion should be one of overall benefit rather han one of Pareto optima.
I'm concerned that backporting more changes to Python 2.2 will become difficult in that area, if the GC implementations vary significantly.
Maintaining multiple branches is always a PITA.
Yes, but the degree of pain varies with the branches' separation.
Maybe this can be reconsidered when there actually is another change to backport.
Anyone who is so inclined is welcome to reconsider it non-stop <wink>.
I suspect we'll indeed reconsider it. Whether we do something about it after the reconsideration will depend on cost-benefit analysis... Alex
On Mon, 1 Jul 2002, Tim Peters wrote:
I checked in a surprisingly large patch to change the way we collect a generation. [...] The point was that almost everything is reachable in the end, and moving an object between lists costs six pointer stores (updating prev and next pointers in the object, and in each of the two lists). So if most stuff is doomed to be reachable in the end, better to move the unreachable stuff than to move the reachable stuff. [...] It would be nicer if we could drive scanned there down to 0 <wink>.
This change may be a short-term win if I can get Jeremy's idea working. It involves temporarily untracking objects with known external roots, so many more objects become unreachable. These include objects stored on the c-eval stack, local variables in the current frame, and possibly other select places. I have no idea if this approach will make enough of a difference to be worthwhile, but it seems like a worthy experiment. -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com
participants (12)
-
Alex Martelli -
barry＠zope.com -
Fredrik Lundh -
Gerhard Häring -
Gordon McMillan -
Greg Ward -
Guido van Rossum -
Kevin Jacobs -
M.-A. Lemburg -
martin＠v.loewis.de -
Tim Peters -
Tim Peters