PEP: 0??? Title: Support for System Upgrades Version: $Revision: 0.0 $ Author: mal@lemburg.com (Marc-Andr? Lemburg) Status: Draft Type: Standards Track Python-Version: 2.3 Created: 19-Jul-2001 Post-History: Abstract This PEP proposes strategies to allow the Python standard library to be upgraded in parts without having to reinstall the complete distribution or having to wait for a new patch level release. Problem Python currently does not allow overriding modules or packages in the standard library per default. Even though this is possible by defining a PYTHONPATH environment variable (the paths defined in this variable are prepended to the Python standard library path), there is no standard way of achieving this without changing the configuration. Since Python's standard library is starting to host packages which are also available separately, e.g. the distutils, email and PyXML packages, which can also be installed independently of the Python distribution, it is desireable to have an option to upgrade these packages without having to wait for a new patch level release of the Python interpreter to bring along the changes. Proposed Solutions This PEP proposes two different but not necessarily conflicting solutions: 1. Adding a new standard search path to sys.path: $stdlibpath/system-packages just before the $stdlibpath entry. This complements the already existing entry for site add-ons $stdlibpath/site-packages which is appended to the sys.path at interpreter startup time. To make use of this new standard location, distutils will need to grow support for installing certain packages in $stdlibpath/system-packages rather than the standard location for third-party packages $stdlibpath/site-packages. 2. Tweaking distutils to install directly into $stdlibpath for the system upgrades rather than into $stdlibpath/site-packages. The first solution has a few advantages over the second: * upgrades can be easily identified (just look in $stdlibpath/system-packages) * upgrades can be deinstalled without affecting the rest of the interpreter installation * modules can be virtually removed from packages; this is due to the way Python imports packages: once it finds the top-level package directory it stay in this directory for all subsequent package submodule imports * the approach has an overall much cleaner design than the hackish install on top of an existing installation approach The only advantages of the second approach are that the Python interpreter does not have to changed and that it works with older Python versions. Both solutions require changes to distutils. These changes can also be implemented by package authors, but it would be better to define a standard way of switching on the proposed behaviour. Scope Solution 1: Python 2.3 and up Solution 2: all Python versions supported by distutils Credits None References None Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil End: -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
Any chance of getting one of the proposed solution into Python 2.3 ? http://www.python.org/peps/pep-0297.html -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
Any chance of getting one of the proposed solution into Python 2.3 ?
I'd be in favor of having an extra path item in front of the default path. Maybe instead of system-packages you could call it site-upgrades? We could also add this to Python 2.2.3 when it comes out and to Python 2.1.4 if and when it is released. --Guido van Rossum (home page: http://www.python.org/~guido/)
MAL, I think an idea like this is good, but I have one concern. Let's say Python 2.3 comes out with email pkg 2.5. Later I find some bugs and release email 2.5.1 and also fix the Python 2.3 maint branch. Now Python 2.3.1 comes out. At best the email package in system-packages should be ignored. At worse, it /has/ to be ignored. E.g. Python 2.3.2 comes out with email 2.5.2. My suggestion is that the Python micro-release number be included in the path to system-packages. IOW, system-packages must exactly match the Python version number, not just the maj.min number. -Barry
MAL, I think an idea like this is good, but I have one concern. Let's say Python 2.3 comes out with email pkg 2.5. Later I find some bugs and release email 2.5.1 and also fix the Python 2.3 maint branch. Now Python 2.3.1 comes out.
At best the email package in system-packages should be ignored. At worse, it /has/ to be ignored. E.g. Python 2.3.2 comes out with email 2.5.2.
My suggestion is that the Python micro-release number be included in the path to system-packages. IOW, system-packages must exactly match the Python version number, not just the maj.min number.
Excellent point. Or a micro-release should clear out the system-packages directory. --Guido van Rossum (home page: http://www.python.org/~guido/)
"GvR" == Guido van Rossum <guido@python.org> writes:
GvR> Excellent point. GvR> Or a micro-release should clear out the system-packages GvR> directory. The only reason I'd rather not do that is so that if a package still needs an update for the new Python micro release, a sysadmin could at least copy the package over from one version to the next. -Barry
Barry A. Warsaw wrote:
"GvR" == Guido van Rossum <guido@python.org> writes:
GvR> Excellent point.
GvR> Or a micro-release should clear out the system-packages GvR> directory.
The only reason I'd rather not do that is so that if a package still needs an update for the new Python micro release, a sysadmin could at least copy the package over from one version to the next.
+1 Ok, then, let's call the dir "site-upgrades-<pyversion>" with <pyversion> being major.minor.patchlevel. It seems that only site.py needs to be changed. Hmm, but what happens if someone invokes Python with -S (don't load site.py) ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
GvR> Or a micro-release should clear out the system-packages GvR> directory.
The only reason I'd rather not do that is so that if a package still needs an update for the new Python micro release, a sysadmin could at least copy the package over from one version to the next.
+1
Ok, then, let's call the dir "site-upgrades-<pyversion>" with <pyversion> being major.minor.patchlevel.
+1
It seems that only site.py needs to be changed. Hmm, but what happens if someone invokes Python with -S (don't load site.py) ?
They deserve what they get; they'll have to do their own sys.path manipulation. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
GvR> Or a micro-release should clear out the system-packages GvR> directory.
The only reason I'd rather not do that is so that if a package still needs an update for the new Python micro release, a sysadmin could at least copy the package over from one version to the next.
+1
Ok, then, let's call the dir "site-upgrades-<pyversion>" with <pyversion> being major.minor.patchlevel.
+1
It seems that only site.py needs to be changed. Hmm, but what happens if someone invokes Python with -S (don't load site.py) ?
They deserve what they get; they'll have to do their own sys.path manipulation.
Ok, I've started looking at adding support for this. Here's a couple of things I found: * getpath.c: Some of the '/' path delimiters are hard coded; shouldn't these be replaced with SEP ? * There's no easy way to find the first item on sys.path which starts the default path added by Python at startup time. It seems that a suffix search for "python23.zip" gives the best hint. The only other possibility I see is writing the support code directly into getpath.c. * site.py contains code which prefixes "site-packages" with both sys.prefix and sys.exec_prefix. Is this really used anywhere ? (distutils and the old Makefile.pre.in both install to sys.prefix per default) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
Ok, I've started looking at adding support for this. Here's a couple of things I found:
* getpath.c: Some of the '/' path delimiters are hard coded; shouldn't these be replaced with SEP ?
All the platforms that I'm awware of that don't use '/' have their own getpath.c copy anyway (the one for Windows is PC/getpathp.c).
* There's no easy way to find the first item on sys.path which starts the default path added by Python at startup time. It seems that a suffix search for "python23.zip" gives the best hint. The only other possibility I see is writing the support code directly into getpath.c.
That's where I'd put it, yes.
* site.py contains code which prefixes "site-packages" with both sys.prefix and sys.exec_prefix. Is this really used anywhere ? (distutils and the old Makefile.pre.in both install to sys.prefix per default)
I thought they might install extension modules in exec_prefix. But maybe it's a YAGNI. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
Ok, I've started looking at adding support for this. Here's a couple of things I found:
* getpath.c: Some of the '/' path delimiters are hard coded; shouldn't these be replaced with SEP ?
All the platforms that I'm awware of that don't use '/' have their own getpath.c copy anyway (the one for Windows is PC/getpathp.c).
But can't hurt to change these in the standard getpath.c, right ? (reduce() is looking for SEP, so on platforms which do use the standard getpath.c but have a different os.sep could be mislead by the hardcoded slash in some constants)
* There's no easy way to find the first item on sys.path which starts the default path added by Python at startup time. It seems that a suffix search for "python23.zip" gives the best hint. The only other possibility I see is writing the support code directly into getpath.c.
That's where I'd put it, yes.
You mean "put it into getpath.c" or "put it in front of .../python23.zip" ?
* site.py contains code which prefixes "site-packages" with both sys.prefix and sys.exec_prefix. Is this really used anywhere ? (distutils and the old Makefile.pre.in both install to sys.prefix per default)
I thought they might install extension modules in exec_prefix. But maybe it's a YAGNI.
Hmm, I've just built a Python interpreter with different prefix and exec_prefix settings: using such an interpreter lets distutils default to the exec_prefix subtree. However, Python itself does not create a site-packages directory in that tree (make install creates this directory in $(prefix)/lib/pythonX.X/ and not $(exec_prefix)/lib/pythonX.X/). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
Ok, I've started looking at adding support for this. Here's a couple of things I found:
* getpath.c: Some of the '/' path delimiters are hard coded; shouldn't these be replaced with SEP ?
All the platforms that I'm awware of that don't use '/' have their own getpath.c copy anyway (the one for Windows is PC/getpathp.c).
But can't hurt to change these in the standard getpath.c, right ? (reduce() is looking for SEP, so on platforms which do use the standard getpath.c but have a different os.sep could be mislead by the hardcoded slash in some constants)
-0; it's too painful to think about all the places that would have to be fixed and how to fix them.
* There's no easy way to find the first item on sys.path which starts the default path added by Python at startup time. It seems that a suffix search for "python23.zip" gives the best hint. The only other possibility I see is writing the support code directly into getpath.c.
That's where I'd put it, yes.
You mean "put it into getpath.c" or "put it in front of .../python23.zip" ?
Put it in getpath.c.
* site.py contains code which prefixes "site-packages" with both sys.prefix and sys.exec_prefix. Is this really used anywhere ? (distutils and the old Makefile.pre.in both install to sys.prefix per default)
I thought they might install extension modules in exec_prefix. But maybe it's a YAGNI.
Hmm, I've just built a Python interpreter with different prefix and exec_prefix settings: using such an interpreter lets distutils default to the exec_prefix subtree. However, Python itself does not create a site-packages directory in that tree (make install creates this directory in $(prefix)/lib/pythonX.X/ and not $(exec_prefix)/lib/pythonX.X/).
Probably an oversight in the Makefile. Doesn't distutils create a needed directory if it doesn't exist? --Guido van Rossum (home page: http://www.python.org/~guido/)
"MAL" == M <mal@lemburg.com> writes:
MAL> Ok, then, let's call the dir "site-upgrades-<pyversion>" with MAL> <pyversion> being major.minor.patchlevel. +1 MAL> It seems that only site.py needs to be changed. Hmm, but MAL> what happens if someone invokes Python with -S (don't load MAL> site.py) ? They lose, just like they lose site-packages if they do it. -Barry
Guido van Rossum wrote My suggestion is that the Python micro-release number be included in the path to system-packages. IOW, system-packages must exactly match the Python version number, not just the maj.min number.
+1
Excellent point. Or a micro-release should clear out the system-packages directory.
-1. That's just a bit too sudden and nasty. If this is the way to go, make a subdirectory of system-packages/site-upgrades and move the old files there (and warn the installer that they've been moved) Anthony
participants (4)
-
Anthony Baxter -
barry＠python.org -
Guido van Rossum -
M.-A. Lemburg