Order that site-packages is added to sys.path
I have a bit of a dilemma when it comes to sys.path and the location of the site-packages directory. The problem comes when someone is using Mailman 2.1 with Python 2.2. The latter comes with the email package, which is in the standard library. Through some contributions, my standalone email package now supports multibyte character sets in RFC-compliant ways (e.g. splitting long headers correctly). The question is, how do I get the updated package to Python 2.2 users? The standalone email package is a simple distutils thingie with a directory and a bunch of .py files. distutils sticks this in site-packages. But an "import email" will always get the standard library version instead of the site-packages version because site.py /appends/ site-packages to sys.path instead of prepending it. I can work around this by adding my own path-hacking code before any import of email.* modules. This is a bit ugly because now it means that the proper functioning of the application depends on import order, and that's nasty. So the question is: why does site.py append site-packages instead of prepending it to sys.path? If there's a valid reason, I don't remember it, and I'm currently blind to any valuable use case. If there's no good reason, it would seem to me that the following use case is better served by prepending: - We want to provide an enhanced version, or a fixed version of a module or package. Distribute it w/distutils and do a normal install. As long as you don't start Python w/ -S, you'll always get the improved version. Don't want the improved version? Start Python w/ -S or just don't ever install the new package. I'm mostly looking for rationale right now, before I try to decide whether it's something worth debating and/or changing. Thanks, -Barry
barry@zope.com (Barry A. Warsaw) writes:
So the question is: why does site.py append site-packages instead of prepending it to sys.path?
I think the rationale was that you are precisely not supposed to override any of the standard modules. It was considered a good thing that if you do "import string" in some version of Python, you know exactly what you will get. There is currently one exception to that rule, which is the xml module and PyXML: the standard xml module allows being replaced by add-on (later, better) packages. However, there have been complaints that this is so: One of Paul Prescod's applications would break if PyXML was installed, since PyXML performed some stricter argument checking in certain cases. The same problem would occur more frequently if you have site-packages in front of the path: The add-on package may behave worse than the standard package in some cases (especially after installing a Python bugfix release); this problem is hard to track. In the specific case, I'd propose the following strategy: - Get the fixes to the Email package into the 2.2 maintainance branch, in addition to getting them into the trunk. This assumes that the patches really do fix bugs and are suitable for the general public etc. - If Python 2.2.1 is released before Mailman 2.1, you are done: Just tell your users that they need 2.2.1 or 2.1.2, but cannot use 2.2 (or need to live with limitations in MIME processing). - If this is not possible, rename the email package inside mailman (e.g. xemail). It then appears that the standard library package is not suitable for mailman, so just ignore its presence, and use your own (under a different name). - As a compromise, you might consider falling back to the email package if you determine it is good enough at installation time, by playing with xemail.__init__.__path__, or even replacing xemail with email in the same way that xml is replaced with _xmlplus. Regards, Martin
"Barry A. Warsaw" wrote:
I have a bit of a dilemma when it comes to sys.path and the location of the site-packages directory.
The problem comes when someone is using Mailman 2.1 with Python 2.2. The latter comes with the email package, which is in the standard library. Through some contributions, my standalone email package now supports multibyte character sets in RFC-compliant ways (e.g. splitting long headers correctly). The question is, how do I get the updated package to Python 2.2 users?
The standalone email package is a simple distutils thingie with a directory and a bunch of .py files. distutils sticks this in site-packages. But an "import email" will always get the standard library version instead of the site-packages version because site.py /appends/ site-packages to sys.path instead of prepending it.
I can work around this by adding my own path-hacking code before any import of email.* modules. This is a bit ugly because now it means that the proper functioning of the application depends on import order, and that's nasty.
Why not put put the updated email package into the Mailman package (is it a package?) ? That way you can update whatever part you want from the Python lib or replace it with something else.
So the question is: why does site.py append site-packages instead of prepending it to sys.path? If there's a valid reason, I don't remember it, and I'm currently blind to any valuable use case.
I guess this is done for the same reason that e.g. /usr/local is last in PATH on Unix: system top level programs and libs should always have top priority. Otherwise, a user could easily override a system program/lib by placing a new version into the local dir which then gets picked up by other system programs. I'd suggest to better be explicit about what you do and to put the new code in the package (which is completely under your control). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
"Martin v. Loewis" wrote:
- As a compromise, you might consider falling back to the email package if you determine it is good enough at installation time, by playing with xemail.__init__.__path__, or even replacing xemail with email in the same way that xml is replaced with _xmlplus.
Those kind of hacks should not be needed if Barry puts his own email package inside the Mailman package. All local imports will pick up his version automatically; even though I'd suggest to use explicit imports for it in the Mailman code to avoid magical problems ;-) Hacking __path__ should really only be the last resort... it (usually) breaks installers, gives importers a hard time, etc. We should not consider this good practice even though it may be needed sometimes (e.g. by PyXML). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
On 12 February 2002, Barry A. Warsaw said:
The standalone email package is a simple distutils thingie with a directory and a bunch of .py files. distutils sticks this in site-packages. But an "import email" will always get the standard library version instead of the site-packages version because site.py /appends/ site-packages to sys.path instead of prepending it.
I can work around this by adding my own path-hacking code before any import of email.* modules. This is a bit ugly because now it means that the proper functioning of the application depends on import order, and that's nasty.
Looong ago, I tried to persuade Guido that giving the Distutils the power to override standard library modules would, on rare occasions, be a good and useful thing. (Yet another idea stolen from Perl's MakeMaker, which can do precisely that. Sometimes, it's useful.) Guess who won? Greg -- Greg Ward - just another Python hacker gward@python.net http://starship.python.net/~gward/ A closed mouth gathers no foot.
"MvL" == Martin v Loewis <martin@v.loewis.de> writes:
MvL> I think the rationale was that you are precisely not supposed MvL> to override any of the standard modules. It was considered a MvL> good thing that if you do "import string" in some version of MvL> Python, you know exactly what you will get. Okay, I can see why that's useful. Let's say there was a way to add stuff to the front of sys.path, such that they could override the standard library. This might work just fine on a single user (or single application) system, but might be very broken on a multiuser or multiapp system ("I know what I'm installing in site-packages, so what's the problem?"). Hopefully, any overrides that were installed would be API compatible with the standard. Such overrides would probably be allowed to fix bugs or add functionality, but not remove functionality. This might still get us into trouble and this path leads to module versioning, etc. I don't want to go there now. I know how to handle my specific case (I've done it before), but just to close the loop, I can't wait for Python 2.2.1 because some of the features I'm depending on are new features, not just bug fixes. I think those will have to wait for Python 2.3 to be safe, so until then, I must distribute a separate package. That's fine, I can live with that. I think "python setup.py install --root blah" will do the trick for me, along with some application specific path-hackery. -Barry
"MAL" == M <mal@lemburg.com> writes:
MAL> I guess this is done for the same reason that e.g. /usr/local MAL> is last in PATH on Unix: system top level programs and libs MAL> should always have top priority. Otherwise, a user could MAL> easily override a system program/lib by placing a new version MAL> into the local dir which then gets picked up by other system MAL> programs. Well, hopefully you'd control who can write into /usr/local so that you could trust overrides being installed there. On a single user system, I usually do in fact put /usr/local/bin early in my path specifically because I do want to override older, buggier, system programs. The analogy is similar in the Python situation. When I'm the only person using the system, and I'm in control of everything, being able to override the standard library is a very useful thing to do. When there's less trust in the environment I'm running in, or more sharing of common resources, it can be problematic. -Barry
"GW" == Greg Ward <gward@python.net> writes:
GW> Looong ago, I tried to persuade Guido that giving the GW> Distutils the power to override standard library modules GW> would, on rare occasions, be a good and useful thing. (Yet GW> another idea stolen from Perl's MakeMaker, which can do GW> precisely that. Sometimes, it's useful.) Guess who won? distutils's --root option could be used to specific a different install directory than site-packages right? So conceivably site.py could prepend some directory onto sys.path, and distutils could be coaxed into installing there rather than site-packages. This might provide a principled way to override Python's standard library when you're really sure that's what you want to do. -Barry
"BAW" == Barry A Warsaw <barry@zope.com> writes:
BAW> distutils's --root option could be used to specific a different BAW> install directory than site-packages right? So conceivably BAW> site.py could prepend some directory onto sys.path, and BAW> distutils could be coaxed into installing there rather than BAW> site-packages. This might provide a principled way to override BAW> Python's standard library when you're really sure that's what BAW> you want to do. Why don't you use "--root /usr/local/lib/python2.2" and *really* override the standard library? It seems fragile to extend Python with yet more directories to search in a special order so that the interpreter picks up the correct copy of somemodule.py from among the four or five copies installed on the system and on the path. Jeremy
On 12 February 2002, Jeremy Hylton said:
Why don't you use "--root /usr/local/lib/python2.2" and *really* override the standard library?
No: --root just lets you replace / with something else. It's mainly so you can build an RPM (eg.) without being superuser. Your example would install to /usr/local/lib/python2.2/usr/local/lib/python2.2/site-packages ...which is probably not what you meant. The distutils install command *is* pretty flexible; if someone cares to sit down and figure it out, I'm sure this is possible. It's just not documented or obvious. Greg -- Greg Ward - Linux weenie gward@python.net http://starship.python.net/~gward/ "He's dead, Jim. You get his tricorder and I'll grab his wallet."
Perhaps --home then? I know there's some command I've used to install Python packages in my Zope lib/python directory by spelling out its full path. Jeremy
"Barry A. Warsaw" wrote:
[distutils --root hackery]
Why not use the subpackage approach I suggested ? It keeps the std lib in a sane state (meaning that the std lib installation only depends on the Python installation and no other hacks on top of it). Since you'll have to ship the complete package anyway, I don't see any win in installing over the std email package. If that's what you really want, I'd suggest to provide the updated email package as separate download and then test inside Mailman for the new version. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
"MAL" == M <mal@lemburg.com> writes:
MAL> "Barry A. Warsaw" wrote: >> [distutils --root hackery] MAL> Why not use the subpackage approach I suggested ? MAL> It keeps the std lib in a sane state (meaning that the std MAL> lib installation only depends on the Python installation and MAL> no other hacks on top of it). MAL> Since you'll have to ship the complete package anyway, I MAL> don't see any win in installing over the std email MAL> package. If that's what you really want, I'd suggest to MAL> provide the updated email package as separate download and MAL> then test inside Mailman for the new version. I'm fine with installing in a Mailman specific location, but I still want to use as much of the distutils machinery as possible. It looks like python setup.py install --home=/some/path gets close enough. This will install the email package into /some/path/lib/python and I can easily arrange for that to be in the right place on sys.path, at least for the mail program and the cgi program. The command line scripts are a bit trickier because you can't wheedle your way into Python's startup machinery without 1) telling your users to setenv PYTHONPATH (yuck) or 2) importing a path-hacking module before any that require the override location. Since I already have to do #2 anyway, this isn't much of a problem, except that some imports will have to be rearranged. It also makes things a little trickier when a user does eventually upgrade to Python 2.3, which will obviate the need for the enhanced package (hopefully). Like everyone else, I'm sure I'll eventually just end up shipping my own complete Python distro to make sure it's got exactly what you need. ;) -Barry
"GW" == Greg Ward <gward@python.net> writes:
GW> The distutils install command *is* pretty flexible; if someone GW> cares to sit down and figure it out, I'm sure this is GW> possible. It's just not documented or obvious. Any hope of actually documenting all this stuff? <wink> -Barry
"Barry A. Warsaw" wrote:
"MAL" == M <mal@lemburg.com> writes:
MAL> "Barry A. Warsaw" wrote: >> [distutils --root hackery]
MAL> Why not use the subpackage approach I suggested ?
MAL> It keeps the std lib in a sane state (meaning that the std MAL> lib installation only depends on the Python installation and MAL> no other hacks on top of it).
MAL> Since you'll have to ship the complete package anyway, I MAL> don't see any win in installing over the std email MAL> package. If that's what you really want, I'd suggest to MAL> provide the updated email package as separate download and MAL> then test inside Mailman for the new version.
I'm fine with installing in a Mailman specific location, but I still want to use as much of the distutils machinery as possible.
It looks like
python setup.py install --home=/some/path
gets close enough.
No, no, no :-) What I am suggesting is to put the email package *inside* the Mailman package: Mailman/__init__.py ... email/__init__.py ... And then use "from Mailman import email" in Mailman source code. That's clean, doesn't interfere with the std lib and it's all your's ;-) (meaning that you have complete control over what email does in the Mailman context). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/
"MAL" == M <mal@lemburg.com> writes:
>> install --home=/some/path gets close enough. MAL> No, no, no :-) MAL> What I am suggesting is to put the email package *inside* the MAL> Mailman package: MAL> Mailman/__init__.py | ... | email/__init__.py | ... What I didn't say was s|/some/path|/path/to/Mailman| so we're saying (nearly) the same thing. MAL> And then use "from Mailman import email" in Mailman source MAL> code. That's clean, doesn't interfere with the std lib MAL> and it's all your's ;-) (meaning that you have complete MAL> control over what email does in the Mailman context). I could do this (and may) or I may use something like from Mailman.pythonlib import email, which is my normal place to put override modules. It's moderately more appealing to put Mailman.pythonlib on sys.path and just leave my "import email"'s alone. I know there are arguments against doing it that way, but I don't want to have to change dozens of files. -Barry
Barry A. Warsaw wrote:
It's moderately more appealing to put Mailman.pythonlib on sys.path and just leave my "import email"'s alone. I know there are arguments against doing it that way, but I don't want to have to change dozens of files.
For shame, Barry, isn't that what Python is for? ;-) -- --- Aahz (@pobox.com) Hugs and backrubs -- I break Rule 6 <*> http://www.rahul.net/aahz/ Androgynous poly kinky vanilla queer het Pythonista We must not let the evil of a few trample the freedoms of the many.
Note that app developers eager to replace standard libraries, and an OS that allowed them to do so, are the causes of the aptly named "DLL Hell" on Windows. It can work fine for a single app, but it's truly hell when multiple apps resort to this, and end users don't have a prayer of sorting out the inevitable, vicious problems. if-you-need-your-own-xxx.py-you-know-where-to-shove-it<wink>-ly y'rs - tim
participants (7)
-
aahz＠rahul.net -
barry＠zope.com -
Greg Ward -
Jeremy Hylton -
M.-A. Lemburg -
martin＠v.loewis.de -
Tim Peters