patch: solving the two major things that people dislike about setuptools, part 1
Folks: There are two things that cause a lot of people to object to the use of setuptools: that it changes the semantics of PYTHONPATH, making it impossible to override packages on the command-line, and that it doesn't work when you run "setup.py install --prefix=./some_dir". This patch addresses the first one. Currently, if you ever allow setuptools to write into your system directory (e.g. /usr/lib/python2.5/site-packages) then it will install a .pth file which changes the behavior of PYTHONPATH. This will prevent you from over-riding the installed package with a specific package on the command-line, such as: PYTHONPATH=./that-version-of-Twisted/ python ./my_script.py Many people find this behavior of setuptools objectionable [1, 2, 3, plus personal correspondance when I did my "Why do you hate setuptools?" survey]. Fortunately it seems possible to preserve the precedence of PYTHONPATH modules while still having installed eggs override installed non-eggs, as motivated in [4]. Sat Nov 15 11:59:32 MST 2008 zooko@zooko.com * leave the PYTHONPATH dirs at the front of the sys.path This is in accordance with http://www.python.org/doc/2.5.2/inst/search-path.html , which says "The PYTHONPATH variable can be set to a list of paths that will be added to the beginning of sys.path.", and it resolves an objection many people have which impels them to ban setuptools from systems they administrate. --- old-dw-0.6c9/setuptools/command/easy_install.py +++ new-dw-0.6c9/setuptools/command/easy_install.py @@ -1364,7 +1364,7 @@ "%s\n" "import sys; new=sys.path[sys.__plen:];" " del sys.path[sys.__plen:];" - " p=getattr(sys,'__egginsert',0); sys.path[p:p]=new;" + " p=getattr(sys,'__egginsert',len(os.environ.get ('PYTHONPATH','').split(os.pathsep))); sys.path[p:p]=new;" " sys.__egginsert = p+len(new)\n" ) % data Regards, Zooko --- http://allmydata.org -- Tahoe, the Least-Authority Filesystem http://allmydata.com -- back up all your files for $10/month [1] http://mail.python.org/pipermail/distutils-sig/2006-July/006492.html [2] https://www.dfwpython.org/pipermail/dfwpython/2007-June/000756.html [3] http://www.rittau.org/blog/20070726-02 [4] http://mail.python.org/pipermail/python-dev/2008-March/077918.html
zooko wrote:
Folks:
There are two things that cause a lot of people to object to the use of setuptools: that it changes the semantics of PYTHONPATH, making it impossible to override packages on the command-line, and that it doesn't work when you run "setup.py install --prefix=./some_dir".
This is #1 on my list of why I hate setuptools. I actually had it installed (we were wondering if we should distribute our software with setuptools), but I removed it when I found out that it broke PYTHONPATH. I have a question about this patch, though: It is somewhat awkward to apply your patch because setuptools is distributed and installed as a .egg file. Rather than trying to figure out how to apply a patch inside a .egg file, I tried to see what the effect on easy_install.pth would be. I changed my easy_install.pth file to look like this: import sys; sys.__plen = len(sys.path) import sys; print "egg insert at",sys.__egginsert ./nose-0.10.4-py2.5.egg import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',len(os.environ.get('PYTHONPATH','').split(os.pathsep))); sys.path[p:p]=new; sys.__egginsert = p+len(new) I think that this is what your patch would do, except for the debugging. In my testing, I found that this easy_install.pth file behaves exactly the same as the old one. That is because sys.__egginsert exists and is always 0; the value you get from looking at the environment is never used. You can verify this from the print statement I included in the .pth file: [banana:~] sienkiew% python egg insert at 0 Python 2.5.1 (r251:54863, Mar 18 2008, 11:29:59) [GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin Type "help", "copyright", "credits" or "license" for more information.
import setuptools ; setuptools.__version__ '0.6c9'
Have I understood your patch correctly? Do you have different behaviour on your system? I think the correct behaviour is to insert the .egg file either just before or just after the directory where we found the .pth file. So, instead of p=getattr(sys,'__egginsert',len(os.environ.get('PYTHONPATH','').split(os.pathsep))); we want p=sys.path.index(sitedir); We can eliminate __egginsert from the code entirely. As far as I can tell, the *only* purpose of __egginsert is to permit the incorrect behaviour that your patch is intended to fix. Does that make sense to you? Mark S.
At 05:46 PM 11/24/2008 -0500, Mark Sienkiewicz wrote:
As far as I can tell, the *only* purpose of __egginsert is to permit the incorrect behaviour that your patch is intended to fix.
The behavior isn't incorrect; it's by design. If you don't want that behavior, just install everything using "-m" (aka --multi-version). The eggs will then not be listed in .pth files, but any scripts installed by setuptools (including those installed via "setup.py develop") will still load the correct eggs at runtime. For code not built with setuptools, you'll need to use pkg_resources.require() to add the eggs to sys.path dynamically. Please note that I've rejected Zooko's patch, because it makes it impossible to ensure that an easy_install-ed package takes precedence over an existing, distutils-installed version of the same package in the same PYTHONPATH directory. This is the intended default behavior of easy_install; if you wish to override it, you should use --multi-version, and explicitly select the egg(s) to be added to sys.path instead.
On Mon, Nov 24, 2008 at 07:20:11PM -0500, Phillip J. Eby wrote:
At 05:46 PM 11/24/2008 -0500, Mark Sienkiewicz wrote:
As far as I can tell, the *only* purpose of __egginsert is to permit the incorrect behaviour that your patch is intended to fix.
The behavior isn't incorrect; it's by design. If you don't want that behavior, just install everything using "-m" (aka --multi-version). The eggs will then not be listed in .pth files, but any scripts installed by setuptools (including those installed via "setup.py develop") will still load the correct eggs at runtime.
For code not built with setuptools, you'll need to use pkg_resources.require() to add the eggs to sys.path dynamically.
Right, setuptools breaks standard Python behavior of a fundamental feature without giving a strong warning and forces people to monkey patch sys.path (which, as we are discussing here, as huge consequences on import semantics). No wonder system administrators (and packagers, which work with them) hate setuptools: it breaks expectation of people, and forces them to learn advanced and specific details of the Python language, which is fairly irrelevant to their work. Moreover, not respecting environment variables is a violation of all Unix good-pratices rules.
Please note that I've rejected Zooko's patch, because it makes it impossible to ensure that an easy_install-ed package takes precedence over an existing, distutils-installed version of the same package in the same PYTHONPATH directory. This is the intended default behavior of easy_install; if you wish to override it, you should use --multi-version, and explicitly select the egg(s) to be added to sys.path instead.
I am not surprised you rejected it. You are not open to any suggestion, and consider only your usecases from your point of view. You act like if what setuptools does is by definition right, no matter how much people dislike these feature, and the bad reputation it has. I have sat and watched discussions on this mailing list for over a year and concluded that the project was not in a state of moving forward, and that discussions happening on this mailing list where a loss of time. I gave up and unsubscribe. The only thing that made me come back was the announcement of a fork. That fork did not go anywhere. I think I should unsubscribe again, because all this mailing list will do is increase my frustration. Setuptools could be a very important piece of software for the future Python. It has some _very_ nice features. Your unwillingness to accept its limitations and fix the problems people are having is sacrificing its future. And yes, I am bitter. Gaël
At 07:39 AM 11/25/2008 +0100, Gael Varoquaux wrote:
I am not surprised you rejected it. You are not open to any suggestion, and consider only your usecases from your point of view.
On the contrary; it's users who *don't* control their systems whose usecase this particular feature supports. It has no personal value to me whatsoever. Until this feature was added to setuptools, the most frequent bug report for easy_install was, "I installed foobar 1.2 with easy_install but I'm still importing foobar 1.1... this program sucks." Personally, I'm just as bitter about people who insist on thinking that their use cases are the only ones that exist, and seem to wilfully ignore my repeated explanations of what groups of people will be hurt by their "fixes".
Phillip J. Eby wrote:
Personally, I'm just as bitter about people who insist on thinking that their use cases are the only ones that exist, and seem to wilfully ignore my repeated explanations of what groups of people will be hurt by their "fixes".
It has nothing to do with what group of people are hurt; setuptools breaks standard behavior of python. If every developer took the same approach, and decided to "fix" their bug reports by breaking the general system behavior, I am sure you would be annoyed by it too. David
At 10:33 PM 11/25/2008 +0900, David Cournapeau wrote:
Phillip J. Eby wrote:
Personally, I'm just as bitter about people who insist on thinking that their use cases are the only ones that exist, and seem to wilfully ignore my repeated explanations of what groups of people will be hurt by their "fixes".
It has nothing to do with what group of people are hurt; setuptools breaks standard behavior of python. If every developer took the same approach, and decided to "fix" their bug reports by breaking the general system behavior, I am sure you would be annoyed by it too.
My point is that people offering "fixes" for setuptools need to take into account those other use cases, or else the improvement for one group comes at the expense of another. For example, in this case, if someone wanted to offer a patch that changed the behavior in such a way that the added eggs came directly before the sys.path directory in which they were contained, but not pushed all the way to the beginning of sys.path, that would be an acceptable way to change the behavior. Alternatively, if someone wanted to come up with a more sophisticated conflict resolution method for handling non-setuptools-installed packages, that might work also. There indeed might be a great many ways to resolve the issue -- but nobody's proposing anything except breaking someone else's use cases for their personal convenience. And I say "personal convenience", because easy_install *does* provide an option to not mess with sys.path. If you use --multi-version to install your eggs, easy_install will NOT affect sys.path in ANY way. So it's not like there isn't already a way for easy_install to be "pure" here.
Phillip J. Eby wrote:
At 05:46 PM 11/24/2008 -0500, Mark Sienkiewicz wrote:
As far as I can tell, the *only* purpose of __egginsert is to permit the incorrect behaviour that your patch is intended to fix.
The behavior isn't incorrect; it's by design.
Here is my reasoning for saying it is broken: [banana:~] sienkiew% which python /usr/stsci/pyssg/Python-2.5.1/bin/python [banana:~] sienkiew% ls -l ~/py/lib/python total 8 drwxr-xr-x 38 sienkiew ststaff 1292 Nov 25 10:07 nose -rw-r--r-- 1 sienkiew ststaff 2102 Nov 25 10:07 nose-0.10.4-py2.5.egg-info [banana:~] sienkiew% ls -l /usr/stsci/pyssg/Python-2.5.1/lib/python2.5/site-packages/ total 680 ...unrelated files here... -rw-r--r-- 1 iraf sienkiew 238 Nov 25 10:06 easy-install.pth drwxr-xr-x 5 iraf sienkiew 170 Nov 25 10:06 nose-0.10.0-py2.5.egg -rwxr-xr-x 1 iraf sienkiew 328025 Oct 7 14:57 setuptools-0.6c9-py2.5.egg -rw-r--r-- 1 iraf sienkiew 29 Nov 25 10:06 setuptools.pth [banana:~] sienkiew% setenv PYTHONPATH ~/py/lib/python [banana:~] sienkiew% python Python 2.5.1 (r251:54863, Mar 18 2008, 11:29:59) [GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin Type "help", "copyright", "credits" or "license" for more information.
import nose nose.__version__ '0.10.0' nose.__file__ '/usr/stsci/pyssg/Python-2.5.1/lib/python2.5/site-packages/nose-0.10.0-py2.5.egg/nose/__init__.pyc'
For example, in this case, if someone wanted to offer a patch that changed the behavior in such a way that the added eggs came directly before the sys.path directory in which they were contained, but not
Look what happened: When I imported "nose", it chose the nose from the system site-packages directory, even though I have a different version that occurs earlier on PYTHONPATH. At this time, I would like to draw your attention to http://www.python.org/doc/2.5.2/tut/node8.html section 6.1.2, where it describes PYTHONPATH. "When PYTHONPATH is not set, or when the file is not found there, the search continues in an installation-dependent default path". That does not appear to be what happened here. It should have found the module that was on PYTHONPATH, not the module that is in the system site-packages. Please explain why you consider this to be the desired behaviour. In a later message, you said: pushed all the way to the beginning of sys.path, that would be an acceptable way to change the behavior. In my comments on zooko's patch:
I think the correct behaviour is to insert the .egg file either just before or just after the directory where we found the .pth file.
So, instead of p=getattr(sys,'__egginsert',len(os.environ.get('PYTHONPATH','').split(os.pathsep)));
we want p=sys.path.index(sitedir);
Does this meet your criterion without breaking anything else? IMHO, this is actually not a great patch because it depends on knowledge that should be private to distutils, but setuptools is already violating that information hiding. (For example, it assumes that it is safe to alter sys.path while distutils is loading the .pth file; I didn't see that documented anywhere, but we can see that it works.) b.t.w. You could make easy_install.pth read: import setuptools; setuptools.prep_sys_path() module-name.egg import setuptools; setuptools.alter_sys_path() and then patches to the first/last line would be easier -- they would just happen when you install a new setuptools. Mark S.
At 10:56 AM 11/25/2008 -0500, Mark Sienkiewicz wrote:
I think the correct behaviour is to insert the .egg file either just before or just after the directory where we found the .pth file.
So, instead of p=getattr(sys,'__egginsert',len(os.environ.get('PYTHONPATH','').spl it(os.pathsep)));
we want p=sys.path.index(sitedir);
Does this meet your criterion without breaking anything else?
The principle is correct; I'm not sure the implementation is. For case-insensitive filesystems, sitedir may or may not be normalized, and sys.path probably isn't. But yes, that's the general idea. The only reason I didn't take that approach myself is because my head hurt trying to come up with a provably-correct way to implement this policy. ;-)
IMHO, this is actually not a great patch because it depends on knowledge that should be private to distutils, but setuptools is already violating that information hiding. (For example, it assumes that it is safe to alter sys.path while distutils is loading the .pth file; I didn't see that documented anywhere, but we can see that it works.)
b.t.w. You could make easy_install.pth read:
import setuptools; setuptools.prep_sys_path() module-name.egg import setuptools; setuptools.alter_sys_path()
and then patches to the first/last line would be easier -- they would just happen when you install a new setuptools.
It would have to be a separate module, added just for this purpose... and of course it would introduce a runtime dependency that would otherwise be unnecessary.
Phillip J. Eby wrote:
At 10:56 AM 11/25/2008 -0500, Mark Sienkiewicz wrote:
I think the correct behaviour is to insert the .egg file either just before or just after the directory where we found the .pth file.
So, instead of p=getattr(sys,'__egginsert',len(os.environ.get('PYTHONPATH','').spl it(os.pathsep)));
we want p=sys.path.index(sitedir);
Does this meet your criterion without breaking anything else?
The principle is correct; I'm not sure the implementation is. For case-insensitive filesystems, sitedir may or may not be normalized, and sys.path probably isn't. But yes, that's the general idea.
The only reason I didn't take that approach myself is because my head hurt trying to come up with a provably-correct way to implement this policy. ;-)
That's a good question about normalized names and case-insensitive filesystems, but setuptools needs to deal with this problem, so we need an answer. There appear to be two cases: 1. in the system site.py I looked around lib/python25/site.py a little. The name of the site-packages directory is created in addsitepackages() then passed to addsitedir() where it is appended to sys.path and then passed to addpackage(). Since the value that was appended to sys.path is the same value that was passed to addpackage() as parameter sitedir, we know that they will match. 2. in the site.py that setuptools put in the directory where setuptools installed something I think the real interesting part is where it says: for item in PYTHONPATH: addsitedir(item) Here, PYTHONPATH was constructed directly from the environment variable, not somehow derived from sys.path. This means that it potentially _could_ contain duplicate and un-normalized directory names. But I see that we already have access to the function makepath() from the system site.py, so we can say: for item in PYTHONPATH: p,np = makepath(item) addsitedir(np) If you do that, then you know that the directory names that you are passing to addsitedir() have been normalized by exactly the same algorithm that normalized them for sys.path. Therefore we know that we can use a string comparison to find a match. So, you know setuptools way better than I do. Have I missed anything? b.t.w. If you have a few minutes, could you write a paragraph about why each of those two big loops in the setuptools site.py are doing what they do? The first one seems to be trying different ways to find the system site.py (why?), but I don't see why you would bother with the second one.
b.t.w. You could make easy_install.pth read:
import setuptools; setuptools.prep_sys_path() module-name.egg import setuptools; setuptools.alter_sys_path()
and then patches to the first/last line would be easier -- they would just happen when you install a new setuptools.
It would have to be a separate module, added just for this purpose... and of course it would introduce a runtime dependency that would otherwise be unnecessary.
Is that a problem? Mark S.
On Nov 24, 2008, at 15:46 PM, Mark Sienkiewicz wrote:
We can eliminate __egginsert from the code entirely. As far as I can tell, the *only* purpose of __egginsert is to permit the incorrect behaviour that your patch is intended to fix.
There are two separable concerns here: 1. Should the act of "easy_install foodistribution" cause subsequent "python -c 'import foomodule'" to get the new, easy-installed foo, instead of the pre-existing, distutils-installed foo which is in the system directory? 2. Should the act of "PYTHONPATH=./my-custom-foo" cause subsequent "python -c 'import foomodule'" to get the foo in ./my-custom-foo instead of the easy-installed foo? PJE has already made clear (see references on the ticket that I filed or in my earlier mail) that he wants the answer to 1 to be "yes", because he is supporting a common use case in which a user doesn't know how to uninstall the old foo but wants to use easy_install to get a new foo. (I personally am kind of ambivalent about #1, but I have no users asking for it to be different so I'm happy to go along with PJE on that.) Issue #2 is actually the one that so many people hate about setuptools, including many of my current and prospective users, and apparently included you, Mark Sienkiewicz. I think it is consistent to answer "yes" to both questions, because in each case the user has expressed intent -- by saying "easy_install foo", you can legitimately be understood as meaning "...and make the one I get over-ride any other ones that might already be here.". Likewise by saying "PYTHONPATH=./my-custom-foo", you are certainly saying "... and make the one I just named override any others". Indeed, the Python docs clearly state that Python will respect that statement of yours: http://www.python.org/doc/2.5.2/inst/search-path.html which says: "The PYTHONPATH variable can be set to a list of paths that will be added to the beginning of sys.path.". My patch is intended to make the answer to #2 be "yes" from now on, while preserving the current behavior of the answer to #1 being "yes". Regards, Zooko --- http://allmydata.org -- Tahoe, the Least-Authority Filesystem http://allmydata.com -- back up all your files for $10/month
At 01:29 PM 11/25/2008 -0700, zooko wrote:
My patch is intended to make the answer to #2 be "yes" from now on, while preserving the current behavior of the answer to #1 being "yes".
It doesn't preserve #1 in the case where the directory where the eggs are installed is *itself* on the PYTHONPATH, which is why I rejected it in its current form.
zooko <zooko@zooko.com> writes:
1. Should the act of "easy_install foodistribution" cause subsequent "python -c 'import foomodule'" to get the new, easy-installed foo, instead of the pre-existing, distutils-installed foo which is in the system directory?
By the defined behaviour of PYTHONPATH and sys.path, the answer to this would be “that depends on where ‘easy_install foodistribution’ installed the distribution”. If ‘easy_install’ wasn't explicitly instructed to install somewhere other than the system directory, then it should behave like distutils in this respect and *replace* the existing instance in the system directory. In other words, by default setuptools should be installing distributions in the same place as distutils. If ‘easy_install’ has been configured to put things in a different place, then at import time the rules of PYTHONPATH and sys.path apply — and setuptools should leave those alone so the user has the behaviour promised in the documentation for PYTHONPATH and sys.path.
PJE has already made clear (see references on the ticket that I filed or in my earlier mail) that he wants the answer to 1 to be "yes", because he is supporting a common use case in which a user doesn't know how to uninstall the old foo but wants to use easy_install to get a new foo.
If the argument is that this “common use case” involves ignoring, overriding, or munging the existing PYTHONPATH or sys.path, that's a bad argument. The way to change that behaviour is *not* to have magic special behaviour in setuptools; it's to change the way Python works so this use case is supported, if you can convince the Python team to do so. Special cases aren't special enough to break the rules. This smells very much like a special case, and it's entirely possible to support it with the existing PYTHONPATH and sys.path behaviour, AFAICT. -- \ “It is hard to believe that a man is telling the truth when you | `\ know that you would lie if you were in his place.” —Henry L. | _o__) Mencken | Ben Finney
Folks: This is just to let you know that my programming partner Brian Warner is one of the many data points I have indicating that respecting the PYTHONPATH would make setuptools more palatable to more hackers. Brian, however, is the most important data point, to me, because I work closely with him every day and I greatly respect his software engineering practices. A few days ago he wrote on the tahoe-dev mailing list: """ "import" should import whatever comes first in the PYTHONPATH, right? So it should be a simple (heh) matter of insuring that the source tree or the newly installed location appears earlier in PYTHONPATH? 'trial' adds the current directory to sys.path before importing the code-under-test, so it's designed to either be run from a source tree (and test the code in that source tree), or from anywhere other than a source tree (and test the code that's installed on your system: whatever get used when you do 'import'). I must say, seeing how badly setuptools futzes around with sys.path, I can't say I feel confident that the ordering of PYTHONPATH matters anymore. I miss this loss of confidence and control. """ http://allmydata.org/pipermail/tahoe-dev/2009-January/001028.html I replied, in part: """ PJE, the maintainer of setuptools rejected my patch, and some other people on the distutils-sig list replied to say that this was the last straw and they were going to fork setuptools. Hopefully after Tahoe-1.3 release we can start using the fork and it will respect the PYTHONPATH. Also, there's a possibility that we could contribute a patch to setuptools that PJE would accept that would also make it respect the PYTHONPATH, but I'm not precisely sure how it would work. I intend to think about that at some point, but I've been awfully busy with Tahoe. Please read the thread rooted at that URL above for full context. See also, if you are curious the [[issue tickets]] tiddler on my klog which contains a (dizzyingly large) list of issue tickets that I'm trying to pay attention to. """ So, this is just to let everyone know that any setuptools variant (including a future release of PJE's setuptools) that respects the PYTHONPATH will get at least two users! Regards, Zooko
At 04:48 PM 1/20/2009 -0700, zooko wrote:
So, this is just to let everyone know that any setuptools variant (including a future release of PJE's setuptools) that respects the PYTHONPATH will get at least two users!
As I've said before, if somebody creates a patch that addresses the use cases I pointed out, I'd be happy to accept it. Please see: http://mail.python.org/pipermail/distutils-sig/2008-November/010533.html and the thread surrounding it for details. Making it sound like I simply reject patches in this area with no reason, isn't at all helpful, and I'd appreciate it if you stop doing so. Perhaps you could solicit volunteers to help develop an appropriate patch, propose possible approaches, etc., or ask for clarification if my previous explanations of what the patch needs to do are not clear.
Phillip J. Eby wrote:
At 04:48 PM 1/20/2009 -0700, zooko wrote:
So, this is just to let everyone know that any setuptools variant (including a future release of PJE's setuptools) that respects the PYTHONPATH will get at least two users!
As I've said before, if somebody creates a patch that addresses the use cases I pointed out, I'd be happy to accept it.
As I understand it, you require that a package installed by setuptools will be discovered before a package installed by distutils >in the same directory<. You do not require that a package installed by setuptools be discovered before a package that is earlier in sys.path. In that case, the .egg file should go on sys.path immediately before the directory where it is stored. This ensures that a setuptools .egg file will always be discovered before a distutils-installed package/module in the same directory, but it will not override packages/modules that occur earlier in sys.path. Do you have any other requirements? In http://mail.python.org/pipermail/distutils-sig/2008-November/010534.html , I described a change to zooko's patch that implements this behaviour. In http://mail.python.org/pipermail/distutils-sig/2008-November/010540.html , I address concerns about case-insensitive filesystems. So, taken together, does my suggestion meet your criteria for an acceptable patch? If so, - I will assemble it into a patch. - I will test it on platforms that I have access to, in ways that I know how to use setuptools. (For the record, that is "python setup.py install --home=/dir"; I never tried the automatic downloads.) If not, - what further requirements do you have? Mark S.
Mark Sienkiewicz <sienkiew@stsci.edu> writes:
In that case, the .egg file should go on sys.path immediately before the directory where it is stored. This ensures that a setuptools .egg file will always be discovered before a distutils-installed package/module in the same directory, but it will not override packages/modules that occur earlier in sys.path.
What of distributions installed via setuptools that are *not* installed as eggs? -- \ “I do not believe in forgiveness as it is preached by the | `\ church. We do not need the forgiveness of God, but of each | _o__) other and of ourselves.” —Robert G. Ingersoll | Ben Finney
Ben Finney wrote:
Mark Sienkiewicz <sienkiew@stsci.edu> writes:
In that case, the .egg file should go on sys.path immediately before the directory where it is stored. This ensures that a setuptools .egg file will always be discovered before a distutils-installed package/module in the same directory, but it will not override packages/modules that occur earlier in sys.path.
What of distributions installed via setuptools that are *not* installed as eggs?
I don't know. Can you tell me more about that? I'm not particularly familiar with setuptools -- I'm just offering to write the patch because I don't see it getting fixed any other way. Every thing that I have ever installed with setuptools has landed in a file or directory named like "*.egg". For example: site-packages/nose-0.10.4-py2.5.egg site-packages/Sphinx-0.5.1-py2.5.egg site-packages/Jinja-1.2-py2.5-macosx-10.3-i386.egg The installation creates a file "easy-install.pth" that looks like this: % cat easy-install.pth import sys; sys.__plen = len(sys.path) ./setuptools-0.6c9-py2.5.egg ./nose-0.10.4-py2.5.egg ./Sphinx-0.5.1-py2.5.egg ./Jinja-1.2-py2.5-macosx-10.3-i386.egg ./Pygments-1.0-py2.5.egg ./docutils-0.4-py2.5.egg import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new) % So, where I said ".egg files", you can substitute the name of the things on line 2 through N-1 in the easy-install.pth file. That last line of easy-install.pth guarantees that the .egg files (in this example, they are all eggs) will be placed in the wrong place on sys.path, and that is all I am proposing to fix. If there is another case that does not involve the easy-install.pth file, that is outside the scope of my patch. So, about distributions that are not installed as eggs: Can you tell me a little about it? Can you give me an example of how I can see it happen? Or does it turn out to be irrelevant to my patch? Mark S.
Mark Sienkiewicz <sienkiew@stsci.edu> writes:
Ben Finney wrote:
What of distributions installed via setuptools that are *not* installed as eggs?
I don't know. Can you tell me more about that?
It's rather unfortunate that the term “egg” has been used to mean two rather different things in setuptools. AIUI, an “egg” is one possible *format* for a distribution: a zipped collection of the distribution files and metadata files <URL:http://peak.telecommunity.com/DevCenter/PythonEggs>. It's supposed to be equivalent to a Java “jar”, in that it can be copied around easily and imported from directly via setuptools's infrastructure. This is offset by the relative opacity of it: it's more difficult to integrate with a proper OS-level package manager, so many developers deliberately *avoid* building or distributing eggs. When building a distribution via setuptools, an egg is *optional*; the distribution may be in an egg, or it may be in a regular tarball or whatever. When installed, it may or may not be installed as an egg <URL:http://peak.telecommunity.com/DevCenter/setuptools#install-run-easy-install-...>. The confusion is compounded by the fact that setuptools creates its metadata in a directory named ‘foo.egg-info/’, even if the distribution is not built to an egg and never will be. Any of the above could be wrong: I'm not a setuptools expert either, and have only rubbed up against its rough edges trying to get things working. -- \ “I installed a skylight in my apartment. The people who live | `\ above me are furious!” —Steven Wright | _o__) | Ben Finney
Packages that aren't installed in eggs don't need to be "before their own directory". The only reason that .egg files or directories do, is because if there's a non-egg package in the same directory, it could conflict with or override the .egg version otherwise. At 05:43 PM 1/23/2009, Mark Sienkiewicz wrote:
Ben Finney wrote:
Mark Sienkiewicz <sienkiew@stsci.edu> writes:
In that case, the .egg file should go on sys.path immediately before the directory where it is stored. This ensures that a setuptools .egg file will always be discovered before a distutils-installed package/module in the same directory, but it will not override packages/modules that occur earlier in sys.path.
What of distributions installed via setuptools that are *not* installed as eggs?
I don't know. Can you tell me more about that? I'm not particularly familiar with setuptools -- I'm just offering to write the patch because I don't see it getting fixed any other way.
Every thing that I have ever installed with setuptools has landed in a file or directory named like "*.egg". For example: site-packages/nose-0.10.4-py2.5.egg site-packages/Sphinx-0.5.1-py2.5.egg site-packages/Jinja-1.2-py2.5-macosx-10.3-i386.egg
The installation creates a file "easy-install.pth" that looks like this:
% cat easy-install.pth import sys; sys.__plen = len(sys.path) ./setuptools-0.6c9-py2.5.egg ./nose-0.10.4-py2.5.egg ./Sphinx-0.5.1-py2.5.egg ./Jinja-1.2-py2.5-macosx-10.3-i386.egg ./Pygments-1.0-py2.5.egg ./docutils-0.4-py2.5.egg import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new) %
So, where I said ".egg files", you can substitute the name of the things on line 2 through N-1 in the easy-install.pth file. That last line of easy-install.pth guarantees that the .egg files (in this example, they are all eggs) will be placed in the wrong place on sys.path, and that is all I am proposing to fix. If there is another case that does not involve the easy-install.pth file, that is outside the scope of my patch.
So, about distributions that are not installed as eggs: Can you tell me a little about it? Can you give me an example of how I can see it happen? Or does it turn out to be irrelevant to my patch?
Mark S.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Phillip J. Eby wrote:
Packages that aren't installed in eggs don't need to be "before their own directory". The only reason that .egg files or directories do, is because if there's a non-egg package in the same directory, it could conflict with or override the .egg version otherwise.
In that case, do you find my proposed patch acceptable? Mark S.
What's the status of Mark Sienkiewicz's patch to solve the most widely-hated feature of setuptools (that it disrespects the order of your PYTHONPATH) while also satisfying PJE's requirements for precedence of egg importing? http://mail.python.org/pipermail/distutils-sig/2009-January/010768.html Here's the ticket: http://bugs.python.org/setuptools/issue53 # respect the PYTHONPATH Regards, Zooko
zooko wrote:
What's the status of Mark Sienkiewicz's patch to solve the most widely-hated feature of setuptools (that it disrespects the order of your PYTHONPATH) while also satisfying PJE's requirements for precedence of egg importing?
It would make certain things easier if I could install setuptools, so this patch is not entirely outside my job description, but I have a lot of work with a higher priority. I still intend to get to it at some point. If somebody else wants to write it, go right ahead. Just send me an email so I know you're doing it and don't write a conflicting patch. One of the followups to the message you cited contains requirements (or as close as you're likely to get, anyway) for the patch. I also posted a brief outline of a patch some time before that. As far as I can tell, defining that set of requirements is the hard part of the patch. The actual implementation is not rocket science. Mark S.
At 03:00 PM 1/22/2009 -0500, Mark Sienkiewicz wrote:
Phillip J. Eby wrote:
At 04:48 PM 1/20/2009 -0700, zooko wrote:
So, this is just to let everyone know that any setuptools variant (including a future release of PJE's setuptools) that respects the PYTHONPATH will get at least two users!
As I've said before, if somebody creates a patch that addresses the use cases I pointed out, I'd be happy to accept it.
As I understand it, you require that a package installed by setuptools will be discovered before a package installed by distutils >in the same directory<. You do not require that a package installed by setuptools be discovered before a package that is earlier in sys.path.
In that case, the .egg file should go on sys.path immediately before the directory where it is stored. This ensures that a setuptools .egg file will always be discovered before a distutils-installed package/module in the same directory, but it will not override packages/modules that occur earlier in sys.path.
Do you have any other requirements?
In http://mail.python.org/pipermail/distutils-sig/2008-November/010534.html , I described a change to zooko's patch that implements this behaviour.
In http://mail.python.org/pipermail/distutils-sig/2008-November/010540.html , I address concerns about case-insensitive filesystems. So, taken together, does my suggestion meet your criteria for an acceptable patch?
If so,
- I will assemble it into a patch. - I will test it on platforms that I have access to, in ways that I know how to use setuptools. (For the record, that is "python setup.py install --home=/dir"; I never tried the automatic downloads.)
If not,
- what further requirements do you have?
In principle, this sounds pretty good. The only piece left that I'm worried about is that there needs to be a fallback for the .index() call in the .pth file. Otherwise, if for some reason there is *not* a match, there will be an error and .pth loading will fail. On the other hand, maybe it's better for there to be an error in that case, I'm just a bit worried about the new and uncontrolled possible way for things to break, vs. the known and predictable way they break now. ;-) As for the two "big loops" in site-patch.py, the first one is there to load the existing site module, without altering sys.path in any way. And it has to use both the path importer cache and 'imp', in order to be compatible w/e.g. systems where the stdlib is a zipfile. The second "big loop" is responsible for moving any paths added by .pth files on PYTHONPATH so that they are ahead of the stdlib and site-packages. (Otherwise, if you install something like PIL to PYTHONPATH, it can't override a version installed in site-packages.) It's not clear to me at this moment whether this loop would need to change with the patch. I don't think it will, though. The site-packages eggs would get inserted before site-packages, and would be considered "known paths", and thus have their position left unchanged. Whereas PYTHONPATH eggs would be before the stdlib, and also left alone. The last concern I have is whether you can partially upgrade an installation, i.e., what happens if you have two PYTHONPATH directories, one where this patch has been used, and the other where it has not. The one where it has been used will put its eggs in the "right" place, while the other will blindly insert to the beginning. Hm. Yes, it seems like that would be okay, because the "right" place would never be before sys.__egginsert. Thus, the old code could never stick an egg in a place that would mess something up. And both approaches result in PYTHONPATH eggs being before the stdlib, and thus left alone by the second loop. Okay, yes, I think this approach can work... *except* for perhaps the case where an old site-patch is in effect, due to being earlier on PYTHONPATH, with newer .pth files in a second PYTHONPATH directory. In that case, the addsitedir() calls in site-patch.py will *not* be normalized. So, for this to work, site-patch.py should *not* be changed. Instead, the .pth-embedded code needs to do the makepath() call before the .index() lookup. Otherwise, people with multiple PYTHONPATH directories would need to specifically update the site.py in them, to not have inexplicable breakages. I'll review an updated patch along these lines.
participants (7)
-
Ben Finney -
David Cournapeau -
Gael Varoquaux -
Mark Sienkiewicz -
P.J. Eby -
Phillip J. Eby
-
zooko