Re: [Distutils] Anyone stuck with easy_install / has .pth file issues -- this is for you!

At 02:43 AM 5/4/2009 -0700, Garrett Cooper wrote:
Hi guys, Just thought I'd might provide this script to fellow developers which fixes .pth files (easy-install.pth / .egg was the prime target -- see the comments for more details): <http://yaneurabeya.livejournal.com/3929.html>. Comments are more than welcome.
As far as I can tell, it doesn't do anything that "easy_install -mxN" doesn't, although it appears to also convert paths of this form: /foo/bar/baz/foo/bar/spam into: ./baz./spam if I'm reading the code correctly. It also seems to have no protection against adding multiple versions of the same project to a .pth file, and to ignore development eggs, whether or not their directories still exist. In contrast, easy_install already removes non-existent files/directories whenever it touches easy_install.pth, and if you gave it a command line globbing the same files as this tool (i.e., just "easy_install [list of eggs]"), you'd at least end up without any duplicates in the .pth file. In short, AFAICT, you could replace the entire tool with a short note on how to accomplish the same things using easy_install, or by simply having it invoke easy_install internally.

Hi P.J.! On Mon, May 4, 2009 at 8:20 AM, P.J. Eby <pje@telecommunity.com> wrote:
At 02:43 AM 5/4/2009 -0700, Garrett Cooper wrote:
Hi guys, Just thought I'd might provide this script to fellow developers which fixes .pth files (easy-install.pth / .egg was the prime target -- see the comments for more details): <http://yaneurabeya.livejournal.com/3929.html>. Comments are more than welcome.
As far as I can tell, it doesn't do anything that "easy_install -mxN" doesn't, although it appears to also convert paths of this form:
/foo/bar/baz/foo/bar/spam
into:
./baz./spam
if I'm reading the code correctly. It also seems to have no protection against adding multiple versions of the same project to a .pth file
You're right -- it doesn't protect against the following (><): /full/path/to/package.egg ./package.egg It does protect against ./package.egg in .pth not existing in the filesystem, and vice versa, depending on the glob of choice employed. I'll fix that gap tonight :). BTW, wasn't -m deprecated? I mean, -m's `delivered' functionality doesn't even work because python stores all loaded package / module entries in .pth files as keys in a dictionary _anyhow_ (hence the proposal I made a few weeks ago with __import__ to do version checking).. setuptools can't even fix this behavior (python does a BFS-like package search based on the module / package name and returns the first entry found), even though it's _sort_ of supposed to (look for "A more complete solution" on <http://mail.python.org/pipermail/distutils-sig/2006-March/006123.html>). Feel free to check sys.modules if you don't believe me :). Hence the need for virtualenv. I have no idea how -xN factor into this though -- maybe I'm missing an important easy-install usage note... either way, we're stuck with 0.6c3 at work because the folks we inherited it from hacked the code (in an ugly way... ew) and I don't have time to fix it now, so does this still apply ;\...?
and to ignore development eggs, whether or not their directories still exist.
By default, no. If one wants to avoid a series of directories, they can omit entries with [a] relevant expression(s) via the -g / -r option(s).
In contrast, easy_install already removes non-existent files/directories whenever it touches easy_install.pth, and if you gave it a command line globbing the same files as this tool (i.e., just "easy_install [list of eggs]"), you'd at least end up without any duplicates in the .pth file.
In short, AFAICT, you could replace the entire tool with a short note on how to accomplish the same things using easy_install, or by simply having it invoke easy_install internally.
- The duplicates erasure was just gravy, and purely an exercise on my part getting used to set's in python. The real point was the missing package in .pth functionality, caused by easy_install / setuptools lacking proper MP-safe logic. - It's a simple tool (took a while to think up, but took 1.5 hours to code / validate) designed to function outside of easy_install / setuptools, even though that was the piece I was trying to target... so if someone hand edits a .pth file (heaven forbid) it would note that the file needed to be fixed / fix the file =]. Thanks for the comments! -Garrett

On Mon, May 4, 2009 at 1:11 PM, Garrett Cooper <yanegomi@gmail.com> wrote:
Hi P.J.!
On Mon, May 4, 2009 at 8:20 AM, P.J. Eby <pje@telecommunity.com> wrote:
At 02:43 AM 5/4/2009 -0700, Garrett Cooper wrote:
Hi guys, Just thought I'd might provide this script to fellow developers which fixes .pth files (easy-install.pth / .egg was the prime target -- see the comments for more details): <http://yaneurabeya.livejournal.com/3929.html>. Comments are more than welcome.
As far as I can tell, it doesn't do anything that "easy_install -mxN" doesn't, although it appears to also convert paths of this form:
/foo/bar/baz/foo/bar/spam
into:
./baz./spam
if I'm reading the code correctly. It also seems to have no protection against adding multiple versions of the same project to a .pth file
You're right -- it doesn't protect against the following (><):
/full/path/to/package.egg ./package.egg
It does protect against ./package.egg in .pth not existing in the filesystem, and vice versa, depending on the glob of choice employed. I'll fix that gap tonight :).
BTW, wasn't -m deprecated? I mean, -m's `delivered' functionality doesn't even work because python stores all loaded package / module entries in .pth files as keys in a dictionary _anyhow_ (hence the proposal I made a few weeks ago with __import__ to do version checking)..
setuptools can't even fix this behavior (python does a BFS-like package search based on the module / package name and returns the first entry found), even though it's _sort_ of supposed to (look for "A more complete solution" on <http://mail.python.org/pipermail/distutils-sig/2006-March/006123.html>). Feel free to check sys.modules if you don't believe me :). Hence the need for virtualenv.
I have no idea how -xN factor into this though -- maybe I'm missing an important easy-install usage note... either way, we're stuck with 0.6c3 at work because the folks we inherited it from hacked the code (in an ugly way... ew) and I don't have time to fix it now, so does this still apply ;\...?
and to ignore development eggs, whether or not their directories still exist.
By default, no. If one wants to avoid a series of directories, they can omit entries with [a] relevant expression(s) via the -g / -r option(s).
In contrast, easy_install already removes non-existent files/directories whenever it touches easy_install.pth, and if you gave it a command line globbing the same files as this tool (i.e., just "easy_install [list of eggs]"), you'd at least end up without any duplicates in the .pth file.
In short, AFAICT, you could replace the entire tool with a short note on how to accomplish the same things using easy_install, or by simply having it invoke easy_install internally.
- The duplicates erasure was just gravy, and purely an exercise on my part getting used to set's in python. The real point was the missing package in .pth functionality, caused by easy_install / setuptools lacking proper MP-safe logic.
I do need to check for commented entries though / expand the search criteria or something... excellent point made P.J.
- It's a simple tool (took a while to think up, but took 1.5 hours to code / validate) designed to function outside of easy_install / setuptools, even though that was the piece I was trying to target... so if someone hand edits a .pth file (heaven forbid) it would note that the file needed to be fixed / fix the file =].
Thanks for the comments! -Garrett
participants (2)
-
Garrett Cooper
-
P.J. Eby