[Python-Dev] how to easily consume just the parts of eggs that are good for you

[Chris McDonough]

zooko wrote:
> 
> On Mar 26, 2008, at 7:34 PM, Chris McDonough wrote:
>> zooko wrote:
> 
> http://mail.python.org/pipermail/python-dev/2008-March/078243.html
> 
>>> Here is a simple proposal:  make the standard Python "import"  
>>> mechanism notice eggs on the PYTHONPATH and insert them (into the  
>>> *same* location) on the sys.path.
>>> This eliminates the #1 problem with eggs -- that they don't easily  
>>> work when installing them into places other than your site-packages  
>>> and that if you allow any of them to be installed on your system 
>>> then  they take precedence over your non-egg packages even you 
>>> explicitly  put those other packages earlier in your PYTHONPATH.  
>>> (That latter  behavior is very disagreeable to more than a few 
>>> prorgammers.)
>>
>> Sorry if I'm out of the loop and there's some subtlety here that I'm 
>> disregarding, but it doesn't appear that either of the issues you 
>> mention is a actually problem with eggs.  These are instead problems 
>> with how eggs get installed by easy_install (which uses a .pth file to 
>> extend sys.path).  It's reasonable to put eggs on the PYTHONPATH 
>> manually (e.g. sys.path.append('/path/to/some.egg')) instead of using 
>> easy_install to install them.
> 
> Yes, you are missing something.  While many programmers, such as 
> yourself and Lennart Regebro (who posted to this thread) find the 
> current eggs system to be perfectly convenient and to Just Work, many 
> others, such as Glyph Lefkowitz (who posted to a related thread) find 
> them to be so annoying that they actively ensure that no eggs are ever 
> allowed to touch their system.
> 
> The reasons for this latter problem are two:
> 
> 1.  You can't conveniently install eggs into a non-system directory, 
> such as ~/my-python-stuff.

That's just not true.

$ export PYTHONPATH=/home/you/my-python-stuff/foo-1.3.egg
$ python
 >>> import foo

Eggs are directories (or zipfiles) that contain packages.  They happen to 
contain other metadata directories too, but these can be ignored if you just 
want to *use* them (as opposed to wanting to introspect metadata about them).

> 2.  If you allow even a single egg to be installed into your PYTHONPATH, 
> it will change the semantics of your PYTHONPATH.

I think you are mistaken.  The use of the .pth file that changes sys.path is a 
feature of easy_install, not of eggs.  You don't need to use any .pth file to 
put eggs on your PYTHONPATH.

Note that zc.buildout is a framework that installs eggs, and it doesn't rely at 
all on .pth files to automatically hack sys.path.  Instead, it munges console 
scripts to add each egg dir to sys.path.  This is pretty nasty too, but it does 
prove the point.

It is however true that you need to change sys.path to use an egg.  Is that what 
you're objecting to fundamentally?  You just don't want to have to change 
sys.path at all to use an egg?  Maybe you're objecting to the notion that an egg 
can contain more than one package.  There is functionally no difference between 
an egg and a directory full of packages.

> Both of these problems are directly caused by the need for eggs to hack 
> your site.py.  If Python automatically added eggs found in the 
> PYTHONPATH to the sys.path, both of these problems would go away.

I'm not sure what you mean.  Eggs don't hack site.py.  Eggs are just a packaging 
format.  easy_install doesn't hack site.py either.  It just puts a .pth file 
(the parsing of which is a feature of "core" Python itself, not any setuptools 
magic) in site packages and manages it.

It seems like you're advocating adding magic that you can't turn off (magical 
detection of eggs in an already site.py-approved packages directory) to defeat 
magic that you can turn off (by not using easy_install or .pth files).  At some 
level the magic you want to see built in to Python would almost certainly wind 
up doing what you hate by hacking sys.path unless you wanted to restrict eggs to 
containing a single package only.  And you wouldn't be able to turn it off 
except through some obscure environment variable setting.

> By the way, since I posted my proposal two weeks ago I have pointed a 
> couple of Python hackers who currently refuse to use eggs at the URL:
> 
> http://mail.python.org/pipermail/python-dev/2008-March/078243.html
> 
> They both agreed that it made perfect sense.  I told one of them about 
> the alternate proposal to define a new database file to contain a list 
> of installed packages, and he sighed and rolled his eyes and said "So 
> they are planning to reinvent apt!".

I think changing the Python core is the worst possible answer to this problem. 
"Don't use easy_install" is currently the best, AFAICT.

- C