[Distutils] easy_install - some thoughts

[Phillip J. Eby]

At 10:32 PM 7/11/2005 +0100, Paul Moore wrote:
>1. It would be much better if easy_install was usable via python -m,
>    either as well as or instead of the existing easy_install.py
>    script.

I actually goofed in removing this; I'll put it back in the next release.


>2. It would be useful if the -f option worked with a file: URL (or
>    better still, a simple directory name), to handle local
>    repositories.

A good idea, which I've been thinking about.  I'll add it in the next 
release, such that if an argument to -f is not a URL, it gets checked for 
being an existing directory.  If it's a directory, I'll scan the contents 
as if they were links on a webpage.  If it's not an existing directory, 
I'll print a warning (since errors should not pass silently) and proceed.


>3. I still dislike the fact that I can't get a listing of installed
>    eggs.

I suggest writing a simple script using pkg_resources.find_distributions(), 
and perhaps adding some documentation and contributing it to setuptools.  :)


>And uninstalling by deletion isn't good, IMHO. I am
>    *strongly* averse to using raw file operations (delete) on files
>    in Python\lib\site-packages. Elsewhere, fine, but I view the
>    Python standard library as a "managed" area, and management tools
>    should exist for it. That's the key benefit to me of bdist_wininst,
>    and I don't see eggs providing it.

So write a pkg_delete script and contribute it, too.  I've got no objection 
to these other tools, they just haven't been my top priority and aren't 
likely to be for a while.  I'd like to focus on the core engine/API, in 
such a way that other people can easily write these sorts of management 
tools around it.


>4. --dry-run doesn't work as expected:
>
> >\Apps\Python24\Scripts\easy_install.py --dry-run 
> dist/alarm-1.0-py2.4-win32.egg
>
>Processing alarm-1.0-py2.4-win32.egg
>Copying alarm-1.0-py2.4-win32.egg to C:\Apps\Python24\Lib\site-packages
>zipimport.ZipImportError: not a Zip file

Sadly, this is about what *I* expect, because most distutils commands can 
only get so far into a --dry-run before some step depends on the 
nonexistent output of the preceding step.  I've been making the assumption 
that it's more useful to have a dry run that tells you most of what it does 
before dying, than to have no dry-run capability at all.  It's possible 
that I could make later steps say, "I'm in dry run mode and I don't know 
what the heck to do here", but I'm really not sure how I'd start.  If you 
or someone else has patches, I'll be happy to look at incorporating 
them.  Again, my focus for the next few releases is going to be on 
hardening pkg_resources to make it more robust and to have a more usable API.


>A nice to have feature would be the ability for easy_install to pick
>apart a bdist_wininst installer and build an egg from it.

Then I guess it's a good thing I implemented it already.  :)  Try running 
EasyInstall on a win32.exe file, and you'll find that it actually does a 
pretty good job of turning it into an egg.  If you were to run 
"easy_install pysqlite", for example, (assuming you could get past your 
firewall) you'd notice that it finds and downloads the bdist_wininst 
installer for PySQLite, converts it to an egg, and then installs it.  This 
feature is only obscurely referenced in the docs and changelog, so I don't 
really blame you for missing it.  :)


>On a related note, had you thought of having EasyInstall put eggs in
>a specific subdirectory? Something like site-packages/eggs? As you're
>using .pth files, this wouldn't be any harder to manage, would it?

Yes, it would, actually, especially for non-system installs, as Michael 
Hoffman was asking about here recently.  The main reason for putting eggs 
in sys.path directories, however, is to support automatic discovery and 
adding them to sys.path at *runtime*, dynamically.  I also wanted to take 
advantage of certain properties of sys.path, like the fact that Python's 
start script has its directory added to sys.path.  This then makes eggs in 
the script directory easily discoverable.

In fact, if you set an --install-dir with EasyInstall, and set the 
--always-copy flag, then your installation directory is an "application 
directory".  Scripts and eggs are both copied there, including all eggs 
needed to run the application.  Now you can just ship that directory and it 
can be used anywhere you have Python and setuptools installed.

Anyway, I think putting eggs in directories on sys.path is a natural 
extension of the sys.path concept, that seems to me to avoid the mental 
overhead of having a separate search path that works differently.


>I can see a use case for a "combination" egg. One which packages
>together a group of packages. For example, if I want a standard set
>of packages installed on all machines, so that users can assume they
>are present when writing one-off scripts. If I could package
>*existing* eggs up into a combination site-std.egg

This is called a "basket" (as in putting all your eggs in one basket!), and 
it's actually already supported, as long as you put your hypothetical 
site-std.egg directly on sys.path.  (It ought to be able to be just placed 
in a *directory* on sys.path, but when looking over the code just now I 
spotted a bug that prevents it from working unless you actually put the 
.egg file directly on sys.path.  I'll fix this bug in the next release, so 
that dropping the egg in a sys.path directory is all you should need to do.)

Anyway, the format of it is that you zip up the *unpacked* form of the 
eggs, directory names and all, so that if you have foo-1.5 and bar-2.7 then 
the interior of the combined egg needs to look like this:

     foo-1.5-py2.3-win32.egg/
         # foo egg contents
     bar-2.7-py2.3-win32.egg/
         # bar egg contents

So, the combination process is trivial enough that you can do it by hand, 
or with a simple script.  Just unzip the old eggs, if they weren't 
installed as directories to begin with, then zip the resulting directories.

Anyway, if you end up with something nice and want to contribute it, that's 
great too.  As with the other tools you've suggested, it's something I 
might get around to eventually, but there are some things I'm dying to get 
working first in the core engine, and I also have a lot of stuff I'd like 
to actually *use* the existing tools for right now.  :)