[Distutils] Initial auto-installation support

[Phillip J. Eby]

At 04:10 PM 5/30/2005 -0500, Ian Bicking wrote:
>I've added some very initial support for automatic installation of
>packages in Paste, using easy_install
>(http://peak.telecommunity.com/DevCenter/EasyInstall).  In a
>configuration file you can put:
>
>    use_package(package_name, pkg_download_url=None)
>
>And a couple other options, but we'll ignore those.  It will look for
>the named package, and if not found will install it (generally in
>app-packages).

FYI, just so you know, your implementation won't handle nested 
dependencies, nor allow specifying optional features of requested packages.

I'd suggest that in a later version, you take a look at subclassing 
pkg_resources.AvailableDistributions, and overriding the 'obtain()' method 
to do the search and installation.  In this way, your 'obtain()' method 
will get called for any dependencies that the originally-requested package 
requires.

To see what I mean, take a look at the source of pkg_resources.require(), 
which looks like this:

     requirements = parse_requirements(requirements)

     for dist in AvailableDistributions().resolve(requirements):
         dist.install_on(sys.path)

So, if you subclass AvailableDistributions to define an 'obtain()' method, 
and then write a similar loop using that subclass, you'll be able to 
cleanly integrate the auto-download in a forward-compatible way for 
packages that declare dependencies in their PackageName.egg-info.


>There's another aspect to Paste installation, where some packages
>(plugins) need to write things into Paste.  I'm not sure quite how that
>will work -- maybe use_package() will see if there's a paste_install
>module in the package somewhere, and call that somehow.

That might be a good candidate for egg metadata; if a package is a Paste 
plugin, you could require a 'paste-install' file in the package's 
EGG-INFO.  Note that the AvailableDistributions().resolve() method returns 
a list of Distribution objects, and distribution objects have a 'metadata' 
attribute that implements 'IMetadataProvider'.  So, 
'aDistribution.metdata.has_metadata("paste-install")' will tell you if that 
distribution has a "paste-install" file in its EGG-INFO, and using the 
get_metadata() method will give it to you as a string.  Thus, you can do 
something like:

     for dist in MyDownloadDistributions().resolve(requirements):
         dist.install_on(sys.path)
         if dist.metadata.has_metadata('paste-install'):
             doSomething(dist.metadata.get_metadata('paste-install'))

If you use this algorithm, it will work for all egg varieties: compressed, 
uncompressed, and "development" eggs.  You will, however, need to keep 
track of which paste-install scripts you've already processed, because 
'resolve()' can yield distributions that are already present on 
sys.path.  Also, you may want to create and cache a single instance of your 
MyDownloadDistributions class, because creating one does a lot of 
filesystem stats and listdirs and such.

Of course, if you have people doing TheirPackage.egg-info/paste-install, 
they can also create TheirPackage.egg-info/depends.txt, and list all their 
dependencies there, with no need to use 'use_package'.  It won't help with 
download URLs, though.  But we could perhaps define an 
EGG-INFO/download_urls.txt as a stopgap, that lists known download urls...

Anyway, as you can see, eggs were definitely designed with plugin systems 
like this in mind.  :)


>   But besides
>that, this should work now for any packages with a distutils install, so
>long as those packages are reasonably well behaved.  Hrm... except
>setuptools 0.3a2 doesn't have SourceForge download support, but 0.3a3
>does and I think PJE will release that soon.

Hopefully within the next 24 hours or so.  It will also include sandboxing 
support (automatically aborts the install if the package tries to write to 
the filesystem outside the build directory), and lots of workarounds to 
support various packages out there that have quirky install_data subclasses.

Those items are already done, but items still on my to-do list include:

* help message to explain how to use require() for multi-version/instdir 
installs

* a --build-dir/-b option to set the build directory, that will leave the 
downloaded package and its extracted contents in place after the 
installation is complete.  (So you can read docs, install scripts, or debug 
a failed installation.)

And I'd like to do something about scripts, but I think that's going to get 
left to an 0.4a1 release, assuming there are no further bug fix releases 
needed in the 0.3 line.