[Distutils] EasyInstall --> distutils command + setuptools enhancements?

[Phillip J. Eby]

As I move towards implementing features for EasyInsall like having 
distutils-style configuration files, and refactoring the 'main()' function 
for reusability and extension by other packages, I began to notice 
something: I was basically reinventing a distutils Command object.

Sure, if I subclass Command, I'm stuck with a crummy 'log', and an ancient 
ancestor of 'optparse' (i.e. 'fancy_getopt'), but I get config file parsing 
for free, I get a known structure for arranging things, and I can trivially 
get access to the work done by other distutils commands, like finding out 
where the 'install' command would've installed libraries and scripts.

Not only that, but if I wanted to implement the commands other people have 
been asking for, like to list installed packages, uninstall packages, 
search PyPI, etc., I could just add them to the same harness.  What's more, 
people could add third-party commands using the --command-packages option 
under 2.4 (and I could perhaps backport support into setuptools so that it 
would work with 2.3 as well, at least for EasyInstall).

My main concern with this approach is lengthening the command line, because 
now you'd have to specify a command, resulting in things like this:

     python -m easy_install get SQLObject

The current name seems to clash with the idea of having different commands, 
too.  It almost seems like it should be more like:

     python -m package get SQLObject
     python -m package list
     python -m package delete mechanize
     python -m package check FooBar[someopt,otheropt]
     python -m package download somepkg
     python -m package extract otherpkg

In addition to providing a bit more flexibility, this would let us break up 
the somewhat monolithic structure of the current installer code, so that 
you could use individual pieces as shown, like extracting a package to a 
directory without building or installing it.  Not only would that make 
these commands available to the user, but it would also make them available 
to packages that use setuptools in their setup.py, such that a package 
could e.g. download and extract the source of some C library that it 
depends on, if it's not installed on the user's system.  The 'download' 
command could delegate to 'download_url', 'download_file', and 
'download_req' subcommands, corresponding to the current 
features.  'extract' would delegate to 'download', and so on.  There'd 
probably be a 'setup' command, too, to run a setup script under sandboxing, 
etc.

So what are the downsides to this, apart from needing to turn lots of 
little methods into entire Command subclasses?  Well, I imagine it makes it 
harder for somebody to build-in the commands to non-distutils programs, 
unless there's an easy way to create and run the commands, like if I added 
a class method like 'invoke(**args)' to a base class, so that you could say 
stuff like 'package.download_url.invoke(url="whatever")' in order to call 
'em from inside the program.  I've already added something similar to the 
'bdist_egg' comamnd called 'call_command(name,**args)', so that you can 
more easily invoke other commands as subroutines.

Features like those could well be useful for the distutils as well, so 
perhaps 'setuptools.Command' should include them.  Right now, setuptools 
exports the same Command class as the distutils, but I could easily change 
that, so that Commands can be easily created and invoked from non-distutils 
code.

The other downside that needs mitigation is the logging crud.  Distutils is 
spattered with write() calls to sys.stderr and sys.stdout, and it 
inconsistently uses its own 'log.' calls (e.g., it has messages beginning 
with the word "Warning" that nonetheless use 'log.info()' as the 
level!).  It has stuff that depends on a distutils-specific verbosity 
level, and so on.

All of this can and should be fixed in Python 2.5, but EasyInstall needs it 
now, so setuptools will probably do some monkeypatching here and there to 
get 'distutils.log' rerouted to 'logging.getLogger("distutils")', and 
perhaps hack things in such a way that the distutils verbose/quiet stuff is 
only used to change the log level, and and restores the level when commands 
are finished.  I'm not entirely sure how feasible this is.  Some of the 
sys.stderr writes are also not monkeypatchable without replacing giant 
routines, and there are close to 70 print statements spread across 16 
modules.  Even just going through to verify which ones are debug prints is 
a PITA.  However, if someone were trying to integrate these commands into 
some type of GUI application, it might cause occasional problems unless 
stdout and stderr were replaced or redirected somehow.

Still, the bulk of the logging issues can probably be safely handled in a 
couple of monkeypatches to Command, CCompiler, and text_file.TextFile; most 
of the rest occur under pretty obscure circumstances.  So, I think an 80% 
solution is quite doable.

I don't really want to tangle with the fancy_getopt vs. optparse 
distinction, though.  Or any of the other places where a utility module in 
the distutils has been left behind by a newfangled version in the standard 
library.  :(

I mean, on one level it's always tempting to try to write "Distutils 2", 
but at this point there's so much accumulated knowledge in distutils that I 
don't think it's reasonable to expect a replacement to ever be doable in a 
single rewrite.  Refactoring the existing stuff is the only way to go.

Anyway, now I'm digressing.  Thoughts, anyone?