[Distutils] How we can get rid of eggs for 2.6 and beyond

[Phillip J. Eby]

So, after having some time to absorb the Python-Dev threads about 
setuptools, bootstrap, and all the rest, I think I see an opportunity 
to let people route around the "damage" of eggs, while still making 
it possible for the people who want to use easy_install or to put 
dependencies in their setup.py to get what they want, too.  (And 
without them needing to install eggs, either.)  At the same time, we 
can address the issues that remain around uninstalling packages, 
system vs. user packages, PYTHONPATH and site.py woes, and really 
pretty much every other complaint I've heard in the last few days 
about setuptools stomping on other people's stuff.  (Even Paul's 
Windows issues, hopefully.)

Now, you might be asking, "Okay, so why are you telling me about 
this?  Why not just go fix setuptools?"  Well, I *can't*.  Not 
without some help from Python-Dev and the Distutils-SIG, to create an 
updated standard for installed package metadata, by updating PEP 262 
("A Database of Installed Python Packages") to include input from the 
system packaging folks, support for namespace packages, and support 
for setuptools-compatible dependency information.

What's that got to do with anything?  Well, without it, setuptools 
can't support uninstall or conflict management without using eggs to 
compartmentalize the installed files.  And because it has to use eggs 
to do *that*, it has to munge .pth files and install its own site.py 
when installing to PYTHONPATH.  All of this ugliness follows directly 
from the absence of a PEP 262-style installation database.

Sure, setuptools could create its own version of this, and I almost 
did that four years ago.  (If you look at the "open issues" part of 
PEP 262, you'll see my comments from back then.)  I decided not to 
for two reasons: first, the distutils didn't support it yet, so it 
didn't help for conflict detection and avoidance in the real world at 
that point.

Second, there were no uninstall tools for it, so I'd have had to 
write one myself.  (Zed's "easy_f'ing_uninstall" to the contrary, it 
ain't easy, and I have an aversion to deleting stuff on people's 
systems without knowing what will break.  There's a big difference 
between them typing 'rm -rf' themselves, and me doing it.)

However, if tools exist and are distributed for such a "database", 
and *everybody* agrees to use it as an officially-blessed standard, 
then it should be possible for setuptools to co-exist with that 
framework, and we're all happy campers.

In particular, the "installing eggs sucks" camp should be happy, 
because it'll be possible for me (or anyone else) to write a version 
of easy_install that doesn't install eggs any more, and 
setuptools-based packages can go back to having "setup.py install" 
install things the old way by default.

So, to accomplish this, we (for some value of "we") need to:

1. Hash out consensus around what changes or enhancements are needed 
to PEP 262, to resolve the previously-listed open issues, those that 
have come up since (namespace packages, dependency specifications, 
canonical name/version forms), and anything else that comes up.

2. Update or replace the implementation as appropriate, and modify 
the distutils to support it in Python 2.6 and beyond.  And "support 
it" means, "ensure that 'install' and *all* bdist commands update the 
database".  The bdist_rpm, bdist_wininst, and bdist_msi commands, 
even bdist_dumb.  (This should probably also include the add/remove 
programs stuff in the Windows case.)

3. Create a document for system packagers referencing the PEP and 
introducing them to what/why/how of the standard, in case they 
weren't one of the original participants in creating this.

It will probably take some non-trivial work to do all this for Python 
2.6, but it's probably possible, if we start now.  I don't think it's 
critical to have an uninstall tool distributed with 2.6, as long as 
there's a reasonable way to bootstrap its installation later.

Questions, comments...  volunteers?   :)