[Python-Dev] setuptools in the stdlib ([Python-checkins] r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py)

[Phillip J. Eby]

At 09:02 PM 4/18/2006 +0200, M.-A. Lemburg wrote:
>Phillip J. Eby wrote:
> > At 07:15 PM 4/18/2006 +0200, M.-A. Lemburg wrote:
> >> Why should a 3rd party extension be hot-fixing the standard
> >> Python distribution ?
> >
> > Because setuptools installs things in zip files, and older versions of
> > pydoc don't work for packages zip files.
>
>That doesn't answer my question.

That is the answer to the question you asked: "why hot-fix?"  Because 
setuptools uses zip files, and older pydocs crash when trying to display 
the documentation for a package (not module; modules were fine) that is in 
a zip file.


>Third-party extension *should not do this* !

If you install setuptools, you presumably would like for things to work, 
and the hot fix eliminates a bug that interferes with them working.

I'm not sure, however, what you believe any of that has to do with 
python-checkins or python-dev.  The version of setuptools that will do this 
is not yet released, and the hotfix aspect will be documented.  If it 
causes actual problems for actual setuptools users, it will be actually 
fixed or actually removed.  :)  (Meanwhile, the separately-distributed 
setuptools package is not a part of Python, any more than the Optik is, 
despite Python including the 'optparse' package spun off from Optik.)


>I'm talking about the setuptools package which does apply
>monkey patching and is needed to manage the download and
>installation of plugin eggs, AFAIK.

In which case the monkeypatching is needed, to handle situations like 
building eggs for third-party non-setuptools libraries listed as dependencies.

You can't have it both ways; either you need setuptools or you don't.  If 
you do, you will almost certainly need the monkeypatching, and you'll 
definitely need it to build eggs for non-setuptools-based packages.


>What's good about it, is that this approach doesn't modify anything
>inside distutils at run-time, but does these modifications
>on a per-setup()-call basis.

What's bad about it, is that it provides no options for virtualizing the 
execution of other packages' setup scripts.  The 'run_setup' mechanism was 
horribly broken in the versions of the distutils that I wrestled with.

(I also don't see anybody stepping up to provide alternative 
implementations or patches to third-party setup scripts or distutils 
extensions to deal with this issue.)


>There aren't all that many things that are wrong in setuptools,
>but some of them are essential:
>
>* setuptools should not monkey patch distutils on import

Please propose an alternative, or better still, produce a patch.  Be sure 
that it still allows distutils extensions like py2exe to work.  The only 
real alternative to monkeypatching that I'm aware of for that issue is to 
actually merge setuptools with the distutils, but my guess is that you'll 
like that even less.  :)


>* the standard "python setup.py install" should continue to
>   work as documented in the Python documentation; any new install
>   command should use a different name, e.g. "install_egg"
>
>* placing too many ZIP files on PYTHONPATH is a bad idea
>   since it slows down import searches for all Python
>   applications, not just ones relying on eggs; a solution
>   to this would be to have a PYTHONEGGPATH which is then
>   only scanned by egg-aware modules/applications
>
>* the user should have freedom of choice in whether to
>   have her Python installation rely on eggs or not (and
>   not only --by-using-some-complicated-options)

These questions have been hashed to death on the distutils-sig already, but 
I will try to summarize my responses here as briefly as practical while 
still covering the full spectrum of issues.  I would begin by saying that 
the tradeoffs I've made favor inexperienced users at the expense of experts 
(who I assume are capable of learning to use --root or 
--single-version-externally managed in order to have finer-grained control).

Your proposals, however, generally favor experts at the expense of the 
average user, and treat it as axiomatic that the benefits provided by 
setuptools are not worth having, no matter how small the cost.  By 
contrast, users who *want* to use setuptools for either distribution or 
installation of packages, would rather it "just work", no matter how much 
complexity is required behind-the-scenes.  They don't care about my cost to 
implement, they just care that they type "easy_install foo" to get a 
package from PyPI or "setup.py sdist bdist_egg upload" to put it there.

Therefore, it makes no sense to apply your design approach to setuptools, 
since by your criteria it wouldn't exist in the first place!  After all, 
expert users can munge distutils extensions to their hearts' content (and 
you certainly have done so already).

Next, your own measurements posted to the distutils-sig debunked the 
PYTHONPATH performance question, IMO: if you installed *every Python 
package listed on PyPI at that time* as a zip file (assuming that the 
average zipfile directory size was the same as in your tests, around 1100 
total packages on PyPI, and linear degradation), Python startup time would 
have increased by little more than half a second.  To me, that says the 
performance is pretty darn good, since the average user isn't going to 
install anywhere near that many packages.

Note also that installing eggs with -m (or --multi-version) doesn't put 
them on sys.path to begin with unless they're needed at runtime, so if you 
care about performance tweaking, you can just use -m for everything, or put 
it in your distutils.cfg or ~/.pydistutils.cfg file so it's 
automatic.  Since scripts that depend on eggs can automatically find and 
add the eggs to sys.path at runtime, they don't need the eggs to be on 
sys.path.

The reason they are put on sys.path by default is to make the default case 
of installation "just work".  Specifically, it's so that "setup.py install" 
works the same way as it does with the distutils (in terms of what the 
users can *do* with a package after installation, not in terms of how the 
files are laid out).

As far as I can tell, your remaining arguments can be reduced to this:

    Users who care about how stuff works inside (i.e. experts) will have to 
use command-line options to get the old behaviors or to tweak the new ones, 
when working with packages that *explicitly chose* to use setuptools.

And that is a perfectly valid concern -- for an audience that's quite 
capable of learning to do something new, however.  Also note that nothing 
stops people from creating packages that warp distutils into installing 
things in weird places, so arguing that "setup.py install" should always do 
the same thing is moot anyway; you would not get your wish even if 
setuptools had never existed.

May I suggest that perhaps a better solution to the actual practical 
problem here is to make the distutils "install" command accept the options 
that setuptools supports?  Then, people who want it can just use 
--single-version-externally-managed with any package under Python 2.5 and 
be guaranteed to get the old-fashioned behavior (that can't be cleanly 
upgraded or uninstalled without a package manager).

As far as I know, however, most tools needing the old installation layout 
use "install --root" anyway (including bdist_wininst, bdist_msi, bdist_rpm, 
etc.), and setuptools automatically switches to compatibility mode in that 
case.  So setuptools "just works" with these tools already.