Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 12:59 PM 3/17/2008 -0500, Guido van Rossum wrote:
On Mon, Mar 17, 2008 at 12:45 PM, Phillip J. Eby <pje@telecommunity.com> wrote:
At 12:17 PM 3/17/2008 -0500, Guido van Rossum wrote:
There will be no egg support in the standard library.
Are there any qualifications on that statement, or is this in the same category as "from __future__ import braces"?
IIUC eggs are a method of package management that includes support for dependencies, multiple versions, and C extensions in zip files, as well as conventions for naming these and for encoding metadata (e.g. how to find out the version or the dependencies). This whole set of conventions is IMO too much to include into the stdlib ATM -- if only because it has proved controversial in the past. Maybe a few years from now it will be no longer controversial and then my objections will disappear.
So, does this mean that the bootstrap tool must not use eggs? That seems a little bit odd, in that setuptools will at least need its .egg-info directory to get installed, and all of the people who'll be using this initially will be using it precisely in order to have support for eggs... So, it might be simpler all around to just clear up the "controversy". To the best of my recollection, only MAL and MvL have ever objected on Python-Dev to the idea of supporting eggs. Note: I'm specifically segregating "egg support" from the topic of including setuptools or easy_install in the stdlib directly. There are many legitimate reservations and open questions about the latter, including availability of volunteer support, choice of defaults, whether to replace distutils with setuptools, etc. etc. I recognize and respect the validity of those issues, which is precisely why I withdrew setuptools from inclusion in Python 2.5. However, regarding support for eggs, my understanding is that there were only two objections to eggs -- even at the time of the 2.5 setuptools discussions. And even though MvL objects to the idea of eggs in *principle*, I didn't read his recent posts as objecting to having the bootstrap tool download and install eggs in *practice*. (Although I hope he will clarify that stance one way or the other.) That leaves MAL, whose objections to PEP 365 centered on the API (he said he was "+1 on the concepts being added to the stdlib, -1 on adding the module in its current state"). Among other concerns, he wanted pkg_resources to be split into pkgutil and a new "egglib" module. I don't have a problem with this in principle, if there were a pkg_resources module that reconstituted the merged API. (But there are some practical problems with that approach, such as trying to split namespace package support between two theoretically-unrelated modules.) I would guess, however, that MAL's issues with the pkg_resources API would not apply to a bootstrap module whose sole purpose was to download eggs and put them on sys.path. Or, perhaps he would object *more*, I don't know. We could certainly ask him, though. :) So, was there anyone else you were counting towards "controversy"? The only other person I recall objecting to setuptools in any way on Python-Dev was effbot, and IIUC his objections were practical/administrative re: supporting easy_install and setuptools, not to the idea of .egg support in general. In summary, I think the controversy on Python-Dev regarding .egg support has actually been over for some time now.
On Mon, Mar 17, 2008 at 1:45 PM, Phillip J. Eby <pje@telecommunity.com> wrote:
At 12:59 PM 3/17/2008 -0500, Guido van Rossum wrote:
On Mon, Mar 17, 2008 at 12:45 PM, Phillip J. Eby <pje@telecommunity.com> wrote:
At 12:17 PM 3/17/2008 -0500, Guido van Rossum wrote:
There will be no egg support in the standard library.
Are there any qualifications on that statement, or is this in the same category as "from __future__ import braces"?
IIUC eggs are a method of package management that includes support for dependencies, multiple versions, and C extensions in zip files, as well as conventions for naming these and for encoding metadata (e.g. how to find out the version or the dependencies). This whole set of conventions is IMO too much to include into the stdlib ATM -- if only because it has proved controversial in the past. Maybe a few years from now it will be no longer controversial and then my objections will disappear.
So, does this mean that the bootstrap tool must not use eggs?
Correct.
That seems a little bit odd, in that setuptools will at least need its .egg-info directory to get installed, and all of the people who'll be using this initially will be using it precisely in order to have support for eggs...
I'm okay if setuptools, once it's been installed, runs some setup code that creates the .egg-info directory and whatever else. This means I'm also okay with the bootstrap module finding and invoking that setup code. But I'm *not* okay with building any kind of egg management into the bootstrap module. The bootstrap module must be be neutral w.r.t. the package management style.
So, it might be simpler all around to just clear up the "controversy". To the best of my recollection, only MAL and MvL have ever objected on Python-Dev to the idea of supporting eggs.
You can add my name to the list. I've heard plenty of people speak highly of eggs, but I've *also* heard from plenty of people (besides MAL and MvL) who have serious difficulties with the concept of eggs. I have certainly personally encountered plenty of situations where I wasn't able to complete an egg-based install because some dependency was broken (e.g. not available for the Python version I was using).
Note: I'm specifically segregating "egg support" from the topic of including setuptools or easy_install in the stdlib directly. There are many legitimate reservations and open questions about the latter, including availability of volunteer support, choice of defaults, whether to replace distutils with setuptools, etc. etc. I recognize and respect the validity of those issues, which is precisely why I withdrew setuptools from inclusion in Python 2.5.
However, regarding support for eggs, my understanding is that there were only two objections to eggs -- even at the time of the 2.5 setuptools discussions. And even though MvL objects to the idea of eggs in *principle*, I didn't read his recent posts as objecting to having the bootstrap tool download and install eggs in *practice*. (Although I hope he will clarify that stance one way or the other.)
You can do it in two stages. The bootstrap can download and install egg support, even though the bootstrap itself knows nothing about eggs. Then another bootstrap can download and install eggs.
That leaves MAL, whose objections to PEP 365 centered on the API (he said he was "+1 on the concepts being added to the stdlib, -1 on adding the module in its current state"). Among other concerns, he wanted pkg_resources to be split into pkgutil and a new "egglib" module. I don't have a problem with this in principle, if there were a pkg_resources module that reconstituted the merged API. (But there are some practical problems with that approach, such as trying to split namespace package support between two theoretically-unrelated modules.)
Well, you've heard my position now.
I would guess, however, that MAL's issues with the pkg_resources API would not apply to a bootstrap module whose sole purpose was to download eggs and put them on sys.path. Or, perhaps he would object *more*, I don't know. We could certainly ask him, though. :)
No need. I object to this myself.
So, was there anyone else you were counting towards "controversy"? The only other person I recall objecting to setuptools in any way on Python-Dev was effbot, and IIUC his objections were practical/administrative re: supporting easy_install and setuptools, not to the idea of .egg support in general.
In summary, I think the controversy on Python-Dev regarding .egg support has actually been over for some time now.
Not really. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
On 17/03/2008, Phillip J. Eby <pje@telecommunity.com> wrote:
That leaves MAL, whose objections to PEP 365 centered on the API (he said he was "+1 on the concepts being added to the stdlib, -1 on adding the module in its current state"). Among other concerns, he wanted pkg_resources to be split into pkgutil and a new "egglib" module. I don't have a problem with this in principle, if there were a pkg_resources module that reconstituted the merged API. (But there are some practical problems with that approach, such as trying to split namespace package support between two theoretically-unrelated modules.)
I would personally like to see such a separation. As one of the authors of PEP 302 (well, the documentation - Just did all of the implementation) I have an interest in strengthening the standard library's support for transparent use of PEP 302 importers. To that end, I would like to see such functions as pkg_resources.resource_string() standardised. That covers the pkgutil aspect of pkg_resources. The "egglib" side of things is where the controversy lies, IMHO. I have a (somewhat unfocussed) dislike of eggs, largely because of the lack of a package management tool to handle them. I can live with them or without them, and it doesn't bother me if others use them, but the thing that really, really bothers me is that the controversy (and yes, there is such) over eggs is hampering the adoption of the pkgutil side of pkg_resources. So from me, there's a strong +1 for the split of pkg_resources into additions to the existing pkgutil module, and an independent egglib. You say there are practical problems to doing this. OK, but could we not have a discussion on how to resolve those issues as they come up? Could the split not be initially into 3 parts - pkgutil (eg, resource_string), egglib, and "difficult"? Then there would be something concrete to discuss and resolve. Paul.
After reading all this, I really don't believe that adding egg support to the stdlib at this time is the right thing to do. I am therefore rejecting the PEP. I am hoping that someone will create a simpler bootstrap module that is able to download a file of pure Python code and install it, perhaps by running its setup.py, assuming that it only depends on distutils (or other things previously installed). I will welcome such a module into the stdlib. I'm not sure a PEP is even needed, though interested parties are certainly welcome to write a PEP specifying the behavior first. With 2.6 and 3.0 slated for release in September, there should be enough time to get this done before then. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum writes:
I am hoping that someone will create a simpler bootstrap module
FWIW (I've never tried to implement one of these things) I agree with Phillip. This is not possible in the sense you are advocating. Anything "simpler" is simply an invitation to an unending stream of issues, far more so than adopting eggs as a best current practice would. Better to Just Say No to an installer in the stdlib.
I'm not sure a PEP is even needed
This I simply don't understand. I *have* participated in bolting on features to such systems, and it's a mess. As features are added, existing users will demand that old packages install exactly as they ever did, except that sometimes (but not always!) they want them upgraded to put things in newly blessed places. Features are easy to add, since on the main thread of control installers are basically just a sequence of single commands, sometimes optional. python-dev has some pretty good controls that will help slow such trends. Nonetheless, PEP-less development of an installer system is scary. Installer complexity is a creeping horror, worse than kudzu.[1] Footnotes: [1] http://en.wikipedia.org/wiki/Kudzu#Invasive_species
There seems to be a misunderstanding about what I am proposing we do instead. The boostrap installer should only be powerful enough to allow it to be used to install a real package manager like setuptools. Maybe my use of Django as an example was confusing; I didn't actually mean that it should be possible to support installing Django directly (although I don't understand all the issure that would make it impossible). Only very few people would care about writing a setup script that works with this bootstrap module; basically only package manager implementers. --Guido On Tue, Mar 18, 2008 at 3:45 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Guido van Rossum writes:
I am hoping that someone will create a simpler bootstrap module
FWIW (I've never tried to implement one of these things) I agree with Phillip. This is not possible in the sense you are advocating. Anything "simpler" is simply an invitation to an unending stream of issues, far more so than adopting eggs as a best current practice would. Better to Just Say No to an installer in the stdlib.
I'm not sure a PEP is even needed
This I simply don't understand. I *have* participated in bolting on features to such systems, and it's a mess. As features are added, existing users will demand that old packages install exactly as they ever did, except that sometimes (but not always!) they want them upgraded to put things in newly blessed places. Features are easy to add, since on the main thread of control installers are basically just a sequence of single commands, sometimes optional.
python-dev has some pretty good controls that will help slow such trends. Nonetheless, PEP-less development of an installer system is scary. Installer complexity is a creeping horror, worse than kudzu.[1]
Footnotes: [1] http://en.wikipedia.org/wiki/Kudzu#Invasive_species
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
participants (4)
-
Guido van Rossum -
Paul Moore -
Phillip J. Eby -
Stephen J. Turnbull