Mailman 3 Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module) - Distutils-SIG

newer
Re: [Distutils] [Python-Dev] PEP...

Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)

older
Re: [Distutils] [Python-Dev] PEP...

Phillip J. Eby

17 Mar 2008 17 Mar '08

11:45 a.m.

At 12:59 PM 3/17/2008 -0500, Guido van Rossum wrote:

...

On Mon, Mar 17, 2008 at 12:45 PM, Phillip J. Eby wrote:

...
At 12:17 PM 3/17/2008 -0500, Guido van Rossum wrote:

...
There will be no egg support in the standard library.

Are there any qualifications on that statement, or is this in the same category as "from __future__ import braces"?

IIUC eggs are a method of package management that includes support for dependencies, multiple versions, and C extensions in zip files, as well as conventions for naming these and for encoding metadata (e.g. how to find out the version or the dependencies). This whole set of conventions is IMO too much to include into the stdlib ATM -- if only because it has proved controversial in the past. Maybe a few years from now it will be no longer controversial and then my objections will disappear.

So, does this mean that the bootstrap tool must not use eggs? That seems a little bit odd, in that setuptools will at least need its .egg-info directory to get installed, and all of the people who'll be using this initially will be using it precisely in order to have support for eggs... So, it might be simpler all around to just clear up the "controversy". To the best of my recollection, only MAL and MvL have ever objected on Python-Dev to the idea of supporting eggs. Note: I'm specifically segregating "egg support" from the topic of including setuptools or easy_install in the stdlib directly. There are many legitimate reservations and open questions about the latter, including availability of volunteer support, choice of defaults, whether to replace distutils with setuptools, etc. etc. I recognize and respect the validity of those issues, which is precisely why I withdrew setuptools from inclusion in Python 2.5. However, regarding support for eggs, my understanding is that there were only two objections to eggs -- even at the time of the 2.5 setuptools discussions. And even though MvL objects to the idea of eggs in *principle*, I didn't read his recent posts as objecting to having the bootstrap tool download and install eggs in *practice*. (Although I hope he will clarify that stance one way or the other.) That leaves MAL, whose objections to PEP 365 centered on the API (he said he was "+1 on the concepts being added to the stdlib, -1 on adding the module in its current state"). Among other concerns, he wanted pkg_resources to be split into pkgutil and a new "egglib" module. I don't have a problem with this in principle, if there were a pkg_resources module that reconstituted the merged API. (But there are some practical problems with that approach, such as trying to split namespace package support between two theoretically-unrelated modules.) I would guess, however, that MAL's issues with the pkg_resources API would not apply to a bootstrap module whose sole purpose was to download eggs and put them on sys.path. Or, perhaps he would object *more*, I don't know. We could certainly ask him, though. :) So, was there anyone else you were counting towards "controversy"? The only other person I recall objecting to setuptools in any way on Python-Dev was effbot, and IIUC his objections were practical/administrative re: supporting easy_install and setuptools, not to the idea of .egg support in general. In summary, I think the controversy on Python-Dev regarding .egg support has actually been over for some time now.

Show replies by date

Guido van Rossum

17 Mar 17 Mar

11:59 a.m.

New subject: [Python-Dev] PEP 365 (Adding the pkg_resources module)

On Mon, Mar 17, 2008 at 1:45 PM, Phillip J. Eby wrote:

...

At 12:59 PM 3/17/2008 -0500, Guido van Rossum wrote:

...
On Mon, Mar 17, 2008 at 12:45 PM, Phillip J. Eby wrote:

...
At 12:17 PM 3/17/2008 -0500, Guido van Rossum wrote:

...
There will be no egg support in the standard library.

Are there any qualifications on that statement, or is this in the same category as "from __future__ import braces"?

IIUC eggs are a method of package management that includes support for dependencies, multiple versions, and C extensions in zip files, as well as conventions for naming these and for encoding metadata (e.g. how to find out the version or the dependencies). This whole set of conventions is IMO too much to include into the stdlib ATM -- if only because it has proved controversial in the past. Maybe a few years from now it will be no longer controversial and then my objections will disappear.

So, does this mean that the bootstrap tool must not use eggs?

Correct.

...

That seems a little bit odd, in that setuptools will at least need its .egg-info directory to get installed, and all of the people who'll be using this initially will be using it precisely in order to have support for eggs...

I'm okay if setuptools, once it's been installed, runs some setup code that creates the .egg-info directory and whatever else. This means I'm also okay with the bootstrap module finding and invoking that setup code. But I'm *not* okay with building any kind of egg management into the bootstrap module. The bootstrap module must be be neutral w.r.t. the package management style.

...

So, it might be simpler all around to just clear up the "controversy". To the best of my recollection, only MAL and MvL have ever objected on Python-Dev to the idea of supporting eggs.

You can add my name to the list. I've heard plenty of people speak highly of eggs, but I've *also* heard from plenty of people (besides MAL and MvL) who have serious difficulties with the concept of eggs. I have certainly personally encountered plenty of situations where I wasn't able to complete an egg-based install because some dependency was broken (e.g. not available for the Python version I was using).

...

Note: I'm specifically segregating "egg support" from the topic of including setuptools or easy_install in the stdlib directly. There are many legitimate reservations and open questions about the latter, including availability of volunteer support, choice of defaults, whether to replace distutils with setuptools, etc. etc. I recognize and respect the validity of those issues, which is precisely why I withdrew setuptools from inclusion in Python 2.5.

However, regarding support for eggs, my understanding is that there were only two objections to eggs -- even at the time of the 2.5 setuptools discussions. And even though MvL objects to the idea of eggs in *principle*, I didn't read his recent posts as objecting to having the bootstrap tool download and install eggs in *practice*. (Although I hope he will clarify that stance one way or the other.)

You can do it in two stages. The bootstrap can download and install egg support, even though the bootstrap itself knows nothing about eggs. Then another bootstrap can download and install eggs.

...

That leaves MAL, whose objections to PEP 365 centered on the API (he said he was "+1 on the concepts being added to the stdlib, -1 on adding the module in its current state"). Among other concerns, he wanted pkg_resources to be split into pkgutil and a new "egglib" module. I don't have a problem with this in principle, if there were a pkg_resources module that reconstituted the merged API. (But there are some practical problems with that approach, such as trying to split namespace package support between two theoretically-unrelated modules.)

Well, you've heard my position now.

...

I would guess, however, that MAL's issues with the pkg_resources API would not apply to a bootstrap module whose sole purpose was to download eggs and put them on sys.path. Or, perhaps he would object *more*, I don't know. We could certainly ask him, though. :)

No need. I object to this myself.

...

So, was there anyone else you were counting towards "controversy"? The only other person I recall objecting to setuptools in any way on Python-Dev was effbot, and IIUC his objections were practical/administrative re: supporting easy_install and setuptools, not to the idea of .egg support in general.

In summary, I think the controversy on Python-Dev regarding .egg support has actually been over for some time now.

Not really. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Paul Moore

1:19 p.m.

New subject: [Python-Dev] PEP 365 (Adding the pkg_resources module)

On 17/03/2008, Phillip J. Eby wrote:

...

That leaves MAL, whose objections to PEP 365 centered on the API (he said he was "+1 on the concepts being added to the stdlib, -1 on adding the module in its current state"). Among other concerns, he wanted pkg_resources to be split into pkgutil and a new "egglib" module. I don't have a problem with this in principle, if there were a pkg_resources module that reconstituted the merged API. (But there are some practical problems with that approach, such as trying to split namespace package support between two theoretically-unrelated modules.)

I would personally like to see such a separation. As one of the authors of PEP 302 (well, the documentation - Just did all of the implementation) I have an interest in strengthening the standard library's support for transparent use of PEP 302 importers. To that end, I would like to see such functions as pkg_resources.resource_string() standardised. That covers the pkgutil aspect of pkg_resources. The "egglib" side of things is where the controversy lies, IMHO. I have a (somewhat unfocussed) dislike of eggs, largely because of the lack of a package management tool to handle them. I can live with them or without them, and it doesn't bother me if others use them, but the thing that really, really bothers me is that the controversy (and yes, there is such) over eggs is hampering the adoption of the pkgutil side of pkg_resources. So from me, there's a strong +1 for the split of pkg_resources into additions to the existing pkgutil module, and an independent egglib. You say there are practical problems to doing this. OK, but could we not have a discussion on how to resolve those issues as they come up? Could the split not be initially into 3 parts - pkgutil (eg, resource_string), egglib, and "difficult"? Then there would be something concrete to discuss and resolve. Paul.

Guido van Rossum

10:31 p.m.

New subject: [Python-Dev] PEP 365 (Adding the pkg_resources module)

After reading all this, I really don't believe that adding egg support to the stdlib at this time is the right thing to do. I am therefore rejecting the PEP. I am hoping that someone will create a simpler bootstrap module that is able to download a file of pure Python code and install it, perhaps by running its setup.py, assuming that it only depends on distutils (or other things previously installed). I will welcome such a module into the stdlib. I'm not sure a PEP is even needed, though interested parties are certainly welcome to write a PEP specifying the behavior first. With 2.6 and 3.0 slated for release in September, there should be enough time to get this done before then. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Stephen J. Turnbull

18 Mar 18 Mar

1:45 p.m.

New subject: [Python-Dev] PEP 365 (Adding the pkg_resources module)

Guido van Rossum writes:

...

I am hoping that someone will create a simpler bootstrap module

FWIW (I've never tried to implement one of these things) I agree with Phillip. This is not possible in the sense you are advocating. Anything "simpler" is simply an invitation to an unending stream of issues, far more so than adopting eggs as a best current practice would. Better to Just Say No to an installer in the stdlib.

...

I'm not sure a PEP is even needed

This I simply don't understand. I *have* participated in bolting on features to such systems, and it's a mess. As features are added, existing users will demand that old packages install exactly as they ever did, except that sometimes (but not always!) they want them upgraded to put things in newly blessed places. Features are easy to add, since on the main thread of control installers are basically just a sequence of single commands, sometimes optional. python-dev has some pretty good controls that will help slow such trends. Nonetheless, PEP-less development of an installer system is scary. Installer complexity is a creeping horror, worse than kudzu.[1] Footnotes: [1] http://en.wikipedia.org/wiki/Kudzu#Invasive_species

Guido van Rossum

1:43 p.m.

New subject: [Python-Dev] PEP 365 (Adding the pkg_resources module)

There seems to be a misunderstanding about what I am proposing we do instead. The boostrap installer should only be powerful enough to allow it to be used to install a real package manager like setuptools. Maybe my use of Django as an example was confusing; I didn't actually mean that it should be possible to support installing Django directly (although I don't understand all the issure that would make it impossible). Only very few people would care about writing a setup script that works with this bootstrap module; basically only package manager implementers. --Guido On Tue, Mar 18, 2008 at 3:45 PM, Stephen J. Turnbull wrote:

...

Guido van Rossum writes:

...
I am hoping that someone will create a simpler bootstrap module

FWIW (I've never tried to implement one of these things) I agree with Phillip. This is not possible in the sense you are advocating. Anything "simpler" is simply an invitation to an unending stream of issues, far more so than adopting eggs as a best current practice would. Better to Just Say No to an installer in the stdlib.

...
I'm not sure a PEP is even needed

This I simply don't understand. I *have* participated in bolting on features to such systems, and it's a mess. As features are added, existing users will demand that old packages install exactly as they ever did, except that sometimes (but not always!) they want them upgraded to put things in newly blessed places. Features are easy to add, since on the main thread of control installers are basically just a sequence of single commands, sometimes optional.

python-dev has some pretty good controls that will help slow such trends. Nonetheless, PEP-less development of an installer system is scary. Installer complexity is a creeping horror, worse than kudzu.[1]

Footnotes: [1] http://en.wikipedia.org/wiki/Kudzu#Invasive_species

-- --Guido van Rossum (home page: http://www.python.org/~guido/)

Marius Gedminas

11:51 a.m.

New subject: [Python-Dev] PEP 365 (Adding the pkg_resources module)

On Mon, Mar 17, 2008 at 08:19:11PM +0000, Paul Moore wrote:

...

On 17/03/2008, Phillip J. Eby wrote:

...
That leaves MAL, whose objections to PEP 365 centered on the API (he said he was "+1 on the concepts being added to the stdlib, -1 on adding the module in its current state"). Among other concerns, he wanted pkg_resources to be split into pkgutil and a new "egglib" module. I don't have a problem with this in principle, if there were a pkg_resources module that reconstituted the merged API. (But there are some practical problems with that approach, such as trying to split namespace package support between two theoretically-unrelated modules.)

I would personally like to see such a separation. As one of the authors of PEP 302 (well, the documentation - Just did all of the implementation) I have an interest in strengthening the standard library's support for transparent use of PEP 302 importers. To that end, I would like to see such functions as pkg_resources.resource_string() standardised.

+1 for something like this in the stdlib. os.path.join(os.path.dirname(__file__), 'foo') just has too many problems. Could namespace package support be split into a set of hooks in the stdlib and a concrete implementation in setuptools? Isn't that the way zip importers were originally done? Marius Gedminas -- As an aside, UPnP's implementation (which features SOAP, HTTP over multicast/broadcast UDP, and extremely odd XML) is a must-read for fans of unnatural and baroque network protocols. -- Anthony Baxter

Paul Moore

2:01 p.m.

New subject: [Python-Dev] PEP 365 (Adding the pkg_resources module)

On 18/03/2008, Marius Gedminas wrote:

...

+1 for something like this in the stdlib.

os.path.join(os.path.dirname(__file__), 'foo') just has too many problems.

Hmm, maybe I should put some of my money where my mouth is. I'd be willing to work up a patch to add some key items into the pkgutils module. However, I don't like the thought of reinventing the wheel - Phillip, would you be OK with me stealing code where I can from pkg_resources? Paul.

Jim Fulton

19 Mar 19 Mar

12:37 p.m.

New subject: [Python-Dev] PEP 365 (Adding the pkg_resources module)

On Mar 18, 2008, at 2:51 PM, Marius Gedminas wrote: ...

...

os.path.join(os.path.dirname(__file__), 'foo') just has too many problems.

Could you be specific? It has worked pretty well for us. It makes it hard to provide software in zip files, but that seems overrated. In particular, I recently noticed that it took significantly longer to start an app from zipped eggs than from unzipped eggs. And, of course, zipped eggs are a lot less convenient for looking at code. Jim -- Jim Fulton Zope Corporation

Marius Gedminas

12:52 p.m.

New subject: [Python-Dev] PEP 365 (Adding the pkg_resources module)

On Wed, Mar 19, 2008 at 03:37:59PM -0400, Jim Fulton wrote:

...

On Mar 18, 2008, at 2:51 PM, Marius Gedminas wrote: ...

...
os.path.join(os.path.dirname(__file__), 'foo') just has too many problems.

Could you be specific?

I suppose I was thinking about the stale .pyc file problem, mostly. zc.buildout/setuptools/distutils (I'm not sure which part of the stack is to blame) likes to compile modules in a temporary directory and then move the .py and .pyc files to their final location. As a result, __file__ points to the now-nonexistent directory. Aside: I never understood the reason for storing the value of __file__ in the .pyc/.pyo file itself. Why not set it to the correct value on import time? Marius Gedminas -- If you have to run heating in winter, you don't own enough computers.

Jim Fulton

12:58 p.m.

New subject: [Python-Dev] PEP 365 (Adding the pkg_resources module)

On Mar 19, 2008, at 3:52 PM, Marius Gedminas wrote:

...

On Wed, Mar 19, 2008 at 03:37:59PM -0400, Jim Fulton wrote:

...
On Mar 18, 2008, at 2:51 PM, Marius Gedminas wrote: ...

...
os.path.join(os.path.dirname(__file__), 'foo') just has too many problems.

Could you be specific?

I suppose I was thinking about the stale .pyc file problem, mostly.

zc.buildout/setuptools/distutils (I'm not sure which part of the stack is to blame) likes to compile modules in a temporary directory and then move the .py and .pyc files to their final location. As a result, __file__ points to the now-nonexistent directory.

You are mistaken. The file information in the code object is wrong, which is wildly annoying when looking at traceback or using pdb, but __file__ is correct. BTW, I'm pretty sure the bad code file information is my fault. I definitely plan to fix this.

...

Aside: I never understood the reason for storing the value of __file__ in the .pyc/.pyo file itself. Why not set it to the correct value on import time?

Agreed. But this has no impact on resource loading. Jim -- Jim Fulton Zope Corporation

Marius Gedminas

2:39 p.m.

New subject: [Python-Dev] PEP 365 (Adding the pkg_resources module)

On Wed, Mar 19, 2008 at 03:58:58PM -0400, Jim Fulton wrote:

...

On Mar 19, 2008, at 3:52 PM, Marius Gedminas wrote:

...
On Wed, Mar 19, 2008 at 03:37:59PM -0400, Jim Fulton wrote:

...
On Mar 18, 2008, at 2:51 PM, Marius Gedminas wrote: ...

...
os.path.join(os.path.dirname(__file__), 'foo') just has too many problems.

Could you be specific?

I suppose I was thinking about the stale .pyc file problem, mostly.

zc.buildout/setuptools/distutils (I'm not sure which part of the stack is to blame) likes to compile modules in a temporary directory and then move the .py and .pyc files to their final location. As a result, __file__ points to the now-nonexistent directory.

You are mistaken. The file information in the code object is wrong, which is wildly annoying when looking at traceback or using pdb, but __file__ is correct.

Oops, you're right. My brain stopped working (which often happens lately, just about when I decide to write an email or comment in a bug tracker. Late evenings and lack of sleep may have something to do with it.). I retract my statement. The only problem with os.path.join(os.path.dirname(__file__), 'foo') is that it's long and cumbersome to type (and doesn't work in zip files, but I don't much care about those). Marius Gedminas -- If it weren't for the last minute, nothing would get done.

5879

Age (days ago)

5881

Last active (days ago)

List overview

Download

11 comments

6 participants

participants (6)

Guido van Rossum
Jim Fulton
Marius Gedminas
Paul Moore
Phillip J. Eby
Stephen J. Turnbull

Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)

Marius Gedminas

Marius Gedminas

Marius Gedminas

tags

participants (6)