At 01:43 PM 5/21/2007 +0200, M.-A. Lemburg wrote:

On 2007-05-21 00:07, Talin wrote:

...

Phillip J. Eby wrote:

...

I wanted to get this in before the Py3K PEP deadline, since this is a Python 2.6 PEP that would presumably impact 3.x as well. Feedback welcome.

PEP: 365 Title: Adding the pkg_resources module

I'm really surprised that there hasn't been more comment on this.

True.... both ways, I guess: I'm still waiting for a reply to my comments.

What comments are you talking about? I must've missed them.

I'd also like to see more discussion about adding e.g.:

* support for user packages

(ie. having site.py add a well-defined user home directory based Python path entry to sys.path, e.g. ~/.python/user-packages, much like what MacPython already does now)

* support for having the import mechanism play nice with namespace packages

(ie. packages that may live in different places on the disk, but appear to be in the same Python package as seen by the import mechanism)

I think those two features would go a long way in reducing the number of hacks setuptools currently applies to get this functionality working with code in .pth files, monkey-patching site.py, etc.

These items aren't directly related to the PEP, however. pkg_resources doesn't monkeypatch anything or touch any .pth files. It only changes sys.path at runtime if you explicitly ask it to locate and activate packages for you. As for namespace packages, pkg_resources provides a more PEP 302-compatible alternative to pkgutil.extend_path(). pkgutil doesn't support anything but existing filesystem directories, but the pkg_resources version supports zipfiles and has hooks to allow namespace package support to be registered for any PEP 302 importer. See: http://peak.telecommunity.com/DevCenter/PkgResources#supporting-custom-impor... (specifically, the register_namespace_handler() function.)

At 06:28 PM 5/21/2007 +0200, M.-A. Lemburg wrote:

However, since this is not egg-specific it should probably be moved to pkgutil and get a separate PEP with detailed documentation (the link you provided doesn't really explain the concepts, reading the code helped a bit).

That doesn't really make sense in the context of the current PEP, though, which isn't to provide a general-purpose namespace package API; it's specifically about adding an existing piece of code to the stdlib, with its API intact.

What I don't understand about your approach is why importers would have to register with the namespace implementation.

This doesn't seem necessary, since the package __path__ attribute already provides all functionality needed for redirecting lookups to different paths.

The registration is so that when new paths are *added* to sys.path at runtime (e.g. by activating a plugin), the necessary __path__ lists are automatically updated. Similarly, when new namespace packages are declared, they need their __path__ updated for everything that's currently on sys.path. Finally, there is no requirement that PEP 302 importer strings use filesystem-path syntax; a handler has to be registered so that the necessary string transformation can be done according to that particular importer's string format. It just happens that zipfiles and regular files have a common syntax. But a URL-based importer, LDAP-DN based importer, SQL importer, or other exotica might require entirely different string transformations. All that PEP 302 guarantees is that sys.path and __path__ lists contain strings, not what the format of those strings is.

Could you add a section that explains the side effects of importing pkg_resources ?

The short answer is, there are no side effects, unless __main__.__requires__ exists. If that variable exists, pkg_resources attempts to adjust sys.path to contain the requested package versions, even if it requires re-ordering the current sys.path contents to prevent conflicting versions from being imported.

The documentation of the module doesn't mention any, but the code suggests that you are installing (some form of) import hooks.

There are no import hooks, just a registry for registering handlers to support other importer types (as seen at the doc link I gave previously).

Some other comments:

* Wouldn't it be better to factor out all the meta-data access code that's not related to eggs into pkgutil ?!

* How about then renaming the remaining module to egglib ?!

These changes in particular would negate the primary purpose of the PEP: to provide a migration path for existing packages using the pkg_resources API, including setuptools, workingenv, zc.buildout, etc.

* The get_*_platform() should probably use the platform module which is a lot more flexible than distutils' get_platform() (which should probably use the platform module as well in the long run)

Please feel free to propose specific improvements to the distutils-sig. But keep in mind that the platform information is specifically for supporting .egg filename platform tags. Where the information comes from is less relevant than defining a framework for determining compatibility. I first tried to get some kind of traction on this issue in 2004: http://mail.python.org/pipermail/distutils-sig/2004-December/004355.html And so far, the only platform for which something better has emerged is Mac OS X, due largely to Bob Ippolito stepping up and submitting some code.

* The module needs some reorganization: imports, globals and constants at the top, maybe a few comments delimiting the various sections,

I'm not sure I follow you. Globals such as registries are usually defined close to the functions that provide the API for manipulating them. And the rest of the globals (such as working_set), can't be defined until the class they're implemented with has been defined.

At 08:56 PM 5/21/2007 +0200, M.-A. Lemburg wrote:

On 2007-05-21 20:01, Phillip J. Eby wrote:

...

At 06:28 PM 5/21/2007 +0200, M.-A. Lemburg wrote:

...

However, since this is not egg-specific it should probably be moved to pkgutil and get a separate PEP with detailed documentation (the link you provided doesn't really explain the concepts, reading the code helped a bit).

That doesn't really make sense in the context of the current PEP, though, which isn't to provide a general-purpose namespace package API; it's specifically about adding an existing piece of code to the stdlib, with its API intact.

You seem to indicate that you're not up to discussing the concepts implemented by the module and *integrating* them with the Python stdlib.

No, I'm saying something else. I'm saying it: 1. has nothing to do with the PEP, 2. isn't something I'm volunteering to do, and 3. would only make sense to do as part of Python 3 stdlib reorganization, if it were done at all. Now, the code is certainly under an open license, and the concepts are entirely free for anyone to use. If somebody wishes to do what you're describing, they're certainly welcome to take on that thankless task. But I personally don't see the point, since by definition that new API would have *no current users*. And the purpose of the PEP is to serve the (rather large) audience that would like to take advantage of existing software that uses the API. Thus, any proposal to alter that API faces a high entry barrier to show how the proposed changes would provide a signficant practical benefit to users. That's not even remotely similar to "take it or leave it". It might *seem* that way, of course, simply because in any proposal to change the API, there's an implicit question of why nobody proposed the change via the Distutils-SIG, sometime during the last 2+ years of discussions around that API. I remain open-minded and curious as to the possibility that someone *could* propose a meaningful change, but am also rationally skeptical that someone actually *will* come up with something that would outweigh the user benefit of keeping the already published, already discussed, already field-tested, already in-use API. For that matter, I remain open-minded and curious as to the possibility of whether someone could propose a reasonable justification for *not* including the module in the stdlib. After all, last year Fredrik Lundh surprised me with a convincing rationale for *not* including setuptools in the stdlib, which is why I backed off on doing so in 2.5, and am now proffering a much-reduced-in-scope proposal for 2.6. So, I'm perfectly willing and able to change my mind, given convincing reasons to do so. So far, though, your change suggestions haven't even explained why *you* want them, let alone why anybody else should agree. We can hardly discuss what you haven't yet said.

At 11:44 PM 5/21/2007 +0200, M.-A. Lemburg wrote:

On 2007-05-21 22:48, Phillip J. Eby wrote:

...

At 08:56 PM 5/21/2007 +0200, M.-A. Lemburg wrote:

...

On 2007-05-21 20:01, Phillip J. Eby wrote:

...

At 06:28 PM 5/21/2007 +0200, M.-A. Lemburg wrote:

...

However, since this is not egg-specific it should probably be moved to pkgutil and get a separate PEP with detailed documentation (the link you provided doesn't really explain the concepts, reading the code helped a bit).

That doesn't really make sense in the context of the current PEP, though, which isn't to provide a general-purpose namespace package API; it's specifically about adding an existing piece of code to the stdlib, with its API intact.

You seem to indicate that you're not up to discussing the concepts implemented by the module and *integrating* them with the Python stdlib.

No, I'm saying something else. I'm saying it:

1. has nothing to do with the PEP, 2. isn't something I'm volunteering to do, and 3. would only make sense to do as part of Python 3 stdlib reorganization, if it were done at all.

I don't understand that last part: how can adding a new module or set of modules require waiting for reorganization of the stdlib ?

I'm saying that an API reorganization that split up the pkg_resources API might be appropriate at that time, in much the same way that say, merging "dl" and "ctypes" (or dropping "dl") might take place during such a reorganization. (And would thus go along with the 2to3 conversion or whatever other mechanism will be used for porting Python 2 code to Python 3 code.)

All I'm suggesting is to reorganize the code in pkg_resources.py a bit and move the relevant bits into pkgutil.py and into a new eggutil.py. ... You can easily provide a pkg_resource.py module with your old API that interfaces to the new reorganized code in the stdlib.

...and are you proposing that this other module *also* be included in the stdlib? If so, that was not clear from your previous messages.

...

That's not even remotely similar to "take it or leave it". It might *seem* that way, of course, simply because in any proposal to change the API, there's an implicit question of why nobody proposed the change via the Distutils-SIG, sometime during the last 2+ years of discussions around that API.

This doesn't have anything to do with distutils. It's entirely about the egg distribution format.

Huh? I'm saying that the pkg_resources API has been being discussed on the Distutils-SIG for a good 2 years now. If there was something that users *wanted* to change in the API, surely it would've been discussed by now?

I'm not sure what you want to hear from me.

A good place to start would be the rationale for your proposed API changes.

You asked for comments, I wrote back and gave you comments.

Yep, and I explained my take on them. You then brought up this whole "take it or leave it" thing, in response to which I attempted to provide further clarification of my reasoning -- none of which involves any "take it or leave it"-ness, from my perspective.

I also made it clear why I think that breaking up the addition into different PEPs makes a lot of sense and why separating the code into different modules for the same reason makes a lot of sense as well.

Actually, no, you didn't. You simply asserted that certain things would be "better" (without any mention of *how* they would be better), and that other things "should probably" be done (without any explanation of *why*). So, I simply responded with information of why I did *not* see these changes to be useful in any self-evident way, and why I'd want to see some justification that would weigh against the PEP's raison d'etre -- supporting the existing user base and making it easier for more people to join that user base.

I also tried to stir up some discussion to make life easier for setuptools by suggesting a user-package directory on sys.path and adding support for namespace packages as general Python feature instead of hiding it away in pkg_resources.py.

The first is a great discussion topic, but unrelated to the PEP. I'll be happy to participate in that discussion, if I have any input to offer. The second, I don't see as "hiding away"; I would actually suggest that pkgutil.extend_path simply be deprecated in favor of the pkg_resources API for namespace packages. I'm not aware of there being a particularly large user base for the pkgutil.extend_path, so I don't see a need to change an API that's used a lot, to match one that's not used as much. AFAIK, extend_path was originally created for Zope Corp., and also AFAIK, they are now using pkg_resources instead. See also this previous distutils-sig discussion (as I said, the API *does* get discussed there): http://mail.python.org/pipermail/distutils-sig/2006-August/006608.html You'll notice that in my response, I also left the door open to supporting .pkg files (an extend_path feature not supported by pkg_resources), if somebody could tell me what they're used for. Nobody has responded to that, so far.

You should see this as chance to introduce new concepts to Python. Instead you seem to feel offended every time someone suggests a change in your design.

I'm not offended. I've simply commented on your comments, and suggested that some of them are off-topic in relation to the PEP, or shown why they would be counter to the expressed purposes of the PEP. As for introducing new concepts to Python, IIRC, Guido and Jim Fulton co-invented namespace packages and pkgutil.extend_path() quite a few years ago, so I'm not really the introducer there, nor is it a particularly new concept. So I'm not sure what new concepts you're talking about.

That's also the reason why I stopped discussing things with you on the distutils list. There was simply no way of getting through to you.

We simply have different goals. In the case of the distutils-sig, my perception is that your proposals were aimed at preserving the investment of a small number of expert distutils users, at the expense of the broader public who knew next to nothing about the distutils. Those proposals "got through to me" just fine; I just didn't (and don't) agree with them as goals for setuptools, in any place where they conflicted with setuptools' primary goals (which included adoption by *new*, distutils-ignorant and/or distutils-disliking users).