PEP 0365: Adding the pkg_resources module
I wanted to get this in before the Py3K PEP deadline, since this is a Python 2.6 PEP that would presumably impact 3.x as well. Feedback welcome. PEP: 365 Title: Adding the pkg_resources module Version: $Revision: 55032 $ Last-Modified: $Date: 2007-04-30 20:24:48 -0400 (Mon, 30 Apr 2007) $ Author: Phillip J. Eby <pje@telecommunity.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30-Apr-2007 Post-History: 30-Apr-2007 Abstract ======== This PEP proposes adding an enhanced version of the ``pkg_resources`` module to the standard library. ``pkg_resources`` is a module used to find and manage Python package/version dependencies and access bundled files and resources, including those inside of zipped ``.egg`` files. Currently, ``pkg_resources`` is only available through installing the entire ``setuptools`` distribution, but it does not depend on any other part of setuptools; in effect, it comprises the entire runtime support library for Python Eggs, and is independently useful. In addition, with one feature addition, this module could support easy bootstrap installation of several Python package management tools, including ``setuptools``, ``workingenv``, and ``zc.buildout``. Proposal ======== Rather than proposing to include ``setuptools`` in the standard library, this PEP proposes only that ``pkg_resources`` be added to the standard library for Python 2.6 and 3.0. ``pkg_resources`` is considerably more stable than the rest of setuptools, with virtually no new features being added in the last 12 months. However, this PEP also proposes that a new feature be added to ``pkg_resources``, before being added to the stdlib. Specifically, it should be possible to do something like:: python -m pkg_resources SomePackage==1.2 to request downloading and installation of ``SomePackage`` from PyPI. This feature would *not* be a replacement for ``easy_install``; instead, it would rely on ``SomePackage`` having pure-Python ``.egg`` files listed for download via the PyPI XML-RPC API, and the eggs would be placed in the ``$PYTHONEGGS`` cache, where they would **not** be importable by default. (And no scripts would be installed) However, if the download egg contains installation bootstrap code, it will be given a chance to run. These restrictions would allow the code to be extremely simple, yet still powerful enough to support users downloading package management tools such as ``setuptools``, ``workingenv`` and ``zc.buildout``, simply by supplying the tool's name on the command line. Rationale ========= Many users have requested that ``setuptools`` be included in the standard library, to save users needing to go through the awkward process of bootstrapping it. However, most of the bootstrapping complexity comes from the fact that setuptools-installed code cannot use the ``pkg_resources`` runtime module unless setuptools is already installed. Thus, installing setuptools requires (in a sense) that setuptools already be installed. Other Python package management tools, such as ``workingenv`` and ``zc.buildout``, have similar bootstrapping issues, since they both make use of setuptools, but also want to provide users with something approaching a "one-step install". The complexity of creating bootstrap utilities for these and any other such tools that arise in future, is greatly reduced if ``pkg_resources`` is already present, and is also able to download pre-packaged eggs from PyPI. (It would also mean that setuptools would not need to be installed in order to simply *use* eggs, as opposed to building them.) Finally, in addition to providing access to eggs built via setuptools or other packaging tools, it should be noted that since Python 2.5, the distutils install package metadata (aka ``PKG-INFO``) files that can be read by ``pkg_resources`` to identify what distributions are already on ``sys.path``. In environments where Python packages are installed using system package tools (like RPM), the ``pkg_resources`` module provides an API for detecting what versions of what packages are installed, even if those packages were installed via the distutils instead of setuptools. Implementation and Documentation ================================ The ``pkg_resources`` implementation is maintained in the Python SVN repository under ``/sandbox/trunk/setuptools/``; see ``pkg_resources.py`` and ``pkg_resources.txt``. Documentation for the egg format(s) supported by ``pkg_resources`` can be found in ``doc/formats.txt``. HTML versions of these documents are available at: * http://peak.telecommunity.com/DevCenter/PkgResources and * http://peak.telecommunity.com/DevCenter/EggFormats (These HTML versions are for setuptools 0.6; they may not reflect all of the changes found in the Subversion trunk's ``.txt`` versions.) Copyright ========= This document has been placed in the public domain.
On 2007-05-01 02:29, Phillip J. Eby wrote:
I wanted to get this in before the Py3K PEP deadline, since this is a Python 2.6 PEP that would presumably impact 3.x as well. Feedback welcome.
Could you add a section that explains the side effects of importing pkg_resources ? The documentation of the module doesn't mention any, but the code suggests that you are installing (some form of) import hooks. Some other comments: * Wouldn't it be better to factor out all the meta-data access code that's not related to eggs into pkgutil ?! * How about then renaming the remaining module to egglib ?! * The module needs some reorganization: imports, globals and constants at the top, maybe a few comments delimiting the various sections, * The get_*_platform() should probably use the platform module which is a lot more flexible than distutils' get_platform() (which should probably use the platform module as well in the long run) Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 04 2007)
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611
On 2007-05-21 00:07, Talin wrote:
True.... both ways, I guess: I'm still waiting for a reply to my comments. I'd also like to see more discussion about adding e.g.: * support for user packages (ie. having site.py add a well-defined user home directory based Python path entry to sys.path, e.g. ~/.python/user-packages, much like what MacPython already does now) * support for having the import mechanism play nice with namespace packages (ie. packages that may live in different places on the disk, but appear to be in the same Python package as seen by the import mechanism) I think those two features would go a long way in reducing the number of hacks setuptools currently applies to get this functionality working with code in .pth files, monkey-patching site.py, etc. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 21 2007)
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611
At 01:43 PM 5/21/2007 +0200, M.-A. Lemburg wrote:
What comments are you talking about? I must've missed them.
These items aren't directly related to the PEP, however. pkg_resources doesn't monkeypatch anything or touch any .pth files. It only changes sys.path at runtime if you explicitly ask it to locate and activate packages for you. As for namespace packages, pkg_resources provides a more PEP 302-compatible alternative to pkgutil.extend_path(). pkgutil doesn't support anything but existing filesystem directories, but the pkg_resources version supports zipfiles and has hooks to allow namespace package support to be registered for any PEP 302 importer. See: http://peak.telecommunity.com/DevCenter/PkgResources#supporting-custom-impor... (specifically, the register_namespace_handler() function.)
On 2007-05-21 16:05, Phillip J. Eby wrote:
I've attached the email. Please see below.
Right. I wasn't referring to this PEP. I think we should have two more PEPs covering the above points, since they offer benefits for all users, not just setuptools users.
Looking at the code it appears as if you've already formalized an implementation for this. However, since this is not egg-specific it should probably be moved to pkgutil and get a separate PEP with detailed documentation (the link you provided doesn't really explain the concepts, reading the code helped a bit). What I don't understand about your approach is why importers would have to register with the namespace implementation. This doesn't seem necessary, since the package __path__ attribute already provides all functionality needed for redirecting lookups to different paths. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 21 2007)
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611
At 06:28 PM 5/21/2007 +0200, M.-A. Lemburg wrote:
That doesn't really make sense in the context of the current PEP, though, which isn't to provide a general-purpose namespace package API; it's specifically about adding an existing piece of code to the stdlib, with its API intact.
The registration is so that when new paths are *added* to sys.path at runtime (e.g. by activating a plugin), the necessary __path__ lists are automatically updated. Similarly, when new namespace packages are declared, they need their __path__ updated for everything that's currently on sys.path. Finally, there is no requirement that PEP 302 importer strings use filesystem-path syntax; a handler has to be registered so that the necessary string transformation can be done according to that particular importer's string format. It just happens that zipfiles and regular files have a common syntax. But a URL-based importer, LDAP-DN based importer, SQL importer, or other exotica might require entirely different string transformations. All that PEP 302 guarantees is that sys.path and __path__ lists contain strings, not what the format of those strings is.
Could you add a section that explains the side effects of importing pkg_resources ?
The short answer is, there are no side effects, unless __main__.__requires__ exists. If that variable exists, pkg_resources attempts to adjust sys.path to contain the requested package versions, even if it requires re-ordering the current sys.path contents to prevent conflicting versions from being imported.
There are no import hooks, just a registry for registering handlers to support other importer types (as seen at the doc link I gave previously).
These changes in particular would negate the primary purpose of the PEP: to provide a migration path for existing packages using the pkg_resources API, including setuptools, workingenv, zc.buildout, etc.
Please feel free to propose specific improvements to the distutils-sig. But keep in mind that the platform information is specifically for supporting .egg filename platform tags. Where the information comes from is less relevant than defining a framework for determining compatibility. I first tried to get some kind of traction on this issue in 2004: http://mail.python.org/pipermail/distutils-sig/2004-December/004355.html And so far, the only platform for which something better has emerged is Mac OS X, due largely to Bob Ippolito stepping up and submitting some code.
* The module needs some reorganization: imports, globals and constants at the top, maybe a few comments delimiting the various sections,
I'm not sure I follow you. Globals such as registries are usually defined close to the functions that provide the API for manipulating them. And the rest of the globals (such as working_set), can't be defined until the class they're implemented with has been defined.
On 2007-05-21 20:01, Phillip J. Eby wrote:
You seem to indicate that you're not up to discussing the concepts implemented by the module and *integrating* them with the Python stdlib. Please correct me if I'm wrong, but if the whole point of the PEP is a take it or leave it decision, then I don't see the point of discussing it. I'm -1 on adding the module in its current state; I'd be +1 on integrating the concepts with the Python stdlib. Hope I'm wrong, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 21 2007)
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611
At 08:56 PM 5/21/2007 +0200, M.-A. Lemburg wrote:
No, I'm saying something else. I'm saying it: 1. has nothing to do with the PEP, 2. isn't something I'm volunteering to do, and 3. would only make sense to do as part of Python 3 stdlib reorganization, if it were done at all. Now, the code is certainly under an open license, and the concepts are entirely free for anyone to use. If somebody wishes to do what you're describing, they're certainly welcome to take on that thankless task. But I personally don't see the point, since by definition that new API would have *no current users*. And the purpose of the PEP is to serve the (rather large) audience that would like to take advantage of existing software that uses the API. Thus, any proposal to alter that API faces a high entry barrier to show how the proposed changes would provide a signficant practical benefit to users. That's not even remotely similar to "take it or leave it". It might *seem* that way, of course, simply because in any proposal to change the API, there's an implicit question of why nobody proposed the change via the Distutils-SIG, sometime during the last 2+ years of discussions around that API. I remain open-minded and curious as to the possibility that someone *could* propose a meaningful change, but am also rationally skeptical that someone actually *will* come up with something that would outweigh the user benefit of keeping the already published, already discussed, already field-tested, already in-use API. For that matter, I remain open-minded and curious as to the possibility of whether someone could propose a reasonable justification for *not* including the module in the stdlib. After all, last year Fredrik Lundh surprised me with a convincing rationale for *not* including setuptools in the stdlib, which is why I backed off on doing so in 2.5, and am now proffering a much-reduced-in-scope proposal for 2.6. So, I'm perfectly willing and able to change my mind, given convincing reasons to do so. So far, though, your change suggestions haven't even explained why *you* want them, let alone why anybody else should agree. We can hardly discuss what you haven't yet said.
On 2007-05-21 22:48, Phillip J. Eby wrote:
I don't understand that last part: how can adding a new module or set of modules require waiting for reorganization of the stdlib ? All I'm suggesting is to reorganize the code in pkg_resources.py a bit and move the relevant bits into pkgutil.py and into a new eggutil.py.
Why is that ? You can easily provide a pkg_resource.py module with your old API that interfaces to the new reorganized code in the stdlib.
This doesn't have anything to do with distutils. It's entirely about the egg distribution format.
I'm not sure what you want to hear from me. You asked for comments, I wrote back and gave you comments. I also made it clear why I think that breaking up the addition into different PEPs makes a lot of sense and why separating the code into different modules for the same reason makes a lot of sense as well. I also tried to stir up some discussion to make life easier for setuptools by suggesting a user-package directory on sys.path and adding support for namespace packages as general Python feature instead of hiding it away in pkg_resources.py. You should see this as chance to introduce new concepts to Python. Instead you seem to feel offended every time someone suggests a change in your design. That's also the reason why I stopped discussing things with you on the distutils list. There was simply no way of getting through to you. Perhaps we should just meet up for a beer in London sometime and sort things out ;-) Cheers, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 21 2007)
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611
At 11:44 PM 5/21/2007 +0200, M.-A. Lemburg wrote:
I'm saying that an API reorganization that split up the pkg_resources API might be appropriate at that time, in much the same way that say, merging "dl" and "ctypes" (or dropping "dl") might take place during such a reorganization. (And would thus go along with the 2to3 conversion or whatever other mechanism will be used for porting Python 2 code to Python 3 code.)
...and are you proposing that this other module *also* be included in the stdlib? If so, that was not clear from your previous messages.
Huh? I'm saying that the pkg_resources API has been being discussed on the Distutils-SIG for a good 2 years now. If there was something that users *wanted* to change in the API, surely it would've been discussed by now?
I'm not sure what you want to hear from me.
A good place to start would be the rationale for your proposed API changes.
You asked for comments, I wrote back and gave you comments.
Yep, and I explained my take on them. You then brought up this whole "take it or leave it" thing, in response to which I attempted to provide further clarification of my reasoning -- none of which involves any "take it or leave it"-ness, from my perspective.
Actually, no, you didn't. You simply asserted that certain things would be "better" (without any mention of *how* they would be better), and that other things "should probably" be done (without any explanation of *why*). So, I simply responded with information of why I did *not* see these changes to be useful in any self-evident way, and why I'd want to see some justification that would weigh against the PEP's raison d'etre -- supporting the existing user base and making it easier for more people to join that user base.
The first is a great discussion topic, but unrelated to the PEP. I'll be happy to participate in that discussion, if I have any input to offer. The second, I don't see as "hiding away"; I would actually suggest that pkgutil.extend_path simply be deprecated in favor of the pkg_resources API for namespace packages. I'm not aware of there being a particularly large user base for the pkgutil.extend_path, so I don't see a need to change an API that's used a lot, to match one that's not used as much. AFAIK, extend_path was originally created for Zope Corp., and also AFAIK, they are now using pkg_resources instead. See also this previous distutils-sig discussion (as I said, the API *does* get discussed there): http://mail.python.org/pipermail/distutils-sig/2006-August/006608.html You'll notice that in my response, I also left the door open to supporting .pkg files (an extend_path feature not supported by pkg_resources), if somebody could tell me what they're used for. Nobody has responded to that, so far.
I'm not offended. I've simply commented on your comments, and suggested that some of them are off-topic in relation to the PEP, or shown why they would be counter to the expressed purposes of the PEP. As for introducing new concepts to Python, IIRC, Guido and Jim Fulton co-invented namespace packages and pkgutil.extend_path() quite a few years ago, so I'm not really the introducer there, nor is it a particularly new concept. So I'm not sure what new concepts you're talking about.
We simply have different goals. In the case of the distutils-sig, my perception is that your proposals were aimed at preserving the investment of a small number of expert distutils users, at the expense of the broader public who knew next to nothing about the distutils. Those proposals "got through to me" just fine; I just didn't (and don't) agree with them as goals for setuptools, in any place where they conflicted with setuptools' primary goals (which included adoption by *new*, distutils-ignorant and/or distutils-disliking users).
On 2007-05-01 02:29, Phillip J. Eby wrote:
I wanted to get this in before the Py3K PEP deadline, since this is a Python 2.6 PEP that would presumably impact 3.x as well. Feedback welcome.
Could you add a section that explains the side effects of importing pkg_resources ? The documentation of the module doesn't mention any, but the code suggests that you are installing (some form of) import hooks. Some other comments: * Wouldn't it be better to factor out all the meta-data access code that's not related to eggs into pkgutil ?! * How about then renaming the remaining module to egglib ?! * The module needs some reorganization: imports, globals and constants at the top, maybe a few comments delimiting the various sections, * The get_*_platform() should probably use the platform module which is a lot more flexible than distutils' get_platform() (which should probably use the platform module as well in the long run) Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 04 2007)
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611
On 2007-05-21 00:07, Talin wrote:
True.... both ways, I guess: I'm still waiting for a reply to my comments. I'd also like to see more discussion about adding e.g.: * support for user packages (ie. having site.py add a well-defined user home directory based Python path entry to sys.path, e.g. ~/.python/user-packages, much like what MacPython already does now) * support for having the import mechanism play nice with namespace packages (ie. packages that may live in different places on the disk, but appear to be in the same Python package as seen by the import mechanism) I think those two features would go a long way in reducing the number of hacks setuptools currently applies to get this functionality working with code in .pth files, monkey-patching site.py, etc. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 21 2007)
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611
At 01:43 PM 5/21/2007 +0200, M.-A. Lemburg wrote:
What comments are you talking about? I must've missed them.
These items aren't directly related to the PEP, however. pkg_resources doesn't monkeypatch anything or touch any .pth files. It only changes sys.path at runtime if you explicitly ask it to locate and activate packages for you. As for namespace packages, pkg_resources provides a more PEP 302-compatible alternative to pkgutil.extend_path(). pkgutil doesn't support anything but existing filesystem directories, but the pkg_resources version supports zipfiles and has hooks to allow namespace package support to be registered for any PEP 302 importer. See: http://peak.telecommunity.com/DevCenter/PkgResources#supporting-custom-impor... (specifically, the register_namespace_handler() function.)
On 2007-05-21 16:05, Phillip J. Eby wrote:
I've attached the email. Please see below.
Right. I wasn't referring to this PEP. I think we should have two more PEPs covering the above points, since they offer benefits for all users, not just setuptools users.
Looking at the code it appears as if you've already formalized an implementation for this. However, since this is not egg-specific it should probably be moved to pkgutil and get a separate PEP with detailed documentation (the link you provided doesn't really explain the concepts, reading the code helped a bit). What I don't understand about your approach is why importers would have to register with the namespace implementation. This doesn't seem necessary, since the package __path__ attribute already provides all functionality needed for redirecting lookups to different paths. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 21 2007)
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611
At 06:28 PM 5/21/2007 +0200, M.-A. Lemburg wrote:
That doesn't really make sense in the context of the current PEP, though, which isn't to provide a general-purpose namespace package API; it's specifically about adding an existing piece of code to the stdlib, with its API intact.
The registration is so that when new paths are *added* to sys.path at runtime (e.g. by activating a plugin), the necessary __path__ lists are automatically updated. Similarly, when new namespace packages are declared, they need their __path__ updated for everything that's currently on sys.path. Finally, there is no requirement that PEP 302 importer strings use filesystem-path syntax; a handler has to be registered so that the necessary string transformation can be done according to that particular importer's string format. It just happens that zipfiles and regular files have a common syntax. But a URL-based importer, LDAP-DN based importer, SQL importer, or other exotica might require entirely different string transformations. All that PEP 302 guarantees is that sys.path and __path__ lists contain strings, not what the format of those strings is.
Could you add a section that explains the side effects of importing pkg_resources ?
The short answer is, there are no side effects, unless __main__.__requires__ exists. If that variable exists, pkg_resources attempts to adjust sys.path to contain the requested package versions, even if it requires re-ordering the current sys.path contents to prevent conflicting versions from being imported.
There are no import hooks, just a registry for registering handlers to support other importer types (as seen at the doc link I gave previously).
These changes in particular would negate the primary purpose of the PEP: to provide a migration path for existing packages using the pkg_resources API, including setuptools, workingenv, zc.buildout, etc.
Please feel free to propose specific improvements to the distutils-sig. But keep in mind that the platform information is specifically for supporting .egg filename platform tags. Where the information comes from is less relevant than defining a framework for determining compatibility. I first tried to get some kind of traction on this issue in 2004: http://mail.python.org/pipermail/distutils-sig/2004-December/004355.html And so far, the only platform for which something better has emerged is Mac OS X, due largely to Bob Ippolito stepping up and submitting some code.
* The module needs some reorganization: imports, globals and constants at the top, maybe a few comments delimiting the various sections,
I'm not sure I follow you. Globals such as registries are usually defined close to the functions that provide the API for manipulating them. And the rest of the globals (such as working_set), can't be defined until the class they're implemented with has been defined.
On 2007-05-21 20:01, Phillip J. Eby wrote:
You seem to indicate that you're not up to discussing the concepts implemented by the module and *integrating* them with the Python stdlib. Please correct me if I'm wrong, but if the whole point of the PEP is a take it or leave it decision, then I don't see the point of discussing it. I'm -1 on adding the module in its current state; I'd be +1 on integrating the concepts with the Python stdlib. Hope I'm wrong, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 21 2007)
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611
At 08:56 PM 5/21/2007 +0200, M.-A. Lemburg wrote:
No, I'm saying something else. I'm saying it: 1. has nothing to do with the PEP, 2. isn't something I'm volunteering to do, and 3. would only make sense to do as part of Python 3 stdlib reorganization, if it were done at all. Now, the code is certainly under an open license, and the concepts are entirely free for anyone to use. If somebody wishes to do what you're describing, they're certainly welcome to take on that thankless task. But I personally don't see the point, since by definition that new API would have *no current users*. And the purpose of the PEP is to serve the (rather large) audience that would like to take advantage of existing software that uses the API. Thus, any proposal to alter that API faces a high entry barrier to show how the proposed changes would provide a signficant practical benefit to users. That's not even remotely similar to "take it or leave it". It might *seem* that way, of course, simply because in any proposal to change the API, there's an implicit question of why nobody proposed the change via the Distutils-SIG, sometime during the last 2+ years of discussions around that API. I remain open-minded and curious as to the possibility that someone *could* propose a meaningful change, but am also rationally skeptical that someone actually *will* come up with something that would outweigh the user benefit of keeping the already published, already discussed, already field-tested, already in-use API. For that matter, I remain open-minded and curious as to the possibility of whether someone could propose a reasonable justification for *not* including the module in the stdlib. After all, last year Fredrik Lundh surprised me with a convincing rationale for *not* including setuptools in the stdlib, which is why I backed off on doing so in 2.5, and am now proffering a much-reduced-in-scope proposal for 2.6. So, I'm perfectly willing and able to change my mind, given convincing reasons to do so. So far, though, your change suggestions haven't even explained why *you* want them, let alone why anybody else should agree. We can hardly discuss what you haven't yet said.
On 2007-05-21 22:48, Phillip J. Eby wrote:
I don't understand that last part: how can adding a new module or set of modules require waiting for reorganization of the stdlib ? All I'm suggesting is to reorganize the code in pkg_resources.py a bit and move the relevant bits into pkgutil.py and into a new eggutil.py.
Why is that ? You can easily provide a pkg_resource.py module with your old API that interfaces to the new reorganized code in the stdlib.
This doesn't have anything to do with distutils. It's entirely about the egg distribution format.
I'm not sure what you want to hear from me. You asked for comments, I wrote back and gave you comments. I also made it clear why I think that breaking up the addition into different PEPs makes a lot of sense and why separating the code into different modules for the same reason makes a lot of sense as well. I also tried to stir up some discussion to make life easier for setuptools by suggesting a user-package directory on sys.path and adding support for namespace packages as general Python feature instead of hiding it away in pkg_resources.py. You should see this as chance to introduce new concepts to Python. Instead you seem to feel offended every time someone suggests a change in your design. That's also the reason why I stopped discussing things with you on the distutils list. There was simply no way of getting through to you. Perhaps we should just meet up for a beer in London sometime and sort things out ;-) Cheers, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 21 2007)
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611
At 11:44 PM 5/21/2007 +0200, M.-A. Lemburg wrote:
I'm saying that an API reorganization that split up the pkg_resources API might be appropriate at that time, in much the same way that say, merging "dl" and "ctypes" (or dropping "dl") might take place during such a reorganization. (And would thus go along with the 2to3 conversion or whatever other mechanism will be used for porting Python 2 code to Python 3 code.)
...and are you proposing that this other module *also* be included in the stdlib? If so, that was not clear from your previous messages.
Huh? I'm saying that the pkg_resources API has been being discussed on the Distutils-SIG for a good 2 years now. If there was something that users *wanted* to change in the API, surely it would've been discussed by now?
I'm not sure what you want to hear from me.
A good place to start would be the rationale for your proposed API changes.
You asked for comments, I wrote back and gave you comments.
Yep, and I explained my take on them. You then brought up this whole "take it or leave it" thing, in response to which I attempted to provide further clarification of my reasoning -- none of which involves any "take it or leave it"-ness, from my perspective.
Actually, no, you didn't. You simply asserted that certain things would be "better" (without any mention of *how* they would be better), and that other things "should probably" be done (without any explanation of *why*). So, I simply responded with information of why I did *not* see these changes to be useful in any self-evident way, and why I'd want to see some justification that would weigh against the PEP's raison d'etre -- supporting the existing user base and making it easier for more people to join that user base.
The first is a great discussion topic, but unrelated to the PEP. I'll be happy to participate in that discussion, if I have any input to offer. The second, I don't see as "hiding away"; I would actually suggest that pkgutil.extend_path simply be deprecated in favor of the pkg_resources API for namespace packages. I'm not aware of there being a particularly large user base for the pkgutil.extend_path, so I don't see a need to change an API that's used a lot, to match one that's not used as much. AFAIK, extend_path was originally created for Zope Corp., and also AFAIK, they are now using pkg_resources instead. See also this previous distutils-sig discussion (as I said, the API *does* get discussed there): http://mail.python.org/pipermail/distutils-sig/2006-August/006608.html You'll notice that in my response, I also left the door open to supporting .pkg files (an extend_path feature not supported by pkg_resources), if somebody could tell me what they're used for. Nobody has responded to that, so far.
I'm not offended. I've simply commented on your comments, and suggested that some of them are off-topic in relation to the PEP, or shown why they would be counter to the expressed purposes of the PEP. As for introducing new concepts to Python, IIRC, Guido and Jim Fulton co-invented namespace packages and pkgutil.extend_path() quite a few years ago, so I'm not really the introducer there, nor is it a particularly new concept. So I'm not sure what new concepts you're talking about.
We simply have different goals. In the case of the distutils-sig, my perception is that your proposals were aimed at preserving the investment of a small number of expert distutils users, at the expense of the broader public who knew next to nothing about the distutils. Those proposals "got through to me" just fine; I just didn't (and don't) agree with them as goals for setuptools, in any place where they conflicted with setuptools' primary goals (which included adoption by *new*, distutils-ignorant and/or distutils-disliking users).
participants (3)
-
M.-A. Lemburg -
Phillip J. Eby -
Talin