[Python-Dev] PEP 382: Namespace Packages
glyph at divmod.com
glyph at divmod.com
Fri Apr 17 05:58:22 CEST 2009
On 16 Apr, 03:36 pm, pje at telecommunity.com wrote:
>At 03:46 AM 4/16/2009 +0000, glyph at divmod.com wrote:
>>On 15 Apr, 09:11 pm, pje at telecommunity.com wrote:
>>Twisted has its own system for "namespace" packages, and I'm not
>>really sure where we fall in this discussion. I haven't been able to
>>follow the whole thread, but my original understanding was that the
>>PEP supports "defining packages", which we now seem to be calling
>>"base packages", just fine.
>Yes, it does. The discussion since the original proposal, however, has
>been dominated by MAL's counterproposal, which *requires* a defining
>Does that all make sense now?
Yes. Thank you very much for the detailed explanation. It was more
than I was due :-).
>MAL's proposal requires a defining package, which is counterproductive
>if you have a pure package with no base, since it now requires you to
>create an additional project on PyPI just to hold your defining
Just as a use-case: would the Java "com.*" namespace be an example of a
"pure package with no base"? i.e. lots of projects are in it, but no
project owns it?
>>I'd appreciate it if the PEP could also be extended cover Twisted's
>>very similar mechanism for namespace packages,
>>"twisted.plugin.pluginPackagePaths". I know this is not quite as
>>widely used as setuptools' namespace package support, but its
>>existence belies a need for standardization.
>>The PEP also seems a bit vague with regard to the treatment of other
>>directories containing __init__.py and *.pkg files.
>Do you have a clarification to suggest? My understanding (probably a
>projection) is that to be a nested namespace package, you have to have
>a parent namespace package.
Just to clarify things on my end: "namespace package" to *me* means
"package with modules provided from multiple distributions (the
distutils term)". The definition provided by the PEP, that a package is
spread over multiple directories on disk, seems like an implementation
Entries on __path__ slow down import, so my understanding of the
platonic ideal of a system python installation is one which has a single
directory where all packages reside, and a set of metadata off to the
side explaining which files belong to which distributions so they can be
uninstalled by a package manager.
Of course, for a development installation, easy uninstallation and quick
swapping between different versions of relevant dependencies is more
important than good import performance. So in that case, you would want
to optimize differently by having all of your distributions installed
into separate directories, with a long PYTHONPATH or lots of .pth files
to point at them.
And of course you may want a hybrid of the two.
So another clarification I'd like in the PEP is an explanation of
motivation. For example, it comes as a complete surprise to me that the
expectation of namespace packages was to provide only single-source
namespaces like zope.*, peak.*, twisted.*. As I mentioned above, I
implicitly thought this was more for com.*, twisted.plugins.*.
Right now it just says that it's a package which resides in multiple
directories, and it's not made clear why that's a desirable feature.
>> The concept of a "defining package" seems important to avoid
>>conflicts like this one:
[snip some stuff about plugins and package layout]
>Namespaces are not plugins and vice versa. The purpose of a namespace
>package is to allow projects managed by the same entity to share a
>namespace (ala Java "package" names) and avoid naming conflicts with
I think this is missing a key word: *separate* projects managed by the
Hmm. I thought I could illustrate that the same problem actually occurs
without using a plugin system, but I can actually only come up with an
example if an application implements multi-library-version compatibility
from bad_old_name import bad_old_feature as feature
from good_new_name import good_new_feature as feature
rather than the other way around; and that's a terrible idea for other
reasons. Other than that, you'd have to use
pkg_resources.resource_listdir or somesuch, at which point you pretty
much are implementing a plugin system.
So I started this reply disagreeing but I think I've convinced myself
that you're right.
>Precisely. Note, however, that neither is twisted.plugins a namespace
>package, and it should not contain any .pkg files. I don't think it's
>reasonable to abuse PEP 382 namespace packages as a plugin system. In
>setuptools' case, a different mechanism is provided for locating plugin
>code, and of course Twisted already has its own system for the same
>thing. It would be nice to have a standardized way of locating plugins
>in the stdlib, but that will need to be a different PEP.
Okay. So what I'm hearing is that Twisted should happily continue using
our own wacky __path__-calculation logic for twisted.plugins, but that
*twisted* should be a namespace package so that our separate
distributions (TwistedCore, TwistedWeb, TwistedConch, et. al.) can be
installed into separate directories.
More information about the Python-Dev