[Python-Dev] PEP 382: Namespace Packages
pje at telecommunity.com
Thu Apr 16 17:36:18 CEST 2009
At 03:46 AM 4/16/2009 +0000, glyph at divmod.com wrote:
>On 15 Apr, 09:11 pm, pje at telecommunity.com wrote:
>>I think that there is some confusion here. A "main" package or
>>buildout that assembles a larger project from components is not the
>>same thing as having a "base" package for a namespace package.
>I'm certainly confused.
>Twisted has its own system for "namespace" packages, and I'm not
>really sure where we fall in this discussion. I haven't been able
>to follow the whole thread, but my original understanding was that
>the PEP supports "defining packages", which we now seem to be
>calling "base packages", just fine.
Yes, it does. The discussion since the original proposal, however,
has been dominated by MAL's counterproposal, which *requires* a
There is a slight distinction between "base package" and "defining
package", although I suppose I've been using them a bit interchangeably.
Base package describes a use case: you have a base package which is
extended in the same namespace. In that use case, you may want to
place your base package in the defining package.
In contrast, setuptools does not support a defining package, so if
you have a base package, you must place it in a submodule or
subpackage of the namespace.
Does that all make sense now?
MAL's proposal requires a defining package, which is
counterproductive if you have a pure package with no base, since it
now requires you to create an additional project on PyPI just to hold
your defining package.
>I'd appreciate it if the PEP could also be extended cover Twisted's
>very similar mechanism for namespace packages,
>"twisted.plugin.pluginPackagePaths". I know this is not quite as
>widely used as setuptools' namespace package support, but its
>existence belies a need for standardization.
>The PEP also seems a bit vague with regard to the treatment of other
>directories containing __init__.py and *.pkg files.
Do you have a clarification to suggest? My understanding (probably a
projection) is that to be a nested namespace package, you have to
have a parent namespace package.
> The concept of a "defining package" seems important to avoid
> conflicts like this one:
>More specifically I don't quite understand the PEP's intentions
>towards hierarchical packages. It says that all of sys.path will be
>searched, but what about this case?
>In Twisted, the suggested idiom to structure a project which wants
>to provide Twisted plugins is to have a directory structure like this:
>If you then put MyProject on PYTHONPATH, MyProject/twisted/plugins
>will be picked up automatically by the plugin machinery.
Namespaces are not plugins and vice versa. The purpose of a
namespace package is to allow projects managed by the same entity to
share a namespace (ala Java "package" names) and avoid naming
conflicts with other authors.
A plugin system, by contrast, is explicitly intended for use by
multiple authors, so the use case is rather different... and using
namespace packages for plugins actually *increases* the possibility
of naming conflicts, unless you add back in another level of
hierarchy. (As apparently you are recommending via "myproject_plugin".)
> However, as "twisted" is *not* a "namespace" package in the same
> way, .py files in MyProject/twisted/ would not be picked up - this
> is very much intentional, since the "twisted" namespace is intended
> to be reserved for packages that we actually produce. If either
> MyProject/twisted or MyProject/twisted/plugins/ had an __init__.py,
> then no modules in MyProject/twisted/plugins/ would be picked up,
> because it would be considered a conflicting package.
Precisely. Note, however, that neither is twisted.plugins a
namespace package, and it should not contain any .pkg files. I don't
think it's reasonable to abuse PEP 382 namespace packages as a plugin
system. In setuptools' case, a different mechanism is provided for
locating plugin code, and of course Twisted already has its own
system for the same thing. It would be nice to have a standardized
way of locating plugins in the stdlib, but that will need to be a
>I hope this all makes sense. As I understand it, both setuptools
>and the proposed standard would either still have the bug described
>by ticket 2339 above, or would ignore twisted/plugins/ as a
>namespace package because its parent isn't a namespace package.
If twisted/ lacked an __init__.py, then setuptools would ignore
it. Under PEP 382, the same, unless it had .pkg files. (Again,
setuptools explicitly does not support using namespace packages as a
>P.S.: vendor packaging systems *ARE* a major use case for just about
>any aspect of Python's package structure. I really liked MvL's
>coverage of "vendor packages", in the PEP, since this could equally
>well apply to MSIs, python libraries distributed in bundles on OS X,
>debs, or RPMs. If this use-case were to be ignored, as one
>particular fellow seems to be advocating, then the broken packages
>and user confusion that has been going on for the last 5 years or so
>is just going to get worse.
More information about the Python-Dev