[Python-Dev] PEP 382: Namespace Packages

[P.J. Eby]

At 03:46 AM 4/16/2009 +0000, glyph at divmod.com wrote:

>On 15 Apr, 09:11 pm, pje at telecommunity.com wrote:
>>I think that there is some confusion here.  A "main" package or 
>>buildout that assembles a larger project from components is not the 
>>same thing as having a "base" package for a namespace package.
>
>I'm certainly confused.
>
>Twisted has its own system for "namespace" packages, and I'm not 
>really sure where we fall in this discussion.  I haven't been able 
>to follow the whole thread, but my original understanding was that 
>the PEP supports "defining packages", which we now seem to be 
>calling "base packages", just fine.

Yes, it does.  The discussion since the original proposal, however, 
has been dominated by MAL's counterproposal, which *requires* a 
defining package.

There is a slight distinction between "base package" and "defining 
package", although I suppose I've been using them a bit interchangeably.

Base package describes a use case: you have a base package which is 
extended in the same namespace.  In that use case, you may want to 
place your base package in the defining package.

In contrast, setuptools does not support a defining package, so if 
you have a base package, you must place it in a submodule or 
subpackage of the namespace.

Does that all make sense now?

MAL's proposal requires a defining package, which is 
counterproductive if you have a pure package with no base, since it 
now requires you to create an additional project on PyPI just to hold 
your defining package.


>I'd appreciate it if the PEP could also be extended cover Twisted's 
>very similar mechanism for namespace packages, 
>"twisted.plugin.pluginPackagePaths".  I know this is not quite as 
>widely used as setuptools' namespace package support, but its 
>existence belies a need for standardization.
>
>The PEP also seems a bit vague with regard to the treatment of other 
>directories containing __init__.py and *.pkg files.

Do you have a clarification to suggest?  My understanding (probably a 
projection) is that to be a nested namespace package, you have to 
have a parent namespace package.


>   The concept of a "defining package" seems important to avoid 
> conflicts like this one:
>
>    http://twistedmatrix.com/trac/ticket/2339
>
>More specifically I don't quite understand the PEP's intentions 
>towards hierarchical packages.  It says that all of sys.path will be 
>searched, but what about this case?
>
>In Twisted, the suggested idiom to structure a project which wants 
>to provide Twisted plugins is to have a directory structure like this:
>
>  MyProject/
>    myproject/
>      __init__.py
>    twisted/
>      plugins/
>        myproject_plugin.py
>
>If you then put MyProject on PYTHONPATH, MyProject/twisted/plugins 
>will be picked up automatically by the plugin machinery.

Namespaces are not plugins and vice versa.  The purpose of a 
namespace package is to allow projects managed by the same entity to 
share a namespace (ala Java "package" names) and avoid naming 
conflicts with other authors.

A plugin system, by contrast, is explicitly intended for use by 
multiple authors, so the use case is rather different...  and using 
namespace packages for plugins actually *increases* the possibility 
of naming conflicts, unless you add back in another level of 
hierarchy.  (As apparently you are recommending via "myproject_plugin".)


>   However, as "twisted" is *not* a "namespace" package in the same 
> way, .py files in MyProject/twisted/ would not be picked up - this 
> is very much intentional, since the "twisted" namespace is intended 
> to be reserved for packages that we actually produce.  If either 
> MyProject/twisted or MyProject/twisted/plugins/ had an __init__.py, 
> then no modules in MyProject/twisted/plugins/ would be picked up, 
> because it would be considered a conflicting package.

Precisely.  Note, however, that neither is twisted.plugins a 
namespace package, and it should not contain any .pkg files.  I don't 
think it's reasonable to abuse PEP 382 namespace packages as a plugin 
system.  In setuptools' case, a different mechanism is provided for 
locating plugin code, and of course Twisted already has its own 
system for the same thing.  It would be nice to have a standardized 
way of locating plugins in the stdlib, but that will need to be a 
different PEP.


>I hope this all makes sense.  As I understand it, both setuptools 
>and the proposed standard would either still have the bug described 
>by ticket 2339 above, or would ignore twisted/plugins/ as a 
>namespace package because its parent isn't a namespace package.

If twisted/ lacked an __init__.py, then setuptools would ignore 
it.  Under PEP 382, the same, unless it had .pkg files.  (Again, 
setuptools explicitly does not support using namespace packages as a 
plugin mechanism.)


>P.S.: vendor packaging systems *ARE* a major use case for just about 
>any aspect of Python's package structure.  I really liked MvL's 
>coverage of "vendor packages", in the PEP, since this could equally 
>well apply to MSIs, python libraries distributed in bundles on OS X, 
>debs, or RPMs. If this use-case were to be ignored, as one 
>particular fellow seems to be advocating, then the broken packages 
>and user confusion that has been going on for the last 5 years or so 
>is just going to get worse.

Indeed.