[Python-Dev] PEP 382: Namespace Packages

[glyph at divmod.com]

On 16 Apr, 03:36 pm, pje at telecommunity.com wrote:
>At 03:46 AM 4/16/2009 +0000, glyph at divmod.com wrote:
>>On 15 Apr, 09:11 pm, pje at telecommunity.com wrote:

>>Twisted has its own system for "namespace" packages, and I'm not 
>>really sure where we fall in this discussion.  I haven't been able to 
>>follow the whole thread, but my original understanding was that the 
>>PEP supports "defining packages", which we now seem to be calling 
>>"base packages", just fine.
>
>Yes, it does.  The discussion since the original proposal, however, has 
>been dominated by MAL's counterproposal, which *requires* a defining 
>package.

[snip clarifications]
>Does that all make sense now?

Yes.  Thank you very much for the detailed explanation.  It was more 
than I was due :-).
>MAL's proposal requires a defining package, which is counterproductive 
>if you have a pure package with no base, since it now requires you to 
>create an additional project on PyPI just to hold your defining 
>package.

Just as a use-case: would the Java "com.*" namespace be an example of a 
"pure package with no base"?  i.e. lots of projects are in it, but no 
project owns it?
>>I'd appreciate it if the PEP could also be extended cover Twisted's 
>>very similar mechanism for namespace packages,

>>"twisted.plugin.pluginPackagePaths".  I know this is not quite as 
>>widely used as setuptools' namespace package support, but its 
>>existence belies a need for standardization.
>>
>>The PEP also seems a bit vague with regard to the treatment of other 
>>directories containing __init__.py and *.pkg files.
>
>Do you have a clarification to suggest?  My understanding (probably a 
>projection) is that to be a nested namespace package, you have to have 
>a parent namespace package.

Just to clarify things on my end: "namespace package" to *me* means 
"package with modules provided from multiple distributions (the 
distutils term)".  The definition provided by the PEP, that a package is 
spread over multiple directories on disk, seems like an implementation 
detail.

Entries on __path__ slow down import, so my understanding of the 
platonic ideal of a system python installation is one which has a single 
directory where all packages reside, and a set of metadata off to the 
side explaining which files belong to which distributions so they can be 
uninstalled by a package manager.

Of course, for a development installation, easy uninstallation and quick 
swapping between different versions of relevant dependencies is more 
important than good import performance.  So in that case, you would want 
to optimize differently by having all of your distributions installed 
into separate directories, with a long PYTHONPATH or lots of .pth files 
to point at them.

And of course you may want a hybrid of the two.

So another clarification I'd like in the PEP is an explanation of 
motivation.  For example, it comes as a complete surprise to me that the 
expectation of namespace packages was to provide only single-source 
namespaces like zope.*, peak.*, twisted.*.  As I mentioned above, I 
implicitly thought this was more for com.*, twisted.plugins.*.

Right now it just says that it's a package which resides in multiple 
directories, and it's not made clear why that's a desirable feature.
>>   The concept of a "defining package" seems important to avoid 
>>conflicts like this one:
>>
>>    http://twistedmatrix.com/trac/ticket/2339

[snip some stuff about plugins and package layout]
>Namespaces are not plugins and vice versa.  The purpose of a namespace 
>package is to allow projects managed by the same entity to share a 
>namespace (ala Java "package" names) and avoid naming conflicts with 
>other authors.

I think this is missing a key word: *separate* projects managed by the 
same entity.

Hmm.  I thought I could illustrate that the same problem actually occurs 
without using a plugin system, but I can actually only come up with an 
example if an application implements multi-library-version compatibility 
by doing

    try:
        from bad_old_name import bad_old_feature as feature
    except ImportError:
        from good_new_name import good_new_feature as feature

rather than the other way around; and that's a terrible idea for other 
reasons.  Other than that, you'd have to use 
pkg_resources.resource_listdir or somesuch, at which point you pretty 
much are implementing a plugin system.

So I started this reply disagreeing but I think I've convinced myself 
that you're right.
>Precisely.  Note, however, that neither is twisted.plugins a namespace 
>package, and it should not contain any .pkg files.  I don't think it's 
>reasonable to abuse PEP 382 namespace packages as a plugin system.  In 
>setuptools' case, a different mechanism is provided for locating plugin 
>code, and of course Twisted already has its own system for the same 
>thing. It would be nice to have a standardized way of locating plugins 
>in the stdlib, but that will need to be a different PEP.

Okay.  So what I'm hearing is that Twisted should happily continue using 
our own wacky __path__-calculation logic for twisted.plugins, but that 
*twisted* should be a namespace package so that our separate 
distributions (TwistedCore, TwistedWeb, TwistedConch, et. al.) can be 
installed into separate directories.