[Python-Dev] PEP 382: Namespace Packages

[glyph at divmod.com]

On 15 Apr, 09:11 pm, pje at telecommunity.com wrote:
>I think that there is some confusion here.  A "main" package or 
>buildout that assembles a larger project from components is not the 
>same thing as having a "base" package for a namespace package.

I'm certainly confused.

Twisted has its own system for "namespace" packages, and I'm not really 
sure where we fall in this discussion.  I haven't been able to follow 
the whole thread, but my original understanding was that the PEP 
supports "defining packages", which we now seem to be calling "base 
packages", just fine.  I don't understand the controversy over the 
counterproposal, since it seems roughly functionally equivalent to me.

I'd appreciate it if the PEP could also be extended cover Twisted's very 
similar mechanism for namespace packages, 
"twisted.plugin.pluginPackagePaths".  I know this is not quite as widely 
used as setuptools' namespace package support, but its existence belies 
a need for standardization.

The PEP also seems a bit vague with regard to the treatment of other 
directories containing __init__.py and *.pkg files.  The concept of a 
"defining package" seems important to avoid conflicts like this one:

    http://twistedmatrix.com/trac/ticket/2339

More specifically I don't quite understand the PEP's intentions towards 
hierarchical packages.  It says that all of sys.path will be searched, 
but what about this case?

In Twisted, the suggested idiom to structure a project which wants to 
provide Twisted plugins is to have a directory structure like this:

  MyProject/
    myproject/
      __init__.py
    twisted/
      plugins/
        myproject_plugin.py

If you then put MyProject on PYTHONPATH, MyProject/twisted/plugins will 
be picked up automatically by the plugin machinery.  However, as 
"twisted" is *not* a "namespace" package in the same way, .py files in 
MyProject/twisted/ would not be picked up - this is very much 
intentional, since the "twisted" namespace is intended to be reserved 
for packages that we actually produce.  If either MyProject/twisted or 
MyProject/twisted/plugins/ had an __init__.py, then no modules in 
MyProject/twisted/plugins/ would be picked up, because it would be 
considered a conflicting package.

This is important so that users can choose not to load the system- 
installed Twisted's plugins when they have both a system-installed 
version of Twisted and a non-installed development version of Twisted 
found first on their PYTHONPATH, and switch between them to indicate 
which version they want to be the "base" or "defining" package for the 
twisted/plugins/ namespace.

Developers might also want to have a system-installed Twisted, but a 
non-installed development version of MyProject on PYTHONPATH.

I hope this all makes sense.  As I understand it, both setuptools and 
the proposed standard would either still have the bug described by 
ticket 2339 above, or would ignore twisted/plugins/ as a namespace 
package because its parent isn't a namespace package.  I apologize for 
not testing with current setuptools before asking, but I'm not sure my 
experiments would be valid given that my environment is set up with 
assumptions from Twisted's system.

P.S.: vendor packaging systems *ARE* a major use case for just about any 
aspect of Python's package structure.  I really liked MvL's coverage of 
"vendor packages", in the PEP, since this could equally well apply to 
MSIs, python libraries distributed in bundles on OS X, debs, or RPMs. 
If this use-case were to be ignored, as one particular fellow seems to 
be advocating, then the broken packages and user confusion that has been 
going on for the last 5 years or so is just going to get worse.