[Python-Dev] Implementing PEP 382, Namespace Packages

[P.J. Eby]

At 06:18 PM 5/30/2010 -0700, Brett Cannon wrote:
>On Sun, May 30, 2010 at 00:40, P.J. Eby <pje at telecommunity.com> wrote:
> >
> > Which would completely break one of the major use cases of the 
> PEP, which is
> > precisely to ensure that you can install two pieces of code to the same
> > namespace without either one overwriting the other's files.
>
>The PEP says the goal is to span packages across directories.

The goal of namespace packages is to allow separately-distributed 
pieces of code to live in the same package namespace.  That this is 
sometimes achieved by installing them to different paths is an 
implementation detail.

In the case of e.g. Linux distributions and other system packaging 
scenarios, the code will all be installed to the *same* directory -- 
so there cannot be any filename collisions among the 
separately-distributed modules.  That's why we want to get rid of the 
need for an __init__.py to mark the directory as a package: it's a 
collision point for system package management tools.


> > pkgutil doesn't have such a limitation, except in the case extend_path, and
> > that limitation is one that PEP 382 intends to remove.
>
>It's because pkgutil.extend_path has that limitation I am asking as
>that's what the PEP refers to. If the PEP wants to remove the
>limitation it should clearly state how it is going to do that.

I'm flexible on it either way.  The only other importer I know of 
that does anything else is one that actually allows (unsafely) 
importing from URLs.

If we allow for other things, then we need to extend the PEP 302 
protocol to have a way to ask an importer for a subpath string.



>As for adding to the PEP 302 protocols, it's a question of how much we
>want importer implementors to have control over this versus us. I
>personally would rather keep any protocol extensions simple and have
>import handle as many of the details as possible.

I lean the other way a bit, in that the more of the importer 
internals you expose, the harder you make it for an importer to be 
anything other than a mere virtual file system.  (As it is, I think 
there is too much "file-ness" coupling in the protocol already, what 
with file extensions and the like.)

Indeed, now that I'm thinking about it, it actually seems to make 
more sense to just require the importers to implement PEP 382, and 
provide some common machinery in imp or pkgutil for reading .pth 
strings, setting up __path__, and hunting down all the other directories.