[Python-Dev] Implementing PEP 382, Namespace Packages

[Brett Cannon]

On Sun, May 30, 2010 at 22:03, P.J. Eby <pje at telecommunity.com> wrote:
> At 06:18 PM 5/30/2010 -0700, Brett Cannon wrote:
>>
>> On Sun, May 30, 2010 at 00:40, P.J. Eby <pje at telecommunity.com> wrote:
>> >
>> > Which would completely break one of the major use cases of the PEP,
>> > which is
>> > precisely to ensure that you can install two pieces of code to the same
>> > namespace without either one overwriting the other's files.
>>
>> The PEP says the goal is to span packages across directories.
>
> The goal of namespace packages is to allow separately-distributed pieces of
> code to live in the same package namespace.  That this is sometimes achieved
> by installing them to different paths is an implementation detail.
>
> In the case of e.g. Linux distributions and other system packaging
> scenarios, the code will all be installed to the *same* directory -- so
> there cannot be any filename collisions among the separately-distributed
> modules.  That's why we want to get rid of the need for an __init__.py to
> mark the directory as a package: it's a collision point for system package
> management tools.
>
>
>> > pkgutil doesn't have such a limitation, except in the case extend_path,
>> > and
>> > that limitation is one that PEP 382 intends to remove.
>>
>> It's because pkgutil.extend_path has that limitation I am asking as
>> that's what the PEP refers to. If the PEP wants to remove the
>> limitation it should clearly state how it is going to do that.
>
> I'm flexible on it either way.  The only other importer I know of that does
> anything else is one that actually allows (unsafely) importing from URLs.
>
> If we allow for other things, then we need to extend the PEP 302 protocol to
> have a way to ask an importer for a subpath string.
>
>
>
>> As for adding to the PEP 302 protocols, it's a question of how much we
>> want importer implementors to have control over this versus us. I
>> personally would rather keep any protocol extensions simple and have
>> import handle as many of the details as possible.
>
> I lean the other way a bit, in that the more of the importer internals you
> expose, the harder you make it for an importer to be anything other than a
> mere virtual file system.  (As it is, I think there is too much "file-ness"
> coupling in the protocol already, what with file extensions and the like.)
>

Yes, there is a VERY strong file path connection in loaders and they
have essentially become simple virtual file systems. But that does
make implementation of alternative back-ends easier which seems to be
the common case. Plus pre-existing code already mutates __path__,
__file__, etc. as if they are file paths using os.path.join. And imp's
new source_from_cache and cache_from_source are not weakening the tie
to file paths either.

But as long as whatever mechanism gets exposed allows people to work
from a module name that will be enough. The path connection is not
required as load_module is the end-all-be-all method. If we have a
similar API added for .pth files that works off of module names then
those loaders that don't want to work from file paths don't have to.