Re: [Python-Dev] Implementing PEP 382, Namespace Packages

On Sun, May 30, 2010 at 22:03, P.J. Eby wrote:

At 06:18 PM 5/30/2010 -0700, Brett Cannon wrote:

...

On Sun, May 30, 2010 at 00:40, P.J. Eby wrote:

...

Which would completely break one of the major use cases of the PEP, which is precisely to ensure that you can install two pieces of code to the same namespace without either one overwriting the other's files.

The PEP says the goal is to span packages across directories.

The goal of namespace packages is to allow separately-distributed pieces of code to live in the same package namespace.  That this is sometimes achieved by installing them to different paths is an implementation detail.

In the case of e.g. Linux distributions and other system packaging scenarios, the code will all be installed to the *same* directory -- so there cannot be any filename collisions among the separately-distributed modules.  That's why we want to get rid of the need for an __init__.py to mark the directory as a package: it's a collision point for system package management tools.

...

...

pkgutil doesn't have such a limitation, except in the case extend_path, and that limitation is one that PEP 382 intends to remove.

It's because pkgutil.extend_path has that limitation I am asking as that's what the PEP refers to. If the PEP wants to remove the limitation it should clearly state how it is going to do that.

I'm flexible on it either way.  The only other importer I know of that does anything else is one that actually allows (unsafely) importing from URLs.

If we allow for other things, then we need to extend the PEP 302 protocol to have a way to ask an importer for a subpath string.

...

As for adding to the PEP 302 protocols, it's a question of how much we want importer implementors to have control over this versus us. I personally would rather keep any protocol extensions simple and have import handle as many of the details as possible.

I lean the other way a bit, in that the more of the importer internals you expose, the harder you make it for an importer to be anything other than a mere virtual file system.  (As it is, I think there is too much "file-ness" coupling in the protocol already, what with file extensions and the like.)

Yes, there is a VERY strong file path connection in loaders and they have essentially become simple virtual file systems. But that does make implementation of alternative back-ends easier which seems to be the common case. Plus pre-existing code already mutates __path__, __file__, etc. as if they are file paths using os.path.join. And imp's new source_from_cache and cache_from_source are not weakening the tie to file paths either. But as long as whatever mechanism gets exposed allows people to work from a module name that will be enough. The path connection is not required as load_module is the end-all-be-all method. If we have a similar API added for .pth files that works off of module names then those loaders that don't want to work from file paths don't have to.