[Import-SIG] PEP 420 issue: extend_path

[Eric V. Smith]

On 5/7/2012 9:50 PM, Nick Coghlan wrote:
> However, I think you're right that suddenly returning a new type
> (*any* new type) from an interface that was previously documented as
> solely returning either a loader or None is too large a breach of
> backwards compatibility to be acceptable.

I'm now of this belief, too.

> So, here's my proposal: we instead build the PEP 420 namespace package
> construction algorithm *into a loader*.
> 
> What my scheme would involve is this:
> 
> - in find_module, when the *first* namespace portion is found, a new
> PackageLoader is created and is initialised with a copy of the
> trailing portions of the path being searched, as well as the
> "packages" suffix list from the FileFinder instance. We *do not* check
> for an __init__.py at this point - we only check for the existence of
> the directory (this directory existence check already exists in
> FileFinder [1]).
> 
> - PackageLoader.load_module() would then be responsible for scanning
> the relevant sections of the path for additional portions. If it finds
> an __init__.py in *any* portion, then it immediately stops the scan
> and returns an ordinary self-contained package by creating a new
> loader of the appropriate type and using *that* to load the package.
> Otherwise it creates a namespace package directly.

I don't think you can do this, at least without losing the ability to
create a namespace package where portions exist in different path_hook
loaders. Currently (in the pep-420 branch) you can have a portion in the
filesystem (FilePath loader), and a portion in a zip file (zipimport
loader). See the SeparatedNestedZipNamespacePackages test in
test_namespace_pkgs.py.

I believe what you're suggesting requires the logic be moved from
PathFinder (which is a meta path hook) in to FileFinder (which is a path
hook). That's why it would break the cross-finder use case.

Note that the meta path hook PathFinder doesn't know anything about
directories or filesystems. That's why it currently (in the pep-420
branch) delegates everything to the path hook finders.

I think the better solution is to create a new finder method, called
something like find_module_or_namespace_portion (but obviously with a
better name). If this exists, then it would be called and allowed to
return a loader, string, or None. If it doesn't exist, find_module would
be called. It could not participate in namespace packages and could only
return a loader or None.

I think the use case of being able to have namespace package portions
returned by different path hooks is important. Imagine a case where the
namespace "encodings" takes off. Who's to same some portions don't ship
as zip files, some as regular files, and maybe some with a hypothetical
http loader?

Eric.