[Import-SIG] PEP 420 issue: extend_path

[Nick Coghlan]

On Tue, May 8, 2012 at 10:27 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Tue, 8 May 2012 08:56:26 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> Given that sys.path already holds strings, as do __path__ attributes, I
>> don't see any value in adding a separate PathEntry type just for signalling
>> purposes.
>
> How should I know, if load_module() returns a string, that it's supposed
> to denote a possible namespace package? sys.path and __path__ don't
> have anything to do with that.

Because it would be documented that any string that find_module()
returns is an entry for the namespace package's __path__. It's an
extension of the existing meaning of strings in the module search
algorithm, rather than anything radically new.

However, I think you're right that suddenly returning a new type
(*any* new type) from an interface that was previously documented as
solely returning either a loader or None is too large a breach of
backwards compatibility to be acceptable.

So, here's my proposal: we instead build the PEP 420 namespace package
construction algorithm *into a loader*.

What my scheme would involve is this:

- in find_module, when the *first* namespace portion is found, a new
PackageLoader is created and is initialised with a copy of the
trailing portions of the path being searched, as well as the
"packages" suffix list from the FileFinder instance. We *do not* check
for an __init__.py at this point - we only check for the existence of
the directory (this directory existence check already exists in
FileFinder [1]).

- PackageLoader.load_module() would then be responsible for scanning
the relevant sections of the path for additional portions. If it finds
an __init__.py in *any* portion, then it immediately stops the scan
and returns an ordinary self-contained package by creating a new
loader of the appropriate type and using *that* to load the package.
Otherwise it creates a namespace package directly.

The rest of importlib should then remain largely untouched - all that
should be necessary is the definition of PackageLoader and the update
to FileFinder to return it when appropriate.

Since PackageLoader would need its own variant of FileFinder in this
case, I suggest refactoring a bit so that there are two classes:
FileFinder and PortionFinder (with the latter being a subclass of the
former).

No magic return values, no backwards compatibility problem.

Regards,
Nick.

[1] http://hg.python.org/cpython/file/default/Lib/importlib/_bootstrap.py#l833

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia