[Python-Dev] Draft PEP: "Simplified Package Layout and Partitioning"

[P.J. Eby]

At 03:22 PM 7/20/2011 -0600, Eric Snow wrote:
>On Wed, Jul 20, 2011 at 2:44 PM, P.J. Eby <pje at telecommunity.com> wrote:
> >
> > So, yeah, actually, that's looking pretty sweet.  Basically, we 
> just have to
> > throw a virtual_package_paths dict into the sys module, and do the above
> > along with the get_virtual_path() function and add get_subpath() to the
> > importer objects, in order to get the PEP's core functionality working.
>
>
>Exactly.  That's part of why the importlib approach is so appealing to
>me.

Actually, it turns out I was a little too optimistic -- the sketch I 
gave doesn't work right for anything but top-level virtual packages, 
because I didn't take into account the part where get_virtual_path() 
needs a parent path.

Fixing *that* error then leads to a really nasty bit of mutual 
recursion in which the parent module imports are attempted over and 
over again in something like O(N**2), I think.  In order to get rid 
of that, _gcd_import would have to grow some internal memoization so 
it doesn't retry the same imports repeatedly.

Ironically enough, this is because _gcd_import() is recursive, and 
thus attempts the imports in the opposite order (sort of) than 
import.c does, which means that you can't get hold of the parent's 
__path__ without recursing (again).  :-(

And trying to work around that with memoization, led me to the 
realization that you actually can't implement PEP 402 using that type 
of recursion.  That is, to implement the spec correctly, _gcd_import 
is going to have to be refactored to iterate left-to-right over 
module name parts, rather than recursing right-to-left.

That's because PEP 402 only allows for processing a virtual path if a 
module is not found, *not* if a module is found but can't be loaded.

But, with importlib currently being recursive, it only knows that a 
parent import failed via ImportError, not whether that error arose 
from failing to find the module, or failing to load the module!

So, the core part of the _gcd_import() function will need to be 
rewritten to iterate instead of recursing.

(Still, it's probably not going to be *terribly* difficult.  I'll 
take a look at doing a sketch of that next, but if I do one I'll send 
it to Import-SIG instead of here; it's not a detail that matters to 
the general PEP discussion.)