[Import-SIG] New PEP draft: "Simplified Package Layout and Partitioning"

[P.J. Eby]

At 10:15 PM 8/15/2011 +0200, Greg Slodkowicz wrote:
>Dear all,
>I've been working on a proof-of-concept implementation of PEP 402 as
>part of my import-related Google Summer of Code project (the
>repository is at https://bitbucket.org/jergosh/pep-402). I made the
>changes to my ImportEngine-based import but the logic should be the
>same as the original __import__ implemented in importlib. I haven't
>studied the CPython implementation but I'm assuming it's similar.

It's not.  The C implementation is iterative rather than recursive, 
so it's actually more amenable to the approach described in the PEP.


>I've ran into the following problem: prior to PEP-402, if we import
>Foo.Bar.Baz, first Foo and then Foo.Bar are imported by recursion
>finally, if these succeed, also Foo.Bar.Baz. Then __import__ returns
>the root module, in this case Foo. But suppose Foo and Bar are virtual
>packages, i. e. there is a file Foo/Bar/Baz.py somewhere on the path,
>lacking any __init__.py files. As far as I understand the proposal,
>then there would be no Foo to return.

That's correct.  It's understood (or at least I understood) that this 
meant importlib could not continue using a recursive approach.  I do 
have a rough (i.e. utterly untested and probably bug-ridden) draft 
rewrite of _gcd_import() and associated functions, if you'd like to 
take a look at it:

     http://pastebin.com/6e29v8LR


It even includes a VirtualPath proxy for automatic recalculation of 
virtual modules' __path__ attributes when sys.path (or any parent 
__path__ object) is changed.


>One way to get around this would be to create an empty module on each
>import level (Foo and Foo.Bar in this case) and set appropriate
>__path__ on each of them. To my mind, this would require changing the
>way get_subpath() works.

If I understand your proposal correctly, this introduces the problem 
of having to get rid of these dummy modules if the import fails -- 
and with no guarantee that you won't wind up with dangling references 
somewhere.