[Python-Dev] Draft PEP: "Simplified Package Layout and Partitioning"

[Eric Snow]

On Wed, Jul 20, 2011 at 2:44 PM, P.J. Eby <pje at telecommunity.com> wrote:
> At 01:35 PM 7/20/2011 -0600, Eric Snow wrote:
>>
>> This is a really nice solution.  So a virtual package is not imported
>> until a submodule of the virtual package is successfully imported
>
> Correct...
>
>> (except for direct import of pure virtual packages).
>
> Not correct.  ;-)  What we do is avoid creating a parent module or altering
> its __path__ until a submodule/subpackage import is just about to be
> successfully completed.

Good point, though I was talking about direct imports of pure virtual
packages (which you've indicated are disallowed by the current draft).

>> Also, it makes sense that the above applies to all virtual packages,
>> not just pure ones.
>
> Well, if the package isn't "pure" then what you've imported is really just
> an ordinary module, not a package at all.  ;-)

I meant that if the submodule import fails in the "impure" case, the
existing module does not end up with a __path__.

>> When a pure virtual package is directly imported, a new [empty] module
>> is created and its __path__ is set to the matching value in
>> sys.virtual_packages.  However, an "impure" virtual package is not
>> created upon direct import, and its __path__ is not updated until a
>> submodule import is attempted.  Even the sys.virtual_packages entry is
>> not generated until the submodule attempt, since the virtual package
>> mechanism doesn't kick in until the point that an ImportError is
>> currently raised.
>>
>> This isn't that big a deal, but it would be the one behavioral
>> difference between the two kinds of virtual packages.  So either leave
>> that one difference, disallow direct import of pure virtual packages,
>> or attempt to make virtual packages for all non-package imports.  That
>> last one would impose the virtual package overhead on many more
>> imports so it is probably too impractical.  I'm fine with leaving the
>> one difference.
>
> At this point, I've updated the PEP to disallow direct imports of pure
> virtual packages.  AFAICT it's the only approach that ensures you can't get
> false positive imports by having unrelated-but-similarly-named directories
> floating around.

I see what you mean.  That case is probably more important than the
case of having a package that fails to import but submodules of the
package that succeed.

>> FYI, last night I started on an importlib-based implementation for the
>> PEP and the above solution would be really easy to incorporate.
>
> Well, you might want to double-check that now that I've updated the spec.
>  ;-)  In the new approach, you cannot rely on parent modules existing before
> proceeding to the submodule import.
>
> However, I've just glanced at the importlib trunk, and I think I see what
> you mean.  It's already using a recursive approach, rather than an iterative
> one, so the change should be a lot simpler there than in import.c.
>
<snip>
>
> So, yeah, actually, that's looking pretty sweet.  Basically, we just have to
> throw a virtual_package_paths dict into the sys module, and do the above
> along with the get_virtual_path() function and add get_subpath() to the
> importer objects, in order to get the PEP's core functionality working.


Exactly.  That's part of why the importlib approach is so appealing to
me.  Brett really did a nice job.

-eric