[Import-SIG] PEP 420 outstanding issues

[PJ Eby]

On Fri, May 11, 2012 at 2:20 PM, Eric V. Smith <eric at trueblade.com> wrote:

> The only thing I'm aware of is that I need to look at Martin's issue of
> how do we gradually migrate to PEP 420 namespace packages from the
> existing pkgutil and pkg_resources versions of namespace packages. I'll
> do that this weekend.
>

FWIW, setuptools and distribute (and maybe pip) already do the right thing
when installing in "single version" mode, which is usually what the distros
use.  They would simply need to stop also creating a .pth file (or dummy
__init__.py files) when running under 3.3.  They'd also need to be updated
to support new-style namespace packages in their source trees.

The implementation of declare_namespace() for 3.3+ would then just be to
create a module object (if not present) and set its __path__ to a virtual
path object based on the parent.  (Sadly, this can't be backported since
older Pythons require an actual list object...  hmm...  or do they?   Maybe
a list *subclass* would work....  must check into that.  If it works back
to 2.3 I could backport...  darn, it uses PyList_Size() and
PyList_Getitem(), so I'd also need a meta_path hook to trigger updates.
Hm.  Got to think about that some more.)

The only corner case I can think of is mixing __init__ portions with
non-__init__ portions.  If you have both, it's not sufficient to create a
simple PEP 420 virtual path, because it won't include the __init__
portions.  A backport or "transition support" version needs a way to force
__init__ portions to be included in the resulting virtual path.  This can't
be done with the current find_loader() protocol, because the finder doesn't
distinguish between package and non-package cases.

If find_loader() always returned a path for a package (even non-namespace
packages), then this would allow virtual paths to be made either inclusive
or exclusive of __init__ segments.  That is, it would let there be a
transition period where you could explicitly declare a namespace to get a
mixed namespace, but by default the paths would be exclusive.

I'm not sure if anything I just said is clear without an example, so I'll
throw one in.  Let's say somebody's writing code that spans multiple Python
versions, and they want their __init__-based namespace packages to work,
but be forward compatible with new subpackages using PEP 420 portions.
Basically, they write some code that calls declare_namespace(), which then
sets the module's __path__ to be an "inclusive virtual" path.  This path
object is similar to the current virtual path object, except that it
*always* uses the second find_loader() return value, even if the first
value returned is not None.  Poof!  Instant "transitional" namespace
package, backward-compatible with older Python versions, and
forward-compatible with PEP 420.

Okay, technically that was more of a rationale than an example, but I hope
it's a bit clearer anyway.  ;-)  For purposes of the PEP, all I'd request
changing is asking that find_loader() always return the path of an existing
directory in the second return value, even if it's also returning a
loader.  More precisely, if it returns a loader for a package, it should
also return the package directory in the second argument.  importlib can
still ignore this second argument, but a transitional version of
declare_namespace() can use it to implement "mixed mode" namespace
packages.  (Which facilitates backporting the mechanism to older setuptools
as well - I'll change the nspkg.pth files to do something like 'import
pep420; pep420.declare_namespace("foo")' and my pep420 module will include
its own mixed-mode virtual path support, and emulate find_loader() for the
builtin importers in older Pythons.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/import-sig/attachments/20120511/f82cef51/attachment.html>