[Import-SIG] New draft revision for PEP 382

[Barry Warsaw]

On Jul 08, 2011, at 08:01 PM, P.J. Eby wrote:

>At 06:31 PM 7/8/2011 -0400, Barry Warsaw wrote:
>>Thanks!  I've been trying to catch up on the mailing list traffic today, and
>>grabbed your prototype code.  I plan on committing it to MvL's pep382 hg
>>branch so we have a place to play with it.
>
>You should probably start from this version instead:
>
>   http://pastebin.com/Wv77WYyb
>
>It's got some work on other things like iter_modules, extend_namespaces, etc.

Are you working from a publicly available repo?  If not, would you like to be?
<wink>.  It will make collaboration easier, and MvL's hg branch is already
available and I think entirely appropriate for this code, at least until its
merged back into trunk.

>> >Portion
>> >     A set of files in a single directory (possibly inside a zip file
>> >     or other storage mechanism) that contribute modules or subpackages
>> >     to a namespace package.  The contents of each portion ``sys.path``
>>
>>This one got cut off.
>
>Oops.  A bad edit; ignore that sentence fragment, it was replaced by language
>in the definition that followed it.

Cool.

>>Do you need to explain a little more why __path__ is significant, and why the
>>registration function is required?
>
>Revsed paragraph:
>
>====
>In current Python versions, however, a registration function (such as
>``pkgutil.extend_path()`` or ``pkg_resources.declare_namespace()``)
>must be explicitly invoked in order to set up the package's
>``__path__``.  (By default, a package's ``__path__`` lists only one
>directory, so to allow imports from more than one directory, the
>``__path__`` must be explicitly extended in code.)
>====

I'd only add something like "Python searches a package's __path__ instead of
sys.path when it's looking for subpackages.  Yes, I know this is covered in
PEP 302, but I think it couldn't hurt a little extra text here.  Your call
though.

>> >Vendor packages typically must not provide overlapping files, and an
>> >attempt to install a vendor package that has a file already on disk
>> >will fail or cause unpredictable behavior.  As vendors might choose to
>> >package distributions such that they will end up all in a single
>> >directory for the namespace package, all portions would contribute
>> >conflicting ``__init__.py`` files.
>>
>>I might word this a little differently.  Perhaps:
>>
>>Vendor packaging standards require every file on disk to be owned by exactly
>>one vendor package.  But because each portion of a namespace package may be
>>contained in a separate vendor package, multiple vendor packages would have to
>>own the namespace package's __init__.py file.  For example, would the
>>``zope.interface`` vendor package own ``zope/__init__.py`` or would the
>>``zope.component`` vendor package own it?  Different vendors handle this
>>conflict differently, and in fact, different packaging tools from the same
>>vendor can handle this differently, which can cause consistency problems.
>
>I took the original wording as directly as practical from MvLs, but I agree
>yours is clearer.  OTOH, I think the "fail or cause unpredictable behavior is
>a much stronger motivator than, "it's nonstandard and confusing".  ;-)
>
>Did you have a specific rationale for your choice?  I mean, what did you want
>to gain or avoid by the change?

The confusion I had was on "overlapping files", since that doesn't have a
clear meaning to me.  I'm happy to use the stronger language you prefer; maybe
you can work both texts into something even better!

>The problem with this example is that it gives the impression that .ns files
>are named for packages, instead of being named for distributions.  So, I went
>with a more detailed and explict example.

The example you posted in the other thread looks great.

>> >This new method is called just before the importer's ``find_module()``
>> >is normally invoked.  If the importer determines that `fullname` is
>> >a namespace package portion under its jurisdiction, then the importer
>> >returns an importer-specific path to that namespace portion.
>>
>>Please define exactly what ``fullname`` is.
>
>Ugh.  Do I have to?  ;-)
>
>Will it work if I just change that to "just before the importer's
>``find_module(fullname)`` is normally invoked", so it's more clearly implied?

Sure, that'll work.

>> >Standard Library Changes/Additions
>> >----------------------------------
>> >
>> >The ``pkgutil`` module should be updated to handle this
>> >specification appropriately, including any necessary changes to
>> >``extend_path()``, ``iter_modules()``, etc.  A new generic API for
>> >calling ``namespace_subpath()`` on importers should be added as well.
>>
>>Is there any reason not to put extend_path() on the road to deprecation?
>
>I don't know.  Is there?  As I said, I considered that an open question.

I think we should.

>> >Specifically the proposed changes and additions are:
>> >
>> >* A new ``namespace_subpath(importer, fullname)`` generic, allowing
>> >   implementations to be registered for existing importers.
>>
>>Is this the registration mechanism?
>
>Registration for what?  I meant that this is analogous to other pkgutil
>generic functions that let you call a PEP 302 extension protocol on an
>importer, whether or not the importer directly implements that protocol.  For
>example, pkgutil.iter_importer_modules() is a generic function that lets you
>ask an importer to iterate over available modules, whether it actually
>implements its own "iter_modules()" method or not.  The
>pkgutil.namespace_subpath() function would do the same for the
>(possibly-absent) namespace_subpath() method on existing importers, and allow
>third parties to register namespace support for custom importers that can't
>be directly modified to support namespace packages.
>
>Any thoughts on how better to word that bit, without necessarily going into
>that much detail?  ;-)

I guess part of the problem is that generics like iter_importer_modules()
isn't actually documented in pkgutils, no (in Python 2.7), even included in
the __all__.  So you can't just say something like:

* A new ``namespace_subpath(importer, fullname)`` generic, analogous to other
  existing generics in the pkgutil package.

or maybe you can when you file a bug to get the existing ones
documented. <wink>.

>We won't be opening the files at all, so the contents will be ignored.

Does your rewrite make that explicit?

I'd like to either have a strong recommendation for the file being empty, or
specify the syntax we'd likely support in any extension to PEP 382.  In all
likelihood that would be to ignore lines with only whitespace, or that begin
with a `#`.

>>I'd be a little more forceful; the PEP should strongly recommend against
>>including namespace package __init__.py files.
>
>As I said, it's controversial.  Some people really want those __init__
>modules, and setuptools sort-of supports them now.  I can make it a bit more
>forceful, though.

I think you're rewrite looked good here.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/import-sig/attachments/20110710/5dd292e7/attachment-0001.pgp>