[Python-Dev] PEP 402: Simplified Package Layout and Partitioning

[P.J. Eby]

At 01:09 PM 8/12/2011 -0400, Glyph Lefkowitz wrote:
>Upon further reflection, PEP 402 _will_ make dealing with namespace 
>packages from this code considerably easier: we won't need to do AST 
>analysis to look for a __path__ attribute or anything gross like 
>that improve correctness; we can just look in various directories on 
>sys.path and accurately predict what __path__ will be synthesized to be.

The flip side of that is that you can't always know whether a 
directory is a virtual package without deep inspection: one 
consequence of PEP 402 is that any directory that contains a Python 
module (of whatever type), however deeply nested, will be a valid 
package name.  So, you can't rule out that a given directory *might* 
be a package, without walking its entire reachable subtree.  (Within 
the subset of directory names that are valid Python identifiers, of course.)

However, you *can* quickly tell that a directory *might* be a package 
or is *probably* one: if it contains modules, or is the same name as 
an already-discovered module, it's a pretty safe bet that you can 
flag it as such.

In any case, you probably should *not* do the building of a virtual 
path yourself; the protocols and APIs added by PEP 402 should allow 
you to simply ask for the path to be constructed on your 
behalf.  Otherwise, you are going to be back in the same business of 
second-guessing arbitrary importer backends again!

(E.g. note that PEP 402 does not say virtual package subpaths must be 
filesystem or zipfile subdirectories of their parents - an importer 
could just as easily allow you to treat subdirectories named 
'twisted.python' as part of a virtual package with that name!)

Anyway, pkgutil defines some extra methods that importers can 
implement to support module-walking, and part of the PEP 402 
implementation should be to make this support virtual packages as well.


>This code still needs to support Python 2.4, but I will make a note 
>of this for future reference.

A suggestion: just take the pkgutil code and bundle it for Python 2.4 
as something._pkgutil.  There's very little about it that's 2.5+ 
specific, at least when I wrote the bits that do the module walking.

Of course, the main disadvantage of pkgutil for your purposes is that 
it currently requires packages to be imported in order to walk their 
child modules.  (IIRC, it does *not*, however, require them to be 
imported in order to discover their existence.)


>In that case, I guess it's a good thing; these bugs should be dealt 
>with.  Thanks for pointing them out.  My opinion of PEP 402 has been 
>completely reversed - although I'd still like to see a section about 
>the module system from a library/tools author point of view rather 
>than a time-traveling perl user's narrative :).

LOL.

If you will propose the wording you'd like to see, I'll be happy to 
check it for any current-and-or-future incorrect assumptions.  ;-)