[Distutils] namespace packages (and PEP 382)

[P.J. Eby]

At 02:45 PM 4/24/2010 +0900, David Cournapeau wrote:
>Is there a
>design document or something else decribing how the namespace
>mechanism works for setuptools ? I would like to support namespace
>package in my own packaging project, but it is not clear to me what
>needs to be done on my side of things.

Namespace packages in *Python* are the result of adding entries to a 
package's '__path__' attribute.  By default, the list only has one 
entry - if you make it have one for each directory (or zipfile 
subdirectory) that contains modules for that package, then presto, 
you have a namespace package.

How you get those entries in __path__ is the only real difference 
between pkgutil, pkg_resources, and the upcoming PEP 382.  (Well, 
there's also the nspkg.pth hack, which I'd just as soon do away with 
altogether.)

I'm thinking that in 0.7 I may add a PEP 382 emulation module 
(separate from pkg_resources) for older versions of Python, as it 
would allow some odd bits to be dispensed with in pkg_resources.

For example, declare_namespace could simply add a '*' to the __path__ 
of the right package(s), and the nspkg.pth files could just make sure 
the PEP 382 emulation was loaded, without needing to set up fake 
package modules.  Other pkg_resources APIs that rely on an internal 
list of namespace packages could just look in sys.namespace_packages, 
and so on.

One thing that's not entirely clear at the moment in PEP 382, though, 
is that it doesn't really specify how the other sys.path entries are 
found and mapped -- presumably it's something like 
os.path.join(syspathentry, *pkgname.split('.')) -- but the handling 
for such path entries isn't really described.  In a Python version 
where NullImporter exists (2.5+), though, the cost of extra 
nonexistent __path__ entries is negligible, so I suppose filtering 
the entries for directory existence and the presence of other .pth 
files isn't necessary.

The PEP should probably be clearer on this point, though - i.e., both 
on what exactly goes in __path__, as well as whether .pth files are 
read in the other paths, and whether .pth files are needed on any 
other paths.  (I also wonder if other .pth contents besides '*' 
really need to be supported anyway; I expect it would depend on 
whether anybody is using the .pkg files allowed by extend_path()...)