[Python-Dev] PEP 382: Namespace Packages

[P.J. Eby]

At 10:33 PM 4/2/2009 +0200, M.-A. Lemburg wrote:
>That's going to slow down Python package detection a lot - you'd
>replace an O(1) test with an O(n) scan.

I thought about this too, but it's pretty trivial considering that 
the only time it takes effect is when you have a directory name that 
matches the name you're importing, and that it will only happen once 
for that directory, unless there is no package on sys.path with that 
name, and the program tries to import the package multiple times.  In 
other words, the overhead isn't likely to be much, compared to the 
time needed to say, open and marshal even a trivial __init__.py file.


>Alternative Approach:
>---------------------
>
>Wouldn't it be better to stick with a simpler approach and look for
>"__pkg__.py" files to detect namespace packages using that O(1) check ?

I thought the same thing (or more precisely, a single .pkg file), but 
when I got lower in the PEP I saw the reason was to support system 
packages not having overlapping filenames.  The PEP could probably be 
a little clearer about the connection between needing *.pkg and the 
system-package use case.


>One of the namespace packages, the defining namespace package, will have
>to include a __init__.py file.

Note that there is no such thing as a "defining namespace package" -- 
namespace package contents are symmetrical peers.


>The above mechanism allows the same kind of flexibility we already
>have with the existing normal __init__.py mechanism.
>
>* It doesn't add yet another .pth-style sys.path extension (which are
>difficult to manage in installations).
>
>* It always uses the same naive sys.path search strategy. The strategy
>is not determined by some file contents.

The above are also true for using only a '*' in .pkg files -- in that 
event there are no sys.path changes.  (Frankly, I'm doubtful that 
anybody is using extend_path and .pkg files to begin with, so I'd be 
fine with a proposal that instead used something like '.nsp' files 
that didn't even need to be opened and read -- which would let the 
directory scan stop at the first .nsp file found.


>* The search is only done once - on the first import of the package.

I believe the PEP does this as well, IIUC.


>* It's possible to have a defining package dir and add-one package
>dirs.

Also possible in the PEP, although the __init__.py must be in the 
first such directory on sys.path.  (However, such "defining" packages 
are not that common now, due to tool limitations.)