[Import-SIG] Do we really need to read .pth files in PEP 382?

[P.J. Eby]

At 05:18 PM 6/24/2011 -0400, Barry Warsaw wrote:
>On Jun 24, 2011, at 05:15 PM, P.J. Eby wrote:
> >I'm fine with another name. (namespace_subpath(), perhaps?)
>
>Done!  Bikeshed painted! :)
>
> >I've coded up a sketch of a 2.x implementation (enabled via "from pep382
> >import meta_importer; meta_importer.install()") using that basic API
> >approach, and it's pretty clean.
>
>I would love to be able to back port this to Python 3.2.

Well, my sketch is probably forward-portable, but my brain hasn't 
been ported to 3.x yet, so you'll have to translate it yourself.  ;-)


>Supporting clean namespace packages in Python 2.{6,7} would be double win.

I guess supporting it for 2.3, 2.4, and 2.5 would be triple win, then.  ;-)


>Do you want to push the code to Cheeseshop, or make a branch available
>somewhere?

When I say "sketch", I mean it's just some code typed into a file 
where I sketch bits of Python code - it's not been tested or even 
compiled, let alone run.  ;-)

I'll stick it in a pastebin for you, though; it's barely more than 150 lines:

    http://pastebin.com/uFQ9iwXQ

Some bits of the code are a little hairy, since they're basically 
dynamically adding namespace_subpath() to existing importer objects, 
or working around the fact that existing loaders expect that *they* 
will set the module __path__.  There's then some further hairiness 
created by the need to set __loader__ and __file__ correctly when the 
workaround for the preceding problem is applied.  ;-)

For Python 2.5+, this code can stand alone, but for 2.3 and 2.4 it 
uses pkg_resources from setuptools, as those versions of the Python 
stdlib don't include pkgutil.ImpImporter and pkgutil.get_importer.

Finally, note that the code is designed to be used with explicit 
registration by default: unless you call meta_importer.install(), 
namespace packages have to be registered with 
'pep382.register_namespace()'.  This is a compromise for import 
performance reasons, as the search for '.ns'-containing 
subdirectories adds extra stat() calls per sys.path entry per import, 
which can be hugely slow on a network.

Explicit registration avoids this overhead for anything that's not 
already known ahead of time to be a namespace package, and 
pkg_resources already uses explicit registration, so it's no big deal 
to hook it up there.  The plan would be for setuptools 0.7 to write 
.pth files that "import pep382; 
pep382.register_namespace('whatever')" in place of its existing 
__path__-hackery, as well as for setuptools' declare_namespace() et 
al to use pep382 code behind-the-scenes.

However, for non-setuptools uses, a simple "meta_importer.install()" 
suffices to enable the module in a fully PEP-compatible mode, at the 
cost of added stat() calls.  (The official PEP 382 implementation can 
avoid the extra stat() calls because it'll be directly modifying the 
stdlib importers, rather than working around them.)

Anyway... reminder: this code is an UNTESTED SKETCH with LIMITED 
DOCUMENTATION, and it might contain errors ranging from simple typos 
to subtle-yet-awful design flaws that won't surface for months or 
years.  You have been warned.  ;-)

(Hm, actually, I just thought of a possible bug/oversight -- this 
code doesn't either check or ensure that parent packages of a 
namespace package are registered as namespaces.  That's okay for 
setuptools' purposes, but might lead to incompatibilities between 
third-party code registering packages and third-party code inspecting 
namespace packages and expecting the "parent of a namespace is also a 
namespace" invariant to apply.)