[Python-Dev] Proposal: expose PEP 302 facilities via 'imp' and 'pkgutil'

[Nick Coghlan]

Phillip J. Eby wrote:
> I propose to create a new API, 'imp.find_loader()' and have it return a PEP 
> 302-compatible loader object, even for cases that would normally be handled 
> via 'imp.load_module()'.  In such cases, the loader returned would be an 
> instance of one of a loader class similar to those in runpy, 
> test_importhooks, and setuptools (which also has similar code).

runpy tries to retrieve "imp.get_loader()" before falling back to its own 
emulation - switching that to look for "find_loader()" instead would be fine 
(although I'd probably leave the emulation in place, since it allows the 
module to be used on 2.4 as well as 2.5).

Also, 'find' more accurately reflects the search effort that goes on.

> What I'm not sure of is where to put the loader class.  It seems to me 
> there should be a stdlib module, but it doesn't seem worth writing in C, 
> especially with so many Python implementations floating around.
> 
> I could create a new Python module for them, but we already have so many 
> import-related modules floating around.  Would it be reasonable to add them 
> to 'pkgutil', which until now has contained only one function?  This would 
> help cut down on some of the code duplication, without adding yet another 
> module to the stdlib.

Would it be worth the effort to switch 'imp' to being a hybrid module?

> An additional issue: "pydoc" needs to be able to determine what submodules, 
> if any, exist in a package, but the PEP 302 protocol does not provide for 
> this ability.  I'd like to add optional additional methods to the PEP 302 
> "importer" protocol (and to any stdlib importer objects) to support the 
> type of filesystem-like queries performed by pydoc.  This should allow 
> pydoc to be compatible with zip files as well as regular files, and any 
> future PEP 302 importers that provide the necessary features.

runpy needs a get_filename() method, so it knows what to set __file__ too - 
currently its emulation supports that, but it isn't officially part of the PEP 
302 API.

> Having these features would also let me cut some code out of setuptools' 
> "pkg_resources" module, that adds some of these features using adapter 
> registries.
> 
> It may be too late for me to be able to implement all of this in time for 
> alpha 2, but at minimum I think the 'find_loader()' addition and the move 
> of the import wrapper classes to 'pkgutil' could be accomplished.
> 
> Replacing pydoc's myriad stat(), listdir() and other path hacking with 
> clean importer protocols might be out of reach, but I think it's worth a 
> try.  For one thing, it would mean that shipping the Python stdlib in a 
> zipfile would not inhibit pydoc's ability to display help for stdlib 
> modules and packages.  (Currently, pydoc does not work correctly with 
> packages that are in zipfiles or which are spread across multiple 
> directories, and it is unable to "discover" modules that are in zipfiles.)

FWIW, I've already had this experience between the original version of runpy, 
and the PEP 302 based version. As I recall, the "real" code in that module is 
now only a few dozen lines - everything else is there to work around the 
absence of imp.find_loader(). Prior to the rewrite, I don't think the module 
was that much shorter overall, and it only supported file system based 
single-directory packages.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org