[Python-Dev] PEP 302 support for pydoc, runpy, etc.

[Phillip J. Eby]

Okay, so I've spent a bit more time thinking this through, and I think I 
have a plan that can work, and deals with all the weird little corner 
issues, including overlapping code with pkg_resources (a new module that's 
to-be-added by setuptools).

My basic idea is as follows:

1. Move the support code from runpy to pkgutil and expose most of it as 
actual APIs.

2. Move additional code from pkg_resources to pkgutil and merge with the 
stuff brought over from runpy, fleshing it out enough so that pydoc can use 
it to find modules, subpackages, etc., and so that pkg_resources' other 
code uses pkgutil instead of having that stuff "in-house".

3. Write up docs for the expanded pkgutil

4. Add find_loader([path]), get_loader(module), and 
get_importer(string[,path_hooks]) functions to the 'imp' module, with 
pure-Python versions in pkgutil that kick in if the C ones are 
missing.  (See next item for why.)

5. Distribute the Python 2.5 'pkgutil' module with setuptools releases for 
older Python versions, so that setuptools won't have to keep duplicates of 
the code I pulled out in item #2 above.

Part of what would be being copied to pkgutil from pkg_resources is the 
"resource provider API", that allows you to treat a module, package, or 
sys.path entry as a kind of virtual filesystem.  pydoc needs this in order 
to be able to document with zipped packages, and to find modules or 
packages on sys.path that are in zipfiles.  (E.g. to document a zipped 
standard library package or module.)

pkg_resources has been out there for almost a year now, and is widely used 
as part of easy_install and the egg runtime facilities, but it's a very 
large module and contains lots of other code besides the virtual filesystem 
stuff.  So, this seems like the simplest way to share the virtual 
filesystem code between pydoc and setuptools, without making the former 
depend on the latter.

It does also mean creating an addition to PEP 302: an optional method on 
importers and loaders that would allow obtaining a virtual filesystem for 
that path location or package.

I believe this and the setuptools checkin can be completed by Monday 
evening, although the documentation might not be fancy at that point.  I 
don't know when the documentation cutoff for alpha 2 is, but I assume I'd 
have time to finish fleshing it out, at least for the pkgutil bit and the 
PEP update.

Anybody have any thoughts, comments, or questions?