[Python-Dev] PEP 302, PEP 338 and imp.getloader (was Re: a Python interface for the AST (WAS: DRAFT: python-dev...)

[Nick Coghlan]

Phillip J. Eby wrote:
> At 06:32 PM 11/22/2005 -0800, Brett Cannon wrote:
>>>  Hmm, it would be nice to give a function a module
>>> name (like from an import statement) and have Python resolve it using
>>> the normal sys.path iteration.
>>>
>> Yep, import path -> filename path would be cool.
> 
> Zipped and frozen modules don't have filename paths, so I'd personally 
> rather see fewer stdlib modules making the assumption that modules are 
> files.  Instead, extensions to the PEP 302 loader protocol should be used 
> to support introspection, assuming there aren't already equivalent 
> capabilities available.  For example, PEP 302 allows a 'get_source()' 
> method on loaders, and I believe the zipimport loader supports that.  (I 
> don't know about frozen modules.)
> 
> The main barrier to this being really usable is the absence of loader 
> objects for the built-in import process.  This was proposed by PEP 302, but 
> never actually implemented, probably due to time constraints on the Python 
> 2.3 release schedule.
> 
> It's relatively easy to implement this "missing loader class" in Python, 
> though, and in fact the PEP 302 regression test in the stdlib does exactly 
> that.  Some work, however, would be required to port this to C and expose 
> it from an appropriate module (imp?).

Prompted by this, I finally got around to reading PEP 302 to see how it 
related to PEP 338 (which is intended to fix the current limitations of the 
'-m' switch by providing a Python fallback when the basic C code can't find 
the module to be run).

The key thing that is missing is the "imp.getloader" functionality discussed 
at the end of PEP 302.

Using that functionality and the exec statement, PEP 338 could easily be 
modified to support any module accessed via a loader which supports get_code() 
(and it could probably also get rid of all of the current cruft dealing with 
normal filesystem packages).

So with that in mind, I'm thinking of updating PEP 338 to propose the following:

1. A new pure Python module called "runpy"

2. A function called "runpy.execmodule" that is very similar to execfile, but 
takes a module reference instead of a filename. It will NOT support 
modification of the caller's namespace (based on recent discussions regarding 
the exec statement). argv[0] and the name __file__ in the execution dictionary 
will be set to the file name for real files (those of type PY_SOURCE or 
PY_COMPILED), and the module reference otherwise. An optional argument will 
permit argv[0] (and __file__) to be forced to a specific value.**

3. A function called "runpy.get_source" that, given a module reference, 
retrieves the source code for that module via loader.get_source()

4. A function called "runpy.get_code" that, given a module reference, 
retrieves the code object for that module via loader.get_code()

5. A function called "runpy.is_runnable" that, given a module reference, 
determines if execmodule will work on that module (e.g. by checking that the 
loader provides the getcode method, that loader.is_package returns false, etc)

6. If invoked as a script, runpy interprets argv[1] as the module to run

7. If the '-m' switch fails to find a module, it invokes runpy as a fallback.

To make PEP 338 independent of the C implementation of imp.getloader for PEP 
302 being finished, it would propose two private elements in runpy: 
runpy._getloader and runpy._StandardImportMetaHook

If imp.getloader was available, it would be assigned to runpy._getloader, 
otherwise runpy would fall back on the Python equivalents.

** I'm open to suggestions on how to deal with argv[0] and __file__. They 
should be set to whatever __file__ would be set to by the module loader, but 
the Importer Protocol in PEP 302 doesn't seem to expose that information. The 
current proposal is a compromise that matches the existing behaviour of -m 
(which supports scripts like regrtest.py) while still giving a meaningful 
value for scripts which are not part of the normal filesystem.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org