[Python-3000] Changing the import machinery

[Phillip J. Eby]

At 10:42 AM 4/21/2006 +0100, Guido van Rossum wrote:
>I think I like this. I wonder if there's a parallel with my preference
>for strings as paths instead of path objects...

PEP 302 still has things analagous to path objects, it's just that you're 
allowed to define the mapping between strings and objects.


> > I would also suggest that we ask Just if he can come up with some elegant
> > extension that would allow PEP 302 to support pluggable import *formats* as
> > well as locations.  :)
>
>I guess you'll need to explain this particular problem a bit better...

Some users of templating systems want to be able to put template files in 
their packages, and then import them as Python objects, especially if it's 
a templating language that compiles to Python code anyway (e.g. Kid, 
PTL).  However, because you can't use PEP 302 hooks to do this, each 
package that does is forced to write a full-blown import hook or use one of 
the other import hook frameworks -- and they generally these extensions 
don't "play nice" with each other.

So basically what they would want is a way to add new file extensions that 
are recognized as "modules" and can be loaded by the import 
machinery.  Think of it as a way to add things to imp.get_suffixes(), if 
imp.get_suffixes() also gave you handlers instead of just metadata.

So, what I'd suggest is that the expanded import mechanism would provide a 
registry of extensions and handlers, defining how you get from file 
extension to module, in a way that would allow PEP 302 importers and 
loaders to delegate this machinery to a handler.  Python would also provide 
built-in handlers for .py, .pyc, .pyo, and so on, so that everybody's 
importers wouldn't have to reimplement these features, the way they do now.

There are a few details to work out, as some formats simply can't work 
without a real filesystem file (C extensions, for example), so the handler 
registry would have to contain information about that.  Also, some file 
formats are generated from others, and so the registry needs to give some 
info about that, and provide hooks for writing back generated 
files.  (Which would have to be optional, since e.g. zip files wouldn't 
allow it.)

All in all, there's nothing particularly revolutionary there, it's just 
working out some tedious details with a registry and interface.  Then, we 
change the built-in machinery to just be some entries in this registry, and 
change zipimport and imp to use that registry.  And that should even be 
able to be done in the 2.x series, as it doesn't create any backward 
incompatibilities, it just enables zipimport and user importers to call 
back to pieces of the machinery (e.g. to load/check/write Python modules) 
rather than reimplementing them.