[Python-3000] Changing the import machinery

[Phillip J. Eby]

At 10:05 PM 4/21/2006 +1000, Nick Coghlan wrote:
>Phillip J. Eby wrote:
>>There are only two things wrong with PEP 302 IMO, and neither is its "fault".
>>The first is that the "classic" import machinery isn't on sys.meta_path, 
>>and the 'imp' API isn't defined in terms of PEP 302.  Those two things 
>>can't change without introducing backward incompatibilities.
>>The second is that PEP 302 only covers "location" importers, not "format" 
>>importers.  That is, if you want to do something like make Kid or Cheetah 
>>templates importable, you have to replace things with new 
>>machinery.  This is a more important problem to solve, IMO, than coming 
>>up with yet another set of semantics for the things that *are* covered by 
>>PEP 302.
>
>Do you mean an API so you can associate loader objects with new 
>extensions, allowing find_module and load_module to deal with them?

More precisely, to allow PEP 302 importers and loaders to deal with 
them.  find_module and load_module should go away, replaced with a PEP 302 
importer and loader, respectively.


>What about something like a sys.load_hooks list? The default contents of 
>sys.load_hooks (on Windows) would be:
>
>   PythonPackageLoader (handles directories with __init__.py files)
>   PythonDLLLoader (handles .pyd files)
>   PythonSourceLoader (handles .py and .pyw files, checks for compiled file)
>   PythonCompiledLoader (handles either .pyc and .pyo files, depending on -O)

Well, it'd be more like sys.module_extensions or some such, but yes.  I 
also don't think there should be a package loader as such.  Instead, the 
other handlers would be applied to the name __init__ within a package.


>That would let us get rid of a lot of ifdef hackery from import.c, and 
>also permit things like:
>
>   sys.load_hooks.append(ZipLoader)
>
>where "ZipLoader" handles files ending in .zip or .egg, and treats them as 
>packages.

No, let's keep PEP 302, which works just fine for this part.  Zip files 
aren't packages, anyway, and definitely shouldn't be conflated with them.


>(Random aside: why the heck is all this machinery stored in sys, anyway? 
>Having it in imp would make far more sense. Was it just because sys.path 
>was already there?)

Well, without sys there are no modules yet, so how would you import 
imp?  But you'd have to ask Just for his reasons.


>>PEP 302 is reasonably clear and concise, although there are occasional 
>>loopholes.  At least three people have re-implemented parts of it in pure 
>>Python from the spec: Just van Rossum, Neal Norwitz, and myself (twice, 
>>in my case).  It is easy and elegant -- the only hard parts are when you 
>>have to wedge all the "classic" imports in, or if you want to implement 
>>"format" importers.
>
>Make that 4 - I had to reimplement a chunk of it for PEP 338.

Oops, I must've got you and Neal mixed up; I just remembered it was an "N" 
name, and he has two "N" names.  :)  Sorry.