[Python-Dev] Write All New Import Hooks (PEP 302) in Python, Not C

[Just van Rossum]

James C. Ahlstrom wrote:

> Let's focus on PEP 302 as a new import hook mechanism, not
> as a way to implement zip imports.  Just's PEP 302 code can
> be used to implement zip imports without making his import
> hooks part of the sys module and thus making them a public
> feature of Python forever.  THIS IS NOT ABOUT ZIP IMPORTS!

Indeed it isn't.

> Python already has an import hook, namely __import__. 

Yes, and the PEP explains clearly what the problems with __import__ (as
a hook) are.

> PEP 302 adds three more hooks: sys.path_hooks, sys.meta_path and
> sys.path_importer_hooks. 

The latter does not exist. There are *two* new hooks: sys.meta_path and
sys.path_hooks. Then there is sys.path_importer_cache, which caches the
results of sys.path_hooks.

> The idea of four import hooks is already fishy.

Well, __import__ is fishy, the new hooks

> PEP 302 enables non-strings on sys.path, and adds two new
> Python objects "importer" and "loader".  It changes the meaning
> of imp.find_module() and imp.load_module(), and adds a new
> imp.find_module2(). 

Yes. The latter remains to be seen (it's not even implemented yet), and
I see Guido just proposed an alternative. I haven't had time yet to read
his full post.

> It changes the meaning of __import__.

Where did you get that idea?

> It proposes to deprecate __path__ manipulations.

It certainly does not. *I* proposed that, but not in PEP 302. Guido is
against it, so that settles that.

> That is a lot of external changes.  That is a lot of code
> written in C.

Except it's not a lot of code.

> I think the proper import hook design is to write Python's
> import mechanism in Python along the lines of Greg's imputil.py
> and Gordon's iu.py. 

This is what iu.py *is*. But iu.py is more: while being a close
reimplementation of all the semantic details, it's a better abstraction
of the Python import mechanism than import.c is. PEP 302 attempts to
expose the key benefits of this better abstraction to Python.

> Import.c would be responsible for flat
> non-package imports from directories and zip files, including
> the import of the real importer iu.py.  The imp module would be
> extended with simple C import utilities that can be used to
> speed up iu.  Once iu.py is imported (probably from site),
> all imports are satisfied using the iu module.

You can do that now by simply using iu.py.

> To provide custom import hooks, the user overrides the iu
> module by writing Python code.  For example, site.py can
> attempt an import of custom_iu before importing iu, and the
> user provides custom_iu.py, probably by copying iu.py.
> I am not sure of the best way to do the override, but I am
> sure it is done in Python code. That enables the user to
> create custom hooks in Python, while relying on the source
> iu.py and the utilities in the imp module for basic
> functionality.
> 
> If sys.path_hooks and the other hooks described in PEP 302
> are approved, they can be implemented in iu.py in Python.

This is backwards: iu.py implements a superset of PEP 302 (with
different details). So you have that now.

> This design still requires C support for zip imports, but
> there are two implementations for that available (from JimA
> and Just).  Other problems such as bootstrapping are easily
> solved.
> 
> I am tired of these endless import hook discussions, which
> always seem to start from nifty ideas instead of from a
> formal solicitation of hook requirements. 

PEP 302 started from the real requirements of a real use case:
zipimport. Added to that are ideas taken from an extremely well though
out model that was developed to solve real problems (iu.py).

> I don't object to
> useful features, but I don't think anything other than easy
> replacement of the Python import mechanism will bring this
> perennial topic to an end.

This is no improvement. You can replace the import mechanism by
overriding __import__. This sucks for most purposes as the import
mechanism is very complicated and contains many subtle details and
pitfalls. PEP 302 allows customizing *parts* of the import mechanism
without having to deal with most of these pitfalls and complications. It
allows completely independend components to add hooks that will work
together seamlessy. This is not true for replacing __import__.

Just