[Python-Dev] PEP: Post import hooks

[Phillip J. Eby]

At 03:20 AM 1/10/2008 +0100, Christian Heimes wrote:
>PyObject* PyImport_NotifyModuleLoaded(PyObject *module)
>    Notify the post import system that a module was requested. Returns the
>    module or NULL if an error has occured.

The big problem here is that the interaction with laziness is 
actually pretty complex, when it comes to re-entrancy and 
initialization order.  A package can actually import a submodule, and 
not yet be finished importing, for example.  So you can actually have 
child hooks executing before parent hooks in this case.

The "Importing" package prevents this by not registering child hooks 
until a parent is actually imported, thus guaranteeing a sane hook 
execution order.  Relative order for hooks targeting the same module 
is maintained, but parent module hooks are guaranteed to execute 
before child hooks, even if the child finishes importing before the parent.

This would be a bit trickier to implement with your C API, since 
"Importing" does this by registering a lot of lambdas.

But, now that I've reviewed my own code and pieced back together the 
rationale for it doing things in this seemingly-odd way, it makes sense.

There's also one twist that I haven't sorted out yet: "Importing" 
guarantees that a parent module 'foo' will have a 'bar' attribute for 
the 'foo.bar' module, if 'foo.bar' is lazy.  It does this by 
registering a callback, ideally *before* any other callback is 
registered for 'foo' or 'foo.bar' that would look at 'foo.bar'.  I 
don't see how to maintain this condition in a world where import 
callbacks can be registered independently.

Bleah.  All of the above isn't really a good explanation of the 
problem.  Let me try to simplify it:

* Lazy importing needs to guarantee that foo.bar = 
sys.modules['foo.bar'], when callbacks for 'foo.bar' execute (in case 
they refer to foo.bar)

* To do this, it registers a callback that sets foo.bar = 
sys.modules['foo.bar'], and does not actually register any foo.bar 
callbacks until 'foo' is really imported (and thus foo.bar gets set 
by that callback)

In the case of the PEP, it's harder for me to figure out what 
happens, because you might not have any lazy modules around, and the 
foo.bar issue would then not come up.  You also have the possibility 
of a problem where a lazy import callback occurs in 3rd party code, 
while callbacks are occurring from the import machinery.  (Which 
means that the notification API should probably set the hooks entry 
to None while it's running, so that if it's called from inside a 
hook, it will not double-run the hooks, and new hooks registered 
while hooks are running will get run immediately as they are 
encountered, instead of getting added to the list.)