[Python-Dev] PEP: Post import hooks

[Christian Heimes]

Phillip J. Eby wrote:
> when_imported('a.b')(func_ab1)
> when_imported('a.b')(func_ab2)
> 
> @when_imported('a')
> def func_a1(module_a):
>     when_imported('a.b')(func_ab3)
>     notify_module('a.b')   # <- this is here to foil trivial
> implementations
> 
> when_imported('a')(func_a2)
> notify_module('a.b')
> 
> This should produce the calling sequence:
> 
> func_a1, func_a2, func_ab1, func_ab2, func_ab3.

My implementation calls the hooks in the right order but I had to insert
fake modules into sys.path.

        # insert the modules into sys.modules to fake a 3rd party import
        a = imp.new_module('a')
        ab = imp.new_module('a.b')
        a.b = ab
        sys.modules["a"] = a
        sys.modules["a.b"] = ab
        # notify
        imp.notify_module_loaded('a.b')

Otherwise the code fails to fetch the module objects from sys.path and
an exception is raised.

> 1. notification is only done once for a given module, ever, even if the
> notification function is called more than once, even if it's called
> during notifications for that module

The latter is not yet implemented. notify_module('a.b') currently fails
with a recursion limit exceeded. It's on my todo list.

> 2. notifications for a child module/package may not begin until the
> notifications for the parent package have begun

This is already implemented for notification by name but not for
notification by module object. It's also on my todo list.

> 3. two registrations for the same module must always be invoked in the
> same order as they were registered, even if some of the registrations
> are done during notification.

I'm pretty but not yet absolute sure that my implementation guarantees
the last invariant. I've to go through my code several times and play
through use cases.

> In order to implement these invariants, you will have to have a way to
> know whether notifications have been begun for a given module.  In
> peak.util.imports, the module objects effectively keep track of this,
> although they don't have a specific flag.  For the Python
> implementation, you could add a __notified__ field to module objects,
> and implement the notify function thus:

That's a nice idea to fix the recursion issue with nested notifications.

> Of course, __notified__ would actually be a structure slot, rather than
> an attribute, so as to avoid any attribute lookup issues with module
> subtypes (e.g. lazy modules).

Sure, I can use a slot for PyModule and its subclasses. Unfortunately
other implementations of lazy imports are adding a proxy object which is
not a subclass of PyModule. I've to use an attribute if not
isinstance(mod, moduletype).

> The register function would simply append a hook to the entry in
> post_import_hooks if it's not None, or call the hook otherwise.

My code queues up new hooks while a sequence of hooks is processed. It
makes sure that hooks for a parent aren't called in the middle of a
child's hook chain.

notification_in_progress = False
queue = []

def register_hook(hook, name):
    if notification_in_progress:
        queue.append((hook, name))
        return

    hooks = sys.post_import_hook_register(name, None)

    if hooks is None:
        module = sys.modules.get(name, None)
        if modules:
            sys.post_import_hook_register[name] = None
            hook(module)
        else:
            hooks = []
            sys.post_import_hook_register[name] = hooks

    hooks.append(hook)

def notify_loaded(mod_or_name):
    notification_in_progress = True
    try:
        ...
    finally:
        notification_in_progress = False
        while queue:
            hook, name = queue.pop(0)
            hook(sys.modules[name])

> With this implementation, I could make a version of peak.util.imports
> that did its own lazy modules, but used the base system for all the hooks.

Very good!

Christian