[Python-3000] Interest in PEP for callbacks on module import

[Phillip J. Eby]

At 12:33 PM 12/7/2007 +0100, Christian Heimes wrote:
>Nick Coghlan wrote:
> > For example, to handle the commented out case above:
> >
> >    @imp.imported('decimal')
> >    def register(decimal):
> >        Inexact.register(decimal.Decimal)
>
>I like the syntax assuming that imp.imported(name) expects a method that
>accepts the module object as argument.

I prefer something like 'when_imported' or 'upon_import' or something 
else that more specifically refers to the idea that this is 
registering a handler to be called.  imported() by itself sounds like 
a query as to whether the module is currently imported.


> > I think a PEP would be needed to decide whether to handle this in a
> > fashion similar to that of PJE's Importing toolkit [1] (i.e., using lazy
> > imports where the actual loading of the module code is deferred until
> > the first access to an attribute of the module), or else to add a new
> > mechanism directly to the interpreter code, where the registered
> > callbacks would be called as soon as the specified module was loaded.
>
>I find a new mechanism easier to implement. Lazy or deferred imports
>require frame hacks

No, they don't.  See:

http://svn.eby-sarna.com/Importing/peak/util/imports.py?view=markup

The mechanism I used is to create a subclass of ModuleType and put it 
in sys.modules as a placeholder for the "real" module.  When the 
module is touched, I effectively change the type back to a regular 
module, and call reload() to actually import the module.

Of course, since Python 2.3 (this code was originally written for 
2.2), you can't change a subclass of ModuleType back to ModuleType 
itself, so I actually change the custom subclass on the fly.  This is 
easy enough to do in Python, but I'd hate to do it in C.

If I were doing it in C, I'd just add a laziness flag and callback 
list to the base ModuleType.  The tp_getattro could then check the 
laziness flag and do the rest.



> > Does anyone else think this is an issue worth pursuing?

A qualified yes: the Importing package took a long time to get 
correct under all the crazy little twists and turns of importing, 
including correctly handling things like the link between packages 
and their child packages/modules, the import order of same, race 
conditions, import locks, clearing out callbacks, etc.  I'm somewhat 
hesitant as to whether a from-scratch reimplementation is wise.

I'm also concerned about interoperability.  If you add the equivalent 
of Importing's whenImported, but not lazyModule, then I'll still need 
Importing itself -- and the builtin implementation of whenImported 
might conflict with lazyModule.  In particular, the builtin 
when_imported might confuse a lazyModule with something that's 
already imported, and fire off callbacks when it shouldn't.

So, I vote for making module laziness part of the implementation, if 
there is one.

There also should be APIs provided by imp to inspect (or change) the 
laziness of a module without causing it to become un-lazy.  This is 
needed so that other ModuleType subclasses can play too.  For 
example, some tools use ModuleType subclasses to implement 
attribute-level laziness instead of whole-module laziness.

There may be other interop issues with libraries that use ModuleType 
subclasses or do lazy import, deprecation, and similar module content 
manipulation, so a publicized PEP would be a good idea to get their 
developers' input.