[Python-3000] Interest in PEP for callbacks on module import

[Phillip J. Eby]

At 03:56 PM 12/8/2007 +1000, Nick Coghlan wrote:
>Phillip J. Eby wrote:
> >> > Does anyone else think this is an issue worth pursuing?
> >
> > A qualified yes: the Importing package took a long time to get correct
> > under all the crazy little twists and turns of importing, including
> > correctly handling things like the link between packages and their child
> > packages/modules, the import order of same, race conditions, import
> > locks, clearing out callbacks, etc.  I'm somewhat hesitant as to whether
> > a from-scratch reimplementation is wise.
>
>I'm far from convinced that a from-scratch rewrite (particularly in C)
>is a great idea myself - that's why I suggested a PEP to look at some of
>the issues.
>
>Some curly questions I thought of myself:
>
>- What do we do if something is inserted in sys.modules directly, and
>then imported later? Do registered callbacks for that module trigger at
>all? If so, when?

The Importing package doesn't have this problem, because it inserts 
the module itself the first time you call whenImported for that 
module.  In other words, any time you call whenImported, either the 
module is not there (in which case it's added, and the callback 
registered), or it is (in which case the callback is invoked immediately).

Of course, if somebody *replaces* the lazy module, all bets are off, 
but that seems like a consenting adults thing to me.


>- Does the lazy importing code need to reacquire the import lock before
>loading the module on attribute access?

If it's written in Python, yes, in order to avoid race conditions 
around the manipulation of the custom type.  In C, however, you 
wouldn't need to because the GIL can prevent that portion of the race 
condition, and then the reload() operation will hold the import lock 
around the Python code execution.

The callbacks should probably be invoked under import lock protection, though.


> > There may be other interop issues with libraries that use ModuleType
> > subclasses or do lazy import, deprecation, and similar module content
> > manipulation, so a publicized PEP would be a good idea to get their
> > developers' input.
>
>Yeah, the more I think about it, the more I think a new PEP for lazy and
>weak importing is the right way to go (although it would be nice if
>using the latter didn't automatically cause the former).

If it doesn't, then you'll need more complex mechanisms to support 
both.  If you want a C implementation, the simplest way I can think 
of to make it work is to give module objects a lazy flag and a 
callback list.  If you don't, then you have to deal with more corner 
cases and interop trickiness.

Misc. notes for the PEP: there should be a way to determine the 
laziness of a module object without causing it to load.  I also think 
you should be able to set a module's __loader__, __path__, and 
__file__, without causing it to load.  That way, you can create a 
lazy module that is loaded by a custom loader.  (Does reload() honor 
__loader__, or does it start the import process over?)