[Python-Dev] PEP: Lazy module imports and post import hook

Christian Heimes lists at cheimes.de
Wed Jan 9 06:01:01 CET 2008


glyph at divmod.com wrote:
> and very recently, I implemented similar functionality myself (though it 
> isn't in use in Twisted yet):
> 
>     http://codebrowse.launchpad.net/~glyph/+junk/pyexport/files

I'm going to study your implementation tomorrow.

> Something that I notice about every other implementation of this 
> functionality is that they're all in Python.  But the proposed 
> implementation here is written in C, and therefore more prone to 
> crashing bugs.  Looking at the roundup log, I can see that several 
> refcounting bugs have already been found and fixed.  Perhaps the post- 
> import hooks, being a modification to the import mechanism itself, needs 
> to be in C, but given that lazy imports *have* been implemented before 
> without using C code (and without post import hooks), I can't see why 
> they need to be in the same PEP.

Correct, the first few patches were rough implementation sketches. I
wanted to discuss implementation details with PJE and it was the easiest
way to exchange code. A distributed VCS would have made the task easier
but that's a different story. Eventually I focused on the reference
counter and I straightened out some ref count bugs.

Deferred loading of modules can be implemented without post import hooks
and vice versa. But PJE urged me to - at least - consider lazy modules /
deferred importing in the post import hook implementation. 3rd party
tools need a way to interact with the hooks and notify the system when
their own lazy module implementation loads a real module.

I figured out I could write a PEP for both features in one PEP and
implement them in a combined patch. I could separate them and create two
PEPs.

> Also, as is my custom, I'd like to suggest that, rather than designing 
> something new, one of the third-party implementations be adopted (or at 
> least its interface) since these have been demonstrated to work in real 
> application code already.  Happily, I can escape the charge of bias this 
> time since I don't think my own implementation should be taken seriously 
> in this case :).

:)

I *did* mention that my implementation of lazy imports is based on PJE's
peak.util.import:

"""
PJE's peak.util.imports [3] supports lazy modules an post load hooks. My
implementation shares a lot with his and it's partly based on his ideas.
"""

Finally I like to say something about my choice to implement the lazy
loading in C. Most implementation like zope.deferredimport, Hg's
demandimport and bzr's lazy_import are using the proxy design pattern.
They are storing a wrapper object in the module and sys.modules instead
of the real module and they are loading the real module when an
attribute is accessed. Some of some (brz) are even using some strange
scope replacing. I've to study the implementation more carefully to
understand what's going on.

peak.util.imports' way is slightly different. It return the instance of
a module subclass. When the real module is loaded it redirect name
lookup of the lazy module to the real module by replacing the
__getattribute__ and __setattr__ methods of the lazy module with the
methods of the real module.

My implementation uses a similar approach. But I've the luxury that I
can modify the C implementation of the module class. It makes the
load-real-module-when-attribute-is-accessed dance much easier. I don't
have to write a proxy or deal with name spaces.

However the PEP covers only the basic infrastructure for lazy imports.
For example imp.lazy_import("a.b.c") doesn't put "a" and "a.b" in
sys.modules. It neither doesn't cover the replacement __import__ hook
provided by Hg nor the deprecation feature of zope.deferredimport. Such
infrastructure functions can be implemented in Python.

Please remember, the PEP is a draft. I'm waiting for comments and
suggestions from users and developers.

Christian



More information about the Python-Dev mailing list