[Python-Dev] Fix import errors to have data

[Phillip J. Eby]

At 09:53 AM 7/28/04 -0700, Guido van Rossum wrote:
> > On Wed, 2004-07-28 at 06:56, Jim Fulton wrote:
> > > Do you think it's practical to limit the effects of module import to
> > > sys.modules, even by convention?  Could we say that it is a bug for
> > > code executed during module import to mutate other modules, including
> > > mutating objects contained in those other modules?
>
>I'm uncomfortable with that; while it's indeed conventional for
>modules to mostly define classes and functions (and constants and
>global variables and ...), there are many other useful things one
>could do at module level, and I hesitate to declare those in any way,
>shape or sense frowned upon.  You can have stricter rules within a
>project, of course, but I don't want to make this part of the general
>Python culture; your concerns are mostly relevant only for very large
>projects.

And for those very large projects, I'm wondering if what is *really* wanted 
is actually to be able to isolate plug-ins or extensions into their own 
"module space", similar to the way the Java "classloader" hierarchy 
works.  In effect, Java can have multiple 'sys' modules, arranged in a 
hierarchy.  Each classloader has its own sys.path equivalent, and can also 
delegate imports to its parent classloader(s), in order to have some 
modules shared between child classloaders.  Classloaders are typically used 
in application servers or other systems that wrap around multiple 
third-party application modules, so that each application unit appears to 
have the entire VM to itself, even though side-by-side there may be other 
applications seeing the same or different versions of various 
modules.  This effect is achieved by creating a classloader for each 
segregated application unit, and then using it to load that application's 
main class.

If a mechanism like this were available for Python, then large systems like 
Zope could simply (well, maybe *simply* is the wrong word!) load extensions 
in a separate "module space".  If there's a problem, just throw the whole 
thing away, with no side effects in the current module space such as 
"insane" half-empty modules.

As a side benefit, individual extensions can depend on/use different 
versions of common packages, as long as they get loaded into separate 
module spaces.  Heck, you can give extensions their own private copy of the 
'logging' module, or anything else that has module-level configuration and 
doesn't "play nice" when multiple extensions want it configured differently.

I don't know for certain that such a thing is possible using just the 
standard import hooks, but it seems to me it might be.  If anybody is 
interested in such an animal, have a look at:

    http://www.eby-sarna.com/pipermail/peak/2004-July/001652.html

and drop me a line.  I presume this is offtopic for further discussion on 
python-dev unless it's proposed as a stdlib addition, and I don't think it 
should be so proposed without first having some field experience in at 
least a couple of those "very large projects".  My personal interest in 
this is mainly aimed at integrating Python with Eclipse, but it seems to me 
that systems like Zope and Chandler that want to support large numbers of 
third-party plugins, probably want to be able to do things like this.  It 
would be nice to have a common implementation, or failing that, at least 
understand what prevents a common implementation from being practical.