Callable modules?

[Bengt Richter]

On Thu, 25 Jul 2002 12:51:31 +0200, holger krekel <pyth at devel.trillke.net> wrote:

>Hi Bengt,
>
>thanks for your nice code samples.  Although i had problems reading
>code containing my name too often (Signal Overflow :-). 
>Especially your experiments with putting the execution
>of the module-file into a class makes some sense to me.  
>Below i tried a similar yet different approach :-)
>
>I really wonder whether there are *any* reasons to have 
>a distinct ModuleType. What does it offer that a plain
>good old (or new) 'class' could not do?
>
Probably some speed for internal access to named module
contents. And write protection for __dict__ itself.
Plus you can't just always create a new object on every
call to __import__.

There is an internal method in moduleobject.c (I only have
2.1 source handy, will have to fix that ;-)
---
static int
module_traverse(PyModuleObject *m, visitproc visit, void *arg)
{
	if (m->md_dict != NULL)
		return visit(m->md_dict, arg);
	return 0;
}
---
which looks like a prime suspect to me, as a reason to have a
built-in module type. The dealloc mechanism might be important too.

But I think I can imagine a way that you could still get that, and
use a custom container (I.e., an instance of an almost arbitrary class).

You could add an extra parameter to __import__, which would be the
container instance, and the current C moduleobject (with support) would be
modified to be able to initialize itself to pass everything through while monitoring
for __getattr__/__setattr__ to __dict__ itself (you could conceivably
allow write, while doing the right thing inbetween) eliminating the old and
setting the new, and module_traverse would know to use the latest user container __dict__.

I.e., the C module __getattr__/__setattr__ could potentially keep up with apparent
changes of __dict__ itself, since it would be able to intercept a change, though
that wouldn't be frequent.

A user-friendly interface to this __import__ with the extra parameter might be
a modified import statement, with an optional "using myContainerInstance" at the end, e.g.,

    import test using MyContainer()

Incidentally, this should allow you to define __call__ in the class whose instance
you passed to __import__, with the result of getting a "callable module." Properties too.
With the right proxying, you could presumably import builtin modules and have the apparent
effect of adding methods and properties etc. Almost like subclassing an instance.

But import is a much bigger issue than just the module containers.

>Anyway, here is one interesting small experiment ...
>
Yes, it's interesting, but I really don't think a class is
a viable hook-in for __import__.  It must not always create
a new object, for one thing. My "myImport" class wasn't meant
as a hook-in either, despite the name ;-) Just a separate way to
do something import-like with class body source.

OTOH, it should be possible to write a Python function that could
plug in. It looks like imp/imputils were created to help do that,
and rexec stuff apparently depends on that. I've not been into that yet.
But this should be orthogonal to custom containers (though rexec might
want to control use of custom containers).

The kind of object surgery after creation done below makes me uneasy ;-)
>** start Module.py **
>import __builtin__
>
>class Module(object):
>    oldimporter = __builtin__.__import__
>
>    def __init__(self, name, globals={}, locals={}, fromlist=[]):
A nit, but mutable defaults scare me. They tend to get shared in surprising ways ;-)
I would use None and create new instances explicitly instead, like
         if globals is None globals={} # etc.
Also, it's a bit worrisome re-using so many names with builtin meanings ;-)
>        module = self.oldimporter(name, globals, locals, fromlist)
>        self.__dict__ = module.__dict__
>
>    def __getattr__(self, name):
>        getter = self.__dict__.get('__getattr__')
>        return getter(name)
>

>__builtin__.__import__ = Module
I think you only want to do this very carefully ;-)
>
>**end**
>
>** test.py **
>def __getattr__(name):
>    print "__getattr__ called with name",name
>
>import os,sys
 ^^^^^^
Note the danger here! These imports also get captured by Module.
You'd have to be very careful to get the wrapping right. I don't
think it's worth taking on the whole job. Better to mod the
existing mechanism to use custom containers.

>**
>
>and the sesssion with these experimental stuff:
>
>>>> import Module   # wraps __import__
>>>> import test
>>>> test
><Module.Module object at 0x402e5b6c>
>>>> test.os
><Module.Module object at 0x402f6d8c>
I think I'd rather see
    <module 'os' from 'D:\python22\lib\os.pyc'>
but maybe you could fix that with a __repr__

>>>> test.h
>__getattr__ called with name h
>
>I wonder if there are any serious drawbacks of this approach.  
>
Probably ;-) Especially since __import__ doesn't always create new
objects, so a class is not a good fit for a direct replacement. Etc. ;-)

Regards,
Bengt Richter