pythonic way to optimize access to imported value?

[Bengt Richter]

On Mon, 18 Nov 2002 06:47:54 -0600, Jeff Epler <jepler at unpythonic.net> wrote:

>On Mon, Nov 18, 2002 at 04:03:30AM +0000, Bengt Richter wrote:
>> IOW, IMO there ought to be a way to do a temporary global-side-effect-free
>> import and delete all traces of its having been done -- without restarting
>> the interpreter or invoking a second interpreter instance.
>
>Let us know when you've implemented it.
>
>Some of the problems:
>
>Modules need not be side-effect-free.  For instance, the following is a
That's not a problem, that's a feature ;-)
I didn't mean that all modules should be free of side effects or bindings
that force their persistence. Most of the time you want some side effects,
but you may want only the side effect to persist, not the module.

<eg>
E.g., if you had a program that needs a lot of memory and does processing
in distinct phases. Obviously you can split the job into separate programs
that use temp files. But what if you want use memory to communicate between
phases? E.g., load a module that retrieves data from the net, another that
parses it and builds convenient data structures, another that requires an
interactive GUI to classify visually presented data, another to create a backup
of clean data and editing log, another to generate pdf reports, etc. etc.
The succeeding phases are going to suffer from memory squeeze if you can't
unload modules that you no longer need.

If you load a module that needs another, of course reference counting should
work as it does, but when you unload a module explicitly, IMO it ought to be
a bit like clearing a dict. A lot of references will get decremented, and
some will go to zero. Why not a module's reference count, to let it disappear
when no longer needed (or have it held in an LRU cache to be bumped as necessary)?
</eg>

>(very harmful) legal Python module:
>    from os import system
>    system("rm -rf /")
>
>Modules may refer to other modules, so you'll need to make any other
>"import"s that happen during that time special.
Yes.

>
>Unloading C modules may not work on all platforms, and may never be
>possible when there are references to objects in the module.
>
Sure, there are all kinds of circumstances when objects legitimately keep
others alive, and that would have to include modules. But I don't see that
as a problem for cases where you program not to do that.

>If any nontrivial object from the module is kept, the module will be around
>anyway, even if it's not listed in sys.modules.  Functions hold a reference
>to the module's globals, for instance.  Any non-trivial class has methods,
>and any interesting object is an instance.
Why would functions *always* hold a reference to the module's global name space,
if it made no use of it? In anticipation of an exec that might? I'm willing for
that to raise an exception if I've unloaded the module. Likewise I'll accept '?'
for various inspection trails that I've cut off. (Maybe some e.g. inspection-supporting
stuff would have to get weak references for private imports, or soemthing
like that).

>
>I think that the presence of modules in sys.modules is used by the system
>to break cycles like that of functions.  At cleanup, a module's dictionary
>is cleared which breaks the module->function->module cycles and allows
>collection.
>
>It's still not exactly clear to me what all these complications would gain
>me anyway...

Mainly recovering memory space for re-use.
See <eg></eg> above, and I'm sure you can
think of other scenarios. I'm not saying there
are no workarounds, but I do think there are cases
where a clean unload could be useful.

Regards,
Bengt Richter