[Python-Dev] Dynamic module namspaces

Josiah Carlson jcarlson at uci.edu
Mon Jul 17 08:52:48 CEST 2006


Andrew Bennetts <andrew-pythondev at puzzling.org> wrote:
> On Sat, Jul 15, 2006 at 03:38:04PM -0300, Johan Dahlin wrote:
> > In an effort to reduce the memory usage used by GTK+ applications 
> > written in python I've recently added a feature that allows attributes 
> > to be lazy loaded in a module namespace. The gtk python module contains 
> > quite a few attributes (around 850) of which many are classes or 
> > interfaces (150+)
> 
> Have you seen the "demandload" hack that Mercurial uses?  You can find it here:
>     http://selenic.com/repo/hg?f=cb4715847a81;file=mercurial/demandload.py
> 
> You can see an example use of it here:
>     http://selenic.com/repo/hg?f=d276571f2c4b;file=mercurial/commands.py

The problem with that particular method is that it requires that you use
a string to describe the set of modules you would like to import, rather
than a name.

In the py3k mailing list I recently described a mechanism (though not
implementation) for a somewhat related method to support automatic
reloading when a file has changed (for things like web frameworks), but
by removing that file change check, you get the equivalent to what you
describe, though you don't need to use strings, you can use names...

    from dynamicreload import DR

and used like...

    DR.os.path.join(...)

As long as you use the DR.-prefixed name, you get automatic reloading 
(if desired) on access.


> The advantage for an interactive command line tool isn't so much memory
> consumption as speed.  Why waste hundreds of milliseconds importing code that
> isn't used?  There's an experimental branch to use the same demandload code in
> bzr, the reported results are "400ms for 'bzr rocks' down to 100ms, 'bzr root'
> from 400ms => 200ms, etc." (according to
> http://permalink.gmane.org/gmane.comp.version-control.bazaar-ng.general/13967)
> 
> Over half the runtime wasted on importing unused code!  There's a definite need
> for a nice solution to this, and it should be included in the standard
> batteries that come with Python.

Well, just starting up Python without loading any modules can be time
consuming; perhaps even dwarfing the few hundred ms saved by dynamic
loading.


> If we can address related problems at the same time, like emitting deprecation
> warnings for accessing certain module attributes, then even better!

__deprecated__ = ['name1', ...]

Make your dynamic load/reload aware of the __deprecated__ attribute, and
you are done.


 - Josiah



More information about the Python-Dev mailing list