[Python-Dev] Dynamic module namspaces

[Andrew Bennetts]

On Sun, Jul 16, 2006 at 11:52:48PM -0700, Josiah Carlson wrote:
> Andrew Bennetts <andrew-pythondev at puzzling.org> wrote:
[...]
> > 
> > Have you seen the "demandload" hack that Mercurial uses?  You can find it here:
> >     http://selenic.com/repo/hg?f=cb4715847a81;file=mercurial/demandload.py
> > 
> > You can see an example use of it here:
> >     http://selenic.com/repo/hg?f=d276571f2c4b;file=mercurial/commands.py
> 
> The problem with that particular method is that it requires that you use
> a string to describe the set of modules you would like to import, rather
> than a name.

I agree, it's ugly.  I'd like there to be a nicer solution.

> In the py3k mailing list I recently described a mechanism (though not
> implementation) for a somewhat related method to support automatic
> reloading when a file has changed (for things like web frameworks), but
> by removing that file change check, you get the equivalent to what you
> describe, though you don't need to use strings, you can use names...
> 
>     from dynamicreload import DR
> 
> and used like...
> 
>     DR.os.path.join(...)
> 
> As long as you use the DR.-prefixed name, you get automatic reloading 
> (if desired) on access.

Aside from also being ugly in its own way, a magic prefix like this would add a
small performance penalty to places that use it.  I believe demandload has the
nice feature that once a demandloaded object is accessed, it is replaced with
the actual object rather than the demandload gunk, so there's only a one-off
performance penalty.

> > The advantage for an interactive command line tool isn't so much memory
> > consumption as speed.  Why waste hundreds of milliseconds importing code that
> > isn't used?  There's an experimental branch to use the same demandload code in
> > bzr, the reported results are "400ms for 'bzr rocks' down to 100ms, 'bzr root'
> > from 400ms => 200ms, etc." (according to
> > http://permalink.gmane.org/gmane.comp.version-control.bazaar-ng.general/13967)
> > 
> > Over half the runtime wasted on importing unused code!  There's a definite need
> > for a nice solution to this, and it should be included in the standard
> > batteries that come with Python.
> 
> Well, just starting up Python without loading any modules can be time
> consuming; perhaps even dwarfing the few hundred ms saved by dynamic
> loading.

Well, it's only about 10ms on my laptop running Ubuntu (it varies up to 90ms,
but I expect that's just noise), and the "-S" switch makes no obvious difference
(tested with "python -c ''").  10ms overhead to start python I can live with.
It takes about that long run "svn --version".

> > If we can address related problems at the same time, like emitting deprecation
> > warnings for accessing certain module attributes, then even better!
> 
> __deprecated__ = ['name1', ...]
> 
> Make your dynamic load/reload aware of the __deprecated__ attribute, and
> you are done.

I'm fine with that syntax.  But regardless of how it's spelled, I'd really like
the standard library or interpreter to have support for it, rather than having a
non-standard 3rd-party system for it.  I want there to "be one-- and preferably
only one --obvious way to do it".

-Andrew.