On Mon, Oct 13, 2008 at 1:12 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

Problem: You have a package containing a large number of classes, of which only a few are typically used by any given application.

If you put each class into its own submodule, then client code is required to use a lot of tedious 'from foo.thingy import Thingy' statements to import the classes it wants to use. This also makes all the submodule names part of the API and makes it hard to rearrange the packaging without breaking code.

If you try to flatten the namespace by importing all the classes into the top level module, you end up importing everything even if it won't be used.

What's needed is a way of lazily importing them, so that the import won't actually happen unless the imported names are referenced.

It's possible to hack something like that up now, but then tools such as py2app and py2exe, that try to find modules by statically examining the source looking for import statements, won't be able to accurately determine which modules are used. At best they'll think the whole package is used and incorporate all of it; at worst they'll miss it altogether.

So I think it would be good to have a dedicated syntax for lazy imports, so the top-level foo package can say something like

from foo.thing lazily import Thing from foo.stuff lazily import Stuff ...

Executing a lazy import statement adds an entry to a list of deferred imports attached to the module. Then, the first time the imported name is referenced, the import is performed and the name becomes an ordinary attribute thereafter.

If py2exe et al are taught about lazy imports, they will then be able to determine exactly which submodules are used by an application and exclude the rest.

How is this mechanism supposed to behave in the presence of things like... # baz.py from foo lazy import bar def fcn(): return bar.obj(...) It would seem that py2exe, etc., would need to include the bar module regardless of whether baz.fcn() was called in any particular invocation of a program, and even if baz.fcn() was never called in any invocation. - Josiah

Greg Ewing wrote:

Problem: You have a package containing a large number of classes, of which only a few are typically used by any given application. [...]

So I think it would be good to have a dedicated syntax for lazy imports, so the top-level foo package can say something like

Yes, this would be good to have. There's clearly a need; Bazaar and Mercurial and probably others have invented versions of this. It would be excellent to have a standard way to spell it. -Andrew.

Greg Ewing wrote:

Problem: You have a package containing a large number of classes, of which only a few are typically used by any given application.

If you put each class into its own submodule, then client code is required to use a lot of tedious 'from foo.thingy import Thingy' statements to import the classes it wants to use. This also makes all the submodule names part of the API and makes it hard to rearrange the packaging without breaking code.

If you try to flatten the namespace by importing all the classes into the top level module, you end up importing everything even if it won't be used.

What's needed is a way of lazily importing them, so that the import won't actually happen unless the imported names are referenced.

It's possible to hack something like that up now, but then tools such as py2app and py2exe, that try to find modules by statically examining the source looking for import statements, won't be able to accurately determine which modules are used. At best they'll think the whole package is used and incorporate all of it; at worst they'll miss it altogether.

So I think it would be good to have a dedicated syntax for lazy imports, so the top-level foo package can say something like

from foo.thing lazily import Thing from foo.stuff lazily import Stuff ...

Executing a lazy import statement adds an entry to a list of deferred imports attached to the module. Then, the first time the imported name is referenced, the import is performed and the name becomes an ordinary attribute thereafter.

If py2exe et al are taught about lazy imports, they will then be able to determine exactly which submodules are used by an application and exclude the rest.

FWIW, PyPy can lazily compute objects: <http://codespeak.net/pypy/dist/pypy/doc/getting-started.html#lazily-computed...> I'm don't know how easy it would be to tie that into imports, but I'm sure it would be possible. Coming from the Bazaar and Mercurial communities, I'd like lazy imports too, but I doubt it'll ever happen because it's just a strange performance hack. --