Mailman 3 Add lazy import statement to protect legitimate circular imports from failing - Python-ideas

5 Feb 2021

      I have seen a lot of interest in this topic, and I think the sheer number
of different lazy import implementations (generally using proxy objects for
modules and then loading them when an attribute is accessed) should present
pretty good evidence that this is a very desired feature, particularly for
writers of library code seeking to optimize start-up times and similar.

Unfortunately, all of these implementations have serious limitations in
that often static code analysis tools cannot understand them, resulting in
autocompletions and useful diagnostic errors being lost. Also, they are
fundamentally incapable of dealing with 'from x import y' syntax, because
this is *always *loaded eagerly by the Python interpreter.

With the increasingly widespread adoption of type hints I think it is time
to revive the proposal for lazy imports as an official language feature.

It is very common to type hint the return type or argument type of a
function as a type which cannot be imported at the top of the file inside
the module namespace because it would trigger a circular import, and must
therefore be imported inside the function. In these instances it is
currently recommended to use the 'TYPE_CHECKING' value in the 'typing'
module which is always 'False' at runtime, in order to be able to provide
the return or argument type hint:

some_module.py
...
from typing import TYPE_CHECKING
if TYPE_CHECKING:
    from module_that_depends_on_me import MyClass
def some_func() -> MyClass:
    from module_that_depends_on_me import MyClass
return MyClass()
However, this is unnecessarily verbose and confusing to most users who are
not familiar with this usage pattern. IDEs will also be unable to warn you
that MyClass is undefined if you try to use it inside 'some_func' without
importing it within 'some_func' first, because it appears to be defined
even though it isn't.

It also is fundamentally incapable of handling this other extremely common
usage pattern:

some_library/ __init__.py
...
__all__ = ["SomeClass", "AnotherClass"]
from .extremely_expensive_module import SomeClass
from .another_extremely_expensive_module import AnotherClass
some_module.py
...
from some_library import  SomeClass
Where a library author wants to make several classes in their library's
public API available at package level in the __init__.py, but doesn't want
to eagerly load all of them when many users will only ever use one in a
given program. In this situation if I try to import SomeClass from
some_library/__init__.py I will *always* trigger AnotherClass to be loaded
as well, totally unnecessarily.

My proposed syntax is pretty much the same as was suggested by Nicolas
Cellier in 2017 here:
https://mail.python.org/pipermail/python-ideas/2017-February/044894.html,
but with the additional stipulation that 'from x import y' format imports
should *also* be supported. The two new lazy import statements should
therefore be:
...
lazy import some_module
from  some_module lazy import SomeClass
For the two examples given above the equivalent code should be.

some_module.py
...
from module_that_depends_on_me lazy import MyClass # does not raise
ImportError when module_that_depends_on_me tries to import me, because this
import is deferred
def some_func() -> MyClass:
    return MyClass() # MyClass is not imported until this line, in a
function that isn't called until after module level imports have already
been resolved
some_library /__init__.py
...
__all__ = ["SomeClass", "AnotherClass"]
from .extremely_expensive_module lazy import SomeClass # this import is
deferred
from .another_extremely_expensive_module lazy import AnotherClass # this
import is deferred
some_module.py
...
from some_library import AnotherClass # this causes SomeClass to be
imported from some_library/__init__.py which in turn causes the import
statement on line 4 of some_library/__init__.py to finally happen for real.
The import of AnotherClass on line 3 *still* hasn't actually happened, and
may never happen if AnotherClass isn't imported (or lazy imported and then
used) during the lifetime of this program.
These are just some examples that I could think of off the top of my head.
I am certain there are more.

This would be insanely useful. I really hope this sparks some discussion :)

Add lazy import statement to protect legitimate circular imports from failing

Matt del Valle

Matt del Valle

Matt del Valle

Chris Angelico

Matt del Valle

Chris Angelico

Paul Bryan

Chris Angelico

Paul Bryan

tags

participants (3)