Add lazy import statement to protect legitimate circular imports from failing

I have seen a lot of interest in this topic, and I think the sheer number of different lazy import implementations (generally using proxy objects for modules and then loading them when an attribute is accessed) should present pretty good evidence that this is a very desired feature, particularly for writers of library code seeking to optimize start-up times and similar. Unfortunately, all of these implementations have serious limitations in that often static code analysis tools cannot understand them, resulting in autocompletions and useful diagnostic errors being lost. Also, they are fundamentally incapable of dealing with 'from x import y' syntax, because this is *always *loaded eagerly by the Python interpreter. With the increasingly widespread adoption of type hints I think it is time to revive the proposal for lazy imports as an official language feature. It is very common to type hint the return type or argument type of a function as a type which cannot be imported at the top of the file inside the module namespace because it would trigger a circular import, and must therefore be imported inside the function. In these instances it is currently recommended to use the 'TYPE_CHECKING' value in the 'typing' module which is always 'False' at runtime, in order to be able to provide the return or argument type hint: some_module.py
However, this is unnecessarily verbose and confusing to most users who are not familiar with this usage pattern. IDEs will also be unable to warn you that MyClass is undefined if you try to use it inside 'some_func' without importing it within 'some_func' first, because it appears to be defined even though it isn't. It also is fundamentally incapable of handling this other extremely common usage pattern: some_library/ __init__.py
some_module.py
from some_library import SomeClass
Where a library author wants to make several classes in their library's public API available at package level in the __init__.py, but doesn't want to eagerly load all of them when many users will only ever use one in a given program. In this situation if I try to import SomeClass from some_library/__init__.py I will *always* trigger AnotherClass to be loaded as well, totally unnecessarily. My proposed syntax is pretty much the same as was suggested by Nicolas Cellier in 2017 here: https://mail.python.org/pipermail/python-ideas/2017-February/044894.html, but with the additional stipulation that 'from x import y' format imports should *also* be supported. The two new lazy import statements should therefore be:
lazy import some_module from some_module lazy import SomeClass
For the two examples given above the equivalent code should be. some_module.py
some_library /__init__.py
some_module.py
These are just some examples that I could think of off the top of my head. I am certain there are more. This would be insanely useful. I really hope this sparks some discussion :)

Since this hasn't actually received moderator approval yet, I'd be happy if you declined this post and let me resubmit it without those typos, since they actually deal with specifics of the proposal and could be confusing if left in. I'm super sorry, my mistake! On Fri, Feb 5, 2021 at 12:39 PM Matt del Valle <matthewgdv@gmail.com> wrote:

On Sat, Feb 6, 2021 at 12:51 AM Matt del Valle <matthewgdv@gmail.com> wrote:
Unfortunately, [lazy importers are] fundamentally incapable of dealing with 'from x import y' syntax, because this is always loaded eagerly by the Python interpreter.
This is because it's fundamentally hard.
With the increasingly widespread adoption of type hints I think it is time to revive the proposal for lazy imports as an official language feature.
Are you writing this proposal assuming PEP 563 behaviour? If type hints are the primary justification, it may be less necessary. But lazy importing does have other value.
Hmm. Okay. Let's talk semantics, not syntax. What exactly should the interpreter do with this line? Is there an actual object in the module dictionary under the name "SomeClass", and if so, what sort of object? One possible meaning for this kind of "lazy from import" would be to maintain a table of import definitions, just like the module dictionary. (I would be inclined to say that this can ONLY be done at module level; lazy imports inside a class or function seem less useful and more hassle than they're worth. But if you disagree, feel free to expand this to other namespaces.) Whenever a module name is looked up, the interpreter looks first in the module dictionary, and then if it's not found, tries the lazy imports; upon finding the import definition, it removes it, performs the import, and stuffs the object into the module dictionary. This would be a fair amount of hassle, and it'd have a performance impact on EVERY module name lookup (and everything that goes beyond that to the builtins). But it would, I believe, give you an actual lazy import system. So the question is: how often is this going to be useful?
Where a library author wants to make several classes in their library's public API available at package level in the __init__.py, but doesn't want to eagerly load all of them when many users will only ever use one in a given program. In this situation if I try to import SomeClass from some_library/__init__.py I will always trigger AnotherClass to be loaded as well, totally unnecessarily.
I wonder if this particular use-case is better served by a module subclass that does the lazy loading. Effectively, you get to break up a module into a package of N separately-loadable modules, and on first use of anything from a particular shard, that shard gets loaded.
This would be insanely useful. I really hope this sparks some discussion :)
Not sure HOW useful it would be in general, but yes, there definitely are use-cases for it. I'm pretty dubious about having a separate lookup on every name load just in case it's been lazily imported, though. ChrisA

I don't really have any knowledge of the Python interpreter's internals so I'm afraid any suggestions of mine on implementation details are going to be shots in the dark. That said, my intuition tells me that something similar to the proxy object pattern currently used by many lazy import implementations maybe could work here, with the interpreter creating some sort of 'lazy import proxy' object on this line rather than performing the import:
from .extremely_expensive_module lazy import SomeClass
And then when it actually is used in any way as part of *any code that isn't another lazy import statement* (normal import statements and even assignment statements that don't modify the object or attempt to access its attributes would count) it actually gets fully imported at that point in time, and all references anywhere in the application to this 'lazy import proxy' are swapped to point at the real object. You could have it so that if you tried to lazy import an object that is actually already a 'lazy import proxy' you would just get a reference to it rather than provisioning a new one. This would allow you to 'chain' lazy import statements across several modules to defer the import until the object (be it a module or a class) is actually used. Since I don't know enough about the implementation of the interpreter I couldn't really say if there's a way that normal objects could just be treated precisely as they are now with no loss of performance, while still giving 'lazy import objects' their special treatment. I suppose that's part of the point of this discussion. Hopefully people who know more can chime in. I definitely do think that there would be value in allowing all objects to be lazily imported via support for the 'from x import y' statement, not just modules, but I accept this is something people will have their own opinions on, and that mine isn't special. My earlier examples highlight situations where I think a lazy implementation of the normal 'import x' statement wouldn't be enough, and 'from x import y' would be needed. On Fri, Feb 5, 2021 at 2:13 PM Chris Angelico <rosuav@gmail.com> wrote:

On Fri, Mar 12, 2021 at 8:10 AM Matt del Valle <matthewgdv@gmail.com> wrote:
Seems fairly reasonable, I think. I'm still not convinced of the real value of this, given that you can always just import something inside a function or somesuch, and repeated imports are always going to be cheap; it's going to come down to use-cases. But if lazy imports were to be a thing, the semantics you describe aren't bad. ChrisA

Of course, you could encapsulate your imports inside of functions, that import only once called. However, there are plenty of cases where that is not practical. Probably one of countless examples of performing lazy imports, for precisely the reason of preventing circular imports: https://github.com/fondat/fondat-core/blob/main/fondat/lazy.py I would be happy if such a feature found its way into the language (in the manner like you suggest). Setting aside the complexities in the interpreter, it would possibly require a new keyword, which is a high bar to clear. On Thu, 2021-03-11 at 21:07 +0000, Matt del Valle wrote:

On Fri, Mar 12, 2021 at 8:24 AM Paul Bryan <pbryan@anode.ca> wrote:
Probably one of countless examples of performing lazy imports, for precisely the reason of preventing circular imports: https://github.com/fondat/fondat-core/blob/main/fondat/lazy.py
In the case of circular imports, at what point should the lazy import be resolved? ChrisA

Since this hasn't actually received moderator approval yet, I'd be happy if you declined this post and let me resubmit it without those typos, since they actually deal with specifics of the proposal and could be confusing if left in. I'm super sorry, my mistake! On Fri, Feb 5, 2021 at 12:39 PM Matt del Valle <matthewgdv@gmail.com> wrote:

On Sat, Feb 6, 2021 at 12:51 AM Matt del Valle <matthewgdv@gmail.com> wrote:
Unfortunately, [lazy importers are] fundamentally incapable of dealing with 'from x import y' syntax, because this is always loaded eagerly by the Python interpreter.
This is because it's fundamentally hard.
With the increasingly widespread adoption of type hints I think it is time to revive the proposal for lazy imports as an official language feature.
Are you writing this proposal assuming PEP 563 behaviour? If type hints are the primary justification, it may be less necessary. But lazy importing does have other value.
Hmm. Okay. Let's talk semantics, not syntax. What exactly should the interpreter do with this line? Is there an actual object in the module dictionary under the name "SomeClass", and if so, what sort of object? One possible meaning for this kind of "lazy from import" would be to maintain a table of import definitions, just like the module dictionary. (I would be inclined to say that this can ONLY be done at module level; lazy imports inside a class or function seem less useful and more hassle than they're worth. But if you disagree, feel free to expand this to other namespaces.) Whenever a module name is looked up, the interpreter looks first in the module dictionary, and then if it's not found, tries the lazy imports; upon finding the import definition, it removes it, performs the import, and stuffs the object into the module dictionary. This would be a fair amount of hassle, and it'd have a performance impact on EVERY module name lookup (and everything that goes beyond that to the builtins). But it would, I believe, give you an actual lazy import system. So the question is: how often is this going to be useful?
Where a library author wants to make several classes in their library's public API available at package level in the __init__.py, but doesn't want to eagerly load all of them when many users will only ever use one in a given program. In this situation if I try to import SomeClass from some_library/__init__.py I will always trigger AnotherClass to be loaded as well, totally unnecessarily.
I wonder if this particular use-case is better served by a module subclass that does the lazy loading. Effectively, you get to break up a module into a package of N separately-loadable modules, and on first use of anything from a particular shard, that shard gets loaded.
This would be insanely useful. I really hope this sparks some discussion :)
Not sure HOW useful it would be in general, but yes, there definitely are use-cases for it. I'm pretty dubious about having a separate lookup on every name load just in case it's been lazily imported, though. ChrisA

I don't really have any knowledge of the Python interpreter's internals so I'm afraid any suggestions of mine on implementation details are going to be shots in the dark. That said, my intuition tells me that something similar to the proxy object pattern currently used by many lazy import implementations maybe could work here, with the interpreter creating some sort of 'lazy import proxy' object on this line rather than performing the import:
from .extremely_expensive_module lazy import SomeClass
And then when it actually is used in any way as part of *any code that isn't another lazy import statement* (normal import statements and even assignment statements that don't modify the object or attempt to access its attributes would count) it actually gets fully imported at that point in time, and all references anywhere in the application to this 'lazy import proxy' are swapped to point at the real object. You could have it so that if you tried to lazy import an object that is actually already a 'lazy import proxy' you would just get a reference to it rather than provisioning a new one. This would allow you to 'chain' lazy import statements across several modules to defer the import until the object (be it a module or a class) is actually used. Since I don't know enough about the implementation of the interpreter I couldn't really say if there's a way that normal objects could just be treated precisely as they are now with no loss of performance, while still giving 'lazy import objects' their special treatment. I suppose that's part of the point of this discussion. Hopefully people who know more can chime in. I definitely do think that there would be value in allowing all objects to be lazily imported via support for the 'from x import y' statement, not just modules, but I accept this is something people will have their own opinions on, and that mine isn't special. My earlier examples highlight situations where I think a lazy implementation of the normal 'import x' statement wouldn't be enough, and 'from x import y' would be needed. On Fri, Feb 5, 2021 at 2:13 PM Chris Angelico <rosuav@gmail.com> wrote:

On Fri, Mar 12, 2021 at 8:10 AM Matt del Valle <matthewgdv@gmail.com> wrote:
Seems fairly reasonable, I think. I'm still not convinced of the real value of this, given that you can always just import something inside a function or somesuch, and repeated imports are always going to be cheap; it's going to come down to use-cases. But if lazy imports were to be a thing, the semantics you describe aren't bad. ChrisA

Of course, you could encapsulate your imports inside of functions, that import only once called. However, there are plenty of cases where that is not practical. Probably one of countless examples of performing lazy imports, for precisely the reason of preventing circular imports: https://github.com/fondat/fondat-core/blob/main/fondat/lazy.py I would be happy if such a feature found its way into the language (in the manner like you suggest). Setting aside the complexities in the interpreter, it would possibly require a new keyword, which is a high bar to clear. On Thu, 2021-03-11 at 21:07 +0000, Matt del Valle wrote:

On Fri, Mar 12, 2021 at 8:24 AM Paul Bryan <pbryan@anode.ca> wrote:
Probably one of countless examples of performing lazy imports, for precisely the reason of preventing circular imports: https://github.com/fondat/fondat-core/blob/main/fondat/lazy.py
In the case of circular imports, at what point should the lazy import be resolved? ChrisA
participants (3)
-
Chris Angelico
-
Matt del Valle
-
Paul Bryan