Declarative imports
This is an idea which has been brought up before, sometimes introduced as "heresy". But an interesting twist has surfaced now which is typing. But firstly, let me present the idea. It is very simple, that Python should have declarative imports, usable anywhere using a simple syntax, @<dotted-name>. For example, `some_regex = @re.compile(...)`. What happens then is that before anything else in that module, that symbol is imported: from re import compile as _mangled_re_compile It must be the very first thing (hoisting) because when else would it happen? It's been suggested before to have a shorthand syntax which does a dynamic import at the time of using it but this brings me to the twist: We want typing to pick up these imports. And this twist has a second leg which is that we often need to import symbols simply in order to type some argument type or return type. This leads to a great many more imports to type. (Nevermind that if you want to take typing further, abstract interfaces really make more sense rather than specific implementations, but the point is the same.) A situation where this would come in really handy is in scripting such as how we use Python in Apache Airflow to let users write out simple workflows. A workflow definition which could be a 5-liner quickly becomes a 20-liner – consider for example: default_args = { "start_date": @datetime.datetime(...) } It's a lot more ergonomic from a user perspective (well perhaps for some users and for some programs). Thoughts? Cheers
On Fri, 8 Apr 2022 at 18:29, Malthe <mborch@gmail.com> wrote:
This is an idea which has been brought up before, sometimes introduced as "heresy". But an interesting twist has surfaced now which is typing.
But firstly, let me present the idea. It is very simple, that Python should have declarative imports, usable anywhere using a simple syntax, @<dotted-name>.
For example, `some_regex = @re.compile(...)`.
What happens then is that before anything else in that module, that symbol is imported:
from re import compile as _mangled_re_compile
It must be the very first thing (hoisting) because when else would it happen?
JavaScript has this feature. ES6 modules are always imported declaratively and always before the rest of the script is executed. It is *extremely* annoying. Most of the time, it behaves the same way as Python's dynamic imports, but any time it doesn't, the "imports before the rest of the code" feature is nothing but hassle. Here's an alternative: Have a whitelist of modules that you make use of in this way. Any time you refer to a name that's on that whitelist and doesn't have an assigned meaning in the module, insert an import statement at the top of the script. It's not too hard to set up your editor to be able to do this (eg in SciTE, I can press Ctrl+1 to quickly run a script; a lot of editors let you configure a code formatter that runs automatically on save, which would also work), and then you retain full control of exactly where the importing happens. ChrisA
On Fri, 8 Apr 2022 at 09:29, Malthe <mborch@gmail.com> wrote:
A workflow definition which could be a 5-liner quickly becomes a 20-liner – consider for example:
default_args = { "start_date": @datetime.datetime(...) }
Are you exaggerating for effect here or would this *actually* just expand to from datetime import datetime default_args = { "start_date": datetime(...) } I'm not suggesting that there isn't a genuine benefit here, but your example doesn't demonstrate what it is. Paul
On Fri, 8 Apr 2022 at 08:51, Paul Moore <p.f.moore@gmail.com> wrote:
Are you exaggerating for effect here or would this *actually* just expand to
from datetime import datetime default_args = { "start_date": datetime(...) }
Yes – and of course that is just a snippet, an actual complete script will have lots of such imports. The point is that in a scripting situation (especially one in which you have many small scripts) the top-level import requirement for simple imports like `datetime` becomes rather verbose. Thanks
On Fri, 8 Apr 2022 at 10:27, Malthe <mborch@gmail.com> wrote:
On Fri, 8 Apr 2022 at 08:51, Paul Moore <p.f.moore@gmail.com> wrote:
Are you exaggerating for effect here or would this *actually* just expand to
from datetime import datetime default_args = { "start_date": datetime(...) }
Yes – and of course that is just a snippet, an actual complete script will have lots of such imports.
The point is that in a scripting situation (especially one in which you have many small scripts) the top-level import requirement for simple imports like `datetime` becomes rather verbose.
But Python's origin is in scripting situations, and explicit imports for scripts has always been both the normal approach, and one of Python's *strengths* ("Explicit is better than implicit"). Arguing that explicit imports are a bad thing in (general) scripts is arguing against decades of history and experience. If there's a reason why *your specific context* would benefit from an abbreviated form, you need to present it. But arguing that explicit imports are too verbose for all cases of scripting isn't going to fly, frankly. Paul
On Fri, Apr 08, 2022 at 08:24:40AM +0000, Malthe wrote:
But firstly, let me present the idea. It is very simple, that Python should have declarative imports,
I'm not sure I understand why you consider this "declarative". https://en.wikipedia.org/wiki/Declarative_programming As I see it, this proposal merely adds a unary operator for implicit imports.
For example, `some_regex = @re.compile(...)`.
What happens then is that before anything else in that module, that symbol is imported:
from re import compile as _mangled_re_compile
It must be the very first thing (hoisting) because when else would it happen?
On-demand. As you say yourself, this could be a dynamic import at the time of use.
It's been suggested before to have a shorthand syntax which does a dynamic import at the time of using it but this brings me to the twist:
We want typing to pick up these imports.
At the moment static type checkers have to look for two import statements: `import spam`, and `from spam import eggs`. With this proposal, they will still need to look for those two import statements, but they will also need to parse every expression looking for `@` as a unary operator: result = 2*x**3 - 3*y + @math.sin(x*y) - 5*x*y**2 This does not help typing at all. It just means the type-checker has to do more work to recognise imports.
And this twist has a second leg which is that we often need to import symbols simply in order to type some argument type or return type. This leads to a great many more imports to type.
Code is read much more than it is written. I would much prefer to have explicit imports (by convention, if not necessity) in one place at the top of the page, than to mystery symbols where the implicit import could be buried far away. Right now, this is an error: # Start of module. obj = wibble.foo because wibble is not a built-in nor a global (excluding weird shenanigans committed by other modules), so the name "wibble" doesn't exist. But with hoisting, that import could be *anywhere* in the file. Even in dead code. # Start of module. obj = wibble.foo ... ... # five pages of code later for a in obj.method(): while flag: if condition: @wibble That's right, with hoisting you can use a module before you import it. Mind. Blown. Have pity on beginners, casual Python programmers, and the people who have to help them. Don't implement this horror. If it were merely on-demand imports, then we would know that the import @wibble would have to appear in actual, executed code before you can use wibble. But with hoisting, we don't even have that promise. It is truly spooky action at a distance. And to save nothing but an import line.
A situation where this would come in really handy is in scripting such as how we use Python in Apache Airflow to let users write out simple workflows. A workflow definition which could be a 5-liner quickly becomes a 20-liner – consider for example:
default_args = { "start_date": @datetime.datetime(...) }
That's just a three-liner, which becomes a four-liner: import datetime default_args = { "start_date": datetime.datetime(...) } -- Steve
To me Steve nailed it... static type checker = lean and mean... On Fri, Apr 8, 2022, 5:03 AM Steven D'Aprano <steve@pearwood.info> wrote:
On Fri, Apr 08, 2022 at 08:24:40AM +0000, Malthe wrote:
But firstly, let me present the idea. It is very simple, that Python should have declarative imports,
I'm not sure I understand why you consider this "declarative".
https://en.wikipedia.org/wiki/Declarative_programming
As I see it, this proposal merely adds a unary operator for implicit imports.
For example, `some_regex = @re.compile(...)`.
What happens then is that before anything else in that module, that symbol is imported:
from re import compile as _mangled_re_compile
It must be the very first thing (hoisting) because when else would it happen?
On-demand. As you say yourself, this could be a dynamic import at the time of use.
It's been suggested before to have a shorthand syntax which does a dynamic import at the time of using it but this brings me to the twist:
We want typing to pick up these imports.
At the moment static type checkers have to look for two import statements: `import spam`, and `from spam import eggs`.
With this proposal, they will still need to look for those two import statements, but they will also need to parse every expression looking for `@` as a unary operator:
result = 2*x**3 - 3*y + @math.sin(x*y) - 5*x*y**2
This does not help typing at all. It just means the type-checker has to do more work to recognise imports.
And this twist has a second leg which is that we often need to import symbols simply in order to type some argument type or return type. This leads to a great many more imports to type.
Code is read much more than it is written. I would much prefer to have explicit imports (by convention, if not necessity) in one place at the top of the page, than to mystery symbols where the implicit import could be buried far away.
Right now, this is an error:
# Start of module. obj = wibble.foo
because wibble is not a built-in nor a global (excluding weird shenanigans committed by other modules), so the name "wibble" doesn't exist. But with hoisting, that import could be *anywhere* in the file. Even in dead code.
# Start of module. obj = wibble.foo ... ... # five pages of code later for a in obj.method(): while flag: if condition: @wibble
That's right, with hoisting you can use a module before you import it. Mind. Blown.
Have pity on beginners, casual Python programmers, and the people who have to help them. Don't implement this horror.
If it were merely on-demand imports, then we would know that the import @wibble would have to appear in actual, executed code before you can use wibble. But with hoisting, we don't even have that promise.
It is truly spooky action at a distance. And to save nothing but an import line.
A situation where this would come in really handy is in scripting such as how we use Python in Apache Airflow to let users write out simple workflows. A workflow definition which could be a 5-liner quickly becomes a 20-liner – consider for example:
default_args = { "start_date": @datetime.datetime(...) }
That's just a three-liner, which becomes a four-liner:
import datetime default_args = { "start_date": datetime.datetime(...) }
-- Steve _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GILL25XY... Code of Conduct: http://python.org/psf/codeofconduct/
On Fri, 8 Apr 2022, 09:30 Malthe, <mborch@gmail.com> wrote:
For example, `some_regex = @re.compile(...)`.
I like the idea of import expressions. I pitched it on Discourse recently: https://discuss.python.org/t/import-expressions/11582 However, I do not see hoisting as something that should be done by Python. In that thread it was suggested your IDE could do it, or maybe isort, which seems fine because it's at the programmer's discretion and ends up being explicit. But, your proposed syntax is not usable because it is ambiguous. Exactly what you propose is already used for decorators in a way that the parser would not be able to distinguish a decorator from an import expression. Consider: @ham.spam() # import expression or decorator? def eggs(): ... This currently parses as a decorator and for backwards compatibility that must not change, which means that import expressions would be usable in some contexts and not others purely based on what follows them.
On Fri, 8 Apr 2022 at 10:15, Daniel Pope <lord.mauve@gmail.com> wrote:
But, your proposed syntax is not usable because it is ambiguous. Exactly what you propose is already used for decorators in a way that the parser would not be able to distinguish a decorator from an import expression. Consider:
@ham.spam() # import expression or decorator?
def eggs(): ...
This currently parses as a decorator and for backwards compatibility that must not change, which means that import expressions would be usable in some contexts and not others purely based on what follows them.
Perhaps `some_regex = re::compile(r"...")` could work. That is, <dotted-path>::<name-to-import> to delineate the import. @breakfast.ham::spam def eggs(): pass Cheers
I do not think it's a "typing" improvement. I think what I like most in the proposal (which might be implemented differently - I am not too tied to this) is that it potentially brings huge optimisations in import times when currently a lot of imports are done unnecessarily early - slowing down startup times of tools that require parsing of python code. There are few examples of those: * Airflow DAGs - this is one of the problems we have in Airflow (and this is I think one of the reasons why Malthe brought it) is nicely described here https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#to... * Tools like click-complete (where autocompletion suggestions are generated based on the decorators of Python code) * Static Python Typing checks (mostly when you declare Type returned/used by function that is only used inside the function). In all those cases, you really do not care about some of the imported code that is in some kind of "internal scope" and your python parsing does not **really** need to perform the imports to do the job. * In Airflow DAGs - tasks which are defined as functions (and default arguments from the example from Malthe) are not executed at all when the DAGs are parsed by scheduler. The DAGs are parsed to get the structure of the DAG. In this case we could optimise the time it take to parse the DAG and generate the structure. * In Click-complete - we only really care about names of the parameters for functions, but static typing for the function parmeters or returned value are not needed at all to generate complete suggestions. In this case importing time is essential, because you want to provide autocomplete suggestions **really quickly** * In case of typing of parameters or return values, it leads quite often to circular imports generated. If - in your module - the only place you use a type is inside one or two functions, adding static typing involves adding top-level import for that type. And that has huge potential of generating circular imports. Before static typing in parameters or return values of those, local imports would do the job nicely, but with typing added to parameters and return values, you cannot do local imports in those functions that have those parameters or return values. This often leads to adding local imports elsewhere, but in fact a "logical" thing is that those imports should be local for the functions and any of the scopes that the functions are called. Top-level imports in this case are causing unnecessary cross-module dependencies and circular imports. And some of those problems can currently be solved with local imports - but not all (default_values case by Malthe cannot). I think this idea has really good "need" behind. Maybe the way to implement could be improved on, but I see clearly the use cases and needs the "general idea" aims to solve.
On Fri, 8 Apr 2022 at 11:26, Malthe <mborch@gmail.com> wrote:
Perhaps `some_regex = re::compile(r"...")` could work.
That is, <dotted-path>::<name-to-import> to delineate the import.
As I mentioned in the Discourse thread, this syntax chimes with Rust (and C++, and also PHP, triggering fond memories of Paamayim Nekudotayim). There are two problems, sort of on the minor side: First, there's one place in Python where this can occur and that is in slices: xs[start::step] means the slice from start to the end with a step of step. But xs[(module::name)] is currently a syntax error, so you could just require parentheses to disambiguate the case. Still, it is confusing if you write something slightly more complex like xs[1 + math::factorial(n)] and you get NameError: name 'math' is not defined. Secondly, it precludes importing just a module object, you have to import a name from a module. Maybe using a module object in an expression is niche and can be disregarded. Or a solution could be to allow an expression like (re::) to import a module object, which I guess we could get used to. Also fun would be to consider what ::name does - access to current globals without the global keyword? I think this discussion is better suited to the python-ideas mailing list, or the Discourse thread I linked earlier.
I've read the rest of the thread so far, and agree strongly that we can't do this at the language/runtime level. However ... On 4/8/2022 9:24 AM, Malthe wrote:
This is an idea which has been brought up before, sometimes introduced as "heresy". But an interesting twist has surfaced now which is typing.
[SNIP]
We want typing to pick up these imports. And this twist has a second leg which is that we often need to import symbols simply in order to type some argument type or return type. This leads to a great many more imports to type.
Type checkers can do whatever they like. The syntax has to remain a valid Python expression, but the semantics of it can include implicit imports if that's what you want it to do. I'd certainly make use of it on those occasions I actually put type annotations in code (rather than .pyi files). But it doesn't need the runtime to do anything differently if the module isn't really going to be used at that point. Cheers, Steve
On Fri, 8 Apr 2022 at 12:23, Steve Dower <steve.dower@python.org> wrote:
I've read the rest of the thread so far, and agree strongly that we can't do this at the language/runtime level.
You mean the hoisting, right? I don't see any reason why an import expression without hoisting would be impractical. But I'd like to hear your thoughts if you think it is. Desirability on the other hand is subjective. I think I actually do desire it, others are not bothered. I don't see strong arguments as to why you definitely wouldn't want it in your language. Design is hard, but designing this is definitely not as hard as designing match/case or except* statements.
On Fri, 8 Apr 2022 at 12:57, Daniel Pope <lord.mauve@gmail.com> wrote:
On Fri, 8 Apr 2022 at 12:23, Steve Dower <steve.dower@python.org> wrote:
I've read the rest of the thread so far, and agree strongly that we can't do this at the language/runtime level.
You mean the hoisting, right?
I don't see any reason why an import expression without hoisting would be impractical. But I'd like to hear your thoughts if you think it is.
Desirability on the other hand is subjective. I think I actually do desire it, others are not bothered. I don't see strong arguments as to why you definitely wouldn't want it in your language.
OK, I'll be explicit. I don't want this in Python. Having imports at the top of a file makes it much easier to see your dependencies. The current language features, combined with community wide style guides that discourage the use of local imports, make this straightforward, while still providing means of doing local imports if needed. This proposal isn't just about having a new syntax to do an on-demand import. It's also about normalising the idea that people can pull code from other modules without declaring the intent to do so up front. While I don't dispute that in some circumstances (notably the REPL, and throwaway scripts[1]) not having to add import statements would be nice, I don't see how we'd limit usage to cases like that - and I *really* don't want to have to work out where some weird inline import happened while debugging a priority 1 bug in a 10,000 line code base at 2am... Also, it should be possible to do something like this using the existing import machinery: from magic_import import on_demand_loader as OD ... # Many lines of code later default_args = { "start_date": OD.datetime.datetime(...) } Trivial proof of concept implementation: class OD: def __getattr__(self, name): return __import__(name) OD = OD() print(OD.datetime.datetime.now()) Paul [1] Of course, today's throwaway script usually turns out to be next month's key component in a mission-critical software stack :-(
On 4/8/2022 12:51 PM, Daniel Pope wrote:
On Fri, 8 Apr 2022 at 12:23, Steve Dower <steve.dower@python.org> wrote:
I've read the rest of the thread so far, and agree strongly that we can't do this at the language/runtime level.
You mean the hoisting, right?
I don't see any reason why an import expression without hoisting would be impractical. But I'd like to hear your thoughts if you think it is.
Sure, __import__() or better yet, importlib.util.import_module() are perfectly good ways to do an import in an expression. Both have exactly the right amount of code-smell as well, and only a few surprising ways to interfere with the behaviour of apparently completely unrelated code ;) As others pointed out, implicitly defining variables that outlive the evaluation of an expression is incredibly messy. It was bad enough getting assignment expressions defined well enough to (a) not be a terrible footgun and (b) be possible for the average developer to predict what will happen when they're used (arguably this isn't true either, but the "average" developer probably isn't doing things that will hit the edge cases). We don't want to have to figure something like that out again. Certainly not for the sake of typing - the SC has already ruled that language development won't be driven by the convenience of static type users. Cheers, Steve
While it's different than you proposal, some people may like this magic "smart imports" project which lazily imports a module when it's used :-) Project: https://pypi.org/project/smart-imports/ Replace: --- import math from my_project import calc # 100500 other imports def my_code(argument, function=calc): return math.log(function(argument)) --- with: --- import smart_imports smart_imports.all() # no any other imports def my_code(argument, function=calc): return math.log(function(argument)) --- I don't like it :-) I prefer explicit imports at the top! Victor
The interesting idea here seems to make "lazy imports" easier to implement by making them explicit in the code. So far, most lazy import frameworks for Python have done hacks with `__getattribute__` overrides. IIRC the Cinder version even modifies the bytecode and/or the interpreter. Disregarding the specific notation proposed, *if* people would be willing to mark the points where they expect lazy imports explicitly, that would make implementation much simpler. The argument that "imports on top" makes code more readable seems pretty weak to me. The current hacks to speed up startup already violate this rule (imports inside functions), and in most cases I start reading or writing code in the middle of a file (having gotten there via a search in my editor) and the meaning of an import is either obvious (e.g. re.match(...)) or requires another annoying search to find the definition of a certain unknown variable. Tools can easily show all imports a module does. The key questions to me are - What should the notation be? - Will users be willing to use it? --Guido On Fri, Apr 8, 2022 at 1:26 AM Malthe <mborch@gmail.com> wrote:
This is an idea which has been brought up before, sometimes introduced as "heresy". But an interesting twist has surfaced now which is typing.
But firstly, let me present the idea. It is very simple, that Python should have declarative imports, usable anywhere using a simple syntax, @<dotted-name>.
For example, `some_regex = @re.compile(...)`.
What happens then is that before anything else in that module, that symbol is imported:
from re import compile as _mangled_re_compile
It must be the very first thing (hoisting) because when else would it happen? It's been suggested before to have a shorthand syntax which does a dynamic import at the time of using it but this brings me to the twist:
We want typing to pick up these imports. And this twist has a second leg which is that we often need to import symbols simply in order to type some argument type or return type. This leads to a great many more imports to type.
(Nevermind that if you want to take typing further, abstract interfaces really make more sense rather than specific implementations, but the point is the same.)
A situation where this would come in really handy is in scripting such as how we use Python in Apache Airflow to let users write out simple workflows. A workflow definition which could be a 5-liner quickly becomes a 20-liner – consider for example:
default_args = { "start_date": @datetime.datetime(...) }
It's a lot more ergonomic from a user perspective (well perhaps for some users and for some programs).
Thoughts?
Cheers _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UX6EJHLJ... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Fri, 8 Apr 2022 at 16:40, Guido van Rossum <guido@python.org> wrote:
The interesting idea here seems to make "lazy imports" easier to implement by making them explicit in the code. So far, most lazy import frameworks for Python have done hacks with `__getattribute__` overrides. IIRC the Cinder version even modifies the bytecode and/or the interpreter. Disregarding the specific notation proposed, *if* people would be willing to mark the points where they expect lazy imports explicitly, that would make implementation much simpler.
Actually, to me the interesting idea is not so much lazy imports – I think they should not be lazy, at least that was my initial thought. I think they should be immediately resolved before anything else in that module: 1. This would settle any discussion about performance impact (there wouldn't be any). 2. This would enable IDEs, typers and other tooling to know the type using existing import logic. 3. Catch errors early! Notation is hard of course. Would users be willing to use it? I think so, at least in Rust, people do use it and I think just in the right places – typically for imports that are either "canonicalized" such as `re.compile` is in Python, or used just once. Cheers
Hi Malthe, On Fri, Apr 8, 2022 at 12:04 PM Malthe <mborch@gmail.com> wrote:
Actually, to me the interesting idea is not so much lazy imports – I think they should not be lazy, at least that was my initial thought. I think they should be immediately resolved before anything else in that module:
I'm +0.25 on your idea as simply a streamlined syntax for inline imports (given actually finding an appropriate syntax, which I haven't thought much about; @ probably doesn't work due to the conflict with decorator syntax, but there might be other options.). If it existed I would probably use it occasionally, but I don't feel a strong need for it. But I think your proposal is much stronger if you eliminate the hoisting from it; with the hoisting I'd be -1. Out-of-source-order execution like this is just quite surprising in the context of Python.
1. This would settle any discussion about performance impact (there wouldn't be any).
If the inline import is actually a performance problem because a certain code path is very hot, the solution is simple: don't use the inline import there, use a top-of-module import instead.
2. This would enable IDEs, typers and other tooling to know the type using existing import logic.
I don't think it enables any such thing. Static-analysis tooling has only the source code to work with, runtime behavior doesn't affect it. If the runtime executes these imports out-of-order, that won't make the slightest difference to how easily IDEs and type checkers can analyze the source code.
3. Catch errors early!
The very strong precedent in Python is that errors in code are caught when the code runs, and the code runs more or less when you'd expect it to, in source order. If you want to catch errors earlier, use a static analysis tool to help catch them. Carl
You only get the ease-of-implementation benefit if you are willing to explicitly mark every _use_ of a lazy-imported name as special (and give the fully qualified name at every usage site). This is rather more cumbersome (assuming multiple uses in a module) than just explicitly marking an import as lazy in one location and then using the imported name in multiple places normally. Other "lazy import" solutions are trying to solve a problem where you want the name to be usable (without special syntax or marking) in many different places in a module, and visible in the module namespace always -- but not actually imported until someone accesses/uses it. The difficulty arises because in this case you need some kind of placeholder for the "deferred import", but you need to avoid this "deferred object" escaping and becoming visible to Python code without being resolved first. Explicitly marking which imports are lazy is fine if you want it (it's just a matter of syntax), but it doesn't do anything to solve the problem of allowing usage of the lazy-imported name to be transparent. I agree that the idea that top-of-module imports help readability is overstated; it sounds slightly Stockholm-syndrome-ish to me :) Top-of-module imports are frankly a pain to maintain and a pain to read (because they are often distant from the uses of the names). But they are a necessary evil if you want a) namespaces and b) not constantly retyping fully-qualified names at every usage site. Python is pretty committed to namespaces at this point (and I wouldn't want to change that), so that leaves the choice between top-of-module imports vs fully qualifying every use of every name; pick your poison. (Inline imports in a scope with multiple uses are a middle ground.) Carl
An interesting point in the lazy imports design space that I hadn't previously considered could be: - lazy imports are explicitly marked and usage of the imported name within the module is transparent, but - lazily imported names are _not_ visible in the module namespace; they can't be accessed by other modules or re-exported; they are internal-use-only within the module This compromise would, I think, make it possible to implement lazy imports entirely in the compiler (effectively as syntax sugar for an inline import at every usage site), which is definitely an implementation improvement. I think in practice explicitly marking lazy imports would make it somewhat harder to gain the benefits of lazy imports for e.g. speeding up startup time in a large CLI, compared to an implicit/automatic approach. But still could be usable to get significant benefits. Carl
Start up overhead due to imports is a real problem for some class of applications, e.g. CLIs, and I’ve seen a lot of hacks implemented to get Python CLIs to be more responsive. E.g. getting from invocation to —help output is a major UX problem. It’s often more complicated than just imports alone though. Expensive module scope initializations and decorators contribute to this problem. Python start up time is one of the main drivers for rewriting CLIs in Go and other languages where I work. Note that this is much less of a problem for things like web services or other long running applications because that start up time is either amortized over the lifetime of the application, or aren’t directly visible to the end user. Lazy imports *might* help with this and seems aligned with the common trick of moving imports into functions rather than at module scope. Faster CPython might help too. But these all feel like they aren’t tackling the start up problem head on[1]. Lots of ideas have been discussed over the years (I remember some in-depth ones at the Microsoft core sprint a few years ago), and I’m sure there are all kinds of other tricks that people use. However, if start up time isn’t a direct benefit of on-demand imports (a.k.a. declarative imports), I’m not sure how actually useful or used they will be. I dunno, top-of-module imports never really bothered me that much. -Barry [1] I could be wrong about Faster CPython; ISTR there are some tickets on that project’s tracker that talk about start up times.
On Apr 8, 2022, at 09:40, Guido van Rossum <guido@python.org> wrote:
The interesting idea here seems to make "lazy imports" easier to implement by making them explicit in the code. So far, most lazy import frameworks for Python have done hacks with `__getattribute__` overrides. IIRC the Cinder version even modifies the bytecode and/or the interpreter. Disregarding the specific notation proposed, *if* people would be willing to mark the points where they expect lazy imports explicitly, that would make implementation much simpler.
The argument that "imports on top" makes code more readable seems pretty weak to me. The current hacks to speed up startup already violate this rule (imports inside functions), and in most cases I start reading or writing code in the middle of a file (having gotten there via a search in my editor) and the meaning of an import is either obvious (e.g. re.match(...)) or requires another annoying search to find the definition of a certain unknown variable. Tools can easily show all imports a module does.
The key questions to me are - What should the notation be? - Will users be willing to use it?
--Guido
On Fri, Apr 8, 2022 at 1:26 AM Malthe <mborch@gmail.com> wrote: This is an idea which has been brought up before, sometimes introduced as "heresy". But an interesting twist has surfaced now which is typing.
But firstly, let me present the idea. It is very simple, that Python should have declarative imports, usable anywhere using a simple syntax, @<dotted-name>.
For example, `some_regex = @re.compile(...)`.
What happens then is that before anything else in that module, that symbol is imported:
from re import compile as _mangled_re_compile
It must be the very first thing (hoisting) because when else would it happen? It's been suggested before to have a shorthand syntax which does a dynamic import at the time of using it but this brings me to the twist:
We want typing to pick up these imports. And this twist has a second leg which is that we often need to import symbols simply in order to type some argument type or return type. This leads to a great many more imports to type.
(Nevermind that if you want to take typing further, abstract interfaces really make more sense rather than specific implementations, but the point is the same.)
A situation where this would come in really handy is in scripting such as how we use Python in Apache Airflow to let users write out simple workflows. A workflow definition which could be a 5-liner quickly becomes a 20-liner – consider for example:
default_args = { "start_date": @datetime.datetime(...) }
It's a lot more ergonomic from a user perspective (well perhaps for some users and for some programs).
Thoughts?
Cheers _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UX6EJHLJ... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) Pronouns: he/him (why is my pronoun here?) _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BTWPAJRV... Code of Conduct: http://python.org/psf/codeofconduct/
Hi Barry, On Fri, Apr 8, 2022 at 12:44 PM Barry Warsaw <barry@python.org> wrote:
Start up overhead due to imports is a real problem for some class of applications, e.g. CLIs, and I’ve seen a lot of hacks implemented to get Python CLIs to be more responsive. E.g. getting from invocation to —help output is a major UX problem.
Definitely, we have this same problem, and also the same symptom of people pushing hard to rewrite Python CLIs in Go for this reason.
It’s often more complicated than just imports alone though. Expensive module scope initializations and decorators contribute to this problem.
One of our projects that can prevent much of this expensive work being done at import time is Strict Modules[1]. Currently it's only available as part of Cinder, though we're hoping to make it pip-installable as part of our project to make Cinder's features more easily accessible. Our experience in practice, though, has been that universally lazy imports is somewhat easier to adopt than Strict Modules, and has had a much bigger overall impact on reducing startup time for big CLIs (and a big web server too; as you note it's not as serious an issue for a web server in production, but restart time still does make a difference to dev speed / experience.) Removing slow stuff happening at import time helps, but it'll never match the speed of not doing the import at all! We've seen startup time improvements up to 70% in real-world CLIs just by making imports lazy. We've also opened an issue to discuss the possibility of upstreaming this. [2] [1] https://github.com/facebookincubator/cinder/#strict-modules [2] https://bugs.python.org/issue46963
Thanks Carl,
On Apr 9, 2022, at 08:25, Carl Meyer <carl@oddbird.net> wrote:
Our experience in practice, though, has been that universally lazy imports is somewhat easier to adopt than Strict Modules, and has had a much bigger overall impact on reducing startup time for big CLIs (and a big web server too; as you note it's not as serious an issue for a web server in production, but restart time still does make a difference to dev speed / experience.)
Excellent point about the impact of restarts and development time. That’s been an issue for us a bit, but not an overwhelming motivation to rewrite in other languages[1].
Removing slow stuff happening at import time helps, but it'll never match the speed of not doing the import at all! We've seen startup time improvements up to 70% in real-world CLIs just by making imports lazy. We've also opened an issue to discuss the possibility of upstreaming this. [2]
[1] https://github.com/facebookincubator/cinder/#strict-modules [2] https://bugs.python.org/issue46963
Post-GH-issues-migration link for the issue: https://github.com/python/cpython/issues/91119 I’ve put some questions and comments there, but I’m also really curious about the technical details for your lazy imports. Have you gotten as far as thinking about a PR or PEP? -Barry [1] Not that there aren’t other reasons folks give for rewriting, such as multicore performance, ecosystem alignment (e.g. SREs being more comfortable in Go), etc.
Breaking out the discussion about lazy imports. It seems somewhat OT in the declarative imports thread. On Mon, Apr 11, 2022 at 10:50 AM Barry Warsaw <barry@python.org> wrote:
Thanks Carl,
On Apr 9, 2022, at 08:25, Carl Meyer <carl@oddbird.net> wrote:
Our experience in practice, though, has been that universally lazy imports is somewhat easier to adopt than Strict Modules, and has had a much bigger overall impact on reducing startup time for big CLIs (and a big web server too; as you note it's not as serious an issue for a web server in production, but restart time still does make a difference to dev speed / experience.)
Excellent point about the impact of restarts and development time. That’s been an issue for us a bit, but not an overwhelming motivation to rewrite in other languages[1].
Lazy imports had been very significant to both CLI startup time, as well as service reload time during local development - we have more details in the docs [1], and a related blog post coming up soon.
Removing slow stuff happening at import time helps, but it'll never match the speed of not doing the import at all! We've seen startup time improvements up to 70% in real-world CLIs just by making imports lazy. We've also opened an issue to discuss the possibility of upstreaming this. [2]
[1] https://github.com/facebookincubator/cinder/#strict-modules [2] https://bugs.python.org/issue46963
Post-GH-issues-migration link for the issue: https://github.com/python/cpython/issues/91119
I’ve put some questions and comments there, but I’m also really curious about the technical details for your lazy imports. Have you gotten as far as thinking about a PR or PEP?
Yes and Yes (at least for the "thinking about" part). Germán Méndez Bravo (Kronuz) will be drafting a PEP, and is leading work internally to port this from 3.8 (first to 3.10, then to the cpython main branch). We are planning to get started on this during PyCon sprints, it would be nice to connect in-person with anyone who's interested in this and happens to be in the sprints too!
-Barry
[1] Not that there aren’t other reasons folks give for rewriting, such as multicore performance, ecosystem alignment (e.g. SREs being more comfortable in Go), etc.
Itamar. [1] https://github.com/facebookincubator/cinder/blob/cinder/3.8/CinderDoc/lazy_i...
On Mon, Apr 11, 2022 at 1:07 PM Itamar O <itamarost@gmail.com> wrote:
Breaking out the discussion about lazy imports. It seems somewhat OT in the declarative imports thread.
On Mon, Apr 11, 2022 at 10:50 AM Barry Warsaw <barry@python.org> wrote:
Thanks Carl,
On Apr 9, 2022, at 08:25, Carl Meyer <carl@oddbird.net> wrote:
Our experience in practice, though, has been that universally lazy imports is somewhat easier to adopt than Strict Modules, and has had a much bigger overall impact on reducing startup time for big CLIs (and a big web server too; as you note it's not as serious an issue for a web server in production, but restart time still does make a difference to dev speed / experience.)
Excellent point about the impact of restarts and development time. That’s been an issue for us a bit, but not an overwhelming motivation to rewrite in other languages[1].
Lazy imports had been very significant to both CLI startup time, as well as service reload time during local development - we have more details in the docs [1], and a related blog post coming up soon.
Removing slow stuff happening at import time helps, but it'll never match the speed of not doing the import at all! We've seen startup time improvements up to 70% in real-world CLIs just by making imports lazy. We've also opened an issue to discuss the possibility of upstreaming this. [2]
[1] https://github.com/facebookincubator/cinder/#strict-modules [2] https://bugs.python.org/issue46963
Post-GH-issues-migration link for the issue: https://github.com/python/cpython/issues/91119
I’ve put some questions and comments there, but I’m also really curious about the technical details for your lazy imports. Have you gotten as far as thinking about a PR or PEP?
Yes and Yes (at least for the "thinking about" part). Germán Méndez Bravo (Kronuz) will be drafting a PEP, and is leading work internally to port this from 3.8 (first to 3.10, then to the cpython main branch). We are planning to get started on this during PyCon sprints, it would be nice to connect in-person with anyone who's interested in this and happens to be in the sprints too!
I am and I will be.
-Barry
[1] Not that there aren’t other reasons folks give for rewriting, such as multicore performance, ecosystem alignment (e.g. SREs being more comfortable in Go), etc.
Itamar.
[1] https://github.com/facebookincubator/cinder/blob/cinder/3.8/CinderDoc/lazy_i... _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NIXH574H... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Apr 11, 2022, at 13:10, Guido van Rossum <guido@python.org> wrote:
On Mon, Apr 11, 2022 at 1:07 PM Itamar O <itamarost@gmail.com> wrote: Breaking out the discussion about lazy imports. It seems somewhat OT in the declarative imports thread.
Yes and Yes (at least for the "thinking about" part). Germán Méndez Bravo (Kronuz) will be drafting a PEP, and is leading work internally to port this from 3.8 (first to 3.10, then to the cpython main branch). We are planning to get started on this during PyCon sprints, it would be nice to connect in-person with anyone who's interested in this and happens to be in the sprints too!
I am and I will be.
Apologies for the icky quoting, but yay! It’s great to hear your intentions, and like Guido, I would like to connect on this at Pycon as well. -Barry
On Fri, 8 Apr 2022, 17:44 Guido van Rossum, <guido@python.org> wrote:
The interesting idea here seems to make "lazy imports" easier to implement by making them explicit in the code. So far, most lazy import frameworks for Python have done hacks with `__getattribute__` overrides.
The value is more than ease of implementation. Having syntax for import expressions makes them statically analysable, which is needed for type checkers and IDE autocompletion. But I also see value in being able to type out code that uses modules not yet imported without breaking my flow to add an import statement. I don't yet trust IDEs to do this because I've been bitten by them doing so incorrectly in the past. The key questions to me are
- What should the notation be?
I would like to bid again for (import package.module) as an expression. Instead of doing the import and assigning package to a variable package it would evaluate to the module object package.module. The `as` form is not needed because no name is assigned and the `from` form isn't as valuable because you can just use attribute access afterwards. It isn't terse but it does make use of the import keyword and is thus instantly recognisable. It is even syntax highlighted correctly by much existing software. If we're using the import keyword then I think it has to look like this. But I concede that it isn't particularly elegant to type hint things with (import collections.abc).Mapping ...but not so inelegant that I couldn't see myself using it for a few one-off imports per module. A quirk is that it means there's a big difference between the statements import foo and (import foo) because one assigns a variable. I don't mind that; I don't think it is too ambiguous to a reader.
On Sun, Apr 10, 2022 at 2:31 AM Daniel Pope <lord.mauve@gmail.com> wrote:
On Fri, 8 Apr 2022, 17:44 Guido van Rossum, <guido@python.org> wrote:
The interesting idea here seems to make "lazy imports" easier to implement by making them explicit in the code. So far, most lazy import frameworks for Python have done hacks with `__getattribute__` overrides.
The value is more than ease of implementation. Having syntax for import expressions makes them statically analysable, which is needed for type checkers and IDE autocompletion.
This has been brought up a few times and I don't get it. Currently a use of an imported module is perfectly analyzable by all the static type checkers I know of (e.g. mypy, Pyre, pyright). For the static analyzer it makes no difference if I have import re . . . def foo(x): if re.match(r"blah", x): ... or the hypothetical inline form: def foo(x): if @re.match(r"blah", x): ...
But I also see value in being able to type out code that uses modules not yet imported without breaking my flow to add an import statement. I don't yet trust IDEs to do this because I've been bitten by them doing so incorrectly in the past.
I have too. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Sun, 10 Apr 2022, 15:53 Guido van Rossum, <guido@python.org> wrote:
On Sun, Apr 10, 2022 at 2:31 AM Daniel Pope <lord.mauve@gmail.com> wrote:
On Fri, 8 Apr 2022, 17:44 Guido van Rossum, <guido@python.org> wrote:
The interesting idea here seems to make "lazy imports" easier to implement by making them explicit in the code. So far, most lazy import frameworks for Python have done hacks with `__getattribute__` overrides.
The value is more than ease of implementation. Having syntax for import expressions makes them statically analysable, which is needed for type checkers and IDE autocompletion.
This has been brought up a few times and I don't get it. Currently a use of an imported module is perfectly analyzable by all the static type checkers I know of (e.g. mypy, Pyre, pyright).
I was comparing a hypothetical import expression syntax with alternatives like __getattribute__ hacks, which I take to mean somemagicobject.package.module.attribute, or as other have suggested, importlib.import_module(). I believe those are not statically analysable at least without special casing them in the type checker. An import expression would be just as statically analysable as a statement, while being more ergonomic in certain situations.
On 10/04/2022 15:52, Guido van Rossum wrote:
On Sun, Apr 10, 2022 at 2:31 AM Daniel Pope <lord.mauve@gmail.com> wrote:
On Fri, 8 Apr 2022, 17:44 Guido van Rossum, <guido@python.org> wrote:
The interesting idea here seems to make "lazy imports" easier to implement by making them explicit in the code. So far, most lazy import frameworks for Python have done hacks with `__getattribute__` overrides.
The value is more than ease of implementation. Having syntax for import expressions makes them statically analysable, which is needed for type checkers and IDE autocompletion.
This has been brought up a few times and I don't get it. Currently a use of an imported module is perfectly analyzable by all the static type checkers I know of (e.g. mypy, Pyre, pyright). For the static analyzer it makes no difference if I have
import re . . . def foo(x): if re.match(r"blah", x): ...
or the hypothetical inline form:
def foo(x): if @re.match(r"blah", x): ...
But I also see value in being able to type out code that uses modules not yet imported without breaking my flow to add an import statement. I don't yet trust IDEs to do this because I've been bitten by them doing so incorrectly in the past.
I have too.
I see no reason why @re.match(...) couldn't just be equivalent to __import__("re").match(...) (or a more optimised compiled version of that). Any side-effects of importing the module (of which there probably shouldn't be any for most modules) just happen at whenever the first time the @re.match(...) is evaluated. After the first call, and unless __import__ is overloaded, I also don't think @re.match(...) would have to be significantly slower than re.match(...). The latter already has to look up a global variable (assuming the module was imported globally, which is almost always the case). I don't think it would be too hard to optimise and Quicken.
On Sun, 10 Apr 2022 at 09:31, Daniel Pope <lord.mauve@gmail.com> wrote:
I would like to bid again for (import package.module) as an expression. Instead of doing the import and assigning package to a variable package it would evaluate to the module object package.module.
I like this proposal and I agree with previous posters about hoisting being undesirable. And if lazy importing is slow then that is probably something that can be optimized away; of course in tight loops or performance-sensitive code you wouldn't use this functionality. But perhaps using "from" makes the behavior more clear (and it is shorter): start_date = from datetime.datetime(2012, 1, 6) I think the behavior could be to simply keep importing until we have a non-module value, rather than rely on putting the import name in parentheses. Cheers
On Sun, Apr 10, 2022 at 2:39 AM Daniel Pope <lord.mauve@gmail.com> wrote:
On Fri, 8 Apr 2022, 17:44 Guido van Rossum, <guido@python.org> wrote:
The interesting idea here seems to make "lazy imports" easier to implement by making them explicit in the code. So far, most lazy import frameworks for Python have done hacks with `__getattribute__` overrides.
The value is more than ease of implementation. Having syntax for import expressions makes them statically analysable, which is needed for type checkers and IDE autocompletion.
But I also see value in being able to type out code that uses modules not yet imported without breaking my flow to add an import statement. I don't yet trust IDEs to do this because I've been bitten by them doing so incorrectly in the past.
The key questions to me are
- What should the notation be?
I would like to bid again for (import package.module) as an expression. Instead of doing the import and assigning package to a variable package it would evaluate to the module object package.module.
That is an extremely subtle shift for what `import x.y` does compared to `(import x.y)`. That requires a context switch of not only seeing `import` in an expression context, but that the statement also acts differently in terms of what is returned by the equivalent statement. I really don't to try and teach that distinction to a newcomer. And I don't think the ergonomics are great enough to warrant the context switch. Plus you can do this today with imortlib.import_module(). Unless you're suggesting the name also be bound in the scope it's run in? In which case that's `(abc := importlib.import_module("collections.abc")).Mapping`. -Brett
The `as` form is not needed because no name is assigned and the `from` form isn't as valuable because you can just use attribute access afterwards.
It isn't terse but it does make use of the import keyword and is thus instantly recognisable. It is even syntax highlighted correctly by much existing software. If we're using the import keyword then I think it has to look like this.
But I concede that it isn't particularly elegant to type hint things with
(import collections.abc).Mapping
...but not so inelegant that I couldn't see myself using it for a few one-off imports per module.
A quirk is that it means there's a big difference between the statements
import foo
and
(import foo)
because one assigns a variable. I don't mind that; I don't think it is too ambiguous to a reader.
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/T6V7WZAF... Code of Conduct: http://python.org/psf/codeofconduct/
On Mon, Apr 11, 2022 at 2:03 PM Brett Cannon <brett@python.org> wrote:
That is an extremely subtle shift for what `import x.y` does compared to `(import x.y)`. That requires a context switch of not only seeing `import` in an expression context, but that the statement also acts differently in terms of what is returned by the equivalent statement. I really don't to try and teach that distinction to a newcomer.
Frankly, there are already an overwhelming-to-a-newcomer number of ways to import modules. We really don't want nore! -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Fri, Apr 8, 2022 at 1:26 AM Malthe <mborch@gmail.com> wrote:
This is an idea which has been brought up before, sometimes introduced as "heresy". But an interesting twist has surfaced now which is typing.
What for? To save a few keystrokes? Can't some IDE's add the import for you? Please don't drag Python in the direction of line noise.
participants (17)
-
Barry Warsaw -
Brett Cannon -
Carl Meyer -
Chris Angelico -
Christopher Barker -
Dan Stromberg -
Daniel Pope -
Guido van Rossum -
Itamar O -
Jarek Potiuk -
Malthe -
Patrick Reader -
Paul Moore -
Sasha Kacanski -
Steve Dower -
Steven D'Aprano -
Victor Stinner