On 2022-06-21 13:53, David Mertz, Ph.D. wrote:
Here is a very rough draft of an idea I've floated often, but not with much specification. Take this as "ideas" with little firm commitment to details from me. PRs, or issues, or whatever, can go to https://github.com/DavidMertz/peps/blob/master/pep-9999.rst as well as mentioning them in this thread.
After looking at this a bit more (with the newer revisions) and following the discussion I think this proposal doesn't really achieve what I would want from deferred evaluation. That may be because what I want is unreasonable, but, well, such is life. :-)
First, it is not clear to me what the real point of this type of deferred evaluation is. The PEP has a "motivation" section that makes a link to Haskell and Dask, but as far as I can see it doesn't explicitly say what is gained by introducing this new form of lazy evaluation into Python.
In particular (as I think someone else mentioned on this thread), Dask-style deferred computations are based on explicitly evaluating the thunk, whereas this proposal would automatically evaluate it on reference. I think that in practice this would make many Dask-style usages unwieldy because you would have to keep repeating the `later` keyword in order to gradually build up a complex deferred computation over multiple statements. For such cases it is more natural to explicitly evaluate the whole thing at the end, rather than explicitly not evaluate it until then.
In theory there could be performance gains, as mentioned in the PEP. But again I don't see a huge advantage to this in Python. It might make sense in Haskell where laziness is built into the language at a fundamental level. But in Python, where eager evaluation is the norm, it again seems more natural to me to use "explicit laziness" (i.e., explicit rather than automatic evaluation). It seems rather unusual to have cases where some variable or function argument might contain either a computationally cheap expression or an expensive one; usually for those types of applications you know where you might do something expensive. And even if you don't, I see little downside to requiring an explicit "eval this thunk" step at the end.
In contrast, what I would want out of deferred evaluation is precisely the ability to evaluate the deferred expression in the *evaluating* scope (not the definition scope) --- or in a custom provided namespace. Whether this evaluation is implicit or explicit is less important to me than the ability to control the scope in which it occurs. As others mentioned in early posts on this thread, this could complicate things too much to be feasible, but without it I don't really see the point.
The reason this is key for me is that I'm focused on a different set of motivating use cases. What I'm interested in is "query-type" situations where you want to pass an expression to some sort of "query engine", which will evaluate the expression in a namespace representing the dataset to be queried. One example would be SQL queries, where it would be nice to be able to do things like:
my_sql_table.select(where=thunk (column1 == 2 and column2 > 5))
Likewise this would make pandas indexing less verbose, turning it from:
df[(df.column1 == 2) & (df.column2 > 5)]
df[(column1 == 2) & (column2 > 3)]
or even potentially:
df[column1 == 2 and column2 > 3]
. . . because the evaluator would have control over the evaluation and could provide a namespace in which `column1` and `column2` do not evaluate directly to numpy-like arrays (for which `and` doesn't work), but to some kind of combinable query object which converts the `and` into something that will work with numpy-like elementwise comparison.
In other words, the point here is not performance gains or even laziness, but simply the ability to use ordinary Python expression syntax (not, say, a string) to create an unevaluated chunk which can be passed to some other code which then gets to control its evaluation scope, rather than having that scope locked to where it was defined. Because of this, it is probably okay with me if explicit unwrapping of the thunk is required. You know when you are writing a query handler and so you know that what you want is an unevaluated query expression; you don't need to have an argument whose value might either be an unevaluated expression or a fully-evaluated result.
This would also mean that such deferred objects could handle the late-bound default case, but the function would have to "commit" to explicit evaluation of such defaults. Probably there could be a no-op "unwrapping" operation that would work on non-deferred objects (so that `unwrap()` or whatever would just evaluate to the same regular list you passed in), so you could still pass in a plain list a to an argument whose default was `deferred `, but the function would still have to explicitly evaluate it in its body. Again, I think I'm okay with this, partly because (as I mentioned in the other thread) I don't see PEP 671-style late-bound defaults as a particularly pressing need.
There are definitely some holes in my idea. For one thing, with explicit evaluation required, it is much closer to a regular lambda. The only real difference is that it would involve more flexible scope control (rather than unalterably closing over the defining scope). For another, because it is not lazy, it is closer to being achievable with existing mechanisms, like requiring all "field" references in the query to be specified as attributes on some base object (which is indeed most SQL ORMs and pandas-like data structures do it currently). Other people might not be as annoyed with these existing solutions as I am. :-) There is also the question of whether it would unacceptably slow down name references because functions would no longer know which variables were local; I think I would be okay with saying that the thunk could not mutate the enclosing namespace (so, e.g., walruses inside the thunk would only affect an internal thunk namespace). The point here is for the consumer to *evaluate* the thunk and get the result, not inline it into the surrounding code.
My idea is much more half-baked than David's proto-PEP so this isn't really worthy of being called an alternative proposal right now. But I wanted to mention these ideas here to at least handwave about what to me the gain would be from deferred evaluation, as I'm coming at it from a somewhat different angle than the proto-PEP. I have a suspicion that the response will be a combination of disgust and deafening silence but that's life.