Could this idea be used to specify the default factory of a dataclass field? For example,
@dataclass class X: x: list[int] = deferred 
@dataclass class X: x: list[int] = field(default_factory=list)
If so, it might be worth adding to your proposal as another common motivating example?
On Wednesday, June 22, 2022 at 2:23:13 PM UTC-4 David Mertz, Ph.D. wrote:
On Wed, Jun 22, 2022 at 10:47 AM Martin Di Paola martinp...@gmail.com wrote:
Hi David, I read the PEP and I think it would be useful to expand the Motivation and Examples sections. While indeed Dask uses lazy evaluation to build a complex computation without executing it, I don't think that it is the whole story. Dask takes this deferred complex computation and *plans* how to execute it and then it *executes* it in non-obvious/direct ways.
Dask is very clever about execution planning. Ray is possibly even more clever in that regard.
However, I think that that should be an explicit non-goal of the PEP. DeferredObjects should create a DAG, yes. But I think Python itself should not think about being *clever* in evaluating that DAG, nor in itself think about parallelism. If my PEP were adopted, that would be something other libraries like Dask or Django could build on top of with more elaborate evaluation plans.
But just the DAG itself gets you more than just "wait until needed to do the final computation." It allows for intermediate computation of nodes of the DAG lower down than the final result. For example, imagine dependencies like this (where all the computation steps are expensive):
A -> B B -> Z B -> Y B -> X A -> C C -> X X -> F X -> G C -> W C -> V A -> D D -> V D -> U D -> T
Hopefully you either see my ASCII art in fixed font, or it's at least intelligible. If I want to force evaluation of A, I need to do everything. But if it turns out all I need within my program is C, then I have to do computations C, X, F, G, W, V. Which is maybe still expensive, but at least I don't worry about B, Z, Y, U, T, or A.
Yes, I should add something like this to the PEP.