On Wed, Jun 22, 2022 at 10:47 AM Martin Di Paola <martinp.dipaola@gmail.com> wrote:
Hi David, I read the PEP and I think it would be useful to expand the
Motivation and Examples sections.
While indeed Dask uses lazy evaluation to build a complex computation
without executing it, I don't think that it is the whole story.
Dask takes this deferred complex computation and *plans* how to execute it
and then it *executes* it in non-obvious/direct ways.

Dask is very clever about execution planning.  Ray is possibly even more clever in that regard.

However, I think that that should be an explicit non-goal of the PEP.  DeferredObjects should create a DAG, yes.  But I think Python itself should not think about being *clever* in evaluating that DAG, nor in itself think about parallelism.  If my PEP were adopted, that would be something other libraries like Dask or Django could build on top of with more elaborate evaluation plans.

But just the DAG itself gets you more than just "wait until needed to do the final computation." It allows for intermediate computation of nodes of the DAG lower down than the final result.  For example, imagine dependencies like this (where all the computation steps are expensive):

A -> B
     B -> Z
     B -> Y
     B -> X
A -> C
     C -> X
          X -> F
          X -> G
     C -> W
     C -> V
A -> D
     D -> V
     D -> U
     D -> T
Hopefully you either see my ASCII art in fixed font, or it's at least intelligible.  If I want to force evaluation of A, I need to do everything.  But if it turns out all I need within my program is C, then I have to do computations C, X, F, G, W, V.  Which is maybe still expensive, but at least I don't worry about B, Z, Y, U, T, or A.

Yes, I should add something like this to the PEP.