[Python-ideas] Reduce/fold and scan with generator expressions and comprehensions

[Stephen J. Turnbull]

Danilo J. S. Bellini writes:

No attribution.  Please attribute, at least when you mix quotes from
different people.

 > > As a practicing economist, I wonder what use cases you're referring
 > > to.  I can't think of any use cases where if one previous value is
 > > useful, having all previous values available (ie, an arbitrary
 > > temporal structure, at the modeler's option) isn't vastly more useful.
 > 
 > Well, see the itertools.accumulate examples yourself then, the ones at
 > docs.python.org...

And this: "list(accumulate(data, max))", needs syntax, why?  I scratch
my head over how you can improve over that.

 > We can start with something really simple like interest rates or
 > uniform series, but...

Don't waste the list's time being snide.

My point is that although new syntax may be useful for simple cases,
serious applications will worry about computational accuracy and
likely will provide packages that handle general cases that nest these
simple cases.  Given they exist, most modelers will prefer using those
packages to writing their own comprehensions.  That may not apply in
other fields, but AFAICS it does apply in economics.  So if you can't
handle the complex cases, the syntax is just cognitive overhead:
TOOWTDI will be the high-quality general packages even where they're
theoretically overkill.

The basic list comprehension doesn't need to deal with that kind of
issue.

 > before arguing here, please convince other people to update the
 > Wikipedia:

Irrelevant and rude.  Please, don't.

 > "Recurrence relations, especially linear recurrence relations, are used
 > extensively in both theoretical and empirical economics."
 > https://en.wikipedia.org/wiki/Recurrence_relation#Economics

I didn't contest that, as quoted above.  What I contest is the claim
that in empirical economics syntactic sugar for 'accumulate' would be
particularly useful.  Sure, you *can* express a second-order
difference equation as two first-order equations, and perhaps you
would actually calculate it that way.  But in economics we normally
express a second-order diff eq as a second-order diff eq.  If the
input is a second-order equation that needs to be reformulated as a
system of first-order equations, then with the code required to
implement such transformations, it is not obvious to me that having
syntax for first-order equations is going to be worth the extra syntax
in the language.  Most of the complexity is going to be in the
transformation which AFAICS is likely to be problem-specific, and
therefore unavoidable by the modeler.  OTOH, as PyScanPrev shows, the
complexity of recursion can be hidden in a decorator, which the
modeler can cargo-cult.

Furthermore, in much of modern time-series econometrics, the order of
the equation (number of lagged values to include) will be determined
from the data and may differ from variable to variable and across
equations, in which case you're effectively going to be carrying
around a huge chunk of the data set as "state" (much of it unused) in
each list element, which seems like a pretty clunky way to think about
such problems computationally, however useful it may be in the theory
of mathematical dynamics.

I grant that 40 years in the field studying econometrics in terms of
fixed data matrices has probably caused my synapses to clot -- and
that's precisely why I'm asking *you* to explain to me how to beautify
such code using the constructs you propose.  I think the examples
presented are already quite pretty without new syntax.

As for economic theory, theory papers in economics don't include
Python programs that I've seen.  So having syntax for this feature in
Python seems unlikely to improve presentation of economic theory.  (I
suppose that keeping "theoretical" in the quote was just an accident,
but I could be missing something.)

 > The proposal isn't about quick and dirty code. The State-space example
 > includes a general linear time-varying MIMO simulation implementation
 > trying to keep the syntax as similar as possible to the control theory
 > engineers are used to.

Fine, but I only questioned economics.  I'm not a rocket scientist,
I'll let the rocket scientists question that.  If they don't, *I*
certainly will concede you have a point in those other fields.

 > Also, my goal when I was looking for a scan syntax
 > to solve the conditional toggling example was to make it cleaner.

    >>> @enable_scan("p")
    ... def ltvss(A, B, C, D, u, x0=0):
    ...     Ak, Bk, Ck, Dk = map(iter, [A, B, C, D])
    ...     u1, u2 = itertools.tee(u, 2)
    ...     x = (next(Ak) * p + next(Bk) * uk for uk in prepend(x0, u1))
    ...     y = (next(Ck) * xk + next(Dk) * uk for xk, uk in zip(x, u2))
    ...     return y

And this needs syntax now ... why?

 > And where's the link?

To what?

 > > [...]. From what I remember, the conclusion reached was that
 > > there are too many degrees of freedom to be able to express
 > > reduction operations in a comprehension-like way that's any
 > > clearer

I didn't write this.  Please keep your attributions straight.

 > I can't talk about a discussion I didn't read, it would be unfair,
 > disrespectful.

You're perfectly willing to tell other people what to read, though.  I
realize times have changed since 1983, but as long as I've been on the
'net, it's always been considered polite to investigate previous
discussions, and AFAIK it still is.  Of course that has to be balanced
against search costs, but now that you know that what you're looking
for exists, the expected benefit of search has jumped, and of course
you could ask David for hints to minimize the search costs.  No?

 > Perhaps there are people who prefer masochist rituals instead of
 > using "reduce", who knows? Who cares?

(1) There are.  (2) Evidently you don't.  (3) You should, because the
leading (literally) person who prefers writing code himself to using
"reduce" is Guido van Rossum.

 > This maillist isn't very inviting...

Why do you say that?  Because people aren't falling over themselves to
accept your proposal?

You've been asked a simple question several times: why does this
feature need to be implemented as syntax rather than a function?  The
only answers I've seen are the importance of the applications (not
contested by anybody AFAICS), and your preference for syntax over a
function.  You have provided strong motivation for the feature, but
not for an implementation via new syntax.

 > but I hope some of you at least try to read the rationale and the
 > examples.

I did.  They are very persuasive ... up to the point where you ask for
syntax for something that appears (now that you've done it, kudos!) to
be perfectly do-able with a function.  It's not for me to say yes or
no, but I can tell you that the outcomes of past discussions of this
kind indicate that it will be unlikely that this proposal will be
accepted without better justification for adding new syntax,
preferably something that's impossible to implement performantly with
a function.  Or perhaps a "killer example" that persuades Guido or one
of the other Most Senior Devs that this is way too cool to go without
syntax.

Steve