[Python-ideas] Saving state in list/generator comprehension

Andrew Carter acarter at cs.hmc.edu
Mon Jun 11 00:42:53 CEST 2012

Forgive me for any problems in this e-mail as I'm new to this mailing list.

I thought it might be nice to be able to somehow save a state in
list/generator comprehensions,
a side effect of this (although not the intended goal) is it would make
reduce feasible in a clean manner as the final result would just be the

One mechanism I can think of is to overload the with/as keyword for use
inside of list/generator comprehensions, and using the previous result as
I believe the change to the grammar in python3k would be

comp_iter : comp_for | comp_if | comp_with
comp_with: 'with' testlist 'as' testlist

So something in the form of
  [expr for i in iterable with initializer as accumulator]
would resolve to something like
  result = []
  accumulator = initializer
  for i in iterable:
    accumulator = expr
  return result

For instance reduce could be defined as (assuming all 3 arguments are
  reduce = lambda function, iterable, initializer : ([initializer] +
[function(accumulator, i) for i in iterable with initializer as
Breaking this down, the "with initializer as accumulator" statement means
that when the list comprehension begins accumulator=initializer,
then after each iteration, accumulator = function(accumulator, i), so with
the function f, list [i1,i2,i3,...], and initial value i0, the resulting
list of
"[function(accumulator, i) for i in iterable with initializer as
accumulator]" would be [f(i0,i1), f(f(i0,i1),i2),
f(f(f(i0,i1),i2),i3),...], or in left associative infix form with
the f = "+" operator, [i0+i1,i0+i1+i2,i0+i1+i2+i3,...].
Consing (effectively) initializer to the beginning of the list ensures
clean behavior for empty lists, and indexing [-1] gets the last element
which is really the only
element that matters.

Consider a slightly more complex example of a Fibonacci generator, one
might define it as follows,
  def fibs():
    a, b = 1, 0
    while True:
      a, b = b, a + b
      yield b

Using the with statement, it would require two generator comprehensions
  fibs = lambda : (b for a,b in (b, a+b for i in itertools.cycle((None,))
with a,b = 0,1))
The inner generator comprehension
  (b, a+b for i in itertools.repeat(None) with a,b = 0,1)
creates an infinite generator of tuples which are
consecutive Fibonacci numbers, the outer list comprehension strips off the
unneeded "state".

Some of the pros of doing it this way is that because with/as are already
keywords in python backwards compatibility shouldn't be an issue,
but if one is just mapping with state then an extra list/generator
comprehension block is needed to strip the state from the intermediate list.

I apologize if similar ideas have already been discussed.
-Andrew Carter

p.s. Is there a built-in way to get the last element from a generator
(perhaps even with a default) a quick google search did not reveal one?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120610/c1fee3a2/attachment.html>

More information about the Python-ideas mailing list