Saving state in list/generator comprehension
Forgive me for any problems in this e-mail as I'm new to this mailing list. I thought it might be nice to be able to somehow save a state in list/generator comprehensions, a side effect of this (although not the intended goal) is it would make reduce feasible in a clean manner as the final result would just be the state. One mechanism I can think of is to overload the with/as keyword for use inside of list/generator comprehensions, and using the previous result as state I believe the change to the grammar in python3k would be comp_iter : comp_for | comp_if | comp_with comp_with: 'with' testlist 'as' testlist So something in the form of [expr for i in iterable with initializer as accumulator] would resolve to something like result = [] accumulator = initializer for i in iterable: accumulator = expr result.append(accumulator) return result For instance reduce could be defined as (assuming all 3 arguments are required) reduce = lambda function, iterable, initializer : ([initializer] + [function(accumulator, i) for i in iterable with initializer as accumulator])[-1] Breaking this down, the "with initializer as accumulator" statement means that when the list comprehension begins accumulator=initializer, then after each iteration, accumulator = function(accumulator, i), so with the function f, list [i1,i2,i3,...], and initial value i0, the resulting list of "[function(accumulator, i) for i in iterable with initializer as accumulator]" would be [f(i0,i1), f(f(i0,i1),i2), f(f(f(i0,i1),i2),i3),...], or in left associative infix form with the f = "+" operator, [i0+i1,i0+i1+i2,i0+i1+i2+i3,...]. Consing (effectively) initializer to the beginning of the list ensures clean behavior for empty lists, and indexing [-1] gets the last element which is really the only element that matters. Consider a slightly more complex example of a Fibonacci generator, one might define it as follows, def fibs(): a, b = 1, 0 while True: a, b = b, a + b yield b Using the with statement, it would require two generator comprehensions fibs = lambda : (b for a,b in (b, a+b for i in itertools.cycle((None,)) with a,b = 0,1)) The inner generator comprehension (b, a+b for i in itertools.repeat(None) with a,b = 0,1) creates an infinite generator of tuples which are consecutive Fibonacci numbers, the outer list comprehension strips off the unneeded "state". Some of the pros of doing it this way is that because with/as are already keywords in python backwards compatibility shouldn't be an issue, but if one is just mapping with state then an extra list/generator comprehension block is needed to strip the state from the intermediate list. I apologize if similar ideas have already been discussed. -Andrew Carter p.s. Is there a built-in way to get the last element from a generator (perhaps even with a default) a quick google search did not reveal one?
Andrew Carter wrote:
reduce already exists; in Python 2, it is a built-in available at all times, in Python 3 it has been banished to the functools module. What is your use-case for this? "Saving state" is a means to an end. The beauty of list comprehensions and generator expressions is that they are intentionally quite simple and limited. If you need something more complex, write a function or generator. Not everything has to be a (very-long and unreadable) one-linear. reduce already exists, but if it didn't, you could write it quite easily. Here's a version with optional starting value which yields the intermediate results: import itertools _MISSING = object() # sentinel value def foldl(func, iterable, start=_MISSING): # foldr is left as an exercise :-) if start is _MISSING: it = iter(iterable) else: it = itertools.chain([start], iterable) a = next(it) # raises if iterable is empty and start not given try: b = next(it) except StopIteration: yield a return a = func(a, b) yield a for b in it: a = func(a, b) yield a Modifying this to return just the last value is easy, and in fact is simpler than the above: def foldl(func, iterable, start=_MISSING): if start is _MISSING: it = iter(iterable) else: it = itertools.chain([start], iterable) a = next(it) for b in it: a = func(a, b) return a [...]
Some of the pros of doing it this way is that because with/as are already keywords in python backwards compatibility shouldn't be an issue,
That's not an argument in favour of your request. That's merely the lack of one specific argument against it. There are an infinite number of things which could be done that won't break backwards compatibility, but that doesn't mean we should do them all. What positive arguments in favour of your proposal do you have? What does your proposal allow us to do that we can't already do, or at least do better?
p.s. Is there a built-in way to get the last element from a generator (perhaps even with a default) a quick google search did not reveal one?
The same as you would get the last element from any iterator, not just generators: iterate over it as quickly as possible, keeping only the last value seen. Because generator values are generated lazily as needed, there's no direct way to skip to the last value, or get random access to them. In pure Python: for x in iterator: pass This may be faster: collections.deque(iterator, maxlen=1)[0] Of course, both examples assume that the iterator or generator yields at least one value, and is not infinite. -- Steven
Andrew Carter wrote:
reduce already exists; in Python 2, it is a built-in available at all times, in Python 3 it has been banished to the functools module. What is your use-case for this? "Saving state" is a means to an end. The beauty of list comprehensions and generator expressions is that they are intentionally quite simple and limited. If you need something more complex, write a function or generator. Not everything has to be a (very-long and unreadable) one-linear. reduce already exists, but if it didn't, you could write it quite easily. Here's a version with optional starting value which yields the intermediate results: import itertools _MISSING = object() # sentinel value def foldl(func, iterable, start=_MISSING): # foldr is left as an exercise :-) if start is _MISSING: it = iter(iterable) else: it = itertools.chain([start], iterable) a = next(it) # raises if iterable is empty and start not given try: b = next(it) except StopIteration: yield a return a = func(a, b) yield a for b in it: a = func(a, b) yield a Modifying this to return just the last value is easy, and in fact is simpler than the above: def foldl(func, iterable, start=_MISSING): if start is _MISSING: it = iter(iterable) else: it = itertools.chain([start], iterable) a = next(it) for b in it: a = func(a, b) return a [...]
Some of the pros of doing it this way is that because with/as are already keywords in python backwards compatibility shouldn't be an issue,
That's not an argument in favour of your request. That's merely the lack of one specific argument against it. There are an infinite number of things which could be done that won't break backwards compatibility, but that doesn't mean we should do them all. What positive arguments in favour of your proposal do you have? What does your proposal allow us to do that we can't already do, or at least do better?
p.s. Is there a built-in way to get the last element from a generator (perhaps even with a default) a quick google search did not reveal one?
The same as you would get the last element from any iterator, not just generators: iterate over it as quickly as possible, keeping only the last value seen. Because generator values are generated lazily as needed, there's no direct way to skip to the last value, or get random access to them. In pure Python: for x in iterator: pass This may be faster: collections.deque(iterator, maxlen=1)[0] Of course, both examples assume that the iterator or generator yields at least one value, and is not infinite. -- Steven
participants (2)
-
Andrew Carter -
Steven D'Aprano