[Python-ideas] Saving state in list/generator comprehension

[Andrew Carter]

On Sun, Jun 10, 2012 at 7:01 PM, Steven D'Aprano <steve at pearwood.info>wrote:

> Andrew Carter wrote:
>
>> Forgive me for any problems in this e-mail as I'm new to this mailing
>> list.
>>
>> I thought it might be nice to be able to somehow save a state in
>> list/generator comprehensions,
>> a side effect of this (although not the intended goal) is it would make
>> reduce feasible in a clean manner as the final result would just be the
>> state.
>>
>
> reduce already exists; in Python 2, it is a built-in available at all
> times, in Python 3 it has been banished to the functools module.
>
> I think that it is was banished for good reason. I have found myself
sometimes writing code that needs a reduce, but I feel that using the
reduce function provided isn't very clear, especially if you see it and
aren't familiar with the function. Admittedly writing a function that is a
few short lines is possible, and what I end up doing, it just seems like
there should be a more elegant way to do it than having a bunch of
specialized functions.

What is your use-case for this? "Saving state" is a means to an end. The
> beauty of list comprehensions and generator expressions is that they are
> intentionally quite simple and limited. If you need something more complex,
> write a function or generator. Not everything has to be a (very-long and
> unreadable) one-linear.
>
> reduce already exists, but if it didn't, you could write it quite easily.
> Here's a version with optional starting value which yields the intermediate
> results:
>
As I have mentioned above occasionally I want to turn a list into a single
value by some repeated operation, but actually I think its more common that
I want to map some operation over a list with dependencies of previous
operation passed through. I think my most common use case, is I have a
function that operates on a single value, and also has some state.
Unfortunately leaving my example purposely vague, I was iterating over a
list, and had what was effectively an environment variable (more of state)
initially as a dynamic environment, so it was updated each time the
function was called across the list comprehension. I then for other reasons
wanted to use the environment type as a key for dictionary (which is a
problem if its mutable), but that meant that the original list
comprehension (which I felt was rather simple). Admittedly it didn't take
me any time at all to write a simple function that did the list
comprehension, but it still felt like a simple enough problem that it could
be elegantly solved without resorting to the helper function.


> import itertools
> _MISSING = object()  # sentinel value
>
> def foldl(func, iterable, start=_MISSING):
>    # foldr is left as an exercise :-)
>    if start is _MISSING:
>        it = iter(iterable)
>    else:
>        it = itertools.chain([start], iterable)
>    a = next(it)  # raises if iterable is empty and start not given
>    try:
>        b = next(it)
>    except StopIteration:
>        yield a
>        return
>    a = func(a, b)
>    yield a
>    for b in it:
>        a = func(a, b)
>        yield a
>
>
> Modifying this to return just the last value is easy, and in fact is
> simpler than the above:
>
> def foldl(func, iterable, start=_MISSING):
>    if start is _MISSING:
>        it = iter(iterable)
>    else:
>        it = itertools.chain([start], iterable)
>    a = next(it)
>    for b in it:
>        a = func(a, b)
>    return a
>
>
>
> [...]
>
>  Some of the pros of doing it this way is that because with/as are already
>> keywords in python backwards compatibility shouldn't be an issue,
>>
>
> That's not an argument in favour of your request. That's merely the lack
> of one specific argument against it. There are an infinite number of things
> which could be done that won't break backwards compatibility, but that
> doesn't mean we should do them all. What positive arguments in favour of
> your proposal do you have? What does your proposal allow us to do that we
> can't already do, or at least do better?
>
>
 I feel like there is a need from personal experience of mapping with state
in a short concise way. However it is quite possible that it is just me,
and I need to think about the problem differently, or perhaps live with 4
line functions that are only used once. As for the backwards compatibility
I think was getting ahead of myself, I feel the with/as solution is quite
clunky, but I couldn't come up with a more elegant solution that operated
in a similar vein to how python feels as a language.

>
>  p.s. Is there a built-in way to get the last element from a generator
>> (perhaps even with a default) a quick google search did not reveal one?
>>
>
> The same as you would get the last element from any iterator, not just
> generators: iterate over it as quickly as possible, keeping only the last
> value seen. Because generator values are generated lazily as needed,
> there's no direct way to skip to the last value, or get random access to
> them.
>
> In pure Python:
>
> for x in iterator:
>    pass
>
> This may be faster:
>
> collections.deque(iterator, maxlen=1)[0]
>
> That's a neat solution, a little bit confusing at first glance, but still
very neat, thanks!

> Of course, both examples assume that the iterator or generator yields at
> least one value, and is not infinite.
>
>
>
> --
> Steven
>
> ______________________________**_________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/**mailman/listinfo/python-ideas<http://mail.python.org/mailman/listinfo/python-ideas>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120610/8015f7a9/attachment.html>