On 7 April 2018 at 08:44, Raymond Hettinger raymond.hettinger@gmail.com wrote:

    def prepend(value, iterator):
        "prepend(1, [2, 3, 4]) -> 1 2 3 4"
        return chain([value], iterator)

I don't have much to add here - I typically agree that an explicit loop is simpler, but my code tends not to be the sort that does this type of operation, so my experience is either where it's not appropriate, or where I'm unfamiliar with the algorithms, so terseness is more of a problem to me than it would be to a domain expert.

Having said that, I find that the arguments that it's easy to add and it broadens the applicability of the function to be significant. Certainly, writing a helper is simple, but as Tim pointed out, the trick to writing that helper is obscure. Also, in the light of the itertools design goal to be a toolkit for iterators, I often find that the tools are just slightly too low level for my use case - they are designed to be combined, certainly, but in practice I find that building my own loop is often quicker than working out how to combine them. (I don't have concrete examples, unfortunately - this feeling comes from working back from the question of why I don't use itertools more than I do). So I tend to favour such slight extensions to the use cases of itertools functions.

A recipe would help, but I don't know how much use the recipes see in practice. I see a lot of questions where "there's a recipe for that" is the answer - indicating that people don't always spot the recipes.

So I guess I'm +0 on the change - but as a matter of principle, rather than from a pressing need for it, so don't take that vote too seriously.

Paul

On Wed, Oct 23, 2019 at 7:03 AM Todd toddrjen@gmail.com wrote:

    def prepend(value, iterator):
        "prepend(1, [2, 3, 4]) -> 1 2 3 4"
        return chain([value], iterator)

That can be solved. We could explicitly license the recipes in the docs under a simpler license. Please start a new thread, as this one has attracted too much spam.

-- --Guido van Rossum (python.org/~guido) Pronouns: he/him **(why is my pronoun here?) http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

On 8 April 2018 at 08:09, Tim Peters tim.peters@gmail.com wrote: [Raymond wrote]:

    def prepend(value, iterator):
        "prepend(1, [2, 3, 4]) -> 1 2 3 4"
        return chain([value], iterator)

I didn't have a strong opinion either way until Tim mentioned sum() and then I went and checked the docs for both that and for accumulate.

First sentence of the sum() docs:

Sums *start* and the items of an *iterable* from left to right and

returns the total.

First sentence of the accumulate docs:

Make an iterator that returns accumulated sums, ...

So I now think that having "start" as a parameter to one but not the other, counts as a genuine API discrepancy.

Providing start to accumulate would then mean the same thing as providing it to sum(): it would change the basis point for the first addition operation, but it wouldn't change the number of cumulative sums produced.

By contrast, using the prepend() approach with accumulate() not only changes the starting value, it also changes the number of cumulative sums produced.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 8 April 2018 at 13:17, Tim Peters tim.peters@gmail.com wrote:

Agreed :)

Right, but if itertools.accumulate() had the semantics of starting with a sum() over an empty iterable, then it would always start with an initial zero.

It doesn't - it starts with "0+first_item", so the length of the output iterator matches the number of items in the input iterable:

>>> list(accumulate([]))
[]
>>> list(accumulate([1, 2, 3, 4]))
[1, 3, 6, 10]

That matches the output you'd get from a naive O(n^2) implementation of cumulative sums:

data = list(iterable)
for stop in range(1, len(iterable)):
    yield sum(data[:stop])

So if the new parameter were to be called start, then I'd expect the semantics to be equivalent to:

data = list(iterable)
for stop in range(1, len(iterable)):
    yield sum(data[:stop], start=start)

rather than the version Raymond posted at the top of the thread (where setting start explicitly also implicitly increases the number of items produced).

That concern mostly goes away if the new parameter is deliberately called something other than "start" (e.g. "prepend=value", or "first=value"), but it could also be addressed by offering a dedicated "yield_start" toggle, such that the revised semantics were:

    def accumulate(iterable, func=operator.add, start=0, yield_start=False):
        it = iter(iterable)
        total = start
        if yield_start:
            yield total
        for element in it:
            total = func(total, element)
            yield total

That approach would have the advantage of making the default value of "start" much easier to document (since it would just be zero, the same as it is for sum()), and only the length of the input iterable and "yield_start" would affect how many partial sums were produced.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 8 April 2018 at 14:31, Guido van Rossum guido@python.org wrote:

The potential ambiguity I see is created mainly by calling the proposed parameter "start", while having it do more than just adjust the individual partial sum calculations by adding an extra partial result to the output series.

If it's called something else (e.g. "first_result"), then the potential "sum(iterable, start=start)" misinterpretation goes away, and it can have Tim's desired effect of defining the first output value (effectively prepending it to the input iterable, without the boilerplate and overhead of actually doing so).

A name like "first_result" would also make it clearer to readers that passing that parameter has an impact on the length of the output series (since you're injecting an extra result), and also that the production of the first result skips calling func completely (as can be seen in Tim's str coercion example).

So where I'd be -1 on:

>>> list(accumulate(1, 2, 3))
[1, 3, 6]
>>> list(accumulate(1, 2, 3, start=0))
[0, 1, 3, 6]
>>> list(accumulate(1, 2, 3, start=1))
[1, 2, 4, 7]

I'd be +1 on:

>>> list(accumulate(1, 2, 3))
[1, 3, 6]
>>> list(accumulate(1, 2, 3, first_result=0))
[0, 1, 3, 6]
>>> list(accumulate(1, 2, 3, first_result=1))
[1, 2, 4, 7]

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Well if you can get Raymond to agree on that too I suppose you can go ahead. Personally I'm -0 but I don't really write this kind of algorithmic code enough to know what's useful. I do think that the new parameter name is ugly. But maybe that's the point.

On Sat, Apr 7, 2018 at 10:26 PM, Tim Peters tim.peters@gmail.com wrote:

>>> list(accumulate(1, 2, 3))
[1, 3, 6]
>>> list(accumulate(1, 2, 3, start=0))
[0, 1, 3, 6]
>>> list(accumulate(1, 2, 3, start=1))
[1, 2, 4, 7]

>>> list(accumulate(1, 2, 3))
[1, 3, 6]
>>> list(accumulate(1, 2, 3, first_result=0))
[0, 1, 3, 6]
>>> list(accumulate(1, 2, 3, first_result=1))
[1, 2, 4, 7]

-- --Guido van Rossum (python.org/~guido)

The Bayesian world view isn't much different except they would prefer "prior" instead of "initial" or "start" ;-)

my_changing_beliefs = accumulate(stream_of_new_evidence, bayes_rule,
prior=what_i_used_to_think)

Though the two analogies are cute, I'm not sure they tell us much. In running programs or bayesian analysis, we care more about the result rather than the accumulation of intermediate results.

My own experience with actually using accumulations in algorithmic code falls neatly into two groups. Many years ago, I used APL extensively in accounting work and my recollection is that a part of the convenience of "+" was that the sequence length didn't change (so that the various data arrays could interoperate with one another).

My other common case for accumulate() is building cumulative probability distributions from probability mass functions (see the code for random.choice() for example, or typical code for a K-S test).

For neither of those use case categories did I ever want an initial value and it would have been distracting to even had the option. For example, when doing a discounted cash flow analysis, I was taught to model the various flows as a single sequence of up and down arrows rather than thinking of the initial balance as a distinct concept¹

Because of this background, I was surprised to have the question ever come up at all (other than the symmetry argument that sum() has "start" so accumulate() must as well).

When writing itertools.accumulate(), I started by looking to see what other languages had done. Since accumulate() is primarily a numerical tool, I expected that the experience of numeric-centric languages would have something to teach us. My reasoning was that if the need hadn't arisen for APL, R, Numpy, Matlab², or Mathematica, perhaps it really was just noise.

My views may be dated though. Looking at the wheel sieve and collatz glide record finder, I see something new, a desire to work with lazy, potentially infinite accumulations (something that iterators do well but almost never arises in the world of fixed-length sequences or cumulative probability distributions).

So I had been warming up to the idea, but got concerned that Nick could have had such a profoundly different idea about what the code should do. That cooled my interest a bit, especially when thinking about two key questions, "Will it create more problems than it solves?" and "Will anyone actually use it?".

Raymond

¹ http://www.chegg.com/homework-help/questions-and-answers/solve-present-worth...

² https://www.mathworks.com/help/matlab/ref/accumarray.html

[Raymond]

my_changing_beliefs = accumulate(stream_of_new_evidence, bayes_rule,
prior=what_i_used_to_think)

They're not intended to. They're just intended to shake people loose from picturing nothing subtler than adding a list of 3 integers ;-)

Sure.

So, a question: why wasn't itertools.accumulate() written to accept iterables of _only_ numeric types? Akin to sum(). I gather from one of Nick's messages that it was so restricted in 3.2. Then why was it generalized to allow any 2-argument function?

Given that it was, sum() is no longer particularly relevant: the closest thing by far is now functools.reduce(), which does support an optional initial argument. Which it really should, because it's impossible for the implementation to guess a suitable starting value for an arbitrary user-supplied dyadic function.

My example using accumulate() to generate list prefixes got snipped, but same thing there: it's impossible for that snippet to work unless an empty list is supplied as the starting value. And it's impossible for the accumulate() implementation to guess that.

In short, for _general_ use accumulate() needs initial for exactly the same reasons reduce() needed it.

BTW, the type signatures on the scanl (requires an initial value) and scanl1 (does not support an initial value) implementations I pasted from Haskell's Standard Prelude give a deeper reason: without an initial value, a list of values of type A can only produce another list of values of type A via scanl1. The dyadic function passed must map As to As. But with an initial value supplied of type B, scanl can transform a list of values of type A to a list of values of type B. While that may not have been obvious in the list prefix example I gave, that was at work: a list of As was transformed into a list _of_ lists of As. That's impossible for scanl1 to do, but easy for scanl.

Or, in short, someone coming from a typed functional language background sees all sorts of things that rarely (if ever) come up in number-crunching languages. Their sensibilities should count too - although not much ;-) They should get _some_ extra consideration in this context, though, because itertools is one of the first things they dig into when they give Python a try.

As above, in all your related experiences "0" was a suitable base value, so you had no reason to care.

Distracting for how long? One second or two? ;-)

As above, the real parallel here is to reduce(). sum() became an historical red herring when accumulate() was generalized.

With a different background, you may just as well have been surprised if the question _hadn't_ come up. For example, this is a standard example in the Haskell world for how to define an infinite Fibonacci sequence with the initial two values f0 and f1:

fibs = f0 : scanl (+) f1 fibs

The part from scanl onward would be spelled in Python as

accumulate(fibs, initial=f1)

although it requires some trickery to get the recursive reference to work (details on request, but I'm sure you already know how to do that).

In the itertools context, I also would have looked hard at Haskell experience.

BTW, whoever wrote the current accumulate() docs also found a use for initial, but hacked around it:

""" First-order recurrence relations can be modeled by supplying the initial value in the iterable and using only the accumulated total in func argument: """

followed by:

That would be better on several counts to my eyes as:

inputs = repeat(None, 35)  # no values actually used
... for x in accumulate(inputs, logistic_map, initial=x0)

In particular, filling inputs with None would lead to an exception if the programmer screwed up and the input values actually _were_ being used. I expect we'll both overlook that writing a generator using the obvious loop would be a lot clearer regardless ;-)

Amazingly enough, those are both just doing integer running sums - all the stuff above about catering to general types doesn't apply in these cases. initial isn't needed for _correctness_ in these cases, it would just add convenience and some clarity.

He appeared to withdraw his objections after agreement _not_ to name the argument "start" was reached. Something about sum() also having an optional argument named "start" caused confusion.

Like? It's an optional argument. One brief example in the docs is all it takes to understand what it does.

As above, the docs could change to use it. And I bet the test suite too. How much more could you want from a feature?! ;-)

Prior to 3.2, accumulate() was in the recipes section as pure Python code. It had no particular restriction to numeric types.

I received a number of requests for accumulate() to be promoted to a real itertool (fast, tested, documented C code with a stable API). I agreed and accumulate() was added to itertools in 3.2. It worked with anything supporting __add__, including str, bytes, lists, and tuples. More specifically, accumulate_next() called PyNumber_Add() without any particular type restriction.

Subsequently, I got requests to generalize accumulate() to support any arity-2 function (with operator.mul offered as the motivating example). Given that there were user requests and there were ample precedents in other languages, I acquiesced despite having some reservations (if used with a lambda, the function call overhead might make accumulate() slower than a plain Python for-loop without the function call). So, that generalized API extension went into 3.3 and has remained unchanged ever since.

Afterwards, I was greeted with the sound of crickets. Either it was nearly perfect or no one cared or both ;-)

It remains one of the least used itertools.

Honestly, I couldn't immediately tell what this code was doing:

list(accumulate([8, 4, "k"], lambda x, y: x + [y], first_result=[]))

This may be a case where a person would be better-off without accumulate() at all.

The reduce() function had been much derided, so I've had it mentally filed in the anti-pattern category. But yes, there may be wisdom there.

Thanks for pointing that out. I hadn't considered that someone might want to transform one type into another using accumulate(). That is pretty far from my mental model of what accumulate() was intended for. Also, I'm still not sure whether we would want code like that buried in an accumulate() call rather than as a regular for-loop where I can see the logic and trace through it with pdb.

As for scanl, I'm not sure what this code means without seeing some python equivalent.

scanl            :: (a -> b -> a) -> a -> [b] -> [a]
scanl f q xs     =  q : (case xs of
                           []   -> []
                           x:xs -> scanl f (f q x) xs)


scanl1           :: (a -> a -> a) -> [a] -> [a]
scanl1 f (x:xs)  =  scanl f x xs
scanl1 _ []      =  []

I concur.

Possibly forever. In my experience, if a person initially frames a problem wrong (or perhaps in a hard to solve way), it can take them a long time to recover. For example with discounted cash flows, people who think of the initial value as being special or distinct from the other cash flows will have a hard time adapting to problem variants like annuity due, balloon payments, internal rate of return, coupon stripping, valuing a transaction that takes place in the future, etc.

I don't want to overstate the case, but I do think a function signature that offers a "first_value" option is an invitation to treat the first value as being distinct from the rest of the data stream.

Do we want the tool to encourage such trickery?

Don't get me wrong, I think it is cool that you could write such code, but could and should aren't always the same.

Haskell probably is a good source of inspiration, but I don't know the language and find its docs to be inscrutable.

So, I have to just trust when you say something like, """ Haskell has millions of functions ;-) mapAccumL is a God-awful mixture of Python's map(), reduce(), and accumulate() :-( The examples here should convince you it's nearly incomprehensible: """

In fact, yes, you've convinced me that there is an intelligibility issue ;-)

The winks may reading your posts fun, but I really can't tell whether position is, "yes, let's do this because someone can do wild things with it", or "no, let's don't this because people would commit atrocities with it".

I'm concerned that the total number of actual users will be exactly two (you and the writer of the wheel-sieve) and that you each would have used it exactly once in your life. That's a pretty small user base for a standard library feature ;-)

Tim, if you could muster an honest to goodness, real +1, that would be good enough for me. Otherwise, I'm back to -0 and prefer not to see Pythonistas writing the Haskell magics described in this thread.

If this does go forward, I would greatly prefer "start" rather than "first_value" or "initial".

The conversation has been enjoyable (perhaps because the stakes are so low) and educational (I learn something new every time you post).

I'll leave this will a few random thoughts on itertools that don't seem to fit anywhere else.

1) When itertools was created, they were one of easiest ways to get C-like performance without writing C. However, when PyPy matured we got other ways to do it. And in the world of PyPy, the plain python for-loops outperform their iterator chain equivalents, so we lost one motivate to use itertools.

2) While I personally like function chains operating on iterators, my consulting and teaching experience has convinced me that very few people think that way. Accordingly, I almost never use compress, filterfalse, takewhile, dropwhile, etc. As people started adopting PEP 279 generator expressions, interest in itertool style thinking seems to have waned.

Putting these two together has left me with a preference for itertools to only cover the simplest and most common cases, leaving the rest to be expressed as plain, everyday pure python. (The combinatoric itertools are an exception because they are more algorithmically interesting).

Raymond

On 9 April 2018 at 14:38, Raymond Hettinger raymond.hettinger@gmail.com wrote:

Weirdly (or perhaps not so weirdly, given my tendency to model computational concepts procedurally), I find the operation of reduce() easier to understand when it's framed as "last(accumulate(iterable, binop, initial=value)))".

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Also Tim Peter's one-line example of:

print(list(itertools.accumulate([1, 2, 3], lambda x, y: str(x) + str(y))))

I think makes it clear that itertools.accumulate is not the right vehicle for this change - we should make a new itertools function with a required "initial" argument.

On Mon, Apr 9, 2018 at 1:44 PM, Peter O'Connor peter.ed.oconnor@gmail.com wrote:

[Tim]

[Raymond]

So that's the restriction Nick had in mind: a duck-typing kind, in that it would blow up on types that don't participate in PyNumber_Add():

Sucker ;-)

Or nobody cared _enough_ to endure a 100-message thread arguing about an objectively minor change ;-)

If you can borrow Guido's time machine, I'd suggest going back to the original implementation, except name it cumsum() instead, and leave accumulate() to the 3rd-party itertools packages (at least one of which (itertoolz) has supported an optional "initial" argument all along).

I don't see how that can be known. There are at least tens of thousands of Python programmers nobody on this list has ever heard about - or from - writing code that's invisible to search engines.

I _believe_ it - I just don't see how it can be known.

list(accumulate([8, 4, "k"], lambda x, y: x + [y], first_result=[]))

Of course you couldn't: you think of accumulate() as being about running sums, and _maybe_ some oddball out there using it for running products. But that's a statement about your background, seeing code you've never seen before, not about the function. Nobody knows immediately, at first sight, what

 list(accumulate([8, 4, 6], lambda x, y: x + y, first_result=0))

does either. It's learned. If your background were in, e.g., Haskell instead, then in the latter case you'd picture a list [a, b, c, ...] and figure it out from thinking about what the prefixes of

0 + a + b + c + ....

compute. In exactly the same way, in the former case you'd think about what the prefixes of

[] + [a] + [b] + [c] + ...

compute. They're equally obvious _after_ undertaking that easy exercise, but clear as mud before doing so.

De gustibus non est disputandum.

The current accumulate() isn't just akin to reduce(), it _is_ reduce(), except a drunken reduce() so nauseated it vomits its internal state out after each new element it eats ;-)

It's nevertheless what the current function supports - nothing being suggested changes that one whit. It's "worse" in Python because while only scanl in Haskell can "change types", the current scanl1-like Python accumulate can change types too. Perhaps the easiest way to see that is by noting that

map(f, xs)

is generally equivalent to

accumulate(xs, lambda x, y: f(y))

right now. That is, just ignore the "accumulator" argument, and you're left with a completely arbitrary transformation of the elements. If you like, you can, e.g., use accumulate() right now to generate a a sequence of socket connections from a sequence of deques.

That can't be done by Haskell's scanl1 because that language's strict static type system can't express the notion of that "well, given a list-of-A, the accumulation function f has arguments of types A and A on the first call, but types type-of-f(A) and A on subsequent calls. By supplying an initial value of type B, scanl's accumulation function returns type B and always has arguments of type B and A. The type system has no problem with that.

De gustibus non est disputandum again ;-) These kinds of arguments belong in a style guide, wiki, tutorial, or nagging Stackoverflow answer. They really - to me - have nothing to do with the issue at hand. Again, nothing above depends in any way on whether or not accumulate grows an optional argument. All the things you see as "bad" are already not just possible, but easy.

scanl            :: (a -> b -> a) -> a -> [b] -> [a]
scanl f q xs     =  q : (case xs of
                           []   -> []
                           x:xs -> scanl f (f q x) xs)


scanl1           :: (a -> a -> a) -> [a] -> [a]
scanl1 f (x:xs)  =  scanl f x xs
scanl1 _ []      =  []

My apologies for that - I assumed you had a reading knowledge of Haskell, and were just unaware of these specific functions. The details don't really matter. You can just take my word for that scanl1 is a type-safe-restricted form of Python's current accumulate, and that the more basic scanl is like what accumulate would be if an initial value were a _required_ argument.

BTW, Haskell has no native notion of optional (or default, or variable, or keyword) arguments. All functions are "curried": they have exactly one argument. So, e.g., while there's generally no harm in viewing the Haskell

f x y

as being like the Python

f(x, y)

it's _really_ more like

functools.partial(f, x)(y)

In any case, "all functions have exactly one argument" is why, in Haskell, even the tiniest variation in functionality is usually implemented by creating a function with a new name for each variation, rather than pile on ever-growing sequences of required flag arguments.

I'll also note that scanl and scanl1 are defined in the Standard Prelude, which is akin to being a Python builtin: they're part of the core language, always available. As such, every experienced Haskell programmer is aware of them.

For a _general_ accumulate, which we already have, _sometimes_ the first value really is distinct. Greg Ewing recently made that point eloquently, so I won't elaborate.

But again, one example in the docs is all it takes:

99% of programmers will see that, think "huh! why would I want initial=?" and never give it another thought. 0.001% of the remaining 1% will ask on Stackoverflow, and get an answer showing "advanced" uses for accumulate. The remaining 0.999% of the remaining 1% will eventually find one of those answers.

Sorry, the original point got lost in the weeds there: the point isn't that such code is desirable, it's just that Haskell programmers _are_ aware of scanl. and that one of its very-well-known toy uses exploits that it specifies an initial value. So _if_ you had that background too, it would be natural as death to wonder "huh - so how come Python's accumulate _doesn't_ have such an argument?".

That's what happens when a worldwide network of obsessed PhDs struggles for decades to push the state of the art ;-)

I mean both, of course ;-) The world is absurd, and often all I can do is chuckle and accept that all options suck in their own endearing ways.

Did you write those docs? If so, that's one of life's absurdities: half of the accumulate() docs are devoted to giving a weird example of an application that ignores the input sequence values entirely, taking from the input sequence _only_ its total length.

I enjoyed the example, but I can't believe you'd _recommend_ it as good practice. It's just "a trick" you _can_ do, like (as above) you _can_ ignore the accumulator argument instead to make accumulate() work like map() instead.

You can't know that, though. It's not my only use, but it's the only use I wrote about because I had just done it the day before this thread started. I also have a prime-generating function inspired by Will Ness's. BTW, that's a beautiful thing in real life, but not primarily because of the wheel gimmicks: you don't have to specify an upper limit in advance (or ever), but it nevertheless consumes memory proportional to the number of primes less than the _square root_ of the largest prime you've generated so far.

That's easy if you know the upper limit in advance, but not if you don't. Will's solution to _that_ part is clever, elegant, and eminently practical.

In any case, Will is "a functional language" person, and didn't even know itertools existed. Janne Karila pointed it out to him, and Will was like a kid in a candy store. I expect (but don't know) _he_ wondered "huh - how come accumulate() doesn't have an initial value like scanl has?", but rather than gripe about it on a Python list created the "chain a singleton list" workaround instead.

Regardless, I wouldn't be surprised a bit if Janne Karila also had similar code - or any number of other Python programmers we don't know about writing code we'll never see.

On purely _technical_ grounds, given that accumulate() is already thoroughly general, I think adding an optional start value is not just a +1, but a no-brainer. I've still seen no technical argument against it, and several sound technical arguments in its favor have been made by several people.

OTOH ... meh. There appears to be scant demand, and code churn also carries costs. I've already wormed around it, so it doesn't scratch any practical itch I still have. It's your job to worry about future generations ;-)

Whether or not the argument is added is simply irrelevant to what's possible to do with accumulate() already - it just affects how transparently a special initial value can be supplied when one is desired.

In all of the (few!) "real life" current uses you've seen, it would obviously make already-sane code simpler & clearer. Against that, worrying about masses of Pythonistas who _could_ make absurd (in Python) code a bit _less_ absurd just doesn't engage me.

Bike-shedding the name holds little interest for me. Against "start", Nick vehemently objects to that name. I _expected_ that you would too, because you've generally seen _no_ value to specifying an initial value for sums, and "start" is the name sum() gives to its optional starting value.

In favor of "initial", the current accumulate() _is_ a generalization not of sum() but of reduce(), and:

""" Help on built-in function reduce in module _functools:

reduce(...) reduce(function, sequence[, initial]) -> value """

That is, the reduce docstring has used "initial" forever.

And there's also that the itertoolz accumulate() already has the feature in question, and named its argument "initial".

Yup, I too prefer fighting to the death over things that don't matter ;-)

I should try PyPy again. I got out of the habit years ago, when I kept running out of RAM. I have a lot more RAM now ;-)

Matches my less extensive experience.

While I've never used any of those, except once each when they were new. I'm aware of them, but they just never seem to fit the problem at hand.

So, just to keep things interesting, let's add an optional start= argument to _all_ itertools functions ;-)

Besides all the combinatoric ones, these are the ones I'd sorely miss:

count cycle repeat

accumulate chain[.from_iterable] islice tee

I'm surprised I left groupby() off that list! I always liked that one "in theory", but in practice - for whatever reasons - whenever I use it I usually end up rewriting the code, in such a way that groupby() no longer makes sense to use. Curious!

Nick, sorry, but your arguments still make little sense to me. I think you're pushing an analogy between sum() details and accumulate() waaaaay too far, changing a simple idea into a needlessly complicated one.

accumulate() can do anything at all it wants to do with a start argument (if it grows one), and a "default" of start=0 makes no sense: unlike sum(), accumulate() is not

specifically for use with numeric values and may
reject non-numeric types [from the `sum()` docs]

accumulate() accepts any two-argument function.

Arguing that it "has to do" something exactly the way sum() happens to be implemented just doesn't follow - not even if they happen to give the same name to an optional argument. If the function were named accumulate_sum(), and restricted to numeric types, maybe - but it's not.

[Nick Coghlan ncoghlan@gmail.com]

    def accumulate(iterable, func=operator.add, start=0, yield_start=False):
        it = iter(iterable)
        total = start
        if yield_start:
            yield total
        for element in it:
            total = func(total, element)
            yield total

As above, start=0 is senseless for accumulate (despite that it makes sense for sum). Raymond gave the obvious implementation in his original message.

If you reworked your implementation to accommodate that NO sensible default for start exists except for the one Raymond used (a unique object private to the implementation, so he knows for sure whether or not start was passed), you'd end up with his implementation ;-)

yield_start looks like a nuisance in any case. As already explained, most uses want the start value if it's given, and in cases where it isn't it's trivial to discard by doing next() once on the result. Of course it could be added - but why bother?

Just a nit here:

[Tim]

[Nick]

They're not quite the same today. For example, if you have an object x of a custom numeric class,

sum([x])

invokes x.__radd__ once (it really does do x.__radd__(0)), but

list(itertools.accumulate([x]))

just returns [x] without invoking x.__add__ or x.__radd__ at all.

Yup! That's fine by me too - and preferable even ;-)

On Mon, Apr 9, 2018 at 11:35 PM, Stephen J. Turnbull turnbull.stephen.fw@u.tsukuba.ac.jp wrote:

[OT] How is that human ability tested? I am a visual learner and I would propose that if you have a set of numbers, you can graph it in different ways to make it easier to perceive one or the other (or maybe both):

• to emphasize the average, draw a line graph -- in my mind I draw a line through the average (getting the trend for free)
• to emphasize the sum, draw a histogram -- in my mind I add up the sizes of the bars

-- --Guido van Rossum (python.org/~guido)

On Mon, Jul 8, 2019 at 3:35 PM john alexa alexajohn05@gmail.com wrote: >

This is pure spam - are list admins able to delete it from the archive?

ChrisA

[Tim]

[Greg Ewing greg.ewing@canterbury.ac.nz]

I'm not sure you've read all the messages in this thread, but that's exactly what's being proposed. That. e.g., a new optional argument:

accumulate(xs, func, initial=S)

act like the current

 accumulate(chain([S], xs), func)

Note that in neither case is the original xs modified in any way, and in both cases the first value generated is S.

Note too that the proposal is exactly the way Haskell's scanl works (although scanl always requires specifying an initial value - while the related scanl1 doesn't allow specifying one).

And that's all been so since the thread's first message, in which Raymond gave a proposed implementation:

    _sentinel = object()

    def accumulate(iterable, func=operator.add, start=_sentinel):
        it = iter(iterable)
        if start is _sentinel:
            try:
                total = next(it)
            except StopIteration:
                return
        else:
            total = start
        yield total
        for element in it:
            total = func(total, element)
            yield total

Indeed, something quite similar often applies when parallelizing search loops of the form:

 for candidate in accumulate(chain([starting_value], cycle(deltas))):

For a sequence that eventually becomes periodic in the sequence of deltas it cycles through, multiple processes can run independent searches starting at carefully chosen different starting values "far" apart. In effect, they're each a "balance brought forward" pretending that previous chunks have already been done.

Funny: it's been weeks now since I wrote an accumulate() that _didn't_ want to specify a starting value - LOL ;-)

• correction to brackets from first example:

def iter_cumsum_tolls_from_day(day, toll_amount_so_far): return accumulate(get_tolls_from_day(day), initial=toll_amount_so_far)

On Mon, Apr 9, 2018 at 11:55 PM, Peter O'Connor peter.ed.oconnor@gmail.com wrote:

accumulate(xs, func, initial=S)

 accumulate(chain([S], xs), func)

    _sentinel = object()

    def accumulate(iterable, func=operator.add, start=_sentinel):
        it = iter(iterable)
        if start is _sentinel:
            try:
                total = next(it)
            except StopIteration:
                return
        else:
            total = start
        yield total
        for element in it:
            total = func(total, element)
            yield total

 for candidate in accumulate(chain([starting_value], cycle(deltas))):

[Tim]

sum(xs[i:j])

Which brought to mind a different problem: we have a list of numbers, xs. For each index position i, we want to know the largest sum among all segments ending at xs[i], and the number of elements in a maximal-sum slice ending at xs[i].

accumulate() is a natural way to approach this, for someone with a functional language background. You'll have to trust me on that ;-)

But there are some twists:

• The identity element for max() is minus infinity, which accumulate() can';t know.

• We want to generate a sequence of 2-tuples, despite that xs is a sequence of numbers.

• In this case, we do _not_ want to see the special initial value.

For example, given the input xs

[-10, 3, -1, 7, -9, -7, -9, 7, 4]

we want to generate

(-10, 1), (3, 1), (2, 2), (9, 3), (0, 4), (-7, 1), (-9, 1), (7, 1), (11, 2)

Note: the largest sum across all non-empty slices is then max(that_result)[0]. The code below could be easily changed to keep track off that incrementally too, but this is already so different from "plain old running sum" that I don't want more novelty than necessary to make the points (a special initial value is needed, and it's not necessarily insane to want to produce results of a different type than the inputs).

The key part is the state updating function:

def update(state, x):
    prevmax, count = state
    newsum = prevmax + x
    if newsum > x:
        return newsum, count + 1
    else:
        return x, 1

That's all there is to it!

Then, e.g.,

I quite like it! It handles everything that's come up pretty naturally. A difference from what was discussed before: I opposed having a skipfirst argument because I didn't want an optional argument that _only_ applied if another optional argument was specified. https://www.straic.com/San-diego-seo-expert/

On Fri, Oct 4, 2019 at 7:41 AM trekha89@gmail.com wrote:

Why is this 18-month-old thread in particular attracting spammers? The last four messages to this thread have been three spam links and one complaint about spam. Is there any reason for this thread to be bumped in the future with legitimate discussion posts, and if not, would it make sense for the list admins to just shut this thread down, and is there a way to do that (e.g. by blacklisting or auto-rejecting future messages to the thread)?