itertools.documentation.ncycles is a bit of a sore thumb
The top of the recipe section in the itertools documentation says: "The superior memory performance is kept by processing elements one at a time rather than bringing the whole iterable into memory all at once." but itertools.ncycles doesn't really. It's one line, so I'll inline it here: def ncycles(iterable, n): return chain.from_iterable(repeat(tuple(iterable), n)) Somewhere along the line something was going to need to buffer that, but it'd be nicer if the example didn't have the buffering done all at once at the outset as it is now. Perhaps more importantly, though, is there a specific objection to just adding a count to itertools.cycle much like itertools.repeat has now for this? Thanks. JB
Julian Berman <julian@...> writes:
The top of the recipe section in the itertools documentation says:
"The superior memory performance is kept by processing elements one at a time
rather than bringing the
whole iterable into memory all at once."
but itertools.ncycles doesn't really. It's one line, so I'll inline it here:
In this case, I think it means that iterable*n is never held all in memory.
On Aug 3, 2011, at 9:39 AM, Julian Berman wrote:
def ncycles(iterable, n): return chain.from_iterable(repeat(tuple(iterable), n))
Somewhere along the line something was going to need to buffer that, but it'd be nicer if the example didn't have the buffering done all at once at the outset as it is now.
Perhaps more importantly, though, is there a specific objection to just adding a count to itertools.cycle much like itertools.repeat has now for this?
The optional argument has never been requested and I don't see ncycles() being used enough in-practice to warrant adding complexity to the API. The recipe for ncycles() is included in the docs as a way of teaching how itertools can be composed. Raymond See: http://www.google.com/codesearch#search/&q=ncycles%20lang:%5Epython
On 2011-08-03 17:54, Raymond Hettinger wrote:
On Aug 3, 2011, at 9:39 AM, Julian Berman wrote:
def ncycles(iterable, n): return chain.from_iterable(repeat(tuple(iterable), n))
Somewhere along the line something was going to need to buffer that, but it'd be nicer if the example didn't have the buffering done all at once at the outset as it is now.
Perhaps more importantly, though, is there a specific objection to just adding a count to itertools.cycle much like itertools.repeat has now for this?
The optional argument has never been requested and I don't see ncycles() being used enough in-practice to warrant adding complexity to the API. The recipe for ncycles() is included in the docs as a way of teaching how itertools can be composed.
How about using this alternate recipe then: def ncycles(iterable, n): return chain.from_iterable(tee(iterable, n)) Or even: from copy import copy def ncycles(iterable, n): it, = tee(iterable, 1) return chain.from_iterable( copy(it) if i<n else it for i in xrange(1,n+1) ) - Jacob
On Aug 3, 2011, at 2:41 PM, Jacob Holm wrote:
How about using this alternate recipe then:
def ncycles(iterable, n): return chain.from_iterable(tee(iterable, n))
Really? Think about what that does for a large value of n. Raymond
On 2011-08-03 23:22, Raymond Hettinger wrote:
On Aug 3, 2011, at 2:41 PM, Jacob Holm wrote:
How about using this alternate recipe then:
def ncycles(iterable, n): return chain.from_iterable(tee(iterable, n))
Really? Think about what that does for a large value of n.
It predictably creates an n-tuple of small objects (the cpython "tee" implementation is really quite efficient), and a deque for storing the values from the iterable as it gets them. For small values of n and unknown (potentially large/expensive) iterables this is still an improvement over the recipe in the documentation if you stop iterating before reaching the end of the first repetition. The other recipe I mentioned was specifically designed to handle large n as well. Both my recipes have the nice property requested by the OP that they don't compute values until they are needed. This is relevant if the values are expensive to compute and there is a chance that you might not need them all. In other words, exactly the case where you are most likely to want to use itertools. For iterables with say 50 items or less that are cheap to compute, and/or where you know that you will exhaust the iterable anyway, and where you don't mind "looking ahead", the current recipe in the itertools documentation is just about perfect. - Jacob
participants (4)
-
Benjamin Peterson -
Jacob Holm -
Julian Berman -
Raymond Hettinger