Now I discover itertools I thing your are right, but maybe the pack function could be rename iwinslice (at the end it's its real name), and add it to itertools ?? paul bedaride On Fri, Mar 20, 2009 at 6:01 PM, Raymond Hettinger <python@rcn.com> wrote:

...

...

I propose a new function for list for pack values of a list and sliding over them:

then we can do things like this: for i, j, k in pack(range(10), 3, partialend=False): print i, j, k

. . .

...

...

def pack(l, size=2, slide=2, partialend=True): lenght = len(l) for p in range(0,lenght-size,slide): def packet(): for i in range(size): yield l[p+i] yield packet() p = p + slide if partialend or lenght-p == size: def packet(): for i in range(lenght-p): yield l[p+i] yield packet()

This has been discussed before and rejected.

There were several considerations.  The itertools recipes already include simple patterns for grouper() and pairwise() that are easy to use as primitives in your code or to serve as models for variants. The design of pack() itself is questionable.  It attempts to be a Swiss Army Knife by parameterizing all possible variations (length of window, length to slide, and how to handle end-cases). This design makes the tool harder to learn and use, and it makes the implementation more complex. That complexity isn't necessary.  Use cases would typically fall into grouper cases where the window length equals the slide length or into cases that slide one element at a time.  You don't win anything by combining the two cases except for more making the tool harder to learn and use.

The pairwise() recipe could be generalized to larger windows, but seemed like less of a good idea after closely examining potential use cases.  For cases that used a larger window, there always seemed to be a better solution than extending pairwise().  For instance, a twenty-day moving average is better implemented with a deque(maxlen=20) and a running sum than with an iterator returning tuples of length twenty -- that approach does a lot of unnecessary work shifting elements in the tuple, turning an O(n) process into an O(m*n) process.

For short windows, like pairwise() itself, the issue is not one of total running time; instead, the problem is that almost every proposed use case was better coded as a simple Python loop, saving the value previous values with a step like:  oldvalue = value. Having pairwise() or tripletwise() tended to be a distraction away from better solutions.  Also, the pure python approach was more general as it allowed accumulations:  total += value. While your proposed function has been re-invented a number of times, that doesn't mean it's a good idea.  It is more an exercise in what can be done, not in what should be done.

Raymond

Josiah Carlson wrote:

iwinslice() is just as bad of a name as any of the others.

I have seen the equivalent of window(iterator, size=2, step=1), which works as you would expect (both as the output, as well as the implementation), with size and step both limited to 5 (because if you are doing things with more than 5 items at a time...you probably really want something else, and in certain cases, you can use multiple window calls to compose larger groups).

Oops, I didn't realise this thread had moved over here, so I just repeated what yourself and Raymond said over on python-dev. Oh well...

I'd be a -0 on the feature, because as Raymond says, it's trivial to implement with a deque. And as I've said before, not all x line functions should be built-in.

That does raise the possibility of adding "iterator windowing done right" by including a deque based implementation in itertools (or at least in the itertools recipes page). For example, the following continuously yields the same deque, but each time the contents represent a new window onto the underlying data: from collections import deque def window (iterable, size=2, step=1, overlap=0): itr = iter(iterable) new_per_window = size - overlap contents = deque(islice(itr, 0, size*step, step), size) while True: yield contents new_data = list(islice(itr, 0, new_per_window*step, step)) if len(new_data) < new_per_window: break contents.extend(new_data) (There are other ways of doing it that involve less data copying, but the above way seems to be the most straightforward) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

Now I discover itertools I thing your are right, but maybe the pack function could be rename iwinslice (at the end it's its real name), and add it to itertools ?? paul bedaride On Fri, Mar 20, 2009 at 6:01 PM, Raymond Hettinger <python@rcn.com> wrote:

...

...

I propose a new function for list for pack values of a list and sliding over them:

then we can do things like this: for i, j, k in pack(range(10), 3, partialend=False): print i, j, k

. . .

...

...

def pack(l, size=2, slide=2, partialend=True): lenght = len(l) for p in range(0,lenght-size,slide): def packet(): for i in range(size): yield l[p+i] yield packet() p = p + slide if partialend or lenght-p == size: def packet(): for i in range(lenght-p): yield l[p+i] yield packet()

This has been discussed before and rejected.

There were several considerations.  The itertools recipes already include simple patterns for grouper() and pairwise() that are easy to use as primitives in your code or to serve as models for variants. The design of pack() itself is questionable.  It attempts to be a Swiss Army Knife by parameterizing all possible variations (length of window, length to slide, and how to handle end-cases). This design makes the tool harder to learn and use, and it makes the implementation more complex. That complexity isn't necessary.  Use cases would typically fall into grouper cases where the window length equals the slide length or into cases that slide one element at a time.  You don't win anything by combining the two cases except for more making the tool harder to learn and use.

The pairwise() recipe could be generalized to larger windows, but seemed like less of a good idea after closely examining potential use cases.  For cases that used a larger window, there always seemed to be a better solution than extending pairwise().  For instance, a twenty-day moving average is better implemented with a deque(maxlen=20) and a running sum than with an iterator returning tuples of length twenty -- that approach does a lot of unnecessary work shifting elements in the tuple, turning an O(n) process into an O(m*n) process.

For short windows, like pairwise() itself, the issue is not one of total running time; instead, the problem is that almost every proposed use case was better coded as a simple Python loop, saving the value previous values with a step like:  oldvalue = value. Having pairwise() or tripletwise() tended to be a distraction away from better solutions.  Also, the pure python approach was more general as it allowed accumulations:  total += value. While your proposed function has been re-invented a number of times, that doesn't mean it's a good idea.  It is more an exercise in what can be done, not in what should be done.

Raymond

Josiah Carlson wrote:

iwinslice() is just as bad of a name as any of the others.

I have seen the equivalent of window(iterator, size=2, step=1), which works as you would expect (both as the output, as well as the implementation), with size and step both limited to 5 (because if you are doing things with more than 5 items at a time...you probably really want something else, and in certain cases, you can use multiple window calls to compose larger groups).

Oops, I didn't realise this thread had moved over here, so I just repeated what yourself and Raymond said over on python-dev. Oh well...

I'd be a -0 on the feature, because as Raymond says, it's trivial to implement with a deque. And as I've said before, not all x line functions should be built-in.

That does raise the possibility of adding "iterator windowing done right" by including a deque based implementation in itertools (or at least in the itertools recipes page). For example, the following continuously yields the same deque, but each time the contents represent a new window onto the underlying data: from collections import deque def window (iterable, size=2, step=1, overlap=0): itr = iter(iterable) new_per_window = size - overlap contents = deque(islice(itr, 0, size*step, step), size) while True: yield contents new_data = list(islice(itr, 0, new_per_window*step, step)) if len(new_data) < new_per_window: break contents.extend(new_data) (There are other ways of doing it that involve less data copying, but the above way seems to be the most straightforward) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------