[Python-bugs-list] [ python-Bugs-681003 ] itertools issues

[SourceForge.net]

Bugs item #681003, was opened at 2003-02-05 10:52
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=681003&group_id=5470

Category: Python Library
Group: Python 2.3
>Status: Closed
>Resolution: Fixed
Priority: 5
Submitted By: Greg Chapman (glchapman)
Assigned to: Raymond Hettinger (rhettinger)
Summary: itertools issues

Initial Comment:
I've been checking out the new itertools module.  It's a great 
addition; I look forward to using it.  However, I did notice a 
few things:

1) I believe islice should worry about overflow when it adds 
step to next.  Given a call like islice(seq, 1, 10, sys.maxint), 
it looks like islice will not stop after retrieving item 1 
because this:

  lz->next += lz->step;

will result in lz->next becoming negative.

2) I note that starmap_next passes the args obtained from 
PyIter_Next directly to PyObject_Call.  PyObject_Call 
assumes that its arg param is a tuple (and so do the various 
tp_call slots).  However, there is nothing to ensure that the 
args object obtained from the iterator in starmap_next is in 
fact a tuple.  I believe either a PyTuple_Check or a 
PySequence_Tuple call is in order before passing the args 
to PyObject_Call.

3) I don't think it's safe for imap_next to reuse the same 
argtuple for all the calls.  This tuple is passed directly to C 
functions, and there's nothing to stop them from saving a 
reference to it someplace (unless it's documented 
somewhere that you shouldn't save references to the args 
tuple).

4) I think count_next should worry about overflow also 
(consider count(sys.maxint)).


----------------------------------------------------------------------

>Comment By: Raymond Hettinger (rhettinger)
Date: 2003-02-07 00:44

Message:
Logged In: YES 
user_id=80475

Committed changes as:
Modules/itertoolsmodule.c 1.2
Doc/lib/libitertools.tex 1.2
Lib/test/test_itertools.py 1.3

Documented the overflow behavior for count() but may 
still decide to add overflow handling at a later date.  All of 
the other ideas were implemented.



----------------------------------------------------------------------

Comment By: Christos Georgiou (tzot)
Date: 2003-02-06 12:19

Message:
Logged In: YES 
user_id=539787

Yes, itertools is great, and I wonder how Guido agreed to it 
(since he usually dislikes using FP stuff in Python).  I also 
think that the tuple way checking refcount will work.

Another point comes to mind: in many places, the code 
(Python in general, not only itertools) builds a tuple and calls 
PyEval_CallObject with it and without any keyword 
arguments; I believe in these cases the call should be 
PyObject_Call, since there's no use in checking every time if 
first arg is a tuple and then calling PyObject_Call...

----------------------------------------------------------------------

Comment By: Greg Chapman (glchapman)
Date: 2003-02-06 11:47

Message:
Logged In: YES 
user_id=86307

Just a quick note on imap.  I checked the 2.3a1 map function 
before posting (and just rechecked).  It creates the alist tuple at 
the beginning of the loop, and then eventually passes it to 
PyEval_CallObject before DECREFing it.  So it looks to me like 
the current version of map guards against someone saving a 
reference to alist.


----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2003-02-06 10:54

Message:
Logged In: YES 
user_id=80475

Glad you like the module.

Will implement the limit overflow checks and the tuple 
check.

I'm still thinking about the re-use of the arg tuple in 
imap_next().  The code was lifted directly from 
__builtins__.map() which has worked flawlessly since the 
early versions of Python.   But, if it does need fixing, then 
I'm looking at an approach that verifies the refcnt is one 
before re-using the tuple.  If that works out, then both the 
re-use and the guard can be applied to imap(), map(), izip
(), and enumerate().

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=681003&group_id=5470