[ python-Feature Requests-1326277 ] itertools.count wraps around after maxint

[SourceForge.net]

Feature Requests item #1326277, was opened at 2005-10-13 23:27
Message generated for change (Comment added) made by phr
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1326277&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Extension Modules
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: paul rubin (phr)
Assigned to: Raymond Hettinger (rhettinger)
Summary: itertools.count wraps around after maxint

Initial Comment:
See below.  This goes against the notion of int/long
unification and can cause weird unexpected behavior,
almost like a buffer overflow.  It should promote to a
long at the appropriate time, or that's not feasible,
then raise an exception, don't wrap around silently. 
Xrange is also not so great about this.  It at least
raises an exception if you give it too large an
endpoint, but promoting to long would be better.

Steven D'Aprano and others on clpy pointed this out.

>>> from itertools import count
>>> b=2**31 - 3
>>> c = count(b)
>>> for i in range(5):
...    print c.next()
...
2147483645
2147483646
2147483647
-2147483648
-2147483647
>>>


----------------------------------------------------------------------

>Comment By: paul rubin (phr)
Date: 2005-10-14 15:11

Message:
Logged In: YES 
user_id=72053

You're right, the docs do describe that behavior of
itertools.count (someone on clpy thought otherwise, IIRC). 
I don't see anything in
http://docs.python.org/lib/built-in-funcs.html about what
enumerate does.  It looks my p3-750 can enumerate about 400k
iterations of itertools.repeat(None) per second, so it can
reach maxint in about 1.5 hours, but I don't feel like
running it that long to see what happens. At minimum,
enumerate's doc should be updated to say what it does.

I suppose it's a matter of opinion but I'd take the view
that both of these wraparounds (assuming enumerate wraps
around) are bugs even if they're documented bugs. 

----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2005-10-14 07:12

Message:
Logged In: YES 
user_id=80475

It's not a bug -- it is the documented behavior.

The simple work-around is to roll your own generators:

    def count(n):
        while 1:
             yield n
             n += 1

    def enumerate(iterable):
         c = 0
         for elem in iterable:
               yield (c, elem)
               c += 1

Will look at possibly enhancing this for Py2.5.

----------------------------------------------------------------------

Comment By: paul rubin (phr)
Date: 2005-10-14 05:53

Message:
Logged In: YES 
user_id=72053

If both fixes are feasible then promoting to long is
definitely the correct one.  Raising an exception is just a
kludge to at least not do something horrible silently. 
However, on a fast 32 bit machine, counting past 2**31 or
something is quite realistic.  A web server might send out
that many packets in a few days or weeks.  It shouldn't
crash or go crazy after running for a long time and
overflowing maxint.

It occurs to me, the enumerate iterator should also be
checked and fixed if needed.  It, too, can overflow maxint
if counting something like a network stream.  Maybe there
are some more iterators besides these, which need the same
attention.

----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2005-10-14 05:18

Message:
Logged In: YES 
user_id=33168

I agree something should be done.  Raymond, which behaviour
would you prefer?  I can implement if you want (just let me
know and assign back to me).

BTW, I don't have the same problem.  I need to set b = 2**63
- 3 :-)
(using current CVS).

----------------------------------------------------------------------

Comment By: paul rubin (phr)
Date: 2005-10-13 23:29

Message:
Logged In: YES 
user_id=72053

I forgot to say, the test example is Python 2.4.1 on linux.
 I don't know if 2.4.2 has a fix.  I don't see anything
about it in the database.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1326277&group_id=5470