enumerate overflow

Tim Chase python.list at tim.thechases.com
Wed Oct 3 16:42:48 CEST 2007


>> for lineNum, line in enumerate(f): ...
>>
>> However, lineNum soon overflows and starts counting backwards. How do
>> i force enumerate to return long integer?
>>
> Just how "soon" exactly do you read sys.maxint lines from a file? I 
> should have thought that it would take a significant amount of time to 
> read 2,147,483,647 lines ...

A modestly (but not overwhelmingly) long time:

(defining our own xrange-ish generator that can handle things 
larger than longs)

 >>> def xxrange(x):
...     i = 0
...     while i < x:
...             yield i
...             i += 1
...
 >>> for i,j in enumerate(xxrange(2**33)): assert i==j
...
Traceback (most recent call last):
   File "<stdin>", line 1, in ?
AssertionError


It took me about an 60-90 minutes to hit the assertion on a 
dual-core 2.8ghz machine under otherwise-light-load.  If 
batch-processing lengthy log files or other large data such as 
genetic data, it's entirely possible to hit this limit as the OP 
discovered.

-tkc






More information about the Python-list mailing list