[Python-ideas] Add irange with large integer step support to itertools

[Nick Coghlan]

On Mon, Jan 10, 2011 at 6:27 PM, Mark Dickinson <dickinsm at gmail.com> wrote:
>> Python 3.1.3 (r313:86834, Nov 28 2010, 10:01:07)
>> [GCC 4.4.5] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>>
>>>>> range(10**10000, 10**10000+10**1000, 10**900)[5]
>> Traceback (most recent call last):
>>  File "<stdin>", line 1, in <module>
>> OverflowError: Python int too large to convert to C ssize_t

Note that the problem isn't actually the step value - it's the overall
length of the resulting sequence.

If you make the sequence shorter, it works (at least in 3.2, I didn't
check earlier versions):

>>> x = range(10**10000, 10**10000+(500*10**900), 10**900)
>>> len(x)
500
>>> x[5]
<snip really big number>

> This example strikes me as a bug in range (specifically, in
> range_subscript in Objects/rangeobject.c).

The main issue is actually in range_item rather than range_subscript -
we invoke range_len() there to simplify the bounds checking logic. To
remove this limitation, the C arithmetic and comparison operations in
that function need to be replaced with their PyLong equivalent,
similar to what has been done for compute_range_length().

There's a related bug where range_subscript doesn't support *indices*
greater than sys.maxsize - given an indexing helper function that can
handle a range length that doesn't fit in sys.maxsize, it would be
easy to call that unconditionally rather than indirectly via
range_item, fixing that problem as well.

>> Does such an addition make sense in your eyes?
>
> Wouldn't it be better to fix 'range' to behave as expected?

Agreed. It isn't a deliberate design limitation - it's just a
consequence of the fact that converting from C integer programming to
PyLong programming is a PITA, so it has been a process of progressive
upgrades in range's support for values that don't fit in sys.maxsize.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia