[Python-Dev] Bad interaction of __index__ and sequence repeat

[Guido van Rossum]

On 7/28/06, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Nick Coghlan wrote:
> > Tim Peters wrote:
> >> OTOH, in the long discussion about PEP 357, I'm not sure anyone except
> >> Travis was clear on whether nb_index was meant to apply only to
> >> sequence /slicing/ or was meant to apply "everywhere an object gets
> >> used in an index-like context".  Clipping makes sense only for the
> >> former, but it looks like the implementation treats it more like the
> >> latter.  This was probably exacerbated by:
> >>
> >>     http://mail.python.org/pipermail/python-dev/2006-February/060663.html
> >>
> >>     [Travis]
> >>     There are other places in Python that check specifically for int objects
> >>     and long integer objects and fail with anything else.  Perhaps all of
> >>     these should aslo call the __index__ slot.
> >>
> >>     [Guido]
> >>     Right, absolutely.
> >>
> >> This is a mess :-)
> >
> > I've been trawling through the code a bit, and I don't think it's as bad as
> > all that.
>
> Damn, it really is a mess. . . nb_index returns the Pyssize_t directly, and a
> whole heap of the code expects errors to be signalled via returning -1 before
> checking PyErr_Occurred().
>
> To get it to work without clipping everywhere, wrap_lenfunc (typeobject.c),
> _PyEval_SliceIndex (ceval.c), PyNumber_Index (abstract.c) and sequence_repeat
> (abstract.c) all had to be modified to recognize PY_SSIZE_T_MIN and
> PY_SSIZE_T_MAX as potential error flags (in order to clear the overflow error
> for _PyEval_SliceIndex, and in order to propagate the exception for the other
> three).
>
> And using this approach still means that (2**100).__index__() raises an
> OverflowError.
>
> It would probably be cleaner to change the signature of nb_index to return a
> PyObject *, and let the code that uses it worry about how (or even whether!)
> to convert PyLong results to a Py_ssize_t.

No time to look through the code here, but IMO it's acceptable (at
least for 2.5) if (2**100).__index__() raises OverflowError, as long
as x[:2**100] silently clips. __index__() is primarily meant to return
a value useful for indexing concrete sequences, and 2**100 isn't.

Certainly the exception is preferrable to the silently truncated
result currently returned.

Fortunately there's some extra time since we're now going to do a third beta.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)