On Sat, Mar 3, 2012 at 11:24, Stefan Behnel wrote:

Eli Bendersky, 03.03.2012 09:36:

...

I find a strange discrepancy in Python with regards to slice subscripting of objects, at the C API level. I mean things like obj[start:end:step].

I'd expect slice subscripts to be part of the sequence interface, and yet they are not. In fact, they are part of the mapping interface. For example, the list object has its slice get/set methods assigned to a PyMappingMethods struct. So does a bytes object, and pretty much every other object that wants to support subscripts.

This doesn't align well with the documentation, in at least two places.

1) The library documentation (http://docs.python.org/dev/library/stdtypes.html) in 4.8 says:

    "Mappings are mutable objects. There is currently only one standard mapping type, the dictionary"

Why then does a list implement the mapping interface? Moreover, why does bytes, an immutable object, implement the mapping interface?

I think that's (partly?) for historical reasons. Originally, there were the slicing functions as part of the sequence interface. They took a start and an end index of the slice. Then, extended slicing was added to the language, and that used a slice object, which didn't fit into the sequence slicing interface. So the interface was unified using the existing mapping getitem interface, and the sequence slicing functions were eventually deprecated and removed in Py3.

This make sense. Not that now there's also duplication in almost all objects because the mapping protocol essentially supersedes the sequence protocol for accessing elements. I.e. sq_item and sq_ass_item are no longer needed if an object implements the mapping protocol, because the mapping interface has precedence, and mp_subscript & mp_ass_subscript are called instead, respectively. Because of that, the first thing they do is check whether the index is a simple number and do the work of their sequence protocol cousins. This duplicates code in almost all objects that need to support __getitem__. Eli

...

I'd expect slice subscripts to be part of the sequence interface, and yet they are not. In fact, they are part of the mapping interface. For example, the list object has its slice get/set methods assigned to a PyMappingMethods struct. So does a bytes object, and pretty much every other object that wants to support subscripts.

It comes from: http://hg.python.org/cpython/rev/245224d1b8c9 http://bugs.python.org/issue400998

Written by Michael Hudson and reviewed by Guido. I wonder why this patch chose to add mapping protocol support to tuples and lists, rather than add a tp_ slot for extended slicing.

Why a separate tp_ slot for extended slicing? ISTM slicing pertains to sequences, similarly to other numeric indices. If you look at PySequenceMethods it has these (apparently no longer used fields): void *was_sq_slice; void *was_sq_ass_slice; These were "simple" slices (pairs of numbers). I suppose if any change is considered, these fields can be re-incarnated to accept PyObject* slices similarly to the current mp_subscript and mp_ass_subscript. Eli

On Sat, Mar 3, 2012 at 4:20 AM, Antoine Pitrou wrote:

...

I'd expect slice subscripts to be part of the sequence interface, and yet they are not. In fact, they are part of the mapping interface. For example, the list object has its slice get/set methods assigned to a PyMappingMethods struct. So does a bytes object, and pretty much every other object that wants to support subscripts.

It comes from: http://hg.python.org/cpython/rev/245224d1b8c9 http://bugs.python.org/issue400998

Written by Michael Hudson and reviewed by Guido. I wonder why this patch chose to add mapping protocol support to tuples and lists, rather than add a tp_ slot for extended slicing.

That's long ago... IIRC it was for binary compatibility -- I didn't want to add an extra slot to the sq struct because it would require recompilation of 3rd party extensions. At the time that was an important concern. -- --Guido van Rossum (python.org/~guido)

On Sat, Mar 3, 2012 at 10:18, Eli Bendersky wrote:

On Sat, Mar 3, 2012 at 19:58, Guido van Rossum wrote:

...

On Sat, Mar 3, 2012 at 4:20 AM, Antoine Pitrou wrote:

...

...

I'd expect slice subscripts to be part of the sequence interface, and yet they are not. In fact, they are part of the mapping interface. For example, the list object has its slice get/set methods assigned to a PyMappingMethods struct. So does a bytes object, and pretty much every other object that wants to support subscripts.

It comes from: http://hg.python.org/cpython/rev/245224d1b8c9 http://bugs.python.org/issue400998

Written by Michael Hudson and reviewed by Guido. I wonder why this patch chose to add mapping protocol support to tuples and lists, rather than add a tp_ slot for extended slicing.

That's long ago... IIRC it was for binary compatibility -- I didn't want to add an extra slot to the sq struct because it would require recompilation of 3rd party extensions. At the time that was an important concern.

Perhaps the situation can be fixed now without binary compatibility concerns. PySequenceMethods is:

typedef struct { lenfunc sq_length; binaryfunc sq_concat; ssizeargfunc sq_repeat; ssizeargfunc sq_item; void *was_sq_slice; ssizeobjargproc sq_ass_item; void *was_sq_ass_slice; objobjproc sq_contains;

binaryfunc sq_inplace_concat; ssizeargfunc sq_inplace_repeat; } PySequenceMethods;

The slots "was_sq_slice" and "was_sq_ass_slice" aren't used any longer. These can be re-incarnated to accept a slice object, and sequence objects can be rewritten to use them instead of implementing the mapping protocol (is there any reason listobject implements the mapping protocol, other than to gain the ability to use slices for __getitem__?). Existing 3rd party extensions don't *need* to be recompiled or changed, however. They *can* be, if their authors are interested, of course.

Why even have separate tp_as_sequence and tp_as_mapping anymore? That particular distinction never existed for Python types, so why should it exist for C types at all? I forget if there was ever a real point to it, but all it seems to do now is create confusion, what with many sequence types implementing both, and PyMapping_Check() and PySequence_Check() doing seemingly random things to come up with somewhat sensible answers. Do note that the dict type actually implements tp_as_sequence (in order to support containtment tests) and that PySequence_Check() has to explicitly return 0 for dicts -- which means that it will give the "wrong" answer for another type that behaves exactly like dicts. Getting rid of the misleading distinction seems like a much better idea than trying to re-conflate some of the issues. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

On Sat, Mar 3, 2012 at 13:02, Antoine Pitrou wrote:

On Sat, 3 Mar 2012 12:59:13 -0800 Thomas Wouters wrote:

...

Why even have separate tp_as_sequence and tp_as_mapping anymore? That particular distinction never existed for Python types, so why should it exist for C types at all? I forget if there was ever a real point to it, but all it seems to do now is create confusion, what with many sequence types implementing both, and PyMapping_Check() and PySequence_Check()

doing

...

seemingly random things to come up with somewhat sensible answers.

Ironically, most of the confusion stems from sequence types implementing the mapping protocol for extended slicing.

...

Do note that the dict type actually implements tp_as_sequence (in order to support containtment tests) and that PySequence_Check() has to explicitly return 0 for dicts -- which means that it will give the "wrong" answer for another type that behaves exactly like dicts.

It seems to be a leftover:

int PySequence_Check(PyObject *s) { if (PyDict_Check(s)) return 0; return s != NULL && s->ob_type->tp_as_sequence && s->ob_type->tp_as_sequence->sq_item != NULL; }

Dict objects have a NULL sq_item so even removing the explicit check would still return the right answer.

...

Getting rid of the misleading distinction seems like a much better idea than trying to re-conflate some of the issues.

This proposal sounds rather backwards, given that we now have separate Mapping and Sequence ABCs.

I'm not sure how the ABCs, which are abstract declarations of semantics, tie into this specific implementation detail. ABCs work just as well for Python types as for C types, and Python types don't have this distinction. The distinction in C types has been *practically* useless for years, so why should it stay? What is the actual benefit here? -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

Thomas Wouters, 03.03.2012 21:59:

Why even have separate tp_as_sequence and tp_as_mapping anymore? That particular distinction never existed for Python types, so why should it exist for C types at all? I forget if there was ever a real point to it, but all it seems to do now is create confusion, what with many sequence types implementing both, and PyMapping_Check() and PySequence_Check() doing seemingly random things to come up with somewhat sensible answers. Do note that the dict type actually implements tp_as_sequence (in order to support containtment tests) and that PySequence_Check() has to explicitly return 0 for dicts -- which means that it will give the "wrong" answer for another type that behaves exactly like dicts.

Getting rid of the misleading distinction seems like a much better idea than trying to re-conflate some of the issues.

We're too far away from the release of Python 4 to change something with that kind of impact, though. Stefan

...

Thomas Wouters, 03.03.2012 21:59:

...

Why even have separate tp_as_sequence and tp_as_mapping anymore? That particular distinction never existed for Python types, so why should it exist for C types at all? I forget if there was ever a real point to it, but all it seems to do now is create confusion, what with many sequence types implementing both, and PyMapping_Check() and PySequence_Check() doing seemingly random things to come up with somewhat sensible answers. Do note that the dict type actually implements tp_as_sequence (in order to support containtment tests) and that PySequence_Check() has to explicitly return 0 for dicts -- which means that it will give the "wrong" answer for another type that behaves exactly like dicts.

Getting rid of the misleading distinction seems like a much better idea than trying to re-conflate some of the issues.

We're too far away from the release of Python 4 to change something with that kind of impact, though.

It's not hard to do this in a backward-compatible way. Either grow one of the tp_as_* to include everything a 'unified' tp_as_everything struct would need, or add a new tp_as_everything slot in the type struct. Then add a tp_flag to indicate that the type has this new layout/slot and guard all uses of the new slots with a check for that flag. If the type doesn't have the new layout or doesn't have it or the slots in it set, the code can fall back to the old try-one-and-then-the-other behaviour of dealing with tp_as_sequence and tp_as_mapping.

(Let's not forget about tp_as_sequence.sq_concat, tp_as_number.nb_add, tp_as_sequence.sq_repeat and tp_as_number.nb_mul either.)

There's nothing to unify, really, since PyMappingMethods is just a subset of PySequenceMethods: typedef struct { lenfunc mp_length; binaryfunc mp_subscript; objobjargproc mp_ass_subscript; } PyMappingMethods; with the small difference that in PySequenceMethods sq_item and sq_ass_item just accept numeric indices. However, if PySequenceMethods has the was_sq_sclies and was_sq_ass_slice fields are reinstated to accept a generic PyObject, PyMappingMethods will be a true subset. If we look at the code, this becomes even clearer: in a full grep on the Python 3.3 source, there is no object that defines tp_as_mapping but does not also define tp_as_sequence, except Modules/_sqlite/row.c [I'm not familiar enough with the _sqlite module, but there's a chance it would make sense for the Row to be a sequence too]. Eli

Thomas Wouters wrote:

Why even have separate tp_as_sequence and tp_as_mapping anymore? That particular distinction never existed for Python types, so why should it exist for C types at all? I forget if there was ever a real point to it,

I imagine the original motivation was to provide a fast path for types that take ints as indexes. Also, it dates from the very beginnings of Python, before it had user defined classes. At that time the archetypal sequence (list) and the archetypal mapping (dict) were very distinct -- I don't think dicts supported 'in' then, so there was no overlap. It looks like a case of "it seemed like a good idea at the time". The distinction broke down fairly soon after, but it's so embedded in the extension module API that it's been very hard to get rid of. -- Greg