Are extension types allowed to implement both nb_add and sq_concat?
Hi all, In the `operator` module, both `operator.concat` and `operator.add` are described as performing `a + b`. When defining a type in pure python, it is not possible to give these different meanings. However, when defining a C extension type, it is possible to fill both `tp_as_number->nb_add` and `tp_as_sequence->sq_concat` with different implementations to obtain different behaviors. Is the fact that `operator.concat` (`PySequence_Concat`) and `operator.add` (`PyNumber_Add`) can have different behaviors an implementation detail of CPython, or is it intended behavior in the language itself? I ask because [a recent PR for numpy][1] proposes implementing both of these slots, and discussion turned to whether it was intended to be possible to do this in the first place, and if so whether a patch will be needed to PyPy. It seems that there was at least some vague intent for this to be possible - In bpo-29139, Serhiy said
Third-party classes (maybe NumPy arrays, I don't know) can have different implementations of `sq_concat` and `nb_add`.
It seems to me there are at least three stances that could be taken here: * Specifying both is considered invalid: python should consider emitting a warning in `Type_READY` if both are filled. * Specifying both is considered an implementation detail specific to CPython: the [C API docs for the type slots][2] should indicate this * Specifying both is explicitly allowed and considered a language feature. `__concat__` should be added as a slot_wrapper around `sq_concat` to allow the language feature to be accessed without writing C extensions. Eric Wieser Apologies if this is not the right list - this didn't feel right for python-ideas or bpo. I'm more than happy to repost this elsewhere if asked. [1]: https://github.com/numpy/numpy/issues/16489 [2]: https://docs.python.org/3/c-api/typeobj.html
But how would you invoke them, other than using operator.add and operator.concat? A better UI would be to define new methods with more discoverable names. On Fri, Jun 12, 2020 at 7:29 AM Eric Wieser <wieser.eric+numpy@gmail.com> wrote:
Hi all,
In the `operator` module, both `operator.concat` and `operator.add` are described as performing `a + b`. When defining a type in pure python, it is not possible to give these different meanings. However, when defining a C extension type, it is possible to fill both `tp_as_number->nb_add` and `tp_as_sequence->sq_concat` with different implementations to obtain different behaviors.
Is the fact that `operator.concat` (`PySequence_Concat`) and `operator.add` (`PyNumber_Add`) can have different behaviors an implementation detail of CPython, or is it intended behavior in the language itself?
I ask because [a recent PR for numpy][1] proposes implementing both of these slots, and discussion turned to whether it was intended to be possible to do this in the first place, and if so whether a patch will be needed to PyPy.
It seems that there was at least some vague intent for this to be possible - In bpo-29139, Serhiy said
Third-party classes (maybe NumPy arrays, I don't know) can have different implementations of `sq_concat` and `nb_add`.
It seems to me there are at least three stances that could be taken here:
* Specifying both is considered invalid: python should consider emitting a warning in `Type_READY` if both are filled. * Specifying both is considered an implementation detail specific to CPython: the [C API docs for the type slots][2] should indicate this * Specifying both is explicitly allowed and considered a language feature. `__concat__` should be added as a slot_wrapper around `sq_concat` to allow the language feature to be accessed without writing C extensions.
Eric Wieser
Apologies if this is not the right list - this didn't feel right for python-ideas or bpo. I'm more than happy to repost this elsewhere if asked.
[1]: https://github.com/numpy/numpy/issues/16489 [2]: https://docs.python.org/3/c-api/typeobj.html _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HCXMI7LP... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
But how would you invoke them, other than using operator.add and operator.concat?
The case I was thinking of was just a direct call to `operator.concat`. The motivation here is simply to make `operator.concat` do something non-surprising on numpy arrays (be that `np.concatenate(a, b)`, or simply `raise TypeError`). I doubt numpy would actually advocate using it, but we can at least remove the foot-gun. Eric On Fri, 12 Jun 2020 at 17:18, Guido van Rossum <guido@python.org> wrote:
But how would you invoke them, other than using operator.add and operator.concat? A better UI would be to define new methods with more discoverable names.
On Fri, Jun 12, 2020 at 7:29 AM Eric Wieser <wieser.eric+numpy@gmail.com> wrote:
Hi all,
In the `operator` module, both `operator.concat` and `operator.add` are described as performing `a + b`. When defining a type in pure python, it is not possible to give these different meanings. However, when defining a C extension type, it is possible to fill both `tp_as_number->nb_add` and `tp_as_sequence->sq_concat` with different implementations to obtain different behaviors.
Is the fact that `operator.concat` (`PySequence_Concat`) and `operator.add` (`PyNumber_Add`) can have different behaviors an implementation detail of CPython, or is it intended behavior in the language itself?
I ask because [a recent PR for numpy][1] proposes implementing both of these slots, and discussion turned to whether it was intended to be possible to do this in the first place, and if so whether a patch will be needed to PyPy.
It seems that there was at least some vague intent for this to be possible - In bpo-29139, Serhiy said
Third-party classes (maybe NumPy arrays, I don't know) can have different implementations of `sq_concat` and `nb_add`.
It seems to me there are at least three stances that could be taken here:
* Specifying both is considered invalid: python should consider emitting a warning in `Type_READY` if both are filled. * Specifying both is considered an implementation detail specific to CPython: the [C API docs for the type slots][2] should indicate this * Specifying both is explicitly allowed and considered a language feature. `__concat__` should be added as a slot_wrapper around `sq_concat` to allow the language feature to be accessed without writing C extensions.
Eric Wieser
Apologies if this is not the right list - this didn't feel right for python-ideas or bpo. I'm more than happy to repost this elsewhere if asked.
[1]: https://github.com/numpy/numpy/issues/16489 [2]: https://docs.python.org/3/c-api/typeobj.html _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HCXMI7LP... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Fri, Jun 12, 2020 at 12:56 PM Eric Wieser <wieser.eric+numpy@gmail.com> wrote:
But how would you invoke them, other than using operator.add and operator.concat?
The case I was thinking of was just a direct call to `operator.concat`. The motivation here is simply to make `operator.concat` do something non-surprising on numpy arrays (be that `np.concatenate(a, b)`, or simply `raise TypeError`). I doubt numpy would actually advocate using it, but we can at least remove the foot-gun.
Oh, that makes sense, as a CPython specific behavior. What does it do currently? Is the problem that it does the same thing as add? I'd vote for raising -- you shouldn't encourage or enable code that uses this pattern. (In general I am a big fan of just never using the operator module.) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Jun 12, 2020, at 04:21, Eric Wieser <wieser.eric+numpy@gmail.com> wrote:
It seems to me there are at least three stances that could be taken here:
* Specifying both is considered invalid: python should consider emitting a warning in `Type_READY` if both are filled. * Specifying both is considered an implementation detail specific to CPython: the [C API docs for the type slots][2] should indicate this * Specifying both is explicitly allowed and considered a language feature. `__concat__` should be added as a slot_wrapper around `sq_concat` to allow the language feature to be accessed without writing C extensions.
If you can define the behavior at the C layer and not at the Python layer, then I think almost by definition it’s an implementation detail of CPython. Whether that’s intentional or not would help determine whether this should be demoted to a bug with an emitted warning, or promoted to a full-on feature supported in the language. Cheers, -Barry
On Sat., 13 Jun. 2020, 2:51 am Barry Warsaw, <barry@python.org> wrote:
On Jun 12, 2020, at 04:21, Eric Wieser <wieser.eric+numpy@gmail.com> wrote:
It seems to me there are at least three stances that could be taken here:
* Specifying both is considered invalid: python should consider emitting a warning in `Type_READY` if both are filled. * Specifying both is considered an implementation detail specific to CPython: the [C API docs for the type slots][2] should indicate this * Specifying both is explicitly allowed and considered a language feature. `__concat__` should be added as a slot_wrapper around `sq_concat` to allow the language feature to be accessed without writing C extensions.
If you can define the behavior at the C layer and not at the Python layer, then I think almost by definition it’s an implementation detail of CPython. Whether that’s intentional or not would help determine whether this should be demoted to a bug with an emitted warning, or promoted to a full-on feature supported in the language.
The one part that has already sort of leaked through to the language level is that only "nb_add" fully supports operand coercion. https://bugs.python.org/issue11477 covers some of the oddities that arise when implementing *only* sq_concat. I've mostly given up on ever fixing that, as I'm not sure we can do it without breaking the workarounds that people have in place for the oddities in the status quo (there were enough other projects relying on the existing semantics that the PyPy devs found it necessary to mimic CPython's implementation quirks in this regard). That said, adding an sq_concat that raises a "don't do that" exception to an existing type that already implements nb_add sounds fine to me - from the Python level everything except operator.concat would encounter the nb_add slot first, so the exception should only trip up genuinely problematic code. Cheers, Nick.
Cheers, -Barry
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DO5J63K3... Code of Conduct: http://python.org/psf/codeofconduct/
participants (4)
-
Barry Warsaw -
Eric Wieser -
Guido van Rossum -
Nick Coghlan