Hi all,
In the `operator` module, both `operator.concat` and `operator.add` are described as performing `a + b`. When defining a type in pure python, it is not possible to give these different meanings. However, when defining a C extension type, it is possible to fill both `tp_as_number->nb_add` and `tp_as_sequence->sq_concat` with different implementations to obtain different behaviors.
Is the fact that `operator.concat` (`PySequence_Concat`) and `operator.add` (`PyNumber_Add`) can have different behaviors an implementation detail of CPython, or is it intended behavior in the language itself?
I ask because [a recent PR for numpy][1] proposes implementing both of these slots, and discussion turned to whether it was intended to be possible to do this in the first place, and if so whether a patch will be needed to PyPy.
It seems that there was at least some vague intent for this to be possible - In bpo-29139, Serhiy said
Third-party classes (maybe NumPy arrays, I don't know) can have different implementations of `sq_concat` and `nb_add`.
It seems to me there are at least three stances that could be taken here:
* Specifying both is considered invalid: python should consider emitting a warning in `Type_READY` if both are filled. * Specifying both is considered an implementation detail specific to CPython: the [C API docs for the type slots][2] should indicate this * Specifying both is explicitly allowed and considered a language feature. `__concat__` should be added as a slot_wrapper around `sq_concat` to allow the language feature to be accessed without writing C extensions.
Eric Wieser
Apologies if this is not the right list - this didn't feel right for python-ideas or bpo. I'm more than happy to repost this elsewhere if asked.
[1]: https://github.com/numpy/numpy/issues/16489 [2]: https://docs.python.org/3/c-api/typeobj.html
But how would you invoke them, other than using operator.add and operator.concat? A better UI would be to define new methods with more discoverable names.
On Fri, Jun 12, 2020 at 7:29 AM Eric Wieser wieser.eric+numpy@gmail.com wrote:
Hi all,
In the `operator` module, both `operator.concat` and `operator.add` are described as performing `a + b`. When defining a type in pure python, it is not possible to give these different meanings. However, when defining a C extension type, it is possible to fill both `tp_as_number->nb_add` and `tp_as_sequence->sq_concat` with different implementations to obtain different behaviors.
Is the fact that `operator.concat` (`PySequence_Concat`) and `operator.add` (`PyNumber_Add`) can have different behaviors an implementation detail of CPython, or is it intended behavior in the language itself?
I ask because [a recent PR for numpy][1] proposes implementing both of these slots, and discussion turned to whether it was intended to be possible to do this in the first place, and if so whether a patch will be needed to PyPy.
It seems that there was at least some vague intent for this to be possible
- In bpo-29139, Serhiy said
Third-party classes (maybe NumPy arrays, I don't know) can have
different implementations of `sq_concat` and `nb_add`.
It seems to me there are at least three stances that could be taken here:
- Specifying both is considered invalid: python should consider emitting a
warning in `Type_READY` if both are filled.
- Specifying both is considered an implementation detail specific to
CPython: the [C API docs for the type slots][2] should indicate this
- Specifying both is explicitly allowed and considered a language feature.
`__concat__` should be added as a slot_wrapper around `sq_concat` to allow the language feature to be accessed without writing C extensions.
Eric Wieser
Apologies if this is not the right list - this didn't feel right for python-ideas or bpo. I'm more than happy to repost this elsewhere if asked.
Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HCXMI7LP... Code of Conduct: http://python.org/psf/codeofconduct/
But how would you invoke them, other than using operator.add and
operator.concat?
The case I was thinking of was just a direct call to `operator.concat`. The motivation here is simply to make `operator.concat` do something non-surprising on numpy arrays (be that `np.concatenate(a, b)`, or simply `raise TypeError`). I doubt numpy would actually advocate using it, but we can at least remove the foot-gun.
Eric
On Fri, 12 Jun 2020 at 17:18, Guido van Rossum guido@python.org wrote:
But how would you invoke them, other than using operator.add and operator.concat? A better UI would be to define new methods with more discoverable names.
On Fri, Jun 12, 2020 at 7:29 AM Eric Wieser wieser.eric+numpy@gmail.com wrote:
Hi all,
In the `operator` module, both `operator.concat` and `operator.add` are described as performing `a + b`. When defining a type in pure python, it is not possible to give these different meanings. However, when defining a C extension type, it is possible to fill both `tp_as_number->nb_add` and `tp_as_sequence->sq_concat` with different implementations to obtain different behaviors.
Is the fact that `operator.concat` (`PySequence_Concat`) and `operator.add` (`PyNumber_Add`) can have different behaviors an implementation detail of CPython, or is it intended behavior in the language itself?
I ask because [a recent PR for numpy][1] proposes implementing both of these slots, and discussion turned to whether it was intended to be possible to do this in the first place, and if so whether a patch will be needed to PyPy.
It seems that there was at least some vague intent for this to be possible - In bpo-29139, Serhiy said
Third-party classes (maybe NumPy arrays, I don't know) can have
different implementations of `sq_concat` and `nb_add`.
It seems to me there are at least three stances that could be taken here:
- Specifying both is considered invalid: python should consider emitting
a warning in `Type_READY` if both are filled.
- Specifying both is considered an implementation detail specific to
CPython: the [C API docs for the type slots][2] should indicate this
- Specifying both is explicitly allowed and considered a language
feature. `__concat__` should be added as a slot_wrapper around `sq_concat` to allow the language feature to be accessed without writing C extensions.
Eric Wieser
Apologies if this is not the right list - this didn't feel right for python-ideas or bpo. I'm more than happy to repost this elsewhere if asked.
Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HCXMI7LP... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/
On Fri, Jun 12, 2020 at 12:56 PM Eric Wieser wieser.eric+numpy@gmail.com wrote:
But how would you invoke them, other than using operator.add and
operator.concat?
The case I was thinking of was just a direct call to `operator.concat`. The motivation here is simply to make `operator.concat` do something non-surprising on numpy arrays (be that `np.concatenate(a, b)`, or simply `raise TypeError`). I doubt numpy would actually advocate using it, but we can at least remove the foot-gun.
Oh, that makes sense, as a CPython specific behavior. What does it do currently? Is the problem that it does the same thing as add? I'd vote for raising -- you shouldn't encourage or enable code that uses this pattern. (In general I am a big fan of just never using the operator module.)
On Jun 12, 2020, at 04:21, Eric Wieser wieser.eric+numpy@gmail.com wrote:
It seems to me there are at least three stances that could be taken here:
- Specifying both is considered invalid: python should consider emitting a warning in `Type_READY` if both are filled.
- Specifying both is considered an implementation detail specific to CPython: the [C API docs for the type slots][2] should indicate this
- Specifying both is explicitly allowed and considered a language feature. `__concat__` should be added as a slot_wrapper around `sq_concat` to allow the language feature to be accessed without writing C extensions.
If you can define the behavior at the C layer and not at the Python layer, then I think almost by definition it’s an implementation detail of CPython. Whether that’s intentional or not would help determine whether this should be demoted to a bug with an emitted warning, or promoted to a full-on feature supported in the language.
Cheers, -Barry
On Sat., 13 Jun. 2020, 2:51 am Barry Warsaw, barry@python.org wrote:
On Jun 12, 2020, at 04:21, Eric Wieser wieser.eric+numpy@gmail.com wrote:
It seems to me there are at least three stances that could be taken here:
- Specifying both is considered invalid: python should consider emitting
a warning in `Type_READY` if both are filled.
- Specifying both is considered an implementation detail specific to
CPython: the [C API docs for the type slots][2] should indicate this
- Specifying both is explicitly allowed and considered a language
feature. `__concat__` should be added as a slot_wrapper around `sq_concat` to allow the language feature to be accessed without writing C extensions.
If you can define the behavior at the C layer and not at the Python layer, then I think almost by definition it’s an implementation detail of CPython. Whether that’s intentional or not would help determine whether this should be demoted to a bug with an emitted warning, or promoted to a full-on feature supported in the language.
The one part that has already sort of leaked through to the language level is that only "nb_add" fully supports operand coercion.
https://bugs.python.org/issue11477 covers some of the oddities that arise when implementing *only* sq_concat.
I've mostly given up on ever fixing that, as I'm not sure we can do it without breaking the workarounds that people have in place for the oddities in the status quo (there were enough other projects relying on the existing semantics that the PyPy devs found it necessary to mimic CPython's implementation quirks in this regard).
That said, adding an sq_concat that raises a "don't do that" exception to an existing type that already implements nb_add sounds fine to me - from the Python level everything except operator.concat would encounter the nb_add slot first, so the exception should only trip up genuinely problematic code.
Cheers, Nick.
Cheers, -Barry
Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DO5J63K3... Code of Conduct: http://python.org/psf/codeofconduct/
-
Barry Warsaw -
Eric Wieser -
Guido van Rossum -
Nick Coghlan