**= does not follow the data model
If you read the language reference for augmented arithmetic assignment, you will note that it essentially says, "call __i<op>__, and if that doesn't work call as if you were doing a <op> b". Unfortunately it appears **= does not follow the rule of falling back on the binary arithmetic expression semantics. I have a GitHub gist with demonstration code that shows this happening in both 3.8 and master ( https://gist.github.com/brettcannon/fec4152857e0ed551b4da515dc63e580). This was reported in https://bugs.python.org/issue38302, although initially it didn't cover __pow__, only __rpow__ being skipped. This appears to happen because the opcode for **= calls PyNumber_InPlacePower() ( https://github.com/python/cpython/blob/802726acf6048338394a6a4750835c2cdd6a9...) which calls ternary_op for __ipow__ or __pow__ depending on which is defined, but will never try both ( https://github.com/python/cpython/blob/802726acf6048338394a6a4750835c2cdd6a9...). All of the other augmented arithmetic assignment operators have a special binary_iop() function to call which takes care of the fallback logic, so no other augmented arithmetic assignments appear to have this problem (I tested them all regardless). I think there are two options to fixing this: 1. Add a note to the data model that **= is special and does not fall back (obviously the most backwards-compatible) 2. Fix **= (which makes sense from a language consistency perspective) Personally, my vote is for #2 as I don't want to have to remember that **= is somehow special compared to all other augmented assignments. I also don't think the backwards-compatibility risk is at all large since the semantics of turning `a **= b` into `a = a ** b` shouldn't really be different. P.S. Why are some of the PyNumber_InPlace*() functions handwritten while others are defined using a macro which mirrors the handwritten ones? Just something I noticed while investigating this.
I am also for #2, as I don't think there is any concrete reason for making **= special. On Sun, Aug 23, 2020 at 5:05 PM Brett Cannon <brett@python.org> wrote:
If you read the language reference for augmented arithmetic assignment, you will note that it essentially says, "call __i<op>__, and if that doesn't work call as if you were doing a <op> b". Unfortunately it appears **= does not follow the rule of falling back on the binary arithmetic expression semantics. I have a GitHub gist with demonstration code that shows this happening in both 3.8 and master ( https://gist.github.com/brettcannon/fec4152857e0ed551b4da515dc63e580). This was reported in https://bugs.python.org/issue38302, although initially it didn't cover __pow__, only __rpow__ being skipped.
This appears to happen because the opcode for **= calls PyNumber_InPlacePower() ( https://github.com/python/cpython/blob/802726acf6048338394a6a4750835c2cdd6a9...) which calls ternary_op for __ipow__ or __pow__ depending on which is defined, but will never try both ( https://github.com/python/cpython/blob/802726acf6048338394a6a4750835c2cdd6a9...). All of the other augmented arithmetic assignment operators have a special binary_iop() function to call which takes care of the fallback logic, so no other augmented arithmetic assignments appear to have this problem (I tested them all regardless).
I think there are two options to fixing this:
1. Add a note to the data model that **= is special and does not fall back (obviously the most backwards-compatible) 2. Fix **= (which makes sense from a language consistency perspective)
Personally, my vote is for #2 as I don't want to have to remember that **= is somehow special compared to all other augmented assignments. I also don't think the backwards-compatibility risk is at all large since the semantics of turning `a **= b` into `a = a ** b` shouldn't really be different.
P.S. Why are some of the PyNumber_InPlace*() functions handwritten while others are defined using a macro which mirrors the handwritten ones? Just something I noticed while investigating this. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MJTHPFSH... Code of Conduct: http://python.org/psf/codeofconduct/
On Sun, Aug 23, 2020 at 1:09 PM Brett Cannon <brett@python.org> wrote:
If you read the language reference for augmented arithmetic assignment, you will note that it essentially says, "call __i<op>__, and if that doesn't work call as if you were doing a <op> b". Unfortunately it appears **= does not follow the rule of falling back on the binary arithmetic expression semantics. I have a GitHub gist with demonstration code that shows this happening in both 3.8 and master ( https://gist.github.com/brettcannon/fec4152857e0ed551b4da515dc63e580). This was reported in https://bugs.python.org/issue38302, although initially it didn't cover __pow__, only __rpow__ being skipped.
Wow, very subtle bug. (Note that the issue was initially raised on StackOverflow.)
This appears to happen because the opcode for **= calls PyNumber_InPlacePower() ( https://github.com/python/cpython/blob/802726acf6048338394a6a4750835c2cdd6a9...) which calls ternary_op for __ipow__ or __pow__ depending on which is defined, but will never try both ( https://github.com/python/cpython/blob/802726acf6048338394a6a4750835c2cdd6a9...). All of the other augmented arithmetic assignment operators have a special binary_iop() function to call which takes care of the fallback logic, so no other augmented arithmetic assignments appear to have this problem (I tested them all regardless).
Ugh, for some reason the public C API PyNumber_InPlacePower() takes three arguments, like pow(x, y, z) (which computes x**y % z), even though there is no way to invoke it like that from Python. I'm sure this was set in stone when augmented assignments were first introduced -- long before we even had type slots. But because of this someone (me?) probably was being lazy and thought that implementing the full fallback strategy for **= was more effort than it was worth. (I don't think I have ever in my life used `**=`. :-)
I think there are two options to fixing this:
1. Add a note to the data model that **= is special and does not fall back (obviously the most backwards-compatible)
I think we ought to do this for 3.8 and 3.9 -- it's too late to change in 3.9.0.
2. Fix **= (which makes sense from a language consistency perspective)
We should do this in 3.10.
Personally, my vote is for #2 as I don't want to have to remember that **= is somehow special compared to all other augmented assignments. I also don't think the backwards-compatibility risk is at all large since the semantics of turning `a **= b` into `a = a ** b` shouldn't really be different.
But it is enough to give me pause about doing this in bugfix releases.
P.S. Why are some of the PyNumber_InPlace*() functions handwritten while others are defined using a macro which mirrors the handwritten ones? Just something I noticed while investigating this.
Is this about some using INPLACE_BINOP and others not using it? I can't tell the difference for InPlaceFloorDivide and -TrueDivide, possibly because these were added at a later time? git blame show that the INPLACE_BINOP macro was introduced by Neil Schemenauer in 2001 for PEP 208 (Reworking the Coercion Model, by Neil and MAL). We didn't have truediv and floordiv then, and I guess when they were added later the same year, for PEP 238 (Changing the Division Operator, by Moshe Zadka and myself) we did it differently. FWIW the in-place power glitch also originated in the PEP 208 commit. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Sun, Aug 23, 2020 at 1:57 PM Guido van Rossum <guido@python.org> wrote:
On Sun, Aug 23, 2020 at 1:09 PM Brett Cannon <brett@python.org> wrote:
If you read the language reference for augmented arithmetic assignment, you will note that it essentially says, "call __i<op>__, and if that doesn't work call as if you were doing a <op> b". Unfortunately it appears **= does not follow the rule of falling back on the binary arithmetic expression semantics. I have a GitHub gist with demonstration code that shows this happening in both 3.8 and master ( https://gist.github.com/brettcannon/fec4152857e0ed551b4da515dc63e580). This was reported in https://bugs.python.org/issue38302, although initially it didn't cover __pow__, only __rpow__ being skipped.
Wow, very subtle bug. (Note that the issue was initially raised on StackOverflow.)
Yeah, I independently ran into it while writing a future blog post and it took me a bit to realize that it wasn't me. 😄
This appears to happen because the opcode for **= calls PyNumber_InPlacePower() ( https://github.com/python/cpython/blob/802726acf6048338394a6a4750835c2cdd6a9...) which calls ternary_op for __ipow__ or __pow__ depending on which is defined, but will never try both ( https://github.com/python/cpython/blob/802726acf6048338394a6a4750835c2cdd6a9...). All of the other augmented arithmetic assignment operators have a special binary_iop() function to call which takes care of the fallback logic, so no other augmented arithmetic assignments appear to have this problem (I tested them all regardless).
Ugh, for some reason the public C API PyNumber_InPlacePower() takes three arguments, like pow(x, y, z) (which computes x**y % z), even though there is no way to invoke it like that from Python. I'm sure this was set in stone when augmented assignments were first introduced -- long before we even had type slots. But because of this someone (me?) probably was being lazy and thought that implementing the full fallback strategy for **= was more effort than it was worth. (I don't think I have ever in my life used `**=`. :-)
I suspect very few have, hence how this has managed to exist as a bug for as long as it has.
I think there are two options to fixing this:
1. Add a note to the data model that **= is special and does not fall back (obviously the most backwards-compatible)
I think we ought to do this for 3.8 and 3.9 -- it's too late to change in 3.9.0.
Yes, I agree. Sorry for forgetting to mention I was assuming these options were targeting for 3.10.
2. Fix **= (which makes sense from a language consistency perspective)
We should do this in 3.10.
👍 BTW I don't have the bandwidth to review the PR that is attached to the issue (although it looks like Serhiy has reviewed it; are you up for meringing it, Serhiy?).
Personally, my vote is for #2 as I don't want to have to remember that **= is somehow special compared to all other augmented assignments. I also don't think the backwards-compatibility risk is at all large since the semantics of turning `a **= b` into `a = a ** b` shouldn't really be different.
But it is enough to give me pause about doing this in bugfix releases.
Yeah, I agree that I don't think we can do this as a bugfix because it is such a shift in semantics. -Brett
P.S. Why are some of the PyNumber_InPlace*() functions handwritten while others are defined using a macro which mirrors the handwritten ones? Just something I noticed while investigating this.
Is this about some using INPLACE_BINOP and others not using it? I can't tell the difference for InPlaceFloorDivide and -TrueDivide, possibly because these were added at a later time? git blame show that the INPLACE_BINOP macro was introduced by Neil Schemenauer in 2001 for PEP 208 (Reworking the Coercion Model, by Neil and MAL). We didn't have truediv and floordiv then, and I guess when they were added later the same year, for PEP 238 (Changing the Division Operator, by Moshe Zadka and myself) we did it differently. FWIW the in-place power glitch also originated in the PEP 208 commit.
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
participants (3)
-
Bernardo Sulzbach -
Brett Cannon -
Guido van Rossum