Mailman 3 3.10 change (?) for __bool__ - Python-Dev

3.10 change (?) for bool

Mats Wichmann

8 Jan 2021 8 Jan '21

11:31 p.m.

Someone reported a testsuite break on stuff I work on (scons) with 3.10a4, and it looks similar to this which appears in the changelog at https://docs.python.org/3.10/whatsnew/changelog.html#changelog bpo-23898: Fix inspect.classify_class_attrs() to support attributes with overloaded __eq__ and __bool__. Patch by Mike Bayer. Except when I go look at that BPO issue, it's old - closed 2015. Is this appearing in the 3.10 changelog in error? Sorry - confused !!! The test in question does indeed touch a class which overrides __bool_ in order to raise an exception (to say "don't do that"), and in the test run the (expected) exception is not raised.

Show replies by thread

Mats Wichmann

11 Jan 11 Jan

6:59 p.m.

On 1/8/21 4:31 PM, Mats Wichmann wrote:

...

Someone reported a testsuite break on stuff I work on (scons) with 3.10a4, and it looks similar to this which appears in the changelog at

https://docs.python.org/3.10/whatsnew/changelog.html#changelog

bpo-23898: Fix inspect.classify_class_attrs() to support attributes with overloaded __eq__ and __bool__. Patch by Mike Bayer.

Except when I go look at that BPO issue, it's old - closed 2015. Is this appearing in the 3.10 changelog in error? Sorry - confused !!!

okay, that was silly, I didn't realize the changelog was cumulative over many versions, so that entry was not for 3.10 at all (teach me to do searching in browser window, where it just flies right past any section headings so I miss it was for a different version :) ).

...

The test in question does indeed touch a class which overrides __bool_ in order to raise an exception (to say "don't do that"), and in the test run the (expected) exception is not raised.

So updated information: the test in question is checking if a class (A) has an attribute using a truth test, where the attribute's value is an instance of another class (B) and expecting that that will cause the __bool__ method to be called. [aside: this test is done to validate that a class which really doesn't want this kind of test indeed rejects it] That apparently no longer happens, if it's wrapped in a try block ??? Distilled down to simple case: class A: pass class B: def __bool__(self): raise AttributeError("don't do that!") a = A() b = B() a.b = b # expect this to cause b.__bool__ to be called if a.b: print("Found it!") and it raises the exception. But when protected: try: if a.b: pass except AttributeError: print("Got expected exception") else: print("Missed expected exception") it won't trigger. But if I add a "real" statement in the block following the "if", then it's back to the pre-3.10 behavior of calling __bool__: try: if a.b: dummy = True except AttributeError: print("Got expected exception") else: print("Missed expected exception") Any thoughts on this?

Chris Angelico

7:35 p.m.

On Tue, Jan 12, 2021 at 6:00 AM Mats Wichmann wrote:

...

On 1/8/21 4:31 PM, Mats Wichmann wrote:

...
Someone reported a testsuite break on stuff I work on (scons) with 3.10a4, and it looks similar to this which appears in the changelog at

https://docs.python.org/3.10/whatsnew/changelog.html#changelog

bpo-23898: Fix inspect.classify_class_attrs() to support attributes with overloaded __eq__ and __bool__. Patch by Mike Bayer.

Except when I go look at that BPO issue, it's old - closed 2015. Is this appearing in the 3.10 changelog in error? Sorry - confused !!!

okay, that was silly, I didn't realize the changelog was cumulative over many versions, so that entry was not for 3.10 at all (teach me to do searching in browser window, where it just flies right past any section headings so I miss it was for a different version :) ).

...
The test in question does indeed touch a class which overrides __bool_ in order to raise an exception (to say "don't do that"), and in the test run the (expected) exception is not raised.

So updated information: the test in question is checking if a class (A) has an attribute using a truth test, where the attribute's value is an instance of another class (B) and expecting that that will cause the __bool__ method to be called. [aside: this test is done to validate that a class which really doesn't want this kind of test indeed rejects it] That apparently no longer happens, if it's wrapped in a try block ??? Distilled down to simple case:

class A: pass

class B: def __bool__(self): raise AttributeError("don't do that!")

a = A() b = B() a.b = b # expect this to cause b.__bool__ to be called if a.b: print("Found it!")

and it raises the exception. But when protected:

try: if a.b: pass except AttributeError: print("Got expected exception") else: print("Missed expected exception")

it won't trigger. But if I add a "real" statement in the block following the "if", then it's back to the pre-3.10 behavior of calling __bool__:

try: if a.b: dummy = True except AttributeError: print("Got expected exception") else: print("Missed expected exception")

Any thoughts on this?

Oooh interesting. I tried on a build of 3.10 from October and: 1) The unguarded version bombed out with an exception 2) The "if... pass" version reported that it got the exception 3) The "if... dummy" version reported that it got the exception ie every one of them did indeed raise. But on a fresh build from the master branch, I got the same results you did. That means the change happened some time between commit 497126f7ea and commit ace008c531, an 800ish commit span. I'll start bisecting to try to track this down. It looks like "if a.b: pass" is getting partially optimized out; the disassembly shows a being loaded, its attribute b being looked up, and then it just jumps to the else - there's no POP_JUMP_IF_FALSE as there is when there's a bit of actual code in there. ChrisA

Pablo Galindo Salgado

7:46 p.m.

This may be related to the changes in https://bugs.python.org/issue42246. Could you open a new issue and add Mark Shannon to it if that turns to be the case? Pablo On Mon, 11 Jan 2021 at 19:36, Chris Angelico wrote:

...

On Tue, Jan 12, 2021 at 6:00 AM Mats Wichmann wrote:

...
On 1/8/21 4:31 PM, Mats Wichmann wrote:

...
Someone reported a testsuite break on stuff I work on (scons) with 3.10a4, and it looks similar to this which appears in the changelog at

https://docs.python.org/3.10/whatsnew/changelog.html#changelog

bpo-23898: Fix inspect.classify_class_attrs() to support attributes

with

...
...
overloaded __eq__ and __bool__. Patch by Mike Bayer.

Except when I go look at that BPO issue, it's old - closed 2015. Is this appearing in the 3.10 changelog in error? Sorry - confused !!!

okay, that was silly, I didn't realize the changelog was cumulative over many versions, so that entry was not for 3.10 at all (teach me to do searching in browser window, where it just flies right past any section headings so I miss it was for a different version :) ).

...
The test in question does indeed touch a class which overrides __bool_ in order to raise an exception (to say "don't do that"), and in the test run the (expected) exception is not raised.

So updated information: the test in question is checking if a class (A) has an attribute using a truth test, where the attribute's value is an instance of another class (B) and expecting that that will cause the __bool__ method to be called. [aside: this test is done to validate that a class which really doesn't want this kind of test indeed rejects it] That apparently no longer happens, if it's wrapped in a try block ??? Distilled down to simple case:

class A: pass

class B: def __bool__(self): raise AttributeError("don't do that!")

a = A() b = B() a.b = b # expect this to cause b.__bool__ to be called if a.b: print("Found it!")

and it raises the exception. But when protected:

try: if a.b: pass except AttributeError: print("Got expected exception") else: print("Missed expected exception")

it won't trigger. But if I add a "real" statement in the block following the "if", then it's back to the pre-3.10 behavior of calling __bool__:

try: if a.b: dummy = True except AttributeError: print("Got expected exception") else: print("Missed expected exception")

Any thoughts on this?

Oooh interesting. I tried on a build of 3.10 from October and: 1) The unguarded version bombed out with an exception 2) The "if... pass" version reported that it got the exception 3) The "if... dummy" version reported that it got the exception

ie every one of them did indeed raise. But on a fresh build from the master branch, I got the same results you did. That means the change happened some time between commit 497126f7ea and commit ace008c531, an 800ish commit span.

I'll start bisecting to try to track this down. It looks like "if a.b: pass" is getting partially optimized out; the disassembly shows a being loaded, its attribute b being looked up, and then it just jumps to the else - there's no POP_JUMP_IF_FALSE as there is when there's a bit of actual code in there.

ChrisA _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/K2LD2L5R... Code of Conduct: http://python.org/psf/codeofconduct/

Guido van Rossum

8 p.m.

All that said (I agree it's surprising that 3.10 seems backwards incompatible here) I would personally not raise AttributeError but TypeError in the `__bool__()` method. On Mon, Jan 11, 2021 at 11:51 AM Pablo Galindo Salgado wrote:

...

This may be related to the changes in https://bugs.python.org/issue42246. Could you open a new issue and add Mark Shannon to it if that turns to be the case?

Pablo

On Mon, 11 Jan 2021 at 19:36, Chris Angelico wrote:

...
On Tue, Jan 12, 2021 at 6:00 AM Mats Wichmann wrote:

...
On 1/8/21 4:31 PM, Mats Wichmann wrote:

...
Someone reported a testsuite break on stuff I work on (scons) with 3.10a4, and it looks similar to this which appears in the changelog at

https://docs.python.org/3.10/whatsnew/changelog.html#changelog

bpo-23898: Fix inspect.classify_class_attrs() to support attributes

with

...
...
overloaded __eq__ and __bool__. Patch by Mike Bayer.

Except when I go look at that BPO issue, it's old - closed 2015. Is this appearing in the 3.10 changelog in error? Sorry - confused !!!

okay, that was silly, I didn't realize the changelog was cumulative over many versions, so that entry was not for 3.10 at all (teach me to do searching in browser window, where it just flies right past any section headings so I miss it was for a different version :) ).

...
The test in question does indeed touch a class which overrides __bool_ in order to raise an exception (to say "don't do that"), and in the test run the (expected) exception is not raised.

So updated information: the test in question is checking if a class (A) has an attribute using a truth test, where the attribute's value is an instance of another class (B) and expecting that that will cause the __bool__ method to be called. [aside: this test is done to validate that a class which really doesn't want this kind of test indeed rejects it] That apparently no longer happens, if it's wrapped in a try block ??? Distilled down to simple case:

class A: pass

class B: def __bool__(self): raise AttributeError("don't do that!")

a = A() b = B() a.b = b # expect this to cause b.__bool__ to be called if a.b: print("Found it!")

and it raises the exception. But when protected:

try: if a.b: pass except AttributeError: print("Got expected exception") else: print("Missed expected exception")

it won't trigger. But if I add a "real" statement in the block following the "if", then it's back to the pre-3.10 behavior of calling __bool__:

try: if a.b: dummy = True except AttributeError: print("Got expected exception") else: print("Missed expected exception")

Any thoughts on this?

Oooh interesting. I tried on a build of 3.10 from October and: 1) The unguarded version bombed out with an exception 2) The "if... pass" version reported that it got the exception 3) The "if... dummy" version reported that it got the exception

ie every one of them did indeed raise. But on a fresh build from the master branch, I got the same results you did. That means the change happened some time between commit 497126f7ea and commit ace008c531, an 800ish commit span.

I'll start bisecting to try to track this down. It looks like "if a.b: pass" is getting partially optimized out; the disassembly shows a being loaded, its attribute b being looked up, and then it just jumps to the else - there's no POP_JUMP_IF_FALSE as there is when there's a bit of actual code in there.

ChrisA _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/K2LD2L5R... Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7DMQXD3E... Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

Stestagg

10:27 p.m.

I've raised: https://bugs.python.org/issue42899. The triggering commit was: c71581c7a4192e6ba9a79eccc583aaadab300efa bpo-42615: Delete redundant jump instructions that only bypass empty blocks (GH-23733) On Mon, Jan 11, 2021 at 8:04 PM Guido van Rossum wrote:

...

All that said (I agree it's surprising that 3.10 seems backwards incompatible here) I would personally not raise AttributeError but TypeError in the `__bool__()` method.

On Mon, Jan 11, 2021 at 11:51 AM Pablo Galindo Salgado < pablogsal@gmail.com> wrote:

...
This may be related to the changes in https://bugs.python.org/issue42246. Could you open a new issue and add Mark Shannon to it if that turns to be the case?

Pablo

On Mon, 11 Jan 2021 at 19:36, Chris Angelico wrote:

...
On Tue, Jan 12, 2021 at 6:00 AM Mats Wichmann wrote:

...
On 1/8/21 4:31 PM, Mats Wichmann wrote:

...
Someone reported a testsuite break on stuff I work on (scons) with 3.10a4, and it looks similar to this which appears in the changelog

...
...
https://docs.python.org/3.10/whatsnew/changelog.html#changelog

bpo-23898: Fix inspect.classify_class_attrs() to support attributes

with

...
overloaded __eq__ and __bool__. Patch by Mike Bayer.

Except when I go look at that BPO issue, it's old - closed 2015. Is this appearing in the 3.10 changelog in error? Sorry - confused !!!

okay, that was silly, I didn't realize the changelog was cumulative over many versions, so that entry was not for 3.10 at all (teach me to do searching in browser window, where it just flies right past any section headings so I miss it was for a different version :) ).

...
The test in question does indeed touch a class which overrides __bool_ in order to raise an exception (to say "don't do that"), and in the test run the (expected) exception is not raised.

So updated information: the test in question is checking if a class (A) has an attribute using a truth test, where the attribute's value is an instance of another class (B) and expecting that that will cause the __bool__ method to be called. [aside: this test is done to validate

at that

...
a class which really doesn't want this kind of test indeed rejects it] That apparently no longer happens, if it's wrapped in a try block ??? Distilled down to simple case:

class A: pass

class B: def __bool__(self): raise AttributeError("don't do that!")

a = A() b = B() a.b = b # expect this to cause b.__bool__ to be called if a.b: print("Found it!")

and it raises the exception. But when protected:

try: if a.b: pass except AttributeError: print("Got expected exception") else: print("Missed expected exception")

it won't trigger. But if I add a "real" statement in the block following the "if", then it's back to the pre-3.10 behavior of calling __bool__:

try: if a.b: dummy = True except AttributeError: print("Got expected exception") else: print("Missed expected exception")

Any thoughts on this?

Oooh interesting. I tried on a build of 3.10 from October and: 1) The unguarded version bombed out with an exception 2) The "if... pass" version reported that it got the exception 3) The "if... dummy" version reported that it got the exception

ie every one of them did indeed raise. But on a fresh build from the master branch, I got the same results you did. That means the change happened some time between commit 497126f7ea and commit ace008c531, an 800ish commit span.

I'll start bisecting to try to track this down. It looks like "if a.b: pass" is getting partially optimized out; the disassembly shows a being loaded, its attribute b being looked up, and then it just jumps to the else - there's no POP_JUMP_IF_FALSE as there is when there's a bit of actual code in there.

ChrisA _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/K2LD2L5R... Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7DMQXD3E... Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c... _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VQ5FDBBC... Code of Conduct: http://python.org/psf/codeofconduct/

Mats Wichmann

10:53 p.m.

On 1/11/21 1:00 PM, Guido van Rossum wrote:

...

All that said (I agree it's surprising that 3.10 seems backwards incompatible here) I would personally not raise AttributeError but TypeError in the `__bool__()` method.

eh, that was just me picking a cheap something to demo it. the program raises an application-specific error that I didn't feel like defining to keep the repro as short as possible.

Guido van Rossum

12 Jan 12 Jan

12:45 a.m.

Ah never mind. Seems to be a real bug -- thanks for reporting! On Mon, Jan 11, 2021 at 2:57 PM Mats Wichmann wrote:

...

On 1/11/21 1:00 PM, Guido van Rossum wrote:

...
All that said (I agree it's surprising that 3.10 seems backwards incompatible here) I would personally not raise AttributeError but TypeError in the `__bool__()` method.

eh, that was just me picking a cheap something to demo it. the program raises an application-specific error that I didn't feel like defining to keep the repro as short as possible. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/SVGFN4DC... Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

Mark Shannon

3:53 p.m.

Hi everyone, Should the optimizer eliminate tests that it can prove have no effect on the control flow of the program, even if that may eliminate some side effects in __bool__()? For several years we have converted if a and b: ... to if a: if b: ... which are equivalent, unless bool(a) has side effects the second time it is called. In master we convert `if x: pass` to `pass` which is equivalent, unless bool(x) has side effects the first time it is called. This is a recent change. This is one of those "easy to fix, if we can decide on the semantics" bugs. Submit your thoughts to https://bugs.python.org/issue42899, please. Cheers, Mark. On 12/01/2021 12:45 am, Guido van Rossum wrote:

...

Ah never mind. Seems to be a real bug -- thanks for reporting!

On Mon, Jan 11, 2021 at 2:57 PM Mats Wichmann mailto:mats@wichmann.us> wrote:

On 1/11/21 1:00 PM, Guido van Rossum wrote: > All that said (I agree it's surprising that 3.10 seems backwards > incompatible here) I would personally not raise AttributeError but > TypeError in the `__bool__()` method.

eh, that was just me picking a cheap something to demo it. the program raises an application-specific error that I didn't feel like defining to keep the repro as short as possible. _______________________________________________ Python-Dev mailing list -- python-dev@python.org mailto:python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org mailto:python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/SVGFN4DC... Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido http://python.org/~guido) /Pronouns: he/him //(why is my pronoun here?)/ http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BK4IUDXC... Code of Conduct: http://python.org/psf/codeofconduct/

Richard Damon

5:23 p.m.

On 1/12/21 10:53 AM, Mark Shannon wrote:

...

Hi everyone,

Should the optimizer eliminate tests that it can prove have no effect on the control flow of the program, even if that may eliminate some side effects in __bool__()?

For several years we have converted

if a and b: ...

to

if a: if b: ...

which are equivalent, unless bool(a) has side effects the second time it is called.

In master we convert `if x: pass` to `pass` which is equivalent, unless bool(x) has side effects the first time it is called. This is a recent change.

This is one of those "easy to fix, if we can decide on the semantics" bugs.

Submit your thoughts to https://bugs.python.org/issue42899, please.

Cheers, Mark.

One key point about 'and' and 'or' is that those operators are defined to be 'short circuiting', i.e. that with a and b, that if a is false, then b is not evaluated at all. This can be important if a is a condition that test if the expression b is 'safe' to evaluate. In fact, isn't it true that 'a and b' is defined to be the equivalent to: 'a if not a else b' so b doesn't need to be evaluated unless a is truthy. I know that I would be very surpised if a statement like if foo(): pass ended up optimizing itself and not calling foo(), which is only one step away from the quoted case of if b: pass which is the equivalent to if b.__bool__(): pass Yes, perhaps there is more of an expectation that __bool__() is innocuous, but is that a an assumption that should be baked into the language. -- Richard Damon

Chris Angelico

5:47 p.m.

On Wed, Jan 13, 2021 at 4:24 AM Richard Damon wrote:

...

On 1/12/21 10:53 AM, Mark Shannon wrote:

...
Hi everyone,

Should the optimizer eliminate tests that it can prove have no effect on the control flow of the program, even if that may eliminate some side effects in __bool__()?

For several years we have converted

if a and b: ...

to

if a: if b: ...

which are equivalent, unless bool(a) has side effects the second time it is called.

In master we convert `if x: pass` to `pass` which is equivalent, unless bool(x) has side effects the first time it is called. This is a recent change.

This is one of those "easy to fix, if we can decide on the semantics" bugs.

Submit your thoughts to https://bugs.python.org/issue42899, please.

Cheers, Mark.

One key point about 'and' and 'or' is that those operators are defined to be 'short circuiting', i.e. that with a and b, that if a is false, then b is not evaluated at all. This can be important if a is a condition that test if the expression b is 'safe' to evaluate. In fact, isn't it true that 'a and b' is defined to be the equivalent to: 'a if not a else b' so b doesn't need to be evaluated unless a is truthy.

Yes, the shortcircuiting behaviour isn't in question. But consider: class A: def __bool__(self): print("A().__bool__") return False def f(): print("if A() and A()") if A() and A(): x = 1 print("cond = A() and A()") cond = A() and A() if cond: x = 1 f() And once you've run the code, disassemble the function for extra insight. There's a very definite difference here, and it's the same difference as: for x in thing: and for x in iter(thing): I'd be fine with documenting that __bool__ is (guaranteed to be) called only if the interpreter needs to know the result, leaving open the option for things like this to be optimized out. That'd leave open the option for "foo() if x else foo()" to be optimized down to just "foo()", although I don't think that particular one is needed.

...

I know that I would be very surpised if a statement like

if foo():

pass

ended up optimizing itself and not calling foo(), which is only one step away from the quoted case of

if b:

pass

which is the equivalent to

if b.__bool__():

pass

Yes, they do look similar. The difference is that calling __bool__ is under the interpreter's control, and it's easier to document that it be assumed to be side-effect-free.

...

Yes, perhaps there is more of an expectation that __bool__() is innocuous, but is that a an assumption that should be baked into the language.

I think so. Consider that sometimes other dunders won't be called, if the interpreter believes it's not necessary: class A(float): def __index__(self): print("A().__index__") return 10 class B(int): def __index__(self): print("B().__index__") return 10 print(range(20)[A():B()]) If it's a subclass of float, slicing will call __index__, but if it's a subclass of int, Python knows already that it can use the internal integer value. ChrisA

Brett Cannon

13 Jan 13 Jan

1:39 a.m.

On Tue, Jan 12, 2021 at 9:51 AM Chris Angelico wrote:

...

On Wed, Jan 13, 2021 at 4:24 AM Richard Damon wrote:

...
On 1/12/21 10:53 AM, Mark Shannon wrote:

...
Hi everyone,

Should the optimizer eliminate tests that it can prove have no effect on the control flow of the program, even if that may eliminate some side effects in __bool__()?

For several years we have converted

if a and b: ...

to

if a: if b: ...

which are equivalent, unless bool(a) has side effects the second time it is called.

In master we convert `if x: pass` to `pass` which is equivalent, unless bool(x) has side effects the first time it is called. This is a recent change.

This is one of those "easy to fix, if we can decide on the semantics" bugs.

Submit your thoughts to https://bugs.python.org/issue42899, please.

Cheers, Mark.

One key point about 'and' and 'or' is that those operators are defined to be 'short circuiting', i.e. that with a and b, that if a is false, then b is not evaluated at all. This can be important if a is a condition that test if the expression b is 'safe' to evaluate. In fact, isn't it true that 'a and b' is defined to be the equivalent to: 'a if not a else b' so b doesn't need to be evaluated unless a is truthy.

https://snarky.ca/unravelling-boolean-operations/ if you want the gory details.

...

Yes, the shortcircuiting behaviour isn't in question. But consider:

class A: def __bool__(self): print("A().__bool__") return False

def f(): print("if A() and A()") if A() and A(): x = 1 print("cond = A() and A()") cond = A() and A() if cond: x = 1

f()

And once you've run the code, disassemble the function for extra insight.

There's a very definite difference here, and it's the same difference as:

for x in thing:

and

for x in iter(thing):

I'd be fine with documenting that __bool__ is (guaranteed to be) called only if the interpreter needs to know the result, leaving open the option for things like this to be optimized out. That'd leave open the option for "foo() if x else foo()" to be optimized down to just "foo()", although I don't think that particular one is needed.

I think that's the key question here: it is a language change, but is it one we want (specifically in this case and in general)? It will require a language reference change at least, and as it has been shown, some people don't expect Python to take shortcuts in execution knowing full well that CPython dutifully executes things. So saying we only call __bool__() *if necessary* is definitely a change. But then this begs the question of whether we want to do this more widely in the language in the name of optimization, or not in the name of consistency that a user can easily reason about.

...

...
I know that I would be very surpised if a statement like

if foo():

pass

ended up optimizing itself and not calling foo(), which is only one step away from the quoted case of

if b:

pass

which is the equivalent to

if b.__bool__():

pass

Yes, they do look similar. The difference is that calling __bool__ is under the interpreter's control, and it's easier to document that it be assumed to be side-effect-free.

...
Yes, perhaps there is more of an expectation that __bool__() is innocuous, but is that a an assumption that should be baked into the language.

I think so. Consider that sometimes other dunders won't be called, if the interpreter believes it's not necessary:

class A(float): def __index__(self): print("A().__index__") return 10

class B(int): def __index__(self): print("B().__index__") return 10

print(range(20)[A():B()])

If it's a subclass of float, slicing will call __index__, but if it's a subclass of int, Python knows already that it can use the internal integer value.

https://snarky.ca/unravelling-not-in-python/ 😁 But basically, https://docs.python.org/3.8/reference/datamodel.html#object.__index__ says "called *if *conversion to an int is necessary" which isn't the case when something is already an int.

...

ChrisA _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GHKDF6YE... Code of Conduct: http://python.org/psf/codeofconduct/

Steven D'Aprano

6:03 a.m.

On Wed, Jan 13, 2021 at 04:47:06AM +1100, Chris Angelico wrote:

...

That'd leave open the option for "foo() if x else foo()" to be optimized down to just "foo()", although I don't think that particular one is needed.

That would be an unsafe optimization. Not all objets are representable as truthy/falsey values, e.g. numpy arrays. >>> bool(a) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() Not even all builtin values. This is in Python 3.9: >>> bool(NotImplemented) <stdin>:1: DeprecationWarning: NotImplemented should not be used in a boolean context True I believe that 3.10 makes it an error. If not 3.10, then it will surely happen soon. But even without the change to NotImplemented, it has never been the case that *every* object is guaranteed to be either truthy or falsey. At least not since the Python 1.x `__nonzero__` dunder was put into the language. -- Steve

Chris Angelico

6:37 a.m.

On Wed, Jan 13, 2021 at 5:05 PM Steven D'Aprano wrote:

...

On Wed, Jan 13, 2021 at 04:47:06AM +1100, Chris Angelico wrote:

...
That'd leave open the option for "foo() if x else foo()" to be optimized down to just "foo()", although I don't think that particular one is needed.

That would be an unsafe optimization. Not all objets are representable as truthy/falsey values, e.g. numpy arrays.

>>> bool(a) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Not even all builtin values. This is in Python 3.9:

>>> bool(NotImplemented) <stdin>:1: DeprecationWarning: NotImplemented should not be used in a boolean context True

I believe that 3.10 makes it an error. If not 3.10, then it will surely happen soon. But even without the change to NotImplemented, it has never been the case that *every* object is guaranteed to be either truthy or falsey. At least not since the Python 1.x `__nonzero__` dunder was put into the language.

But this is exactly where we started: with a boolification that fails, in a context where the result wouldn't actually change anything. Actually calling bool() on something will continue to have the behaviour you're describing, but if the truthiness or falsiness would make no difference, is the interpreter required to find out? ChrisA

Ethan Furman

7:10 a.m.

On 1/12/21 10:37 PM, Chris Angelico wrote:

...

On Wed, Jan 13, 2021 at 5:05 PM Steven D'Aprano wrote:

...
On Wed, Jan 13, 2021 at 04:47:06AM +1100, Chris Angelico wrote:

...
That'd leave open the option for "foo() if x else foo()" to be optimized down to just "foo()", although I don't think that particular one is needed.

That would be an unsafe optimization. Not all objets are representable as truthy/falsey values

...

...
Not even all builtin values. This is in Python 3.9:

>>> bool(NotImplemented) <stdin>:1: DeprecationWarning: NotImplemented should not be used in a boolean context True

I believe that 3.10 makes it an error. If not 3.10, then it will surely happen soon. But even without the change to NotImplemented, it has never been the case that *every* object is guaranteed to be either truthy or falsey. At least not since the Python 1.x `__nonzero__` dunder was put into the language.

But this is exactly where we started: with a boolification that fails, in a context where the result wouldn't actually change anything. Actually calling bool() on something will continue to have the behaviour you're describing, but if the truthiness or falsiness would make no difference, is the interpreter required to find out?

Yes. Optimizations are an implementation detail, and implementation details should not change the language. -- ~Ethan~

Chris Angelico

7:27 a.m.

On Wed, Jan 13, 2021 at 6:11 PM Ethan Furman wrote:

...

On 1/12/21 10:37 PM, Chris Angelico wrote:

...
On Wed, Jan 13, 2021 at 5:05 PM Steven D'Aprano wrote:

...
On Wed, Jan 13, 2021 at 04:47:06AM +1100, Chris Angelico wrote:

...
That'd leave open the option for "foo() if x else foo()" to be optimized down to just "foo()", although I don't think that particular one is needed.

That would be an unsafe optimization. Not all objets are representable as truthy/falsey values

...
...
Not even all builtin values. This is in Python 3.9:

>>> bool(NotImplemented) <stdin>:1: DeprecationWarning: NotImplemented should not be used in a boolean context True

I believe that 3.10 makes it an error. If not 3.10, then it will surely happen soon. But even without the change to NotImplemented, it has never been the case that *every* object is guaranteed to be either truthy or falsey. At least not since the Python 1.x `__nonzero__` dunder was put into the language.

But this is exactly where we started: with a boolification that fails, in a context where the result wouldn't actually change anything. Actually calling bool() on something will continue to have the behaviour you're describing, but if the truthiness or falsiness would make no difference, is the interpreter required to find out?

Yes.

Optimizations are an implementation detail, and implementation details should not change the language.

The language can also be defined in an optimization-friendly way, though. Consider how we got positional-only arguments in Python: first they existed in C-implemented functions in CPython, even though they couldn't exist in pure Python code, and then the functionality got added to the language definition, thus permitting the optimization. Or consider dictionary lookup. Most people treat it as "find a key which is equal to the one you're looking for", but the actual definition is "find a key which is identical to, or equal to, the one you're looking for". That's partly to ensure that weird cases like NaN don't break too badly, but also it means that x.__eq__(x) doesn't have to be called all the time. The topic under discussion is a language definition. Choosing to permit the optimization doesn't mean that the implementation detail changes the language. Choosing to deny it means there won't be an optimization. I personally don't see any reason to force Python to calculate something unnecessarily, given that this is *already* happening in other situations (see the "if a and b:" optimization, which doesn't boolify twice). ChrisA

Paul Sokolovsky

8:16 a.m.

Hello, On Wed, 13 Jan 2021 18:27:07 +1100 Chris Angelico wrote: []

...

...
Optimizations are an implementation detail, and implementation details should not change the language.

The language can also be defined in an optimization-friendly way, though. Consider how we got positional-only arguments in Python: first they existed in C-implemented functions in CPython, even though they couldn't exist in pure Python code, and then the functionality got added to the language definition, thus permitting the optimization.

In this case, the culprit seems to be that __bool__() and many other "special" methods may have side effects. So, just as "const" annotation would be useful for variables, "pure" (side-effect free) annotation would be useful for functions. But that alone won't help due to too-dynamic nature of Python. It's not much of an optimization if, at runtime, at each call-site, you need to check whether a function is pure to decide if you have to call it, or may skip it. Instead, that rather be done at compile time. But checking which method (pure or impure) will be called when is again complicated statically in Python. What to do about that? Well, we can introduce, ahem, some "strict" mode, in which *all* __bool__(), __gt__(), and friends must be declared pure, and for good measure, returning bool. Then we know we can skip any such method call. You see, all heresy starts from small things... [] -- Best regards, Paul mailto:pmiscml@gmail.com

Ethan Furman

9:01 a.m.

On 1/12/21 11:27 PM, Chris Angelico wrote:

...

On Wed, Jan 13, 2021 at 6:11 PM Ethan Furman wrote:

...

...
Optimizations are an implementation detail, and implementation details should not change the language.

The language can also be defined in an optimization-friendly way, though. Consider how we got positional-only arguments in Python: first they existed in C-implemented functions in CPython, even though they couldn't exist in pure Python code, and then the functionality got added to the language definition, thus permitting the optimization.

1. What optimization? 2. Did the language change because of the optimization?

...

Or consider dictionary lookup. Most people treat it as "find a key which is equal to the one you're looking for", but the actual definition is "find a key which is identical to, or equal to, the one you're looking for".

Exactly. The definition, i.e. language spec, says identity, then equality.

...

The topic under discussion is a language definition. Choosing to permit the optimization doesn't mean that the implementation detail changes the language. Choosing to deny it means there won't be an optimization.

There are, I am sure, many optimizations that are not possible because of Python's dynamism. `if <something>` is supposed to evaluate `bool(something)`, regardless of what comes after.

...

I personally don't see any reason to force Python to calculate something unnecessarily, given that this is *already* happening in other situations (see the "if a and b:" optimization, which doesn't boolify twice).

Sure, and I agree that calling `bool()` a second time is wasteful, as well as possibly confusing -- Python already has the answer from the first `bool()` call, so why would it need to do it again? That seems a matter of correctness, not optimization. -- ~Ethan~

Chris Angelico

9:27 a.m.

On Wed, Jan 13, 2021 at 8:02 PM Ethan Furman wrote:

...

On 1/12/21 11:27 PM, Chris Angelico wrote:

...
On Wed, Jan 13, 2021 at 6:11 PM Ethan Furman wrote:

...
...
Optimizations are an implementation detail, and implementation details should not change the language.

The language can also be defined in an optimization-friendly way, though. Consider how we got positional-only arguments in Python: first they existed in C-implemented functions in CPython, even though they couldn't exist in pure Python code, and then the functionality got added to the language definition, thus permitting the optimization.

1. What optimization? 2. Did the language change because of the optimization?

It's a lot faster for C-implemented functions to require positional parameters. A number of functions had help text that implied that they accepted keyword-or-positional args, but if you tried to call them with keyword args, they'd error out. And yes. PEP 457 and then PEP 570 introduced real positional-only arguments, and part of the debate surrounded the question "should we require that all C-implemented functions support keyword args?". Why have a performance hit on all C functions just for the sake of an arbitrary consistency? Instead, the language definition was changed to make this possible - and now we have, for instance, "len(obj, /)", which clearly does not accept len(obj="x"), but does accept len("x").

...

...
Or consider dictionary lookup. Most people treat it as "find a key which is equal to the one you're looking for", but the actual definition is "find a key which is identical to, or equal to, the one you're looking for".

Exactly. The definition, i.e. language spec, says identity, then equality.

Right. The definition is set to permit the optimization. Therefore the optimization cannot be in violation of the spec.

...

...
The topic under discussion is a language definition. Choosing to permit the optimization doesn't mean that the implementation detail changes the language. Choosing to deny it means there won't be an optimization.

There are, I am sure, many optimizations that are not possible because of Python's dynamism. `if <something>` is supposed to evaluate `bool(something)`, regardless of what comes after.

It doesn't, though. There is a difference between: if a and b: ... and x = a and b if x: ... The first one will boolify a only once; the second will figure out what bool(a) is before the assignment, and then (if it's false) boolify it again in the 'if' statement. Current builds of Python 3.10 optimize this away too: if a or 1: print(1) Regardless of the truthiness of a, the print call will happen. (Note that a is still *evaluated*. It's just not queried for truthiness. For instance, "if f() or 1:" will still call the function; it just won't call __bool__() on the return value.)

...

...
I personally don't see any reason to force Python to calculate something unnecessarily, given that this is *already* happening in other situations (see the "if a and b:" optimization, which doesn't boolify twice).

Sure, and I agree that calling `bool()` a second time is wasteful, as well as possibly confusing -- Python already has the answer from the first `bool()` call, so why would it need to do it again? That seems a matter of correctness, not optimization.

Is it? It means that there's a visible difference between putting it in a variable and doing it inline. If it's okay to do that, then why is it not okay to remove the bool check when it can't affect the result? In theory, "x = (expr)" followed by "if x:" should perform the exact same checks as "if (expr):"; if it's okay to violate that principle for "if a and b:", then why not for "if a: pass"? Either way, the interpreter knows that any sane __bool__ function cannot affect the result. ChrisA

Emily Bowman

10:08 a.m.

Even if you define __bool__() as returning a bool, and error/undefined behavior otherwise, that doesn't eliminate side effects. Is it even possible to nail down a definition to the point that you can say, "Thou shalt not mutate or cause anything" and have it meaningfully enforced in the compiler or interpreter? -Em On Wed, Jan 13, 2021 at 12:18 AM Paul Sokolovsky wrote:

...

Hello,

On Wed, 13 Jan 2021 18:27:07 +1100 Chris Angelico wrote:

[]

...
...
Optimizations are an implementation detail, and implementation details should not change the language.

The language can also be defined in an optimization-friendly way, though. Consider how we got positional-only arguments in Python: first they existed in C-implemented functions in CPython, even though they couldn't exist in pure Python code, and then the functionality got added to the language definition, thus permitting the optimization.

In this case, the culprit seems to be that __bool__() and many other "special" methods may have side effects. So, just as "const" annotation would be useful for variables, "pure" (side-effect free) annotation would be useful for functions.

But that alone won't help due to too-dynamic nature of Python. It's not much of an optimization if, at runtime, at each call-site, you need to check whether a function is pure to decide if you have to call it, or may skip it. Instead, that rather be done at compile time.

But checking which method (pure or impure) will be called when is again complicated statically in Python. What to do about that? Well, we can introduce, ahem, some "strict" mode, in which *all* __bool__(), __gt__(), and friends must be declared pure, and for good measure, returning bool. Then we know we can skip any such method call.

You see, all heresy starts from small things...

[]

-- Best regards, Paul mailto:pmiscml@gmail.com _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4P52JORZ... Code of Conduct: http://python.org/psf/codeofconduct/

Chris Angelico

10:19 a.m.

On Wed, Jan 13, 2021 at 9:08 PM Emily Bowman wrote:

...

Even if you define __bool__() as returning a bool, and error/undefined behavior otherwise, that doesn't eliminate side effects. Is it even possible to nail down a definition to the point that you can say, "Thou shalt not mutate or cause anything" and have it meaningfully enforced in the compiler or interpreter?

Yes - just say "will be called when the interpreter needs to know the booleanness of the object", and don't mandate that it be called if the interpreter already knows. If someone writes a __bool__ function that decrements a counter and returns True if it's still above zero, then they're shooting themselves in the foot, and we don't have to protect them. It's like having __iter__ return an object on which __iter__ doesn't return self. There's no protection in the language to stop you from creating such a monster, but you'll run into problems sooner or later if you do, and it's not a bug in Python. ChrisA

Larry Hastings

10:50 a.m.

On 1/12/21 11:27 PM, Chris Angelico wrote:

...

The language can also be defined in an optimization-friendly way, though. Consider how we got positional-only arguments in Python: first they existed in C-implemented functions in CPython, even though they couldn't exist in pure Python code, and then the functionality got added to the language definition, thus permitting the optimization.

I wouldn't describe positional-only parameters as an "optimization". They were created to add expressiveness to Python code, not to make Python code faster. The classic example is the dict constructor: technically, you couldn't implement it correctly in pure Python code, because it had a parameter (the iterable) that could not be a named parameter. Any name you gave it would preclude passing in that name as a name=value argument. Parsing positional-only parameters is often faster than parsing named parameters, for obvious reasons. But this is really small stuff, drowned out by almost any other variation in your code. Cheers, //arry/

Paul Sokolovsky

10:57 a.m.

Hello, On Wed, 13 Jan 2021 02:08:01 -0800 Emily Bowman wrote:

...

Even if you define __bool__() as returning a bool, and error/undefined behavior otherwise, that doesn't eliminate side effects. Is it even possible to nail down a definition to the point that you can say, "Thou shalt not mutate or cause anything" and have it meaningfully enforced in the compiler or interpreter?

Yes, sure, "pure" annotation on a function would (ideally/eventually) need "typechecking". We could start with simple rules like: a) a pure function should not mutate any objects which it didn't create; b) for "mutation" also counts passing objects to (other) non-pure functions. The hardest part is again to know which of these other functions is pure or impure. Runtime checking should not be the aim to shoot for. There should be a way to preserve as much as possible of Python's dynamic nature that we all love, yet avoid runtime lookups. The above "purity" conditions are also too restrictive, so would need to be extended/elaborated (which would certainly require more detailed annotations; fortunately, we all already accepted annotations to be a part of Python's nature, so wanting to use them more shouldn't come as surprise).

...

-Em

On Wed, Jan 13, 2021 at 12:18 AM Paul Sokolovsky wrote:

...
Hello,

On Wed, 13 Jan 2021 18:27:07 +1100 Chris Angelico wrote:

[]

...
...
Optimizations are an implementation detail, and implementation details should not change the language.

The language can also be defined in an optimization-friendly way, though. Consider how we got positional-only arguments in Python: first they existed in C-implemented functions in CPython, even though they couldn't exist in pure Python code, and then the functionality got added to the language definition, thus permitting the optimization.

In this case, the culprit seems to be that __bool__() and many other "special" methods may have side effects. So, just as "const" annotation would be useful for variables, "pure" (side-effect free) annotation would be useful for functions.

But that alone won't help due to too-dynamic nature of Python. It's not much of an optimization if, at runtime, at each call-site, you need to check whether a function is pure to decide if you have to call it, or may skip it. Instead, that rather be done at compile time.

But checking which method (pure or impure) will be called when is again complicated statically in Python. What to do about that? Well, we can introduce, ahem, some "strict" mode, in which *all* __bool__(), __gt__(), and friends must be declared pure, and for good measure, returning bool. Then we know we can skip any such method call.

You see, all heresy starts from small things...

[]

-- Best regards, Paul mailto:pmiscml@gmail.com _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4P52JORZ... Code of Conduct: http://python.org/psf/codeofconduct/

-- Best regards, Paul mailto:pmiscml@gmail.com

Steven D'Aprano

4:21 p.m.

On Wed, Jan 13, 2021 at 08:27:46PM +1100, Chris Angelico wrote:

...

It's a lot faster for C-implemented functions to require positional parameters. A number of functions had help text that implied that they accepted keyword-or-positional args, but if you tried to call them with keyword args, they'd error out.

That's an unfortunate limitation of the docs, not a language feature. One might even call it a documentation bug.

...

And yes. PEP 457 and then PEP 570 introduced real positional-only arguments, and part of the debate surrounded the question "should we require that all C-implemented functions support keyword args?". Why have a performance hit on all C functions just for the sake of an arbitrary consistency? Instead, the language definition was changed to make this possible - and now we have, for instance, "len(obj, /)", which clearly does not accept len(obj="x"), but does accept len("x").

Sorry to be pedantic here... oh who am I fooling, I'm not sorry at all *wink* I think that if we are to be precise, and we should be, Python the language has always supported positional-only arguments. What we lacked was syntax for declaring named positional-only **parameters**. There has always been at least two ways to implement positional-only arguments: - write your function in C; - or use `*args` and process them yourself. So that's not a new feature. The new feature gained in 3.8 was syntax to define names for positional-only parameters in def statements. This may not have been really obvious until after PEPs 457 and 570 because we were (and still are) fairly blasé about the difference between arguments and parameters, and of calling things "positional" regardless of whether they were actually positional-or-keyword or positional-only. In any case, this is very much a red herring. Function call syntax is not an optimization, it is a syntactic feature of the language. The ability to pass an argument by position, position-or-keyword, or keyword is an element of language design. The choice of which ones to support may be influenced by concerns about efficiency and speed, but that's as far as it goes. I think your earlier example of short-cutting equality tests with identity tests is more relevant here. [...]

...

In theory, "x = (expr)" followed by "if x:" should perform the exact same checks as "if (expr):"; if it's okay to violate that principle for "if a and b:", then why not for "if a: pass"? Either way, the interpreter knows that any sane __bool__ function cannot affect the result.

Of course they can: the method can fail and raise an exception. That could be due to a bug, or by design, as in numpy arrays and (soon) NotImplemented. Sane `__bool__` methods can have useful side-effects. More than once I've debugged code by sticking some logging inside a method I knew would be called, and that method could as easily be `__bool__` as any other. -- Steve

Steven D'Aprano

4:29 p.m.

On Wed, Jan 13, 2021 at 02:08:01AM -0800, Emily Bowman wrote:

...

Even if you define __bool__() as returning a bool, and error/undefined behavior otherwise, that doesn't eliminate side effects. Is it even possible to nail down a definition to the point that you can say, "Thou shalt not mutate or cause anything" and have it meaningfully enforced in the compiler or interpreter?

No, I don't think it is possible to enforce lack of side-effects. Not without rebuilding the language from the ground up with a clear, definitive and enforcable distinction between pure and impure functions. (I think Haskell does something like that. But trying to retrofit that into Python would probably be about as hard as building a restricted mode, and for similar reasons.) Besides, we probably don't want to prohibit side-effects in `__bool__`. That would prohibit useful tricks such as putting logging calls into a method you are trying to debug. -- Steve

Paul Sokolovsky

5:17 p.m.

Hello, On Thu, 14 Jan 2021 03:29:35 +1100 Steven D'Aprano wrote:

...

On Wed, Jan 13, 2021 at 02:08:01AM -0800, Emily Bowman wrote:

...
Even if you define __bool__() as returning a bool, and error/undefined behavior otherwise, that doesn't eliminate side effects. Is it even possible to nail down a definition to the point that you can say, "Thou shalt not mutate or cause anything" and have it meaningfully enforced in the compiler or interpreter?

No, I don't think it is possible to enforce lack of side-effects. Not without rebuilding the language from the ground up with a clear, definitive and enforcable distinction between pure and impure functions.

(I think Haskell does something like that. But trying to retrofit that into Python would probably be about as hard as building a restricted mode, and for similar reasons.)

Besides, we probably don't want to prohibit side-effects in `__bool__`. That would prohibit useful tricks such as putting logging calls into a method you are trying to debug.

Surely, if we do that, we wouldn't use Haskell's definition of purity ;-). Rather, a practical definition of purity, Python-style. For example, print() would be considered "pure", as its purpose is to provide program output, not arbitrarily change program state (so, all of __str__, __repr__, __format__ would need to be pure, but if someone overrode print() to shoot at random modules/data at them, then they shoot themselves in the feet, nothing else). [] -- Best regards, Paul mailto:pmiscml@gmail.com

Greg Ewing

11:37 p.m.

On 14/01/21 5:29 am, Steven D'Aprano wrote:

...

No, I don't think it is possible to enforce lack of side-effects. Not without rebuilding the language from the ground up with a clear, definitive and enforcable distinction between pure and impure functions.

(I think Haskell does something like that.

All functions are pure in Haskell -- not sure if that counts as "doing something like that". Retrofitting it into Python would turn it into a very different language. -- Greg

Greg Ewing

11:45 p.m.

On 14/01/21 6:17 am, Paul Sokolovsky wrote:

...

For example, print() would be considered "pure", as its purpose is to provide program output, not arbitrarily change program state

That definition of purity wouldn't really help with optimisations, though, because optimising away a print() call would still cause a visible change in the program's behaviour. -- Greg

Paul Sokolovsky

14 Jan 14 Jan

12:13 a.m.

Hello, On Thu, 14 Jan 2021 12:45:11 +1300 Greg Ewing wrote:

...

On 14/01/21 6:17 am, Paul Sokolovsky wrote:

...
For example, print() would be considered "pure", as its purpose is to provide program output, not arbitrarily change program state

That definition of purity wouldn't really help with optimisations, though, because optimising away a print() call would still cause a visible change in the program's behaviour.

But nobody talked about optimizing away generic "pure"-annotated functions (which would differ from "mathematical" definition of purity), only about optimizing "pure" *dunder* methods (which are by definition special methods, which are called implicitly; or not called, which is the aim we discuss). "Pure" annotation on other functions are drawn only to verify that a dunder itself is indeed "pure". And "pure" is of course a pretty rough term meaning "not causing *non-local* side effects". print() in that regard causes only localized side effect of output appearing on terminal. print() also can be redefined to output to io.StringIO, and it still will be "pure", as side effects will be localized to just that io.StringIO object (which has dedicated purpose to be print's buffer in the first place). But if user redefines print() to e.g. manipulate sys.path, it's now causes non-local side effects, and thus breaks API contract re: "purity".

...

-- Greg

[] -- Best regards, Paul mailto:pmiscml@gmail.com

Greg Ewing

1:56 a.m.

On 14/01/21 1:13 pm, Paul Sokolovsky wrote:

...

But nobody talked about optimizing away generic "pure"-annotated functions (which would differ from "mathematical" definition of purity), only about optimizing "pure" *dunder* methods

The same thing applies. If we decide that print() is pure, then a __bool__ that calls print() is also pure, so there's nothing wrong with optimising it away, right? -- Greg

Terry Reedy

2:32 a.m.

On 1/13/2021 8:56 PM, Greg Ewing wrote:

...

On 14/01/21 1:13 pm, Paul Sokolovsky wrote:

...
But nobody talked about optimizing away generic "pure"-annotated functions (which would differ from "mathematical" definition of purity), only about optimizing "pure" *dunder* methods

The same thing applies. If we decide that print() is pure, then a __bool__ that calls print() is also pure, so there's nothing wrong with optimising it away, right?

I say 'yes', because the purpose of logging is to document what happens, and if nothing happens, there is nothing to document. Wrapping a .__bool__ in a logging decorator might be part of testing it. -- Terry Jan Reedy

Greg Ewing

10 a.m.

On 14/01/21 3:32 pm, Terry Reedy wrote:

...

I say 'yes', because the purpose of logging is to document what happens, and if nothing happens, there is nothing to document. Wrapping a .__bool__ in a logging decorator might be part of testing it.

Or it might be something else. It would be fine to *define* __bool__ as something the compiler is allowed to optimise away. But that's a different thing from "purity" (which is hard to pin down in an imperative language). -- Greg

Paul Sokolovsky

10:42 a.m.

Hello, On Thu, 14 Jan 2021 23:00:06 +1300 Greg Ewing wrote:

...

On 14/01/21 3:32 pm, Terry Reedy wrote:

...
I say 'yes', because the purpose of logging is to document what happens, and if nothing happens, there is nothing to document. Wrapping a .__bool__ in a logging decorator might be part of testing it.

Or it might be something else.

It would be fine to *define* __bool__ as something the compiler is allowed to optimise away.

Yes, and the talk was exactly about how to *proceed* with (re)defining __bool__ (and other dunders, then other runtime inefficiencies) as something the compiler is allowed to optimise away. To retrace the outline: 1. Current semantics doesn't say that __bool__ can be optimized away, and it stays that way. (Compare that to the parallel discussion of PEP649 - it tries to change existing semantics by disallowing variable annotations inside the "if" statement, and that's raised as a concern). 2. Instead, a new semantic mode is introduced, which needs to be explicitly enabled. 3. Now the problem comes up about compatibility of the existing source code with the new mode. 4. It's resolved by requiring __bool__ and other dunders to be annotated "pure" (tentative code-name). The code which doesn't have all __bool__, etc. methods annotated as such cannot be run in the new semantic mode (and benefit from any optimizations it may offer), and has to continue be run in the old mode. (And note that annotated code still can run in the old mode either.) 5. Now, stamping annotations is nice, but the code should correspond to them, so there should be ability to "typecheck" such annotated code (more like "behavior-check", though effects tracking is also part of the type theory). It can be handled in the same way as typechecking existing type annotations happen - by external tools.

...

But that's a different thing from "purity" (which is hard to pin down in an imperative language).

At the beginning of this sub-thread, example of Haskell was given. It went the different way - of banning any mutability and side-effects at all, then making heroic efforts of introducing concepts which would make writing programs under such model still surmountable, like monads, etc. That was hard. It didn't stop them. We here talk about evolutionary, and clearly delinated adjustments to Python semantics, based on the experience with efficiently executing Python code. Certainly, much less hard than what Haskell did.

...

-- Greg

[] -- Best regards, Paul mailto:pmiscml@gmail.com

Steven D'Aprano

11:05 a.m.

On Wed, Jan 13, 2021 at 08:17:26PM +0300, Paul Sokolovsky wrote:

...

...
Besides, we probably don't want to prohibit side-effects in `__bool__`. That would prohibit useful tricks such as putting logging calls into a method you are trying to debug.

Surely, if we do that, we wouldn't use Haskell's definition of purity ;-). Rather, a practical definition of purity, Python-style.

It's not *Haskell's* definition, it's pretty much *everyone's* definition of pure. https://en.wikipedia.org/wiki/Pure_function

...

For example, print() would be considered "pure", as its purpose is to provide program output, not arbitrarily change program state

Is writing to an external file a change to program state? Keep in mind that programs can get state from external files. I would say that what you are describing is not a distinction between pure and impure functions, based on whether or not they can be safely optimised away without a change in visible behaviour. The distinction you are proposing is into two arbitrary sets of functions, some of which can be optimized away and some which can't, regardless of whether or not that optimisation leads to change in visible behaviour. If you want to argue for that, fine, do so, but don't call impure functions "pure". -- Steve

Paul Sokolovsky

11:41 a.m.

Hello, On Thu, 14 Jan 2021 22:05:37 +1100 Steven D'Aprano wrote:

...

On Wed, Jan 13, 2021 at 08:17:26PM +0300, Paul Sokolovsky wrote:

...
...
Besides, we probably don't want to prohibit side-effects in `__bool__`. That would prohibit useful tricks such as putting logging calls into a method you are trying to debug.

Surely, if we do that, we wouldn't use Haskell's definition of purity ;-). Rather, a practical definition of purity, Python-style.

It's not *Haskell's* definition, it's pretty much *everyone's* definition of pure.

https://en.wikipedia.org/wiki/Pure_function

...
For example, print() would be considered "pure", as its purpose is to provide program output, not arbitrarily change program state

Is writing to an external file a change to program state? Keep in mind that programs can get state from external files.

Please reread the reply about "not having non-local side effects". NotHavingNonLocalSideEffects is a bit longish for an annotation identifier though...

...

I would say that what you are describing is not a distinction between pure and impure functions,

You say that as if you propose some better identifier for annotation ;-). I personally would be fine if the annotation is: def print(*Any) -> PythoniclyPure: ... I'm talking about small-scale semantic distinctions required for capture needed large-scale semantic effect. Naming, when it comes to it, will be subject of long, looooooong bikeshedding. (Which already started, thanks!)

...

based on whether or not they can be safely optimised away without a change in visible behaviour. The distinction you are proposing is into two arbitrary sets of functions, some of which can be optimized away and some which can't, regardless of whether or not that optimisation leads to change in visible behaviour.

If you want to argue for that, fine, do so, but don't call impure functions "pure".

[] -- Best regards, Paul mailto:pmiscml@gmail.com

Rob Cliffe

15 Jan 15 Jan

11:10 a.m.

On 12/01/2021 15:53, Mark Shannon wrote:

...

Hi everyone,

In master we convert `if x: pass` to `pass` which is equivalent, unless bool(x) has side effects the first time it is called. This is a recent change.

Suppose x is not a currently valid variable name at runtime. Will the NameError still be "optimised" away? Thanks Rob Cliffe

Chris Angelico

11:59 a.m.

On Fri, Jan 15, 2021 at 10:13 PM Rob Cliffe via Python-Dev wrote:

...

On 12/01/2021 15:53, Mark Shannon wrote:

...
Hi everyone,

In master we convert `if x: pass` to `pass` which is equivalent, unless bool(x) has side effects the first time it is called. This is a recent change.

Suppose x is not a currently valid variable name at runtime. Will the NameError still be "optimised" away?

No, it won't. The expression still gets fully evaluated. The ONLY part that gets optimized away is the check "is this thing true?". ChrisA

1190

Age (days ago)

1197

Last active (days ago)

List overview

Download

36 comments

17 participants

participants (17)

Brett Cannon
Chris Angelico
Emily Bowman
Ethan Furman
Greg Ewing
Guido van Rossum
Larry Hastings
Mark Shannon
Mats Wichmann
Mats Wichmann
Pablo Galindo Salgado
Paul Sokolovsky
Richard Damon
Rob Cliffe
Stestagg
Steven D'Aprano
Terry Reedy

3.10 change (?) for __bool__

tags

participants (17)

3.10 change (?) for bool