Hi all, I’d like your comments and feedback on an enhancement that introduces power assertions to the Python language. Proposal -------- This feature is inspired by a similar feature of the Groovy language[1], and is effectively a variant of the `assert` keyword. When an assertion expression evaluates to `False`, the output shows not only the failure, but also a breakdown of the evaluated expression from the inner part to the outer part. For example, a procedure like: ```python class SomeClass: def __init__(self): self.val = {'d': 'e'} def __str__(self): return str(self.val) sc = SomeClass() assert sc.val['d'] == 'f' ``` Will result in the output: ```python Assertion failed: sc.val['d'] == f | | | | e False | {'d': 'e'} ``` See link [2] if the formatting above is screwed up. In the output above we can see the value of every part of the expression from left to right, mapped to their expression fragment with the pipe (`|`). The number of rows that are printed depend on the value of each fragment of the expression. If the value of a fragment is longer than the actual fragment (`{'d': 'e'}` is longer than `sc`), then the next value (`e`) will be printed on a new line which will appear above. Values are appended to the same line until it overflows in length to horizontal position of the next fragment. The information that’s displayed is dictated by the type. If the type is a constant value, it will be displayed as is. If the type implements `__str__`, then the return value of that will be displayed. It is important to note that expressions with side effects are affected by this feature. This is because in order to display this information, we must store references to the instances and not just the values. Rational -------- Every test boils down to the binary statement "Is this true or false?", whether you use the built-in assert keyword or a more advanced assertion method provided by a testing framework. When an assertion fails, the output is binary too — "Expected x, but got y". There are helpful libraries like Hamcrest which give you a more verbose breakdown of the difference and answer the question "What exactly is the difference between x and y?". This is extremely helpful, but it still focuses on the difference between the values. We need to keep in mind that a given state is normally an outcome of a series of states, that is, one outcome is a result of multiple conditions and causes. This is where power assertion comes in. It allows us to better understand what led to the failure. Implementation -------- I’ve already built a fully functional implementation[2] of this feature as part of my Python testing framework - Nimoy[3]. The current implementation uses AST manipulation to remap the expression to a data structure[4] at compile time, so that it can then be evaluated and printed[5] at runtime. [1] http://docs.groovy-lang.org/next/html/documentation/core-testing-guide.html#... [2] https://browncoat-ninjas.github.io/nimoy/examples/#power-assertions-beta [3] https://github.com/browncoat-ninjas/nimoy/ [4] https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/ast_tools/expre... [5] https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/assertions/powe...
This is cool. AFAIK pytest does something like this. How does your implementation differ? What is your argument for making this part of the language? Why not a 3rd party library? What about asserts that are not used for testing, but as classic “unless there’s a bug, this should hold”? Those may not want to incur the extra cost. —Guido On Sun, Sep 12, 2021 at 07:09 <noam@10ne.org> wrote:
Hi all,
I’d like your comments and feedback on an enhancement that introduces power assertions to the Python language.
Proposal -------- This feature is inspired by a similar feature of the Groovy language[1], and is effectively a variant of the `assert` keyword. When an assertion expression evaluates to `False`, the output shows not only the failure, but also a breakdown of the evaluated expression from the inner part to the outer part.
For example, a procedure like: ```python class SomeClass: def __init__(self): self.val = {'d': 'e'}
def __str__(self): return str(self.val)
sc = SomeClass()
assert sc.val['d'] == 'f' ```
Will result in the output:
```python Assertion failed:
sc.val['d'] == f | | | | e False | {'d': 'e'} ``` See link [2] if the formatting above is screwed up.
In the output above we can see the value of every part of the expression from left to right, mapped to their expression fragment with the pipe (`|`). The number of rows that are printed depend on the value of each fragment of the expression. If the value of a fragment is longer than the actual fragment (`{'d': 'e'}` is longer than `sc`), then the next value (`e`) will be printed on a new line which will appear above. Values are appended to the same line until it overflows in length to horizontal position of the next fragment.
The information that’s displayed is dictated by the type. If the type is a constant value, it will be displayed as is. If the type implements `__str__`, then the return value of that will be displayed.
It is important to note that expressions with side effects are affected by this feature. This is because in order to display this information, we must store references to the instances and not just the values.
Rational -------- Every test boils down to the binary statement "Is this true or false?", whether you use the built-in assert keyword or a more advanced assertion method provided by a testing framework. When an assertion fails, the output is binary too — "Expected x, but got y".
There are helpful libraries like Hamcrest which give you a more verbose breakdown of the difference and answer the question "What exactly is the difference between x and y?". This is extremely helpful, but it still focuses on the difference between the values.
We need to keep in mind that a given state is normally an outcome of a series of states, that is, one outcome is a result of multiple conditions and causes. This is where power assertion comes in. It allows us to better understand what led to the failure.
Implementation -------- I’ve already built a fully functional implementation[2] of this feature as part of my Python testing framework - Nimoy[3]. The current implementation uses AST manipulation to remap the expression to a data structure[4] at compile time, so that it can then be evaluated and printed[5] at runtime.
[1] http://docs.groovy-lang.org/next/html/documentation/core-testing-guide.html#... [2] https://browncoat-ninjas.github.io/nimoy/examples/#power-assertions-beta [3] https://github.com/browncoat-ninjas/nimoy/ [4] https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/ast_tools/expre... [5] https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/assertions/powe... _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/T26DR4... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido (mobile)
On 2021-09-12 at 07:28:53 -0700, Guido van Rossum <guido@python.org> wrote:
What about asserts that are not used for testing, but as classic “unless there’s a bug, this should hold”? Those may not want to incur the extra cost.
I was actually thinking exactly the opposite: this would more useful in production than in testing. When I'm testing, tests build on each other. I should know that the inner parts work, and I should be testing specific aspects of the outer parts. If I don't have confidence in those inner parts, then I need to write more tests against them. If I don't "know" where the data comes from in my assertions, then my tests are trying to test too much at once. On the other hand, weird things happen in production, and my first reaction to "this shouldn't happen unless there's a bug" are is to start looking in the logs at how we got there. These power assertions are like a retroactive or JIT logging mechanism (in the sense that I may not have logged enough detail), or a sort of first order run-time debugger that shows me what's relevant at the point of the failure. As far as the extra cost goes, how does that cost compare to the full stack trace I already get from the exception being raised?
2QdxY4RzWzUUiLuE＠potatochowder.com wrote:
What about asserts that are not used for testing, but as classic “unless there’s a bug, this should hold”? Those may not want to incur the extra cost. I was actually thinking exactly the opposite: this would more useful in
On 2021-09-12 at 07:28:53 -0700, Guido van Rossum guido@python.org wrote: production than in testing. When I'm testing, tests build on each other. I should know that the inner parts work, and I should be testing specific aspects of the outer parts. If I don't have confidence in those inner parts, then I need to write more tests against them. If I don't "know" where the data comes from in my assertions, then my tests are trying to test too much at once. On the other hand, weird things happen in production, and my first reaction to "this shouldn't happen unless there's a bug" are is to start looking in the logs at how we got there. These power assertions are like a retroactive or JIT logging mechanism (in the sense that I may not have logged enough detail), or a sort of first order run-time debugger that shows me what's relevant at the point of the failure. As far as the extra cost goes, how does that cost compare to the full stack trace I already get from the exception being raised?
That's an interesting observation. I haven't thought of that. Regarding cost - I haven't profiled my implementation, so I'll have to get back to you on that one.
This is cool. Thank you. Much appreciated.
AFAIK pytest does something like this. How does your implementation differ? The pytest implementation is very powerful in the way of hints and suggestions that point to the difference and source, but when the asserted expression has more than one sub-expression, the thread-style breakdown is unreadable. I believe my chart-style implementation offers better readability.
What is your argument for making this part of the language? Why not a 3rd party library? The first argument is that asserts are fundamental, which is why they are a core part of the language in the first place. If they're already part of the core, we can improve them to provide more constructive feedback, without having to pull in mammoth testing frameworks, which brings me to my second argument.
My understanding of the compiler is that a 3rd party library can't simply rewrite AST during the standard compilation. If one wants to intervene with the AST phase, one must rewrite and execute the python files. This is a very intrusive and impractical process for a scope of a 3rd party library. It's definitely more fitting for frameworks that act as executers such as pytest and my framework Nimoy. Please correct me if I'm mistaken.
What about asserts that are not used for testing, but as classic “unless there’s a bug, this should hold”? Those may not want to incur the extra cost. Correct. I believe that the proper way to handle this is a switch much like `-O`. The feature will be opt-in, and one can opt-in using that switch.
12.09.21 17:28, Guido van Rossum пише:
This is cool.
AFAIK pytest does something like this. How does your implementation differ?
What pytest does is awesome. I though about implementing it in the standard compiler since seen it the first time.
What is your argument for making this part of the language? Why not a 3rd party library?
It needs a support in the compiler. The condition expression should be compiled to keep all immediate results of subexpressions on the stack. If the final result is true, immediate results are dropped. If it is false, the second argument of assert is evaluated and its value together with all immediate results of the first expression, together with references to corresponding subexpressions (as strings, ranges or AST nodes) are passed to the special handler. That handler can be implemented in a third-party library, because formatting and outputting a report is a complex task. The default handler can just raise an AttributeError.
What about asserts that are not used for testing, but as classic “unless there’s a bug, this should hold”? Those may not want to incur the extra cost.
The only extra cost is that immediate results are temporary save on stack instead of just be dropped. It increases the size of bytecode and stack, but I don't think it will be significant.
Maybe you all could collaborate on a PEP? This sounds a worthy topic. On Sun, Sep 12, 2021 at 08:37 Serhiy Storchaka <storchaka@gmail.com> wrote:
12.09.21 17:28, Guido van Rossum пише:
This is cool.
AFAIK pytest does something like this. How does your implementation differ?
What pytest does is awesome. I though about implementing it in the standard compiler since seen it the first time.
What is your argument for making this part of the language? Why not a 3rd party library?
It needs a support in the compiler. The condition expression should be compiled to keep all immediate results of subexpressions on the stack. If the final result is true, immediate results are dropped. If it is false, the second argument of assert is evaluated and its value together with all immediate results of the first expression, together with references to corresponding subexpressions (as strings, ranges or AST nodes) are passed to the special handler. That handler can be implemented in a third-party library, because formatting and outputting a report is a complex task. The default handler can just raise an AttributeError.
What about asserts that are not used for testing, but as classic “unless there’s a bug, this should hold”? Those may not want to incur the extra cost.
The only extra cost is that immediate results are temporary save on stack instead of just be dropped. It increases the size of bytecode and stack, but I don't think it will be significant.
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/S673FX... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido (mobile)
On Mon, Sep 13, 2021 at 1:37 AM Serhiy Storchaka <storchaka@gmail.com> wrote:
12.09.21 17:28, Guido van Rossum пише:
This is cool.
AFAIK pytest does something like this. How does your implementation differ?
What pytest does is awesome. I though about implementing it in the standard compiler since seen it the first time.
What is your argument for making this part of the language? Why not a 3rd party library?
It needs a support in the compiler. The condition expression should be compiled to keep all immediate results of subexpressions on the stack. If the final result is true, immediate results are dropped. If it is false, the second argument of assert is evaluated and its value together with all immediate results of the first expression, together with references to corresponding subexpressions (as strings, ranges or AST nodes) are passed to the special handler. That handler can be implemented in a third-party library, because formatting and outputting a report is a complex task. The default handler can just raise an AttributeError.
I wonder, could this be simplified a bit, on the assumption that a well-written assertion shouldn't have a problem with being executed twice? Instead of keeping all the subexpressions around (a run-time cost), keep the AST of the expression itself (a compile-time cost). Then, when the exception is about to be printed to the console, re-evaluate it and do the display. ChrisA
12.09.21 21:36, Chris Angelico пише:
I wonder, could this be simplified a bit, on the assumption that a well-written assertion shouldn't have a problem with being executed twice? Instead of keeping all the subexpressions around (a run-time cost), keep the AST of the expression itself (a compile-time cost). Then, when the exception is about to be printed to the console, re-evaluate it and do the display.
Yes, it would simplify, but we cannot guarantee this (especially in tests). If we could, we would not need an assert.
I wonder, could this be simplified a bit, on the assumption that a well-written assertion shouldn't have a problem with being executed twice?
While I agree as an engineering principle an assert should not have side effects and hence re-evaluation should be fine in most cases, it is not universal. It is possible for assertions to not have side effects but yet change value between evaluations if they interact with a shared resource such as the file system.. For example consider the following assertion: assert os.path.isdir("config") and os.path.isfile("config/setup.yml") It is completely possible for the value of this expression to change between evaluations. Granted this would like mean their is some more significant issue with my code, however, I would like the interpreter to give me accurate information about why my assertion failed. Bad information is worse than no information. Like imagine that on the first evaluation the directory config does not exist but on the second it has been created by another process. A naive revaluation strategy would likely result in it pointing at the second clause and saying the assertion failed their when it really failed on the first clause. This would send me down a rabbit hole of debugging why setup.yml was not constructed properly instead of debugging why the config directory didn’t exist. Further while it is bad engineering practices to have side effects in an assert it is completely possible. For example consider the following pathological example: class PathologicalFoo: def __init__(self): self._val = 0 def get(self): old_val = self._val self._val = 1 return old_val foo = PathologicalFoo()assert foo.get() == 1 Or worse it is possible for revaluation to cause errors class UniquePtr: def __init__(self, obj): self.set(obj) def get(self): if self._valid: self._valid = False obj = self._obj self._obj = None return obj else: raise ValueError() def set(self, obj): self._obj = obj self._valid = True x = UniquePtr(1)assert x.get() == 0 x.set(0) How would the interpreter handle this? On Sun, Sep 12, 2021 at 11:39 AM Chris Angelico <rosuav@gmail.com> wrote:
On Mon, Sep 13, 2021 at 1:37 AM Serhiy Storchaka <storchaka@gmail.com> wrote:
12.09.21 17:28, Guido van Rossum пише:
This is cool.
AFAIK pytest does something like this. How does your implementation
differ?
What pytest does is awesome. I though about implementing it in the standard compiler since seen it the first time.
What is your argument for making this part of the language? Why not a 3rd party library?
It needs a support in the compiler. The condition expression should be compiled to keep all immediate results of subexpressions on the stack. If the final result is true, immediate results are dropped. If it is false, the second argument of assert is evaluated and its value together with all immediate results of the first expression, together with references to corresponding subexpressions (as strings, ranges or AST nodes) are passed to the special handler. That handler can be implemented in a third-party library, because formatting and outputting a report is a complex task. The default handler can just raise an AttributeError.
I wonder, could this be simplified a bit, on the assumption that a well-written assertion shouldn't have a problem with being executed twice? Instead of keeping all the subexpressions around (a run-time cost), keep the AST of the expression itself (a compile-time cost). Then, when the exception is about to be printed to the console, re-evaluate it and do the display.
ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/IXPY4B... Code of Conduct: http://python.org/psf/codeofconduct/
On Tue, Oct 5, 2021 at 9:02 AM Caleb Donovick <donovick@cs.stanford.edu> wrote:
I wonder, could this be simplified a bit, on the assumption that a well-written assertion shouldn't have a problem with being executed twice?
While I agree as an engineering principle an assert should not have side effects and hence re-evaluation should be fine in most cases, it is not universal. It is possible for assertions to not have side effects but yet change value between evaluations if they interact with a shared resource such as the file system..
For example consider the following assertion:
assert os.path.isdir("config") and os.path.isfile("config/setup.yml")
It is completely possible for the value of this expression to change between evaluations. Granted this would like mean their is some more significant issue with my code, however, I would like the interpreter to give me accurate information about why my assertion failed. Bad information is worse than no information. Like imagine that on the first evaluation the directory config does not exist but on the second it has been created by another process. A naive revaluation strategy would likely result in it pointing at the second clause and saying the assertion failed their when it really failed on the first clause. This would send me down a rabbit hole of debugging why setup.yml was not constructed properly instead of debugging why the config directory didn’t exist.
That seems like an abuse of assertions. If you have assertions that depend on external state that can change that quickly, then the assertion is *already useless*. What do you gain by asserting something that might have changed by the next line of code?
Further while it is bad engineering practices to have side effects in an assert it is completely possible. For example consider the following pathological example:
class PathologicalFoo: def __init__(self): self._val = 0
def get(self): old_val = self._val self._val = 1 return old_val
foo = PathologicalFoo() assert foo.get() == 1
Yes, side effects in assertions are always possible. If someone has assertions with side effects, do we say that python -O is buggy, or the assertion is buggy? In a world in which assertions might and might not be evaluated, is it such a stretch to demand that they can be safely reevaluated (in the same context)? Yes, it's a change to the expectations, but one which well-designed assertions shouldn't be bothered by. My imagining of this is that it'd be handled when an AssertionError reaches top level, and it'd be broadly thus: try: all_your_code() except AssertionError as e: ... reevaluate etc Meaning there are four possibilities: 1) The assertion is consistent, and the extra info is absolutely correct 2) Some OTHER exception occurs on the reevaluation. It's a chained exception like any other. 3) No assertion failure happens (eg PathologicalFoo). Might require a minor special case "if nothing goes wrong, print out the original" but that's the most obvious thing to do. 4) The assertion fails in an inconsistent way, but it still fails. You'll get the second form instead of the first. It's only really the fourth case that would be confusing, and only if the first evaluation actually causes the problem (otherwise it's just an inconsistent assertion and you'd need to debug both parts anyway). This is a pretty narrow problem, and even then, you've been shown a weird assertion that needs to be debugged. Is it that bad to say that an assertion that gives inconsistent results is buggy? ChrisA
2) Some OTHER exception occurs on the reevaluation. It's a chained exception like any other.
Except it's not a chained exception and displaying as such would be VERY confusing IMO. Granted we could easily strip the chained exception and just return the original one. So after reconsideration I agree this is not an issue.
It's only really the fourth case that would be confusing
I generally agree with your analysis but I think this 4th case is more problematic than you think. Given no information I am immediately going to split my assertion so I can see what part is failing. However, if the interpreter gives me incorrect information I am going to be super confused. Most people will not have carefully read section 7.3 of the language reference and will not understand this critical aspect of the execution of assertion statements. They will assume that the interpreter is not lying to them. I think storing the intermediate results on the stack is vastly preferable to revaluation for this reason. On Mon, Oct 4, 2021 at 3:20 PM Chris Angelico <rosuav@gmail.com> wrote:
On Tue, Oct 5, 2021 at 9:02 AM Caleb Donovick <donovick@cs.stanford.edu> wrote:
I wonder, could this be simplified a bit, on the assumption that a well-written assertion shouldn't have a problem with being executed twice?
While I agree as an engineering principle an assert should not have side
and hence re-evaluation should be fine in most cases, it is not universal. It is possible for assertions to not have side effects but yet change value between evaluations if they interact with a shared resource such as the file system..
For example consider the following assertion:
assert os.path.isdir("config") and os.path.isfile("config/setup.yml")
It is completely possible for the value of this expression to change between evaluations. Granted this would like mean their is some more significant issue with my code, however, I would like the interpreter to give me accurate information about why my assertion failed. Bad information is worse than no information. Like imagine that on the first evaluation the directory config does not exist but on
effects the second it has been created by another process.
A naive revaluation strategy would likely result in it pointing at the second clause and saying the assertion failed their when it really failed on the first clause. This would send me down a rabbit hole of debugging why setup.yml was not constructed properly instead of debugging why the config directory didn’t exist.
That seems like an abuse of assertions. If you have assertions that depend on external state that can change that quickly, then the assertion is *already useless*. What do you gain by asserting something that might have changed by the next line of code?
Further while it is bad engineering practices to have side effects in an assert it is completely possible. For example consider the following pathological example:
class PathologicalFoo: def __init__(self): self._val = 0
def get(self): old_val = self._val self._val = 1 return old_val
foo = PathologicalFoo() assert foo.get() == 1
Yes, side effects in assertions are always possible. If someone has assertions with side effects, do we say that python -O is buggy, or the assertion is buggy? In a world in which assertions might and might not be evaluated, is it such a stretch to demand that they can be safely reevaluated (in the same context)? Yes, it's a change to the expectations, but one which well-designed assertions shouldn't be bothered by.
My imagining of this is that it'd be handled when an AssertionError reaches top level, and it'd be broadly thus:
try: all_your_code() except AssertionError as e: ... reevaluate etc
Meaning there are four possibilities: 1) The assertion is consistent, and the extra info is absolutely correct 2) Some OTHER exception occurs on the reevaluation. It's a chained exception like any other. 3) No assertion failure happens (eg PathologicalFoo). Might require a minor special case "if nothing goes wrong, print out the original" but that's the most obvious thing to do. 4) The assertion fails in an inconsistent way, but it still fails. You'll get the second form instead of the first.
It's only really the fourth case that would be confusing, and only if the first evaluation actually causes the problem (otherwise it's just an inconsistent assertion and you'd need to debug both parts anyway). This is a pretty narrow problem, and even then, you've been shown a weird assertion that needs to be debugged. Is it that bad to say that an assertion that gives inconsistent results is buggy?
ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/GYNRJA... Code of Conduct: http://python.org/psf/codeofconduct/
On Tue, Oct 5, 2021 at 9:50 AM Caleb Donovick <donovick@cs.stanford.edu> wrote:
2) Some OTHER exception occurs on the reevaluation. It's a chained exception like any other.
Except it's not a chained exception and displaying as such would be VERY confusing IMO. Granted we could easily strip the chained exception and just return the original one. So after reconsideration I agree this is not an issue.
It's only really the fourth case that would be confusing
I generally agree with your analysis but I think this 4th case is more problematic than you think. Given no information I am immediately going to split my assertion so I can see what part is failing. However, if the interpreter gives me incorrect information I am going to be super confused. Most people will not have carefully read section 7.3 of the language reference and will not understand this critical aspect of the execution of assertion statements. They will assume that the interpreter is not lying to them.
I think storing the intermediate results on the stack is vastly preferable to revaluation for this reason.
The trouble is that storing intermediate results is a price paid by every assertion that succeeds, and reevaluating is a price paid only when assertions fail - and the problem you're referring to ONLY happens with broken assertions. Does every assert need to pay the price for the possibility of buggy use of assert? ChrisA
What about asserts that are not used for testing, but as classic “unless there’s a bug, this should hold”?
To me this relates to the thread about having a structured *assert* that doesn't go away with *-O*. My purpose when addressing *assert* was precisely the *“unless there’s a bug, this should hold”* kind of assertions. In that context, introducing additional yet known costs (as in this "power" idea), providing for different exception types (maybe all descending from AssertError?), and allowing for a block to prepare the exception, are all worth it. Introducing the new syntax for *assert* would imply zero cost for existing assertions. On Sun, Sep 12, 2021 at 10:28 AM Guido van Rossum <guido@python.org> wrote:
This is cool.
AFAIK pytest does something like this. How does your implementation differ?
What is your argument for making this part of the language? Why not a 3rd party library?
What about asserts that are not used for testing, but as classic “unless there’s a bug, this should hold”? Those may not want to incur the extra cost.
—Guido
On Sun, Sep 12, 2021 at 07:09 <noam@10ne.org> wrote:
Hi all,
I’d like your comments and feedback on an enhancement that introduces power assertions to the Python language.
Proposal -------- This feature is inspired by a similar feature of the Groovy language[1], and is effectively a variant of the `assert` keyword. When an assertion expression evaluates to `False`, the output shows not only the failure, but also a breakdown of the evaluated expression from the inner part to the outer part.
For example, a procedure like: ```python class SomeClass: def __init__(self): self.val = {'d': 'e'}
def __str__(self): return str(self.val)
sc = SomeClass()
assert sc.val['d'] == 'f' ```
Will result in the output:
```python Assertion failed:
sc.val['d'] == f | | | | e False | {'d': 'e'} ``` See link [2] if the formatting above is screwed up.
In the output above we can see the value of every part of the expression from left to right, mapped to their expression fragment with the pipe (`|`). The number of rows that are printed depend on the value of each fragment of the expression. If the value of a fragment is longer than the actual fragment (`{'d': 'e'}` is longer than `sc`), then the next value (`e`) will be printed on a new line which will appear above. Values are appended to the same line until it overflows in length to horizontal position of the next fragment.
The information that’s displayed is dictated by the type. If the type is a constant value, it will be displayed as is. If the type implements `__str__`, then the return value of that will be displayed.
It is important to note that expressions with side effects are affected by this feature. This is because in order to display this information, we must store references to the instances and not just the values.
Rational -------- Every test boils down to the binary statement "Is this true or false?", whether you use the built-in assert keyword or a more advanced assertion method provided by a testing framework. When an assertion fails, the output is binary too — "Expected x, but got y".
There are helpful libraries like Hamcrest which give you a more verbose breakdown of the difference and answer the question "What exactly is the difference between x and y?". This is extremely helpful, but it still focuses on the difference between the values.
We need to keep in mind that a given state is normally an outcome of a series of states, that is, one outcome is a result of multiple conditions and causes. This is where power assertion comes in. It allows us to better understand what led to the failure.
Implementation -------- I’ve already built a fully functional implementation[2] of this feature as part of my Python testing framework - Nimoy[3]. The current implementation uses AST manipulation to remap the expression to a data structure[4] at compile time, so that it can then be evaluated and printed[5] at runtime.
[1] http://docs.groovy-lang.org/next/html/documentation/core-testing-guide.html#... [2] https://browncoat-ninjas.github.io/nimoy/examples/#power-assertions-beta [3] https://github.com/browncoat-ninjas/nimoy/ [4] https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/ast_tools/expre... [5] https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/assertions/powe... _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/T26DR4... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido (mobile) _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/EEVHXK... Code of Conduct: http://python.org/psf/codeofconduct/
-- Juancarlo *Añez*
I think that this is a great idea. However, pipes only point to one character, which can be confusing. (Citation: many tracebacks before 3.10.) I'm wondering: Could failed assertions step you into `pdb`, if they are used for testing purposes? Could there be a way to specify different levels of assertions? For example, maybe certain assertions are turned off with -O or -OO, others turned off only with -O, and some that never are? There are lots of ways `assert` could be improved, and the question is how? What is/are the best way(s)? -- Finn Mason On Mon, Sep 13, 2021, 5:36 AM Juancarlo Añez <apalala@gmail.com> wrote:
What about asserts that are not used for testing, but as classic “unless
there’s a bug, this should hold”?
To me this relates to the thread about having a structured *assert* that doesn't go away with *-O*.
My purpose when addressing *assert* was precisely the *“unless there’s a bug, this should hold”* kind of assertions.
In that context, introducing additional yet known costs (as in this "power" idea), providing for different exception types (maybe all descending from AssertError?), and allowing for a block to prepare the exception, are all worth it.
Introducing the new syntax for *assert* would imply zero cost for existing assertions.
On Sun, Sep 12, 2021 at 10:28 AM Guido van Rossum <guido@python.org> wrote:
This is cool.
AFAIK pytest does something like this. How does your implementation differ?
What is your argument for making this part of the language? Why not a 3rd party library?
What about asserts that are not used for testing, but as classic “unless there’s a bug, this should hold”? Those may not want to incur the extra cost.
—Guido
On Sun, Sep 12, 2021 at 07:09 <noam@10ne.org> wrote:
Hi all,
I’d like your comments and feedback on an enhancement that introduces power assertions to the Python language.
Proposal -------- This feature is inspired by a similar feature of the Groovy language[1], and is effectively a variant of the `assert` keyword. When an assertion expression evaluates to `False`, the output shows not only the failure, but also a breakdown of the evaluated expression from the inner part to the outer part.
For example, a procedure like: ```python class SomeClass: def __init__(self): self.val = {'d': 'e'}
def __str__(self): return str(self.val)
sc = SomeClass()
assert sc.val['d'] == 'f' ```
Will result in the output:
```python Assertion failed:
sc.val['d'] == f | | | | e False | {'d': 'e'} ``` See link [2] if the formatting above is screwed up.
In the output above we can see the value of every part of the expression from left to right, mapped to their expression fragment with the pipe (`|`). The number of rows that are printed depend on the value of each fragment of the expression. If the value of a fragment is longer than the actual fragment (`{'d': 'e'}` is longer than `sc`), then the next value (`e`) will be printed on a new line which will appear above. Values are appended to the same line until it overflows in length to horizontal position of the next fragment.
The information that’s displayed is dictated by the type. If the type is a constant value, it will be displayed as is. If the type implements `__str__`, then the return value of that will be displayed.
It is important to note that expressions with side effects are affected by this feature. This is because in order to display this information, we must store references to the instances and not just the values.
Rational -------- Every test boils down to the binary statement "Is this true or false?", whether you use the built-in assert keyword or a more advanced assertion method provided by a testing framework. When an assertion fails, the output is binary too — "Expected x, but got y".
There are helpful libraries like Hamcrest which give you a more verbose breakdown of the difference and answer the question "What exactly is the difference between x and y?". This is extremely helpful, but it still focuses on the difference between the values.
We need to keep in mind that a given state is normally an outcome of a series of states, that is, one outcome is a result of multiple conditions and causes. This is where power assertion comes in. It allows us to better understand what led to the failure.
Implementation -------- I’ve already built a fully functional implementation[2] of this feature as part of my Python testing framework - Nimoy[3]. The current implementation uses AST manipulation to remap the expression to a data structure[4] at compile time, so that it can then be evaluated and printed[5] at runtime.
[1] http://docs.groovy-lang.org/next/html/documentation/core-testing-guide.html#... [2] https://browncoat-ninjas.github.io/nimoy/examples/#power-assertions-beta [3] https://github.com/browncoat-ninjas/nimoy/ [4] https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/ast_tools/expre... [5] https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/assertions/powe... _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/T26DR4... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido (mobile) _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/EEVHXK... Code of Conduct: http://python.org/psf/codeofconduct/
-- Juancarlo *Añez* _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/RHXMGH... Code of Conduct: http://python.org/psf/codeofconduct/
If assertions have an associated block, then `pdb` can be invoked within. I almost never use debuggers, so I don't remember, but I think a recent Python version introduced the likes of `debug()` to step into the pre-configured debugger. About the "power assertions" proposal in this thread, once everything is in place to produce the output as proposed, then libraries may come in to produce different output formats. My personal preference is for a form of assertions that don't go away, ever. Regarding the syntax, I think that should be the last part of the design. If you think about it, a block associated with the assert should execute when the assertion fails, so maybe it should be something like: *assert* *cond* *else:* *block* with an AssertionError raised if *"block"* does not raise. Another proposal out there that has the same semantics is: *unless* *cond*: *block* Yet I find that using a negative keyword makes things ugly. I'd rather stick to "*assert*", or maybe go for "*invariant*" statement, to be rid of the legacy of the meaning of *"assert"* in Python's history. *invariant* *cond*: *block* After all, I think it's invariants what I've been after in my recent posts. It's easy to document that the *"cond"* must be cheap because it will *_always_* be executed, and that the *"block"* can be as complex as required by the severity of the failure. I still think that there's different levels of criticality to failed assertions, and that an exception hierarchy that has levels can help build more reliable systems. On Tue, Sep 14, 2021 at 5:52 PM Finn Mason <finnjavier08@gmail.com> wrote:
I think that this is a great idea. However, pipes only point to one character, which can be confusing. (Citation: many tracebacks before 3.10.)
I'm wondering: Could failed assertions step you into `pdb`, if they are used for testing purposes? Could there be a way to specify different levels of assertions? For example, maybe certain assertions are turned off with -O or -OO, others turned off only with -O, and some that never are? There are lots of ways `assert` could be improved, and the question is how? What is/are the best way(s)?
-- Finn Mason
On Mon, Sep 13, 2021, 5:36 AM Juancarlo Añez <apalala@gmail.com> wrote:
What about asserts that are not used for testing, but as classic “unless
there’s a bug, this should hold”?
To me this relates to the thread about having a structured *assert* that doesn't go away with *-O*.
My purpose when addressing *assert* was precisely the *“unless there’s a bug, this should hold”* kind of assertions.
In that context, introducing additional yet known costs (as in this "power" idea), providing for different exception types (maybe all descending from AssertError?), and allowing for a block to prepare the exception, are all worth it.
Introducing the new syntax for *assert* would imply zero cost for existing assertions.
On Sun, Sep 12, 2021 at 10:28 AM Guido van Rossum <guido@python.org> wrote:
This is cool.
AFAIK pytest does something like this. How does your implementation differ?
What is your argument for making this part of the language? Why not a 3rd party library?
What about asserts that are not used for testing, but as classic “unless there’s a bug, this should hold”? Those may not want to incur the extra cost.
—Guido
On Sun, Sep 12, 2021 at 07:09 <noam@10ne.org> wrote:
Hi all,
I’d like your comments and feedback on an enhancement that introduces power assertions to the Python language.
Proposal -------- This feature is inspired by a similar feature of the Groovy language[1], and is effectively a variant of the `assert` keyword. When an assertion expression evaluates to `False`, the output shows not only the failure, but also a breakdown of the evaluated expression from the inner part to the outer part.
For example, a procedure like: ```python class SomeClass: def __init__(self): self.val = {'d': 'e'}
def __str__(self): return str(self.val)
sc = SomeClass()
assert sc.val['d'] == 'f' ```
Will result in the output:
```python Assertion failed:
sc.val['d'] == f | | | | e False | {'d': 'e'} ``` See link [2] if the formatting above is screwed up.
In the output above we can see the value of every part of the expression from left to right, mapped to their expression fragment with the pipe (`|`). The number of rows that are printed depend on the value of each fragment of the expression. If the value of a fragment is longer than the actual fragment (`{'d': 'e'}` is longer than `sc`), then the next value (`e`) will be printed on a new line which will appear above. Values are appended to the same line until it overflows in length to horizontal position of the next fragment.
The information that’s displayed is dictated by the type. If the type is a constant value, it will be displayed as is. If the type implements `__str__`, then the return value of that will be displayed.
It is important to note that expressions with side effects are affected by this feature. This is because in order to display this information, we must store references to the instances and not just the values.
Rational -------- Every test boils down to the binary statement "Is this true or false?", whether you use the built-in assert keyword or a more advanced assertion method provided by a testing framework. When an assertion fails, the output is binary too — "Expected x, but got y".
There are helpful libraries like Hamcrest which give you a more verbose breakdown of the difference and answer the question "What exactly is the difference between x and y?". This is extremely helpful, but it still focuses on the difference between the values.
We need to keep in mind that a given state is normally an outcome of a series of states, that is, one outcome is a result of multiple conditions and causes. This is where power assertion comes in. It allows us to better understand what led to the failure.
Implementation -------- I’ve already built a fully functional implementation[2] of this feature as part of my Python testing framework - Nimoy[3]. The current implementation uses AST manipulation to remap the expression to a data structure[4] at compile time, so that it can then be evaluated and printed[5] at runtime.
[1] http://docs.groovy-lang.org/next/html/documentation/core-testing-guide.html#... [2] https://browncoat-ninjas.github.io/nimoy/examples/#power-assertions-beta [3] https://github.com/browncoat-ninjas/nimoy/ [4] https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/ast_tools/expre... [5] https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/assertions/powe... _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/T26DR4... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido (mobile) _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/EEVHXK... Code of Conduct: http://python.org/psf/codeofconduct/
-- Juancarlo *Añez* _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/RHXMGH... Code of Conduct: http://python.org/psf/codeofconduct/
-- Juancarlo *Añez*
participants (8)
-
2QdxY4RzWzUUiLuE＠potatochowder.com -
Caleb Donovick -
Chris Angelico -
Finn Mason -
Guido van Rossum -
Juancarlo Añez -
noam＠10ne.org -
Serhiy Storchaka