2QdxY4RzWzUUiLuE＠potatochowder.com wrote:

...

What about asserts that are not used for testing, but as classic “unless there’s a bug, this should hold”? Those may not want to incur the extra cost. I was actually thinking exactly the opposite: this would more useful in

On 2021-09-12 at 07:28:53 -0700, Guido van Rossum guido@python.org wrote: production than in testing. When I'm testing, tests build on each other. I should know that the inner parts work, and I should be testing specific aspects of the outer parts. If I don't have confidence in those inner parts, then I need to write more tests against them. If I don't "know" where the data comes from in my assertions, then my tests are trying to test too much at once. On the other hand, weird things happen in production, and my first reaction to "this shouldn't happen unless there's a bug" are is to start looking in the logs at how we got there. These power assertions are like a retroactive or JIT logging mechanism (in the sense that I may not have logged enough detail), or a sort of first order run-time debugger that shows me what's relevant at the point of the failure. As far as the extra cost goes, how does that cost compare to the full stack trace I already get from the exception being raised?

That's an interesting observation. I haven't thought of that. Regarding cost - I haven't profiled my implementation, so I'll have to get back to you on that one.

12.09.21 17:28, Guido van Rossum пише:

This is cool.

AFAIK pytest does something like this. How does your implementation differ?

What pytest does is awesome. I though about implementing it in the standard compiler since seen it the first time.

What is your argument for making this part of the language? Why not a 3rd party library?

It needs a support in the compiler. The condition expression should be compiled to keep all immediate results of subexpressions on the stack. If the final result is true, immediate results are dropped. If it is false, the second argument of assert is evaluated and its value together with all immediate results of the first expression, together with references to corresponding subexpressions (as strings, ranges or AST nodes) are passed to the special handler. That handler can be implemented in a third-party library, because formatting and outputting a report is a complex task. The default handler can just raise an AttributeError.

What about asserts that are not used for testing, but as classic “unless there’s a bug, this should hold”? Those may not want to incur the extra cost.

The only extra cost is that immediate results are temporary save on stack instead of just be dropped. It increases the size of bytecode and stack, but I don't think it will be significant.

12.09.21 21:20, Guido van Rossum пише:

Maybe you all could collaborate on a PEP? This sounds a worthy topic.

I can write an implementation if we decide on the interface. I am currently have higher priorities of other tasks than to make a research on this feature.

On Mon, Sep 13, 2021 at 1:37 AM Serhiy Storchaka wrote:

12.09.21 17:28, Guido van Rossum пише:

...

This is cool.

AFAIK pytest does something like this. How does your implementation differ?

What pytest does is awesome. I though about implementing it in the standard compiler since seen it the first time.

...

What is your argument for making this part of the language? Why not a 3rd party library?

It needs a support in the compiler. The condition expression should be compiled to keep all immediate results of subexpressions on the stack. If the final result is true, immediate results are dropped. If it is false, the second argument of assert is evaluated and its value together with all immediate results of the first expression, together with references to corresponding subexpressions (as strings, ranges or AST nodes) are passed to the special handler. That handler can be implemented in a third-party library, because formatting and outputting a report is a complex task. The default handler can just raise an AttributeError.

I wonder, could this be simplified a bit, on the assumption that a well-written assertion shouldn't have a problem with being executed twice? Instead of keeping all the subexpressions around (a run-time cost), keep the AST of the expression itself (a compile-time cost). Then, when the exception is about to be printed to the console, re-evaluate it and do the display. ChrisA

I wonder, could this be simplified a bit, on the assumption that a well-written assertion shouldn't have a problem with being executed twice?

While I agree as an engineering principle an assert should not have side effects and hence re-evaluation should be fine in most cases, it is not universal. It is possible for assertions to not have side effects but yet change value between evaluations if they interact with a shared resource such as the file system.. For example consider the following assertion: assert os.path.isdir("config") and os.path.isfile("config/setup.yml") It is completely possible for the value of this expression to change between evaluations. Granted this would like mean their is some more significant issue with my code, however, I would like the interpreter to give me accurate information about why my assertion failed. Bad information is worse than no information. Like imagine that on the first evaluation the directory config does not exist but on the second it has been created by another process. A naive revaluation strategy would likely result in it pointing at the second clause and saying the assertion failed their when it really failed on the first clause. This would send me down a rabbit hole of debugging why setup.yml was not constructed properly instead of debugging why the config directory didn’t exist. Further while it is bad engineering practices to have side effects in an assert it is completely possible. For example consider the following pathological example: class PathologicalFoo: def __init__(self): self._val = 0 def get(self): old_val = self._val self._val = 1 return old_val foo = PathologicalFoo()assert foo.get() == 1 Or worse it is possible for revaluation to cause errors class UniquePtr: def __init__(self, obj): self.set(obj) def get(self): if self._valid: self._valid = False obj = self._obj self._obj = None return obj else: raise ValueError() def set(self, obj): self._obj = obj self._valid = True x = UniquePtr(1)assert x.get() == 0 x.set(0) How would the interpreter handle this? On Sun, Sep 12, 2021 at 11:39 AM Chris Angelico wrote:

On Mon, Sep 13, 2021 at 1:37 AM Serhiy Storchaka wrote:

...

12.09.21 17:28, Guido van Rossum пише:

...

This is cool.

AFAIK pytest does something like this. How does your implementation

differ?

...

What pytest does is awesome. I though about implementing it in the standard compiler since seen it the first time.

...

What is your argument for making this part of the language? Why not a 3rd party library?

It needs a support in the compiler. The condition expression should be compiled to keep all immediate results of subexpressions on the stack. If the final result is true, immediate results are dropped. If it is false, the second argument of assert is evaluated and its value together with all immediate results of the first expression, together with references to corresponding subexpressions (as strings, ranges or AST nodes) are passed to the special handler. That handler can be implemented in a third-party library, because formatting and outputting a report is a complex task. The default handler can just raise an AttributeError.

I wonder, could this be simplified a bit, on the assumption that a well-written assertion shouldn't have a problem with being executed twice? Instead of keeping all the subexpressions around (a run-time cost), keep the AST of the expression itself (a compile-time cost). Then, when the exception is about to be printed to the console, re-evaluate it and do the display.

ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/IXPY4B... Code of Conduct: http://python.org/psf/codeofconduct/

2) Some OTHER exception occurs on the reevaluation. It's a chained exception like any other.

Except it's not a chained exception and displaying as such would be VERY confusing IMO. Granted we could easily strip the chained exception and just return the original one. So after reconsideration I agree this is not an issue.

It's only really the fourth case that would be confusing

I generally agree with your analysis but I think this 4th case is more problematic than you think. Given no information I am immediately going to split my assertion so I can see what part is failing. However, if the interpreter gives me incorrect information I am going to be super confused. Most people will not have carefully read section 7.3 of the language reference and will not understand this critical aspect of the execution of assertion statements. They will assume that the interpreter is not lying to them. I think storing the intermediate results on the stack is vastly preferable to revaluation for this reason. On Mon, Oct 4, 2021 at 3:20 PM Chris Angelico wrote:

On Tue, Oct 5, 2021 at 9:02 AM Caleb Donovick wrote:

...

...

I wonder, could this be simplified a bit, on the assumption that a well-written assertion shouldn't have a problem with being executed twice?

While I agree as an engineering principle an assert should not have side

...

and hence re-evaluation should be fine in most cases, it is not universal. It is possible for assertions to not have side effects but yet change value between evaluations if they interact with a shared resource such as the file system..

For example consider the following assertion:

assert os.path.isdir("config") and os.path.isfile("config/setup.yml")

It is completely possible for the value of this expression to change between evaluations. Granted this would like mean their is some more significant issue with my code, however, I would like the interpreter to give me accurate information about why my assertion failed. Bad information is worse than no information. Like imagine that on the first evaluation the directory config does not exist but on

effects the second it has been created by another process.

...

A naive revaluation strategy would likely result in it pointing at the second clause and saying the assertion failed their when it really failed on the first clause. This would send me down a rabbit hole of debugging why setup.yml was not constructed properly instead of debugging why the config directory didn’t exist.

That seems like an abuse of assertions. If you have assertions that depend on external state that can change that quickly, then the assertion is *already useless*. What do you gain by asserting something that might have changed by the next line of code?

...

Further while it is bad engineering practices to have side effects in an assert it is completely possible. For example consider the following pathological example:

class PathologicalFoo: def __init__(self): self._val = 0

def get(self): old_val = self._val self._val = 1 return old_val

foo = PathologicalFoo() assert foo.get() == 1

Yes, side effects in assertions are always possible. If someone has assertions with side effects, do we say that python -O is buggy, or the assertion is buggy? In a world in which assertions might and might not be evaluated, is it such a stretch to demand that they can be safely reevaluated (in the same context)? Yes, it's a change to the expectations, but one which well-designed assertions shouldn't be bothered by.

My imagining of this is that it'd be handled when an AssertionError reaches top level, and it'd be broadly thus:

try: all_your_code() except AssertionError as e: ... reevaluate etc

Meaning there are four possibilities: 1) The assertion is consistent, and the extra info is absolutely correct 2) Some OTHER exception occurs on the reevaluation. It's a chained exception like any other. 3) No assertion failure happens (eg PathologicalFoo). Might require a minor special case "if nothing goes wrong, print out the original" but that's the most obvious thing to do. 4) The assertion fails in an inconsistent way, but it still fails. You'll get the second form instead of the first.

It's only really the fourth case that would be confusing, and only if the first evaluation actually causes the problem (otherwise it's just an inconsistent assertion and you'd need to debug both parts anyway). This is a pretty narrow problem, and even then, you've been shown a weird assertion that needs to be debugged. Is it that bad to say that an assertion that gives inconsistent results is buggy?

ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/GYNRJA... Code of Conduct: http://python.org/psf/codeofconduct/

What about asserts that are not used for testing, but as classic “unless there’s a bug, this should hold”?

To me this relates to the thread about having a structured *assert* that doesn't go away with *-O*. My purpose when addressing *assert* was precisely the *“unless there’s a bug, this should hold”* kind of assertions. In that context, introducing additional yet known costs (as in this "power" idea), providing for different exception types (maybe all descending from AssertError?), and allowing for a block to prepare the exception, are all worth it. Introducing the new syntax for *assert* would imply zero cost for existing assertions. On Sun, Sep 12, 2021 at 10:28 AM Guido van Rossum wrote:

This is cool.

AFAIK pytest does something like this. How does your implementation differ?

What is your argument for making this part of the language? Why not a 3rd party library?

What about asserts that are not used for testing, but as classic “unless there’s a bug, this should hold”? Those may not want to incur the extra cost.

—Guido

On Sun, Sep 12, 2021 at 07:09 wrote:

...

Hi all,

I’d like your comments and feedback on an enhancement that introduces power assertions to the Python language.

Proposal -------- This feature is inspired by a similar feature of the Groovy language[1], and is effectively a variant of the `assert` keyword. When an assertion expression evaluates to `False`, the output shows not only the failure, but also a breakdown of the evaluated expression from the inner part to the outer part.

For example, a procedure like: ```python class SomeClass: def __init__(self): self.val = {'d': 'e'}

def __str__(self): return str(self.val)

sc = SomeClass()

assert sc.val['d'] == 'f' ```

Will result in the output:

```python Assertion failed:

sc.val['d'] == f | | | | e False | {'d': 'e'} ``` See link [2] if the formatting above is screwed up.

In the output above we can see the value of every part of the expression from left to right, mapped to their expression fragment with the pipe (`|`). The number of rows that are printed depend on the value of each fragment of the expression. If the value of a fragment is longer than the actual fragment (`{'d': 'e'}` is longer than `sc`), then the next value (`e`) will be printed on a new line which will appear above. Values are appended to the same line until it overflows in length to horizontal position of the next fragment.

The information that’s displayed is dictated by the type. If the type is a constant value, it will be displayed as is. If the type implements `__str__`, then the return value of that will be displayed.

It is important to note that expressions with side effects are affected by this feature. This is because in order to display this information, we must store references to the instances and not just the values.

Rational -------- Every test boils down to the binary statement "Is this true or false?", whether you use the built-in assert keyword or a more advanced assertion method provided by a testing framework. When an assertion fails, the output is binary too — "Expected x, but got y".

There are helpful libraries like Hamcrest which give you a more verbose breakdown of the difference and answer the question "What exactly is the difference between x and y?". This is extremely helpful, but it still focuses on the difference between the values.

We need to keep in mind that a given state is normally an outcome of a series of states, that is, one outcome is a result of multiple conditions and causes. This is where power assertion comes in. It allows us to better understand what led to the failure.

Implementation -------- I’ve already built a fully functional implementation[2] of this feature as part of my Python testing framework - Nimoy[3]. The current implementation uses AST manipulation to remap the expression to a data structure[4] at compile time, so that it can then be evaluated and printed[5] at runtime.

[1] http://docs.groovy-lang.org/next/html/documentation/core-testing-guide.html#... [2] https://browncoat-ninjas.github.io/nimoy/examples/#power-assertions-beta [3] https://github.com/browncoat-ninjas/nimoy/ [4] https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/ast_tools/expre... [5] https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/assertions/powe... _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/T26DR4... Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido (mobile) _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/EEVHXK... Code of Conduct: http://python.org/psf/codeofconduct/

-- Juancarlo *Añez*

If assertions have an associated block, then `pdb` can be invoked within. I almost never use debuggers, so I don't remember, but I think a recent Python version introduced the likes of `debug()` to step into the pre-configured debugger. About the "power assertions" proposal in this thread, once everything is in place to produce the output as proposed, then libraries may come in to produce different output formats. My personal preference is for a form of assertions that don't go away, ever. Regarding the syntax, I think that should be the last part of the design. If you think about it, a block associated with the assert should execute when the assertion fails, so maybe it should be something like: *assert* *cond* *else:* *block* with an AssertionError raised if *"block"* does not raise. Another proposal out there that has the same semantics is: *unless* *cond*: *block* Yet I find that using a negative keyword makes things ugly. I'd rather stick to "*assert*", or maybe go for "*invariant*" statement, to be rid of the legacy of the meaning of *"assert"* in Python's history. *invariant* *cond*: *block* After all, I think it's invariants what I've been after in my recent posts. It's easy to document that the *"cond"* must be cheap because it will *_always_* be executed, and that the *"block"* can be as complex as required by the severity of the failure. I still think that there's different levels of criticality to failed assertions, and that an exception hierarchy that has levels can help build more reliable systems. On Tue, Sep 14, 2021 at 5:52 PM Finn Mason wrote:

I think that this is a great idea. However, pipes only point to one character, which can be confusing. (Citation: many tracebacks before 3.10.)

I'm wondering: Could failed assertions step you into `pdb`, if they are used for testing purposes? Could there be a way to specify different levels of assertions? For example, maybe certain assertions are turned off with -O or -OO, others turned off only with -O, and some that never are? There are lots of ways `assert` could be improved, and the question is how? What is/are the best way(s)?

-- Finn Mason

On Mon, Sep 13, 2021, 5:36 AM Juancarlo Añez wrote:

...

What about asserts that are not used for testing, but as classic “unless

...

there’s a bug, this should hold”?

To me this relates to the thread about having a structured *assert* that doesn't go away with *-O*.

My purpose when addressing *assert* was precisely the *“unless there’s a bug, this should hold”* kind of assertions.

In that context, introducing additional yet known costs (as in this "power" idea), providing for different exception types (maybe all descending from AssertError?), and allowing for a block to prepare the exception, are all worth it.

Introducing the new syntax for *assert* would imply zero cost for existing assertions.

On Sun, Sep 12, 2021 at 10:28 AM Guido van Rossum wrote:

...

This is cool.

AFAIK pytest does something like this. How does your implementation differ?

What is your argument for making this part of the language? Why not a 3rd party library?

What about asserts that are not used for testing, but as classic “unless there’s a bug, this should hold”? Those may not want to incur the extra cost.

—Guido

On Sun, Sep 12, 2021 at 07:09 wrote:

...

Hi all,

I’d like your comments and feedback on an enhancement that introduces power assertions to the Python language.

Proposal -------- This feature is inspired by a similar feature of the Groovy language[1], and is effectively a variant of the `assert` keyword. When an assertion expression evaluates to `False`, the output shows not only the failure, but also a breakdown of the evaluated expression from the inner part to the outer part.

For example, a procedure like: ```python class SomeClass: def __init__(self): self.val = {'d': 'e'}

def __str__(self): return str(self.val)

sc = SomeClass()

assert sc.val['d'] == 'f' ```

Will result in the output:

```python Assertion failed:

sc.val['d'] == f | | | | e False | {'d': 'e'} ``` See link [2] if the formatting above is screwed up.

In the output above we can see the value of every part of the expression from left to right, mapped to their expression fragment with the pipe (`|`). The number of rows that are printed depend on the value of each fragment of the expression. If the value of a fragment is longer than the actual fragment (`{'d': 'e'}` is longer than `sc`), then the next value (`e`) will be printed on a new line which will appear above. Values are appended to the same line until it overflows in length to horizontal position of the next fragment.

The information that’s displayed is dictated by the type. If the type is a constant value, it will be displayed as is. If the type implements `__str__`, then the return value of that will be displayed.

It is important to note that expressions with side effects are affected by this feature. This is because in order to display this information, we must store references to the instances and not just the values.

Rational -------- Every test boils down to the binary statement "Is this true or false?", whether you use the built-in assert keyword or a more advanced assertion method provided by a testing framework. When an assertion fails, the output is binary too — "Expected x, but got y".

There are helpful libraries like Hamcrest which give you a more verbose breakdown of the difference and answer the question "What exactly is the difference between x and y?". This is extremely helpful, but it still focuses on the difference between the values.

We need to keep in mind that a given state is normally an outcome of a series of states, that is, one outcome is a result of multiple conditions and causes. This is where power assertion comes in. It allows us to better understand what led to the failure.

Implementation -------- I’ve already built a fully functional implementation[2] of this feature as part of my Python testing framework - Nimoy[3]. The current implementation uses AST manipulation to remap the expression to a data structure[4] at compile time, so that it can then be evaluated and printed[5] at runtime.

[1] http://docs.groovy-lang.org/next/html/documentation/core-testing-guide.html#... [2] https://browncoat-ninjas.github.io/nimoy/examples/#power-assertions-beta [3] https://github.com/browncoat-ninjas/nimoy/ [4] https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/ast_tools/expre... [5] https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/assertions/powe... _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/T26DR4... Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido (mobile) _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/EEVHXK... Code of Conduct: http://python.org/psf/codeofconduct/

-- Juancarlo *Añez* _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/RHXMGH... Code of Conduct: http://python.org/psf/codeofconduct/

-- Juancarlo *Añez*