PEP Draft: Power Assertion
Hi All, Following the discussions on "Power Assertions: Is it PEP-able?", I've drafted this PEP. Your comments are most welcome. PEP: 9999 Title: Power Assertion Author: Noam Tenne <noam@10ne.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 24-Sep-2021 Abstract ======== This PEP introduces a language enhancement named "Power Assertion". The Power Assertion is inspired by a similar feature found in the `Groovy language`_ and is an extension to the core lib's ``assert`` keyword. When an assertion expression evaluates to ``False``, the output shows not only the failure, but also a breakdown of the evaluated expression from the inner part to the outer part. .. _Groovy language: http://docs.groovy-lang.org/next/html/documentation/core-testing-guide.html#... Motivation ========= Every test boils down to the binary statement "Is this true or false?", whether you use the built-in assert keyword or a more advanced assertion method provided by a testing framework. When an assertion fails, the output is binary too — "Expected x, but got y". There are helpful libraries like Hamcrest which give you a more verbose breakdown of the difference and answer the question "What exactly is the difference between x and y?". This is extremely helpful, but it still focuses on the difference between the values. Keep in mind that a given state is normally an outcome of a series of states, that is, one outcome is a result of multiple conditions and causes. This is where Power Assertion comes in. It allows us to better understand what led to the failure. The Community Wants This ------------------------ As mentioned in the abstract, this feature was borrowed from Groovy. It is a very popular feature within the Groovy community, also used by projects such as `Spock`_. On top of that, it is very much needed in the Python community as well: * `Power Assertion was explicitly requested`_ as a feature in the `Nimoy`_ testing framework * There's a `similar feature in pytest`_ And when `discussed in the python-ideas`_ mailing list, the responses were overall positive: * "This is cool." - Guido van Rossum * "I was actually thinking exactly the opposite: this would more useful in production than in testing." - 2QdxY4RzWzUUiLuE＠potatochowder.com * "What pytest does is awesome. I though about implementing it in the standard compiler since seen it the first time." - Serhiy Storchaka .. _Spock: https://spockframework.org/ .. _Power Assertion was explicitly requested: https://stackoverflow.com/a/58536986/198825 .. _similar feature in pytest: https://docs.pytest.org/en/latest/how-to/assert.html .. _discussed in the python-ideas: https://mail.python.org/archives/list/python-ideas@python.org/thread/T26DR4B... Rational ======== Code Example ------------ :: class SomeClass: def __init__(self): self.val = {'d': 'e'} def __str__(self): return str(self.val) sc = SomeClass() assert sc.val['d'] == 'f' This routine will result in the output: :: Assertion failed: sc.val['d'] == f | | | | e False | {'d': 'e'} Display ------- In the output above we can see the value of every part of the expression from left to right, mapped to their expression fragment with the pipe (``|``). The number of rows that are printed depend on the value of each fragment of the expression. If the value of a fragment is longer than the actual fragment (``{'d': 'e'}`` is longer than ``sc``), then the next value (``e``) will be printed on a new line which will appear above. Values are appended to the same line until it overflows in length to horizontal position of the next fragment. This way of presentation is clearer and more human friendly than the output offered by pytest's solution. The information that’s displayed is dictated by the type. If the type is a constant value, it will be displayed as is. If the type implements `__str__`, then the return value of that will be displayed. Mechanics --------- Reference Implementation '''''''''''''''''''''''' The reference implementation uses AST manipulation because this is the only way that this level of involvement can be achieved by a third party library. It iterates over every subexpression in the assert statement. Subexpressions are parts of the expression separated by a lookup (``map[key]``), an attribute reference (``key.value``) or a binary comparison operator (``==``). It then builds an AST in the structure of a tree to maintain the order of the operations in the original code, and tracks the original code of the subexpression together with the AST code of the subexpression and the original indices. It then rewrites the AST of the original expression to call a specialised assertion function, which accepts the tree as a parameter. At runtime the expression is executed. If it fails, a rendering function is called to render the assertion message as per the example above. Actual Implementation ''''''''''''''''''''' To be discussed. In the python-ideas mailing list, Serhiy Storchaka suggests: It needs a support in the compiler. The condition expression should be compiled to keep all immediate results of subexpressions on the stack. If the final result is true, immediate results are dropped. If it is false, the second argument of assert is evaluated and its value together with all immediate results of the first expression, together with references to corresponding subexpressions (as strings, ranges or AST nodes) are passed to the special handler. That handler can be implemented in a third-party library, because formatting and outputting a report is a complex task. The default handler can just raise an AttributeError. Caveats ------- It is important to note that expressions with side effects are affected by this feature. This is because in order to display this information, we must store references to the instances and not just the values. Reference Implementation ======================== There's a `complete implementation`_ of this enhancement in the `Nimoy`_ testing framework. It uses AST manipulation to remap the expression to a `data structure`_ at compile time, so that it can then be `evaluated and printed`_ at runtime. .. _Nimoy: https://browncoat-ninjas.github.io/nimoy/ .. _complete implementation: https://browncoat-ninjas.github.io/nimoy/examples/#power-assertions-beta .. _data structure: https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/ast_tools/expre... .. _evaluated and printed: https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/assertions/powe... Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
On Fri, 24 Sept 2021 at 12:05, Noam Tenne <noam@10ne.org> wrote:
Caveats -------
It is important to note that expressions with side effects are affected by this feature. This is because in order to display this information, we must store references to the instances and not just the values.
One immediate thought. You should give examples of the sort of expressions that are affected, and precisely what the effect is. It's impossible to judge the importance of this point without details. Paul
My first thought when reading this (and the recent discussion on Python ideas) was that pytest already does this. Then I see in the PEP: "What pytest does is awesome“ So what? Well, two points: 1) this should be part of a test framework, not something built in to exceptions. Pytest has shown that it can be done without any language changes. 2) there are a lot of things in Pytest that are very useful. And a lot of problems with unittest. So maybe what we should do now, rather than add one feature, is propose a new test framework/test runner for the stdlib, inspired by pytest. Alternatively, take the approach taken with distutils and setuptools— officially accept that a full featured test framework will be left to third parties. NOTE: if the proposal does require actual language changes beyond the current introspection options, that should be made clear. -CHB On Fri, Sep 24, 2021 at 4:12 AM Paul Moore <p.f.moore@gmail.com> wrote:
On Fri, 24 Sept 2021 at 12:05, Noam Tenne <noam@10ne.org> wrote:
Caveats -------
It is important to note that expressions with side effects are affected by this feature. This is because in order to display this information, we must store references to the instances and not just the values.
One immediate thought. You should give examples of the sort of expressions that are affected, and precisely what the effect is. It's impossible to judge the importance of this point without details.
Paul _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/JJ2CT4... Code of Conduct: http://python.org/psf/codeofconduct/
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Fri, Sep 24, 2021 at 07:45:41PM -0700, Christopher Barker wrote:
1) this should be part of a test framework, not something built in to exceptions. Pytest has shown that it can be done without any language changes.
Pytest needs to take heroic measures in an import hook to re-write your assertions to do what it does: http://pybites.blogspot.com/2011/07/behind-scenes-of-pytests-new-assertion.h... Maybe a language change, with a public API for assertions, would make it much easier and less hacky for test frameworks to do what Pytest does. It wouldn't be the first time that Python added features to the language to make it easier for third-party libraries. Also, Pytest only supports CPython and PyPy. https://pytest.org/en/6.2.x/getting-started.html I know that Jython and IronPython aren't getting all the love that maybe they should be getting (migration to 3.x seems to be pretty slow), but it would be nice if better assertions was a language feature rather than requiring heroic import hook and AST hacks to work.
2) there are a lot of things in Pytest that are very useful.
Will they be less useful if Python gives better diagnostics on assertion failures?
And a lot of problems with unittest.
Such as? The only really unpleasant part of unittest is having to remember the spellings of all the various `self.assertSpam...` methods. Otherwise, it's a pretty nice framework. If assertions were better, perhaps we could fix that. On the other hand, I personally regularly use plain old assert in my unit tests, to distinguish between internal checks of the test logic, versus actual unit tests. Tests are code too, and can be buggy. I think it is important that test code includes assertions that refer to the test code itself. -- Steve
On Fri, Sep 24, 2021 at 8:51 PM Steven D'Aprano <steve@pearwood.info> wrote:
Pytest needs to take heroic measures in an import hook to re-write your assertions to do what it does:
Maybe a language change, with a public API for assertions, would make it much easier and less hacky for test frameworks to do what Pytest does.
Now that's a PEP I could get behind. That's what I was getting at with asking if there were language changes that would be required to really do this right. Adding one feature to unittest doesn't seem worth it to me -- adding something to the language that makes it possible (easier) for all test frameworks to be featureful would be a fine idea. I'd love to hear what the pytest folks think of this. It wouldn't be the first time that Python added features to the language
to make it easier for third-party libraries.
agreed.
Also, Pytest only supports CPython and PyPy.
Well, if the other implementations want a good test framework, that's up to them. it would be nice if better assertions was a language feature rather than
requiring heroic import hook and AST hacks to work.
I suspect that the "heroic measures" pytest takes are the challenge for other implementations, but so would adding this kind of functionality to all implementations -- putting it in official Python would be essentially requiring other implementations to do so -- but the developers of those would still need to do the work. (that being said, one option I suggested what that a shiny new featureful test framework be added to the stdlib :-) )
2) there are a lot of things in Pytest that are very useful.
Will they be less useful if Python gives better diagnostics on assertion failures?
Of course not -- my point was that if you want a better test framework, you will want more than just this one feature -- and it you're going to use pytest (or something else) anyway, then why do you need this in the stdlib.
And a lot of problems with unittest. Such as?
That's a long story that I've been meaning to write up for ages, but I'll make these few points: 1) Many major (and minor) projects use pytest, even though unittest is in the stdlib -- why is that? And why did they even bother writing pytest (and nose, back in the day) 2) unitest is not very open to extending -- no one wants to write many more assertions, and yet, we then don't have many. 3) This is, I suppose, personal taste, but I never liked unitest, it's just too awkward and verbose for me -- the second I discovered pytest I never looked back. And I'm not alone in that. The only really unpleasant part of unittest is having to remember the
spellings of all the various `self.assertSpam...` methods. Otherwise, it's a pretty nice framework.
The way to test Exception raising (assertRaises ?) is painful :-( But I know you have looked at the self.assertSpam methods -- why have them at all? all they do is provide a nice failure message (at least the ones I've looked at) with pytest, you don't need them at all. And there are any number of asserts that don't exist, so you have to use assertTrue, and then you don't get any benefit at all. If assertions were better, perhaps we could fix that.
yup -- then you wouldn't need any assertSpam methods -- then we could get rid of the TestCase classes, and you'd have something like pytest ;-) -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Fri, Sep 24, 2021 at 10:39:32PM -0700, Christopher Barker wrote:
Now that's a PEP I could get behind.
You mean, like this PEP? *wink*
Adding one feature to unittest doesn't seem worth it to me
I don't think this PEP directly proposes any new features to unittest. If it is accepted, who knows what will happen in the future?
Of course not -- my point was that if you want a better test framework, you will want more than just this one feature -- and it you're going to use pytest (or something else) anyway, then why do you need this in the stdlib.
The stdlib has got a test or two. We can't use pytest for the stdlib tests without making pytest a dependency and that would tie pytest to the stdlib's release cycle. This would make assertions more useful to everyone, whatever test framework (or no test framework at all) they use.
And a lot of problems with unittest. Such as?
That's a long story that I've been meaning to write up for ages, but I'll make these few points:
1) Many major (and minor) projects use pytest, even though unittest is in the stdlib -- why is that? And why did they even bother writing pytest (and nose, back in the day)
Good question. My recollection from way back in Python 1.5 days was that the vibe on comp.lang.python was all "Unit test? That's from Java, it's unpythonic!" Besides, everyone has their own opinion on test frameworks, like web frameworks and static code checkers. That's why there are so many of them :-)
2) unitest is not very open to extending -- no one wants to write many more assertions, and yet, we then don't have many.
o_O Most people complain that unittest has *too many* assert methods to remember, not too few. I count 32 assert methods in unittest. How many would you like? For the statistics test suite, I subclassed TestCase and added a new assert method. It's not hard to do: subclass TestCase and add some more methods, and call `self.failureException` with your error message if the test fails. How do you extend pytest's special assertion handling? Anyway, this isn't supposed to be a competition between test frameworks. There is plenty of room in the Python ecosystem for whatever test frameworks people want to run. If it were a competition, they would all lose to the mighty doctest!!! *wink*
The way to test Exception raising (assertRaises ?) is painful :-(
self.assertRaises(SomeException, callable, args, kwargs) Or if you prefer: with self.assertRaises(SomeException): block That's almost identical to the pytest way: https://docs.pytest.org/en/stable/reference.html#pytest-raises (For what it's worth, both pytest and unittest introduced the context manager form at the same time, July 2010.)
But I know you have looked at the self.assertSpam methods -- why have them at all? all they do is provide a nice failure message (at least the ones I've looked at) with pytest, you don't need them at all.
In practice, you don't normally call most of the specialist methods yourself. (Or at least I don't.) For example, assertEqual automatically delegates to one of the type-specific methods (lists, multi-line strings, tuples, etc). I would expect that pytest probably has similar specialised methods for formatting type-specific exceptions, except that they are purely internal while unittest exposes them as public methods. You say tom-a-toe, I say tom-ar-tow. -- Steve
Getting OT here -- a full discussion of unitest is a while other topic. One I do want to have one of these days, but not now. But for closure:
This would make assertions more useful to everyone, whatever test framework (or no test framework at all) they use.
1) Many major (and minor) projects use pytest, even though unittest is in the stdlib -- why is that? And why did they even bother writing pytest (and nose, back in the day)
Good question.
That was a rhetorical question,b ut I'll give the simple answer: Because a lot of people find pytest (and nose before it) some combination of easier, more comfortable and powerful. Which could be just taste, but I'd say writing pytest is an awful lot of work for just taste. And I know once I discovered it I never looked back.
My recollection from way back in Python 1.5 days was that the vibe on comp.lang.python was all "Unit test? That's from Java, it's unpythonic!"
Which is still very true, as Phillip J Eby famously wrote "Python is not Java". I"m not intending to criticise the decision at the time, unit testing was fairly new, and jUnit was successful. And it's not just style -- it's a fundamental approach -- jUnit was designed for a far more static, less introspectable, and more OO language than Python.
2) unitest is not very open to extending -- no one wants to write many more
assertions, and yet, we then don't have many.
o_O
Most people complain that unittest has *too many* assert methods to remember, not too few. I count 32 assert methods in unittest. How many would you like?
This is exactly the problem -- no one wants hundreds of assert Methods, yet there are hundreds (thousands) of things one might want to assert. Requiring a custom assert method for each case, is, shall we say, a bit problematic. But Python is a very introspectable language -- writing a static custom assert so that you can get a nice failure message is unnecessary. For the statistics test suite, I subclassed TestCase and added a new
assert method. It's not hard to do: subclass TestCase and add some more methods, and call `self.failureException` with your error message if the test fails.
But why did you have to do it at all? I'm speaking from my experience -- after PEP 485 was accepted, I sat down at the PyCon sprints to add an assertClose TestCase method. The result was this: https://bugs.python.org/issue27198 In the end, the best solution was to add a new attribute to assertAlmostEqual -- which I never got around to -- I had expended my PyCon sprint time. Why didn't I pick it up later? primarily because I don't use unittest myself anyway. But that conversation sure didn't make me think that unitest was a good API, and I don't know that any of the ideas talked about there have been implemented. How do you extend pytest's special assertion handling?
you don't -- that's the entire point. back to `math.isclose()` -- you simply do: assert isclose(a, b) done. And this is what you get if it fails:
assert isclose(1.0, 1.0000001)
E assert False E + where False = isclose(1.0, 1.0000001) Anyway, this isn't supposed to be a competition between test frameworks. indeed -- it's gotten OT.
The way to test Exception raising (assertRaises ?) is painful :-(
self.assertRaises(SomeException, callable, args, kwargs)
exactly -- painful.
Or if you prefer:
with self.assertRaises(SomeException): block
much better -- didn't exist when I learned unitest :-) In practice, you don't normally call most of the specialist methods
yourself. (Or at least I don't.) For example, assertEqual automatically delegates to one of the type-specific methods (lists, multi-line strings, tuples, etc).
ouch! Exactly the point -- that's all work that should be done, automatically by the test runner. and still we have: self.assertEqual(a, b) when we could have: assert a == b and self.assertTrue(something) rather than assert something I guess it's just taste, but my preference is clear. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Fri, Sep 24, 2021 at 19:49 Christopher Barker <pythonchb@gmail.com> wrote:
Alternatively, take the approach taken with distutils and setuptools— officially accept that a full featured test framework will be left to third parties.
I think this is by far the best option. Pytest can evolve much faster than the stdlib. —Guido
-- --Guido (mobile)
Guido van Rossum writes:
I think this is by far the best option. Pytest can evolve much faster than the stdlib.
Is there no room for making it easier to do this with less invasive changes to the stdlib, or are Steven d'A's "heroic measures in an import hook" the right way to go? Other Steve
On Fri, Sep 24, 2021 at 22:07 Stephen J. Turnbull < stephenjturnbull@gmail.com> wrote:
Guido van Rossum writes:
I think this is by far the best option. Pytest can evolve much faster than the stdlib.
Is there no room for making it easier to do this with less invasive changes to the stdlib, or are Steven d'A's "heroic measures in an import hook" the right way to go?
Other Steve
There’s room for that, but that’s not what’s being proposed (yet :-). —Guido
-- --Guido (mobile)
Is there no room for making it easier to do this with less invasive changes to the stdlib, or are Steven d'A's "heroic measures in an import hook" the right way to go?
I'm not familiar with this, can you please elaborate? From what I understand we can change stdlib to the point of manipulating the AST, and leave rendering to 3rd parties? On Sat, Sep 25, 2021, at 09:23, Guido van Rossum wrote:
On Fri, Sep 24, 2021 at 22:07 Stephen J. Turnbull <stephenjturnbull@gmail.com> wrote:
Guido van Rossum writes:
I think this is by far the best option. Pytest can evolve much faster than the stdlib.
Is there no room for making it easier to do this with less invasive changes to the stdlib, or are Steven d'A's "heroic measures in an import hook" the right way to go?
Other Steve
There’s room for that, but that’s not what’s being proposed (yet :-).
—Guido
-- --Guido (mobile)
On Fri, Sep 24, 2021 at 11:23:00PM -0700, Guido van Rossum wrote:
Is there no room for making it easier to do this with less invasive changes to the stdlib, or are Steven d'A's "heroic measures in an import hook" the right way to go?
Other Steve
There’s room for that, but that’s not what’s being proposed (yet :-).
I'm confused. My reading of the pre-PEP is that that is precisely what it is proposing: changing the way assert works so that the value of each sub-expression is available to be displayed to the user. Presumably any framework or library would be able to access that information. Have I missed something? I don't see anything in the proposal about creating new stdlib frameworks or changing unittest. The reference implementation for these "Power Assertions" uses AST re-writing, which is similar to what pytest does: http://pybites.blogspot.com/2011/07/behind-scenes-of-pytests-new-assertion.h... but the proposal suggests moving that into the compiler (which Serhiy considered doing earlier). I don't know if pytest would actually be able to make use of that, or whether they would keep their own AST re-writing implementation, it would probably be a good thing for Noam to reach out to pytest and other frameworks and see if they could use this feature. Personally, even if all we got out of this was better debugging information in assertions, I would be in favour. But over in Groovy land, the official docs are suggesting that it is often better to rely on power assertions rather than JUnit. http://docs.groovy-lang.org/next/html/documentation/core-testing-guide.html#... -- Steve
On Sat, Sep 25, 2021 at 00:56 Steven D'Aprano <steve@pearwood.info> wrote:
On Fri, Sep 24, 2021 at 11:23:00PM -0700, Guido van Rossum wrote:
Is there no room for making it easier to do this with less invasive changes to the stdlib, or are Steven d'A's "heroic measures in an import hook" the right way to go?
Other Steve
There’s room for that, but that’s not what’s being proposed (yet :-).
I'm confused. My reading of the pre-PEP is that that is precisely what it is proposing: changing the way assert works so that the value of each sub-expression is available to be displayed to the user. Presumably any framework or library would be able to access that information.
Have I missed something? I don't see anything in the proposal about creating new stdlib frameworks or changing unittest.
But others were. Anyway, I admit that I didn’t read the PRP carefully enough and was confused about what it proposes. My next responses: - The name “Power Assertions” is terrible. It sounds like a Microsoft product. :-) - I was envisioning something that provides an API that would allow a test framework to do what pytest does; not a behavior chance to the assert statement by default. - Please, please collaborate with the pytest developers on the design. —Guido -- --Guido (mobile)
On Sat, 25 Sept 2021 at 06:09, Stephen J. Turnbull <stephenjturnbull@gmail.com> wrote:
Guido van Rossum writes:
I think this is by far the best option. Pytest can evolve much faster than the stdlib.
Is there no room for making it easier to do this with less invasive changes to the stdlib, or are Steven d'A's "heroic measures in an import hook" the right way to go?
+1 on this. There are a number of "better exceptions" packages out there (for example, `better_exceptions` :-)) which would benefit from an improved mechanism to introspect failed assertions. A language change to make life easier for all those packages seems like an "obvious" feature to me.From there, it's a relatively small step to having a better *default* assertion reporting mechanism, but the PEP should focus on the machinery first, and the front end second IMO. Paul
My first thought was: it would be very nice, when doing quick and dirty scratch code, to not have to resort to a full fledged test framework to get readable assertions. You could just throw an assert statement in a if __name__ == '__main__' block at the bottom, click the run arrow next to it in pycharm or other IDE, and get more readable assertion errors. It doesn't seem like it is recreating pytest; pytest is far more than rewritten assertions (though it's a key feature of course). On the other hand, providing an API for hooking into the assert mechanism could be more useful. Imagine pytest spinning out their assertion rewriting functionality into a separate project that you could turn into a plugin in VS Code or Pycharm. And it would be much more easily extendable. +1 on the basic idea/desire for rewritten assertions appearing in more places than pytest. +1 on official support for hooking into the assertion machinery and letting a thousand flowers bloom. +0 on the actual proposed PEP. On Sat, Sep 25, 2021, 5:50 AM Paul Moore <p.f.moore@gmail.com> wrote:
On Sat, 25 Sept 2021 at 06:09, Stephen J. Turnbull <stephenjturnbull@gmail.com> wrote:
Guido van Rossum writes:
I think this is by far the best option. Pytest can evolve much faster
than
the stdlib.
Is there no room for making it easier to do this with less invasive changes to the stdlib, or are Steven d'A's "heroic measures in an import hook" the right way to go?
+1 on this. There are a number of "better exceptions" packages out there (for example, `better_exceptions` :-)) which would benefit from an improved mechanism to introspect failed assertions. A language change to make life easier for all those packages seems like an "obvious" feature to me.From there, it's a relatively small step to having a better *default* assertion reporting mechanism, but the PEP should focus on the machinery first, and the front end second IMO.
Paul _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/J3SQIJ... Code of Conduct: http://python.org/psf/codeofconduct/
So maybe what we should do now, rather than add one feature, is propose a new test framework/test runner for the stdlib, inspired by pytest.
I'd actually avoid reinventing a test framework as I think this has proven to be futile. I'm of the opinion that the language should provide a lean infra for useful testing and have libraries take it from there. I don't think this is exclusively testing related, as it can be useful as a debugging mechanism for things that shouldn't happen, but happen in production.
Alternatively, take the approach taken with distutils and setuptools— officially accept that a full featured test framework will be left to third parties.
I'm not sure I'm familiar with this. Can you please elaborate?
NOTE: if the proposal does require actual language changes beyond the current introspection options, that should be made clear.
Note taken. I'm not very familiar with the internals, so I hesitated to write this in the PEP, but from what I understand we can make a language change that will take care of the AST side of things so that 3rd party libraries can focus on the rendering. This way the 3rd party libraries don't have to act as executors. Does that make sense? On Sat, Sep 25, 2021, at 05:45, Christopher Barker wrote:
My first thought when reading this (and the recent discussion on Python ideas) was that pytest already does this.
Then I see in the PEP: "What pytest does is awesome“
So what?
Well, two points:
1) this should be part of a test framework, not something built in to exceptions. Pytest has shown that it can be done without any language changes.
2) there are a lot of things in Pytest that are very useful. And a lot of problems with unittest.
So maybe what we should do now, rather than add one feature, is propose a new test framework/test runner for the stdlib, inspired by pytest.
Alternatively, take the approach taken with distutils and setuptools— officially accept that a full featured test framework will be left to third parties.
NOTE: if the proposal does require actual language changes beyond the current introspection options, that should be made clear.
-CHB
On Fri, Sep 24, 2021 at 4:12 AM Paul Moore <p.f.moore@gmail.com> wrote:
On Fri, 24 Sept 2021 at 12:05, Noam Tenne <noam@10ne.org> wrote:
Caveats -------
It is important to note that expressions with side effects are affected by this feature. This is because in order to display this information, we must store references to the instances and not just the values.
One immediate thought. You should give examples of the sort of expressions that are affected, and precisely what the effect is. It's impossible to judge the importance of this point without details.
Paul _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/JJ2CT4... Code of Conduct: http://python.org/psf/codeofconduct/ -- Christopher Barker, PhD (Chris)
Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Will do! On Fri, Sep 24, 2021, at 14:10, Paul Moore wrote:
On Fri, 24 Sept 2021 at 12:05, Noam Tenne <noam@10ne.org> wrote:
Caveats -------
It is important to note that expressions with side effects are affected by this feature. This is because in order to display this information, we must store references to the instances and not just the values.
One immediate thought. You should give examples of the sort of expressions that are affected, and precisely what the effect is. It's impossible to judge the importance of this point without details.
Paul
This proposal sounds very promising. Some observations/questions follow. On Fri, Sep 24, 2021 at 02:04:21PM +0300, Noam Tenne wrote:
Assertion failed:
sc.val['d'] == f | | | | e False | {'d': 'e'}
Can we see what the output would look like in larger, more complex examples? Especially those that cross line boundaries? strict = False direction = 'left' sym = 'N' count = 2 clusters = [None] assert ((strict and direction is not None) or (sym == 'N' and count == 1 and len(clusters) == 1) ), "description ..." How about examples where one of the values is extremely large? sc.val = {'d': 'e'} sc.val.update(dict.fromkeys(range(100)) assert sc.val['d'] == 'f' If the handler is a hook in sys, like displayhook and except hook, then people can install their own handlers. But I think the default handler (asserthook?) should do the right thing for ugly assertions.
In the python-ideas mailing list, Serhiy Storchaka suggests:
It needs a support in the compiler. The condition expression should be compiled to keep all immediate results of subexpressions on the stack. If the final result is true, immediate results are dropped. If it is false, the second argument of assert is evaluated and its value together with all immediate results of the first expression, together with references to corresponding subexpressions (as strings, ranges or AST nodes) are passed to the special handler. That handler can be implemented in a third-party library, because formatting and outputting a report is a complex task. The default handler can just raise an AttributeError.
Is that supposed to be AssertionError?
It is important to note that expressions with side effects are affected by this feature. This is because in order to display this information, we must store references to the instances and not just the values.
Don't you mean that expressions with side-effects are *not* affected? Or have I misunderstood? You don't have to re-evaluate the expression if the assertion fails, so any side-effects only occur once. greeting = "Hello world" assert print(greeting) Hello world Assertion failed: print(greeting) | | | 'Hello world' | None Rather than the situation where the expression has to be evaluated twice (as older versions of Pytest used to do): greeting = "Hello world" assert print(greeting) Hello world Assertion failed: print(greeting) Hello world | | | 'Hello world' | None -- Steve
So should I just scratch this and rewrite a PEP for an extensible assertion mechanism? On Fri, Sep 24, 2021, at 14:04, Noam Tenne wrote:
Hi All,
Following the discussions on "Power Assertions: Is it PEP-able?", I've drafted this PEP. Your comments are most welcome.
PEP: 9999 Title: Power Assertion Author: Noam Tenne <noam@10ne.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 24-Sep-2021
Abstract ========
This PEP introduces a language enhancement named "Power Assertion".
The Power Assertion is inspired by a similar feature found in the `Groovy language`_ and is an extension to the core lib's ``assert`` keyword.
When an assertion expression evaluates to ``False``, the output shows not only the failure, but also a breakdown of the evaluated expression from the inner part to the outer part.
.. _Groovy language: http://docs.groovy-lang.org/next/html/documentation/core-testing-guide.html#...
Motivation =========
Every test boils down to the binary statement "Is this true or false?", whether you use the built-in assert keyword or a more advanced assertion method provided by a testing framework. When an assertion fails, the output is binary too — "Expected x, but got y".
There are helpful libraries like Hamcrest which give you a more verbose breakdown of the difference and answer the question "What exactly is the difference between x and y?". This is extremely helpful, but it still focuses on the difference between the values.
Keep in mind that a given state is normally an outcome of a series of states, that is, one outcome is a result of multiple conditions and causes. This is where Power Assertion comes in. It allows us to better understand what led to the failure.
The Community Wants This ------------------------
As mentioned in the abstract, this feature was borrowed from Groovy. It is a very popular feature within the Groovy community, also used by projects such as `Spock`_.
On top of that, it is very much needed in the Python community as well:
* `Power Assertion was explicitly requested`_ as a feature in the `Nimoy`_ testing framework * There's a `similar feature in pytest`_
And when `discussed in the python-ideas`_ mailing list, the responses were overall positive:
* "This is cool." - Guido van Rossum * "I was actually thinking exactly the opposite: this would more useful in production than in testing." - 2QdxY4RzWzUUiLuE＠potatochowder.com * "What pytest does is awesome. I though about implementing it in the standard compiler since seen it the first time." - Serhiy Storchaka
.. _Spock: https://spockframework.org/ .. _Power Assertion was explicitly requested: https://stackoverflow.com/a/58536986/198825 .. _similar feature in pytest: https://docs.pytest.org/en/latest/how-to/assert.html .. _discussed in the python-ideas: https://mail.python.org/archives/list/python-ideas@python.org/thread/T26DR4B...
Rational ========
Code Example ------------
::
class SomeClass: def __init__(self): self.val = {'d': 'e'}
def __str__(self): return str(self.val)
sc = SomeClass()
assert sc.val['d'] == 'f'
This routine will result in the output:
::
Assertion failed:
sc.val['d'] == f | | | | e False | {'d': 'e'}
Display -------
In the output above we can see the value of every part of the expression from left to right, mapped to their expression fragment with the pipe (``|``). The number of rows that are printed depend on the value of each fragment of the expression. If the value of a fragment is longer than the actual fragment (``{'d': 'e'}`` is longer than ``sc``), then the next value (``e``) will be printed on a new line which will appear above. Values are appended to the same line until it overflows in length to horizontal position of the next fragment.
This way of presentation is clearer and more human friendly than the output offered by pytest's solution.
The information that’s displayed is dictated by the type. If the type is a constant value, it will be displayed as is. If the type implements `__str__`, then the return value of that will be displayed.
Mechanics ---------
Reference Implementation ''''''''''''''''''''''''
The reference implementation uses AST manipulation because this is the only way that this level of involvement can be achieved by a third party library.
It iterates over every subexpression in the assert statement. Subexpressions are parts of the expression separated by a lookup (``map[key]``), an attribute reference (``key.value``) or a binary comparison operator (``==``).
It then builds an AST in the structure of a tree to maintain the order of the operations in the original code, and tracks the original code of the subexpression together with the AST code of the subexpression and the original indices.
It then rewrites the AST of the original expression to call a specialised assertion function, which accepts the tree as a parameter.
At runtime the expression is executed. If it fails, a rendering function is called to render the assertion message as per the example above.
Actual Implementation '''''''''''''''''''''
To be discussed.
In the python-ideas mailing list, Serhiy Storchaka suggests:
It needs a support in the compiler. The condition expression should be compiled to keep all immediate results of subexpressions on the stack. If the final result is true, immediate results are dropped. If it is false, the second argument of assert is evaluated and its value together with all immediate results of the first expression, together with references to corresponding subexpressions (as strings, ranges or AST nodes) are passed to the special handler. That handler can be implemented in a third-party library, because formatting and outputting a report is a complex task. The default handler can just raise an AttributeError.
Caveats -------
It is important to note that expressions with side effects are affected by this feature. This is because in order to display this information, we must store references to the instances and not just the values.
Reference Implementation ========================
There's a `complete implementation`_ of this enhancement in the `Nimoy`_ testing framework.
It uses AST manipulation to remap the expression to a `data structure`_ at compile time, so that it can then be `evaluated and printed`_ at runtime.
.. _Nimoy: https://browncoat-ninjas.github.io/nimoy/ .. _complete implementation: https://browncoat-ninjas.github.io/nimoy/examples/#power-assertions-beta .. _data structure: https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/ast_tools/expre... .. _evaluated and printed: https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/assertions/powe...
Copyright =========
This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
*Attachments:* * pep-9999.rst
First you need to fond a core dev to sponsor you (Steven D’A?). That person will guide you through the process. On Sat, Sep 25, 2021 at 08:30 Noam Tenne <noam@10ne.org> wrote:
So should I just scratch this and rewrite a PEP for an extensible assertion mechanism?
On Fri, Sep 24, 2021, at 14:04, Noam Tenne wrote:
Hi All,
Following the discussions on "Power Assertions: Is it PEP-able?", I've drafted this PEP. Your comments are most welcome.
PEP: 9999 Title: Power Assertion Author: Noam Tenne <noam@10ne.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 24-Sep-2021
Abstract ========
This PEP introduces a language enhancement named "Power Assertion".
The Power Assertion is inspired by a similar feature found in the `Groovy language`_ and is an extension to the core lib's ``assert`` keyword.
When an assertion expression evaluates to ``False``, the output shows not only the failure, but also a breakdown of the evaluated expression from the inner part to the outer part.
.. _Groovy language: http://docs.groovy-lang.org/next/html/documentation/core-testing-guide.html#...
Motivation =========
Every test boils down to the binary statement "Is this true or false?", whether you use the built-in assert keyword or a more advanced assertion method provided by a testing framework. When an assertion fails, the output is binary too — "Expected x, but got y".
There are helpful libraries like Hamcrest which give you a more verbose breakdown of the difference and answer the question "What exactly is the difference between x and y?". This is extremely helpful, but it still focuses on the difference between the values.
Keep in mind that a given state is normally an outcome of a series of states, that is, one outcome is a result of multiple conditions and causes. This is where Power Assertion comes in. It allows us to better understand what led to the failure.
The Community Wants This ------------------------
As mentioned in the abstract, this feature was borrowed from Groovy. It is a very popular feature within the Groovy community, also used by projects such as `Spock`_.
On top of that, it is very much needed in the Python community as well:
* `Power Assertion was explicitly requested`_ as a feature in the `Nimoy`_ testing framework * There's a `similar feature in pytest`_
And when `discussed in the python-ideas`_ mailing list, the responses were overall positive:
* "This is cool." - Guido van Rossum * "I was actually thinking exactly the opposite: this would more useful in production than in testing." - 2QdxY4RzWzUUiLuE＠potatochowder.com * "What pytest does is awesome. I though about implementing it in the standard compiler since seen it the first time." - Serhiy Storchaka
.. _Spock: https://spockframework.org/ .. _Power Assertion was explicitly requested: https://stackoverflow.com/a/58536986/198825 .. _similar feature in pytest: https://docs.pytest.org/en/latest/how-to/assert.html .. _discussed in the python-ideas: https://mail.python.org/archives/list/python-ideas@python.org/thread/T26DR4B...
Rational ========
Code Example ------------
::
class SomeClass: def __init__(self): self.val = {'d': 'e'}
def __str__(self): return str(self.val)
sc = SomeClass()
assert sc.val['d'] == 'f'
This routine will result in the output:
::
Assertion failed:
sc.val['d'] == f | | | | e False | {'d': 'e'}
Display -------
In the output above we can see the value of every part of the expression from left to right, mapped to their expression fragment with the pipe (``|``). The number of rows that are printed depend on the value of each fragment of the expression. If the value of a fragment is longer than the actual fragment (``{'d': 'e'}`` is longer than ``sc``), then the next value (``e``) will be printed on a new line which will appear above. Values are appended to the same line until it overflows in length to horizontal position of the next fragment.
This way of presentation is clearer and more human friendly than the output offered by pytest's solution.
The information that’s displayed is dictated by the type. If the type is a constant value, it will be displayed as is. If the type implements `__str__`, then the return value of that will be displayed.
Mechanics ---------
Reference Implementation ''''''''''''''''''''''''
The reference implementation uses AST manipulation because this is the only way that this level of involvement can be achieved by a third party library.
It iterates over every subexpression in the assert statement. Subexpressions are parts of the expression separated by a lookup (``map[key]``), an attribute reference (``key.value``) or a binary comparison operator (``==``).
It then builds an AST in the structure of a tree to maintain the order of the operations in the original code, and tracks the original code of the subexpression together with the AST code of the subexpression and the original indices.
It then rewrites the AST of the original expression to call a specialised assertion function, which accepts the tree as a parameter.
At runtime the expression is executed. If it fails, a rendering function is called to render the assertion message as per the example above.
Actual Implementation '''''''''''''''''''''
To be discussed.
In the python-ideas mailing list, Serhiy Storchaka suggests:
It needs a support in the compiler. The condition expression should be compiled to keep all immediate results of subexpressions on the stack. If the final result is true, immediate results are dropped. If it is false, the second argument of assert is evaluated and its value together with all immediate results of the first expression, together with references to corresponding subexpressions (as strings, ranges or AST nodes) are passed to the special handler. That handler can be implemented in a third-party library, because formatting and outputting a report is a complex task. The default handler can just raise an AttributeError.
Caveats -------
It is important to note that expressions with side effects are affected by this feature. This is because in order to display this information, we must store references to the instances and not just the values.
Reference Implementation ========================
There's a `complete implementation`_ of this enhancement in the `Nimoy`_ testing framework.
It uses AST manipulation to remap the expression to a `data structure`_ at compile time, so that it can then be `evaluated and printed`_ at runtime.
.. _Nimoy: https://browncoat-ninjas.github.io/nimoy/ .. _complete implementation: https://browncoat-ninjas.github.io/nimoy/examples/#power-assertions-beta .. _data structure: https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/ast_tools/expre... .. _evaluated and printed: https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/assertions/powe...
Copyright =========
This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
*Attachments:*
- pep-9999.rst
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/DO3ETJ... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido (mobile)
Thank you On Sat, Sep 25, 2021, at 18:53, Guido van Rossum wrote:
First you need to fond a core dev to sponsor you (Steven D’A?). That person will guide you through the process.
On Sat, Sep 25, 2021 at 08:30 Noam Tenne <noam@10ne.org> wrote:
__ So should I just scratch this and rewrite a PEP for an extensible assertion mechanism?
On Fri, Sep 24, 2021, at 14:04, Noam Tenne wrote:
Hi All,
Following the discussions on "Power Assertions: Is it PEP-able?", I've drafted this PEP. Your comments are most welcome.
PEP: 9999 Title: Power Assertion Author: Noam Tenne <noam@10ne.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 24-Sep-2021
Abstract ========
This PEP introduces a language enhancement named "Power Assertion".
The Power Assertion is inspired by a similar feature found in the `Groovy language`_ and is an extension to the core lib's ``assert`` keyword.
When an assertion expression evaluates to ``False``, the output shows not only the failure, but also a breakdown of the evaluated expression from the inner part to the outer part.
.. _Groovy language: http://docs.groovy-lang.org/next/html/documentation/core-testing-guide.html#...
Motivation =========
Every test boils down to the binary statement "Is this true or false?", whether you use the built-in assert keyword or a more advanced assertion method provided by a testing framework. When an assertion fails, the output is binary too — "Expected x, but got y".
There are helpful libraries like Hamcrest which give you a more verbose breakdown of the difference and answer the question "What exactly is the difference between x and y?". This is extremely helpful, but it still focuses on the difference between the values.
Keep in mind that a given state is normally an outcome of a series of states, that is, one outcome is a result of multiple conditions and causes. This is where Power Assertion comes in. It allows us to better understand what led to the failure.
The Community Wants This ------------------------
As mentioned in the abstract, this feature was borrowed from Groovy. It is a very popular feature within the Groovy community, also used by projects such as `Spock`_.
On top of that, it is very much needed in the Python community as well:
* `Power Assertion was explicitly requested`_ as a feature in the `Nimoy`_ testing framework * There's a `similar feature in pytest`_
And when `discussed in the python-ideas`_ mailing list, the responses were overall positive:
* "This is cool." - Guido van Rossum * "I was actually thinking exactly the opposite: this would more useful in production than in testing." - 2QdxY4RzWzUUiLuE＠potatochowder.com * "What pytest does is awesome. I though about implementing it in the standard compiler since seen it the first time." - Serhiy Storchaka
.. _Spock: https://spockframework.org/ .. _Power Assertion was explicitly requested: https://stackoverflow.com/a/58536986/198825 .. _similar feature in pytest: https://docs.pytest.org/en/latest/how-to/assert.html .. _discussed in the python-ideas: https://mail.python.org/archives/list/python-ideas@python.org/thread/T26DR4B...
Rational ========
Code Example ------------
::
class SomeClass: def __init__(self): self.val = {'d': 'e'}
def __str__(self): return str(self.val)
sc = SomeClass()
assert sc.val['d'] == 'f'
This routine will result in the output:
::
Assertion failed:
sc.val['d'] == f | | | | e False | {'d': 'e'}
Display -------
In the output above we can see the value of every part of the expression from left to right, mapped to their expression fragment with the pipe (``|``). The number of rows that are printed depend on the value of each fragment of the expression. If the value of a fragment is longer than the actual fragment (``{'d': 'e'}`` is longer than ``sc``), then the next value (``e``) will be printed on a new line which will appear above. Values are appended to the same line until it overflows in length to horizontal position of the next fragment.
This way of presentation is clearer and more human friendly than the output offered by pytest's solution.
The information that’s displayed is dictated by the type. If the type is a constant value, it will be displayed as is. If the type implements `__str__`, then the return value of that will be displayed.
Mechanics ---------
Reference Implementation ''''''''''''''''''''''''
The reference implementation uses AST manipulation because this is the only way that this level of involvement can be achieved by a third party library.
It iterates over every subexpression in the assert statement. Subexpressions are parts of the expression separated by a lookup (``map[key]``), an attribute reference (``key.value``) or a binary comparison operator (``==``).
It then builds an AST in the structure of a tree to maintain the order of the operations in the original code, and tracks the original code of the subexpression together with the AST code of the subexpression and the original indices.
It then rewrites the AST of the original expression to call a specialised assertion function, which accepts the tree as a parameter.
At runtime the expression is executed. If it fails, a rendering function is called to render the assertion message as per the example above.
Actual Implementation '''''''''''''''''''''
To be discussed.
In the python-ideas mailing list, Serhiy Storchaka suggests:
It needs a support in the compiler. The condition expression should be compiled to keep all immediate results of subexpressions on the stack. If the final result is true, immediate results are dropped. If it is false, the second argument of assert is evaluated and its value together with all immediate results of the first expression, together with references to corresponding subexpressions (as strings, ranges or AST nodes) are passed to the special handler. That handler can be implemented in a third-party library, because formatting and outputting a report is a complex task. The default handler can just raise an AttributeError.
Caveats -------
It is important to note that expressions with side effects are affected by this feature. This is because in order to display this information, we must store references to the instances and not just the values.
Reference Implementation ========================
There's a `complete implementation`_ of this enhancement in the `Nimoy`_ testing framework.
It uses AST manipulation to remap the expression to a `data structure`_ at compile time, so that it can then be `evaluated and printed`_ at runtime.
.. _Nimoy: https://browncoat-ninjas.github.io/nimoy/ .. _complete implementation: https://browncoat-ninjas.github.io/nimoy/examples/#power-assertions-beta .. _data structure: https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/ast_tools/expre... .. _evaluated and printed: https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/assertions/powe...
Copyright =========
This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
*Attachments:* * pep-9999.rst
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/DO3ETJ... Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido (mobile)
participants (8)
-
Christopher Barker -
Guido van Rossum -
Noam Tenne -
Paul Moore -
Ricky Teachey -
Serhiy Storchaka -
Stephen J. Turnbull -
Steven D'Aprano