hookable assertions - to support testing utilities and debugging
Hi, the idea is to have the assert statement call a global method, just like import does one could then do something like with exception.collect_assertions: assert myobj.foo is bar assert myobj.baz == 123 on order to do groups of assertions having more debugging information on failed assertions it could also enable more interesting context managers like with testtool.fail_lazy: ... # this would fail the test but not stop its exeution in the middle with testtools.informative_assert: ... # this will get logged but the test run wont be considered failed
On Thu, Feb 19, 2015 at 08:25:34AM +0100, Ronny Pfannschmidt wrote:
Hi,
the idea is to have the assert statement call a global method, just like import does
one could then do something like
[...] I believe that overloading assert in this fashion is an abuse of assert. If we want custom "raise if not this condition" calls, I think we should copy unittest and define functions that do what we want, not abuse the assert statement. If you want to (mis)use assert for unit and regression testing, you should see the nose third party module and see if that does the sort of thing you want. -- Steve
On 19 February 2015 at 21:05, Steven D'Aprano <steve@pearwood.info> wrote:
On Thu, Feb 19, 2015 at 08:25:34AM +0100, Ronny Pfannschmidt wrote:
Hi,
the idea is to have the assert statement call a global method, just like import does
one could then do something like
[...]
I believe that overloading assert in this fashion is an abuse of assert. If we want custom "raise if not this condition" calls, I think we should copy unittest and define functions that do what we want, not abuse the assert statement.
If you want to (mis)use assert for unit and regression testing, you should see the nose third party module and see if that does the sort of thing you want.
py.test is the one to look at for this: http://pytest.org/latest/assert.html#assert-with-the-assert-statement It (optionally) rewrites the AST for assert statements to make them suitable for testing purposes rather than being optional runtime integrity checks. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Im aware, my intent is to standardize the mechanism and push it into the language, I'm already involved in the py.test development. On Thursday, February 19, 2015 12:12:23 PM CEST, Nick Coghlan wrote:
On 19 February 2015 at 21:05, Steven D'Aprano <steve@pearwood.info> wrote:
On Thu, Feb 19, 2015 at 08:25:34AM +0100, Ronny Pfannschmidt wrote: ...
py.test is the one to look at for this: http://pytest.org/latest/assert.html#assert-with-the-assert-statement
It (optionally) rewrites the AST for assert statements to make them suitable for testing purposes rather than being optional runtime integrity checks.
Cheers, Nick.
On 19 February 2015 at 21:16, Ronny Pfannschmidt <opensource@ronnypfannschmidt.de> wrote:
Im aware, my intent is to standardize the mechanism and push it into the language,
Unfortunately, you're faced with a lot of C programmers firmly of the opinion that if you care about whether or not a piece of code gets executed, you don't put it it in an assert statement. It's a giant flashing red sign that says "This code is just checking something that should always be true, you don't actually need to read it in order to understand what this code is doing, but the assumption it documents may be helpful if something is puzzling you". So from my perspective (being one of the aforementioned C programmers), I see this question as not so much about the assert statement per se, but rather a more philosophical question: * is testing code categorically different from production code? * if yes, should assert statements be handled differently in testing code? * if yes, how should that difference be communicated to readers and to the interpreter? (FWIW, I'm personally inclined to answer the first two questions with "yes", but haven't got any ideas as far as the third one goes) Making the behaviour of assert statements configurable attempts to duck those questions instead of answering them, so I don't see at as a particularly useful approach - it makes the language more complex without providing additional clarity, rather than figuring out how to encode a genuinely important categorical distinction between test code and production code into the design of the language. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Feb 19, 2015, at 6:32 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 19 February 2015 at 21:16, Ronny Pfannschmidt <opensource@ronnypfannschmidt.de> wrote:
Im aware, my intent is to standardize the mechanism and push it into the language,
Unfortunately, you're faced with a lot of C programmers firmly of the opinion that if you care about whether or not a piece of code gets executed, you don't put it it in an assert statement. It's a giant flashing red sign that says "This code is just checking something that should always be true, you don't actually need to read it in order to understand what this code is doing, but the assumption it documents may be helpful if something is puzzling you".
So from my perspective (being one of the aforementioned C programmers), I see this question as not so much about the assert statement per se, but rather a more philosophical question:
* is testing code categorically different from production code? * if yes, should assert statements be handled differently in testing code? * if yes, how should that difference be communicated to readers and to the interpreter?
(FWIW, I'm personally inclined to answer the first two questions with "yes", but haven't got any ideas as far as the third one goes)
Making the behaviour of assert statements configurable attempts to duck those questions instead of answering them, so I don't see at as a particularly useful approach - it makes the language more complex without providing additional clarity, rather than figuring out how to encode a genuinely important categorical distinction between test code and production code into the design of the language.
Speaking as a not C programmer, I think the answers to the first two are absolutely yes, and the third one I dunno, maybe some sort of top level import like from __testing__ import real_assert or something. I absolutely hate the unittest style of assertFoo, it’s a whole bunch of methods that I can never remember the names of and I have to go look them up. I think using the assert statement is completely logical for test code and I believe you can see this by the fact all of the methods to do checks in unittest are named “assert something”. I think it introduces extra cognitive burden for new users to have to use a whole bunch of additional methods to do assertions in tests TBH. The way py.test works composes nicely with the same kinds of expressions that you’re going to need to learn to write any meaningful Python, but the assertFoo methods are special cases that don’t. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 02/19/2015 01:55 PM, Donald Stufft wrote:
So from my perspective (being one of the aforementioned C programmers), I see this question as not so much about the assert statement per se, but rather a more philosophical question:
* is testing code categorically different from production code? * if yes, should assert statements be handled differently in testing code? * if yes, how should that difference be communicated to readers and to the interpreter?
(FWIW, I'm personally inclined to answer the first two questions with "yes", but haven't got any ideas as far as the third one goes)
Making the behaviour of assert statements configurable attempts to duck those questions instead of answering them, so I don't see at as a particularly useful approach - it makes the language more complex without providing additional clarity, rather than figuring out how to encode a genuinely important categorical distinction between test code and production code into the design of the language.
Speaking as a not C programmer, I think the answers to the first two are absolutely yes, and the third one I dunno, maybe some sort of top level import like from __testing__ import real_assert or something.
I absolutely hate the unittest style of assertFoo, it’s a whole bunch of methods that I can never remember the names of and I have to go look them up. I think using the assert statement is completely logical for test code and I believe you can see this by the fact all of the methods to do checks in unittest are named “assert something”. I think it introduces extra cognitive burden for new users to have to use a whole bunch of additional methods to do assertions in tests TBH. The way py.test works composes nicely with the same kinds of expressions that you’re going to need to learn to write any meaningful Python, but the assertFoo methods are special cases that don’t.
+1 Georg
On Feb 19, 2015, at 09:32 PM, Nick Coghlan wrote:
So from my perspective (being one of the aforementioned C programmers), I see this question as not so much about the assert statement per se, but rather a more philosophical question:
* is testing code categorically different from production code? * if yes, should assert statements be handled differently in testing code? * if yes, how should that difference be communicated to readers and to the interpreter?
(FWIW, I'm personally inclined to answer the first two questions with "yes", but haven't got any ideas as far as the third one goes)
I'm of the same mind. asserts in the code are almost like documentation. When I write them I'm usually thinking "this can't possibly happen, but maybe I forgot something". That's very different from an assertion in test code which says "let's make sure the outcome of this contrived call is what I expect it to be". So to me, using the unittest.assert* methods makes that difference obvious, though I'll admit that sometimes I'm in the same boat of having to dip into the documentation to remember if it's assertMultilineEqual() or assertMultiLineEqual(). I find most of the assert* methods fairly easy to remember. Cheers, -Barry
On Thu, Feb 19, 2015 at 5:32 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
* is testing code categorically different from production code? * if yes, should assert statements be handled differently in testing code?
assert statements are compiled out with -O (e.g. PYTHON_OPTIMIZE=2) * https://docs.python.org/3/using/cmdline.html#cmdoption-OO * self.assert statements are not compiled out * nose decorated test cases are not compiled out
* if yes, how should that difference be communicated to readers and to the interpreter?
It's such a frequent pattern. ([declarative] "literate programming" (with objects)) * Is there a better tool than tests for teaching? * https://github.com/taavi/ipython_nose adds an IPython magic command (%nose) that collects test functions and test output w/ IPYNB JSON (-> HTML, PDF, XML) output (e.g. for use with https://pypi.python.org/pypi/runipy) * Justifying multiple failures and assertions per test: "Delayed assert / multiple failures per test" http://pythontesting.net/strategy/delayed-assert/ * (collecting / emitting / yielding (s, p, o) tuples from nested dicts with inferred #URI/paths) * The JSON-LD test suite uses EARL, an RDF schema (instead of xUnit XML) * http://json-ld.org/test-suite/ * http://json-ld.org/test-suite/reports/ * http://www.w3.org/TR/EARL10-Schema/ "Evaluation and Report Language (EARL) 1.0 Schema" (2011)
On Thu, 19 Feb 2015 08:25:34 +0100 Ronny Pfannschmidt <opensource@ronnypfannschmidt.de> wrote:
Hi,
the idea is to have the assert statement call a global method, just like import does
How would that work? Does it give the AST to the global method? Something else? Regards Antoine.
the idea is to call the method with the assertionerror instance so instead of directly doing a raise theerror, python would call __raise_assertion__(theerror) this composes nicely with the proposal for introspectable assertions On Thursday, February 19, 2015 1:54:06 PM CEST, Antoine Pitrou wrote:
On Thu, 19 Feb 2015 08:25:34 +0100 Ronny Pfannschmidt <opensource@ronnypfannschmidt.de> wrote:
Hi,
the idea is to have the assert statement call a global method, just like import does
How would that work? Does it give the AST to the global method? Something else?
Regards
Antoine.
participants (8)
-
Antoine Pitrou -
Barry Warsaw -
Donald Stufft -
Georg Brandl -
Nick Coghlan -
Ronny Pfannschmidt -
Steven D'Aprano -
Wes Turner