the role of assert in the standard library ?
Hello
I removed some assert calls in distutils some time ago because the package was not behaving correctly when people were using Python with the --optimize flag. In other words, assert became a full part of the code logic and removing them via -O was changing the behavior.
In my opinion assert should be avoided completely anywhere else than in the tests. If this is a wrong statement, please let me know why :)
So, I grepped the stdlib for assert calls, and I have found 177 of them and many of them are making Python acts differently depending on the -O flag,
Here's an example on a randomly picked assert in the threading module:
>>>>>>>>>>>>>>>>>>
import threading
class test(threading.Thread): def __init__(self): self.bla = 1
def run(self): print('running')
t = test() print(t) <<<<<<<<<<<<<<<<<<<<<<
The __repr__ method is not behaving the same way depending on the O flag:
$ python3 -O test.py Traceback (most recent call last): File "test.py", line 12, in <module> print(t) File "/usr/local/lib/python3.2/threading.py", line 652, in __repr__ if self._started.is_set(): AttributeError: 'test' object has no attribute '_started'
$ python3 test.py Traceback (most recent call last): File "test.py", line 12, in <module> print(t) File "/usr/local/lib/python3.2/threading.py", line 650, in __repr__ assert self._initialized, "Thread.__init__() was not called" AttributeError: 'test' object has no attribute '_initialized'
$ python test.py Traceback (most recent call last): File "test.py", line 12, in <module> print(t) File "/usr/lib/python2.6/threading.py", line 451, in __repr__ assert self.__initialized, "Thread.__init__() was not called" AssertionError: Thread.__init__() was not called <--- oops different error
$ python -O test.py Traceback (most recent call last): File "test.py", line 12, in <module> print(t) File "/usr/lib/python2.6/threading.py", line 453, in __repr__ if self.__started.is_set(): AttributeError: 'test' object has no attribute '_Thread__started'
I have seen some other places where thing would simply break with -O.
Am I right thinking we should do a pass on those and remove them or turn them into exception that are triggered with -O as well ?
This flag is meant to "optimize generated bytecode slightly", but I am not sure this involves also slightly changing the way the code behaves
Cheers Tarek
On 4/28/2011 3:54 AM, Tarek Ziadé wrote:
Hello
I removed some assert calls in distutils some time ago because the package was not behaving correctly when people were using Python with the --optimize flag. In other words, assert became a full part of the code logic and removing them via -O was changing the behavior.
In my opinion assert should be avoided completely anywhere else than in the tests. If this is a wrong statement, please let me know why :)
My understanding is that assert can be used in production code but only to catch logic errors by testing supposed invariants or postconditions. It should not be used to test usage errors, including preconditions. In other words, assert presence or absence should not affect behavior unless the code has a bug.
So, I grepped the stdlib for assert calls, and I have found 177 of them and many of them are making Python acts differently depending on the -O flag,
Here's an example on a randomly picked assert in the threading module:
This, to me is wrong:
def __init__(self, group=None, target=None, name=None, args=(), kwargs=None, verbose=None): assert group is None, "group argument must be None for now"
That catches a usage error and should raise a ValueError.
This
def _wait(self, timeout): if not self._cond.wait_for(lambda : self._state != 0, timeout): #timed out. Break the barrier self._break() raise BrokenBarrierError if self._state < 0: raise BrokenBarrierError assert self._state == 1
appears to be, or should be, a test of a postcondition that should *always* be true regardless of usage.
On 28/04/2011 09:34, Terry Reedy wrote:
On 4/28/2011 3:54 AM, Tarek Ziadé wrote:
Hello
I removed some assert calls in distutils some time ago because the package was not behaving correctly when people were using Python with the --optimize flag. In other words, assert became a full part of the code logic and removing them via -O was changing the behavior.
In my opinion assert should be avoided completely anywhere else than in the tests. If this is a wrong statement, please let me know why :)
My understanding is that assert can be used in production code but only to catch logic errors by testing supposed invariants or postconditions. It should not be used to test usage errors, including preconditions. In other words, assert presence or absence should not affect behavior unless the code has a bug.
Agreed. We should ideally have buildbots doing test runs with -O and -OO. R. David Murray did a lot of work a year ago (or so) to ensure the test run passes with -OO but it easily degrades..
There are a couple of asserts in unittest (for test discovery) but I only use them to provide failure messages early. The functionality is unchanged (and tests still pass) with -OO.
All the best,
Michael Foord
So, I grepped the stdlib for assert calls, and I have found 177 of them and many of them are making Python acts differently depending on the -O flag,
Here's an example on a randomly picked assert in the threading module:
This, to me is wrong:
def __init__(self, group=None, target=None, name=None, args=(), kwargs=None, verbose=None): assert group is None, "group argument must be None for now"
That catches a usage error and should raise a ValueError.
This
def _wait(self, timeout): if not self._cond.wait_for(lambda : self._state != 0, timeout): #timed out. Break the barrier self._break() raise BrokenBarrierError if self._state < 0: raise BrokenBarrierError assert self._state == 1appears to be, or should be, a test of a postcondition that should *always* be true regardless of usage.
On Thu, Apr 28, 2011 at 12:27 PM, Michael Foord fuzzyman@voidspace.org.uk wrote:
On 28/04/2011 09:34, Terry Reedy wrote:
On 4/28/2011 3:54 AM, Tarek Ziadé wrote:
Hello
I removed some assert calls in distutils some time ago because the package was not behaving correctly when people were using Python with the --optimize flag. In other words, assert became a full part of the code logic and removing them via -O was changing the behavior.
In my opinion assert should be avoided completely anywhere else than in the tests. If this is a wrong statement, please let me know why :)
My understanding is that assert can be used in production code but only to catch logic errors by testing supposed invariants or postconditions. It should not be used to test usage errors, including preconditions. In other words, assert presence or absence should not affect behavior unless the code has a bug.
Agreed. We should ideally have buildbots doing test runs with -O and -OO. R. David Murray did a lot of work a year ago (or so) to ensure the test run passes with -OO but it easily degrades..
There are a couple of asserts in unittest (for test discovery) but I only use them to provide failure messages early. The functionality is unchanged (and tests still pass) with -OO.
All the best,
I'll try to add a useful report on "bad asserts" in the bug tracker.
I am replying again to this on Python-ideas because I want to debate on assert :)
Cheers Tarek
On Thu, 28 Apr 2011 09:54:23 +0200 Tarek Ziadé ziade.tarek@gmail.com wrote:
I have seen some other places where thing would simply break with -O.
Am I right thinking we should do a pass on those and remove them or turn them into exception that are triggered with -O as well ?
Agreed. Argument checking should not depend on the -O flag.
Regards
Antoine.
On Apr 28, 2011, at 04:34 AM, Terry Reedy wrote:
On 4/28/2011 3:54 AM, Tarek Ziadé wrote:
Hello
I removed some assert calls in distutils some time ago because the package was not behaving correctly when people were using Python with the --optimize flag. In other words, assert became a full part of the code logic and removing them via -O was changing the behavior.
In my opinion assert should be avoided completely anywhere else than in the tests. If this is a wrong statement, please let me know why :)
My understanding is that assert can be used in production code but only to catch logic errors by testing supposed invariants or postconditions. It should not be used to test usage errors, including preconditions. In other words, assert presence or absence should not affect behavior unless the code has a bug.
I would agree. Use asserts for "this can't possibly happen <wink>" conditions.
-Barry
On Thu, Apr 28, 2011 at 10:37 AM, Barry Warsaw barry@python.org wrote:
I would agree. Use asserts for "this can't possibly happen <wink>" conditions.
Maybe we should rename "assert" to "wink", just to be clear on the usage. :-)
-Fred
Barry> I would agree. Use asserts for "this can't possibly happen Barry> <wink>" conditions.
Without looking, I suspect that's probably what the author thought he was doing.
Skip
On Apr 28, 2011, at 10:22 AM, skip@pobox.com wrote:
Barry> I would agree. Use asserts for "this can't possibly happen Barry> <wink>" conditions.
Without looking, I suspect that's probably what the author thought he was doing.
BTW, I think it always helps to have a really good assert message, and/or a leading comment to explain *why* that condition can't possibly happen.
-Barry
On Apr 28, 2011, at 11:04 AM, Fred Drake wrote:
On Thu, Apr 28, 2011 at 10:37 AM, Barry Warsaw barry@python.org wrote:
I would agree. Use asserts for "this can't possibly happen <wink>" conditions.
Maybe we should rename "assert" to "wink", just to be clear on the usage. :-)
Off to python-ideas for you! <wink>
-Barry
On Thu, Apr 28, 2011 at 5:26 PM, Barry Warsaw barry@python.org wrote:
On Apr 28, 2011, at 10:22 AM, skip@pobox.com wrote:
Barry> I would agree. Use asserts for "this can't possibly happen Barry> <wink>" conditions.
Without looking, I suspect that's probably what the author thought he was doing.
BTW, I think it always helps to have a really good assert message, and/or a leading comment to explain *why* that condition can't possibly happen.
why bother, it can't happen ;)
-Barry
Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com
On Thu, Apr 28, 2011 at 12:54 AM, Tarek Ziadé ziade.tarek@gmail.com wrote:
In my opinion assert should be avoided completely anywhere else than in the tests. If this is a wrong statement, please let me know why :)
I would turn that around. The assert statement should not be used in unit tests; unit tests should use self.assertXyzzy() always. In regular code, assert should be about detecting buggy code. It should not be used to test for error conditions in input data. (Both these can be summarized as "if you still want the test to happen with -O, don't use assert.)
On Apr 28, 2011, at 12:59 PM, Guido van Rossum wrote:
On Thu, Apr 28, 2011 at 12:54 AM, Tarek Ziadé ziade.tarek@gmail.com wrote:
In my opinion assert should be avoided completely anywhere else than in the tests. If this is a wrong statement, please let me know why :)
I would turn that around. The assert statement should not be used in unit tests; unit tests should use self.assertXyzzy() always. In regular code, assert should be about detecting buggy code. It should not be used to test for error conditions in input data. (Both these can be summarized as "if you still want the test to happen with -O, don't use assert.)
You're both right! :) My take on "assert" is "don't use it, ever".
assert is supposed to be about conditions that never happen. So there are a few cases where I might use it:
If I use it to enforce a precondition, it's wrong because under -OO my preconditions won't be checked and my input might be invalid.
If I use it to enforce a postcondition, then my API's consumers have to occasionally handle this weird error, except it won't be checked under -OO so they won't be able to handle it consistently.
If I use it to try to make assertions about internal state during a computation, then I introduce an additional, untested (at the very least untested under -OO), probably undocumented (did I remember to say "and raises AssertionError when..." in its docstring?) code path where when this "bad" thing happens, I get an exception instead of a result.
If that's an important failure mode, then there ought to be a documented exception, which the computation's consumers can deal with.
If it really should "never happen", then I really should have just written some unit tests verifying that it doesn't happen in any case I can think of. And I shouldn't be writing code to handle cases I can't come up with any way to exercise, because how do I know that it's going to do the right thing? (If I had a dollar for every 'assert' message that didn't have the right number of arguments to its format string, etc.)
Also, when things that should "never happen" do actually happen in real life, is a random exception that interrupts the process actually an improvement over just continuing on with some potentially bad data? In most cases, no, it really isn't, because by blowing up you've removed the ability of the user to take corrective action or do a workaround. (In the cases where blowing up is better because you're about to do something destructive, again, a test seems in order.)
My python code is very well documented, which means that there is sometimes a significant runtime overhead from docstrings. That's really my only interest in -OO: reducing memory footprint of Python processes by dropping dozens of megabytes of library documentation from each process. The fact that it changes the semantics of 'assert' is an unfortunate distraction.
So the only time I'd even consider using 'assert' is in a throwaway script which might be run once, that I'm not going to write any tests for and I'm not going to maintain, but I might care about just enough to want to blow up instead of calling 'os.unlink' if certain conditions are not met.
(But then every time I actually use it that way, I realize that I should have dealt with the error sanely and I probably have to go back and fix it anyway.)
On Thu, Apr 28, 2011 at 6:59 PM, Guido van Rossum guido@python.org wrote:
On Thu, Apr 28, 2011 at 12:54 AM, Tarek Ziadé ziade.tarek@gmail.com wrote:
In my opinion assert should be avoided completely anywhere else than in the tests. If this is a wrong statement, please let me know why :)
I would turn that around. The assert statement should not be used in unit tests; unit tests should use self.assertXyzzy() always.
FWIW this is only true for the unittest module/pkg policy for writing and organising tests. There are other popular test frameworks like nose and pytest which promote using plain asserts within writing unit tests and also allow to write tests in functions. And judging from my tutorials and others places many people appreciate the ease of using asserts as compared to learning tons of new methods. YMMV.
Holger
regular code, assert should be about detecting buggy code. It should not be used to test for error conditions in input data. (Both these can be summarized as "if you still want the test to happen with -O, don't use assert.)
-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/holger.krekel%40gmail.com
On 4/28/2011 11:22 AM, skip@pobox.com wrote:
Barry> I would agree. Use asserts for "this can't possibly happen Barry> <wink>" conditions.Without looking, I suspect that's probably what the author thought he was doing.
You wish: to repeat the example from threading:
def __init__(self, group=None, target=None, name=None, args=(), kwargs=None, verbose=None): assert group is None, "group argument must be None for now"
is something that can easily happen.
On 4/28/11 3:27 PM, Holger Krekel wrote:
On Thu, Apr 28, 2011 at 6:59 PM, Guido van Rossumguido@python.org wrote:
On Thu, Apr 28, 2011 at 12:54 AM, Tarek Ziadéziade.tarek@gmail.com wrote:
In my opinion assert should be avoided completely anywhere else than in the tests. If this is a wrong statement, please let me know why :)
I would turn that around. The assert statement should not be used in unit tests; unit tests should use self.assertXyzzy() always.
FWIW this is only true for the unittest module/pkg policy for writing and organising tests. There are other popular test frameworks like nose and pytest which promote using plain asserts within writing unit tests and also allow to write tests in functions. And judging from my tutorials and others places many people appreciate the ease of using asserts as compared to learning tons of new methods. YMMV.
Holger
regular code, assert should be about detecting buggy code. It should not be used to test for error conditions in input data. (Both these can be summarized as "if you still want the test to happen with -O, don't use assert.)
Regardless of whether those frameworks encourage it, it's still the wrong thing to do for the reason that Guido states. Some bugs only show up under -O, so you ought to be running your test suite under -O, too.
On Apr 28, 2011, at 1:27 PM, Holger Krekel wrote:
On Thu, Apr 28, 2011 at 6:59 PM, Guido van Rossum guido@python.org wrote:
On Thu, Apr 28, 2011 at 12:54 AM, Tarek Ziadé ziade.tarek@gmail.com wrote:
In my opinion assert should be avoided completely anywhere else than in the tests. If this is a wrong statement, please let me know why :)
I would turn that around. The assert statement should not be used in unit tests; unit tests should use self.assertXyzzy() always.
FWIW this is only true for the unittest module/pkg policy for writing and organising tests. There are other popular test frameworks like nose and pytest which promote using plain asserts within writing unit tests and also allow to write tests in functions. And judging from my tutorials and others places many people appreciate the ease of using asserts as compared to learning tons of new methods. YMMV.
I've also observed that people appreciate using asserts with nose.py and py.test.
Raymond
On Thu, Apr 28, 2011 at 2:53 PM, Raymond Hettinger raymond.hettinger@gmail.com wrote:
On Apr 28, 2011, at 1:27 PM, Holger Krekel wrote:
On Thu, Apr 28, 2011 at 6:59 PM, Guido van Rossum guido@python.org wrote:
On Thu, Apr 28, 2011 at 12:54 AM, Tarek Ziadé ziade.tarek@gmail.com wrote:
In my opinion assert should be avoided completely anywhere else than in the tests. If this is a wrong statement, please let me know why :)
I would turn that around. The assert statement should not be used in unit tests; unit tests should use self.assertXyzzy() always.
FWIW this is only true for the unittest module/pkg policy for writing and organising tests. There are other popular test frameworks like nose and pytest which promote using plain asserts within writing unit tests and also allow to write tests in functions. And judging from my tutorials and others places many people appreciate the ease of using asserts as compared to learning tons of new methods. YMMV.
I've also observed that people appreciate using asserts with nose.py and py.test.
They must not appreciate -O. :-)
On Apr 28, 2011, at 3:07 PM, Guido van Rossum wrote:
On Thu, Apr 28, 2011 at 2:53 PM, Raymond Hettinger raymond.hettinger@gmail.com wrote:
On Apr 28, 2011, at 1:27 PM, Holger Krekel wrote:
On Thu, Apr 28, 2011 at 6:59 PM, Guido van Rossum guido@python.org wrote:
On Thu, Apr 28, 2011 at 12:54 AM, Tarek Ziadé ziade.tarek@gmail.com wrote:
In my opinion assert should be avoided completely anywhere else than in the tests. If this is a wrong statement, please let me know why :)
I would turn that around. The assert statement should not be used in unit tests; unit tests should use self.assertXyzzy() always.
FWIW this is only true for the unittest module/pkg policy for writing and organising tests. There are other popular test frameworks like nose and pytest which promote using plain asserts within writing unit tests and also allow to write tests in functions. And judging from my tutorials and others places many people appreciate the ease of using asserts as compared to learning tons of new methods. YMMV.
I've also observed that people appreciate using asserts with nose.py and py.test.
They must not appreciate -O. :-)
It might be nice if there were a pragma or module variable to selectively enable asserts for a given test module so that -O would turn-off asserts in the production code but leave them on in a test_suite.
Raymond
2011/4/28 Raymond Hettinger raymond.hettinger@gmail.com:
It might be nice if there were a pragma or module variable to selectively enable asserts for a given test module so that -O would turn-off asserts in the production code but leave them on in a test_suite.
from __future__ import perfect_code_so_no_asserts
:)
Tarek Ziadé writes:
On Thu, Apr 28, 2011 at 5:26 PM, Barry Warsaw barry@python.org wrote:
BTW, I think it always helps to have a really good assert message, and/or a leading comment to explain *why* that condition can't possibly happen.
why bother, it can't happen ;)
Except before breakfast, says the Red Queen.
On Fri, Apr 29, 2011 at 12:31 AM, Raymond Hettinger raymond.hettinger@gmail.com wrote:
On Apr 28, 2011, at 3:07 PM, Guido van Rossum wrote:
On Thu, Apr 28, 2011 at 2:53 PM, Raymond Hettinger raymond.hettinger@gmail.com wrote:
On Apr 28, 2011, at 1:27 PM, Holger Krekel wrote:
On Thu, Apr 28, 2011 at 6:59 PM, Guido van Rossum guido@python.org wrote:
On Thu, Apr 28, 2011 at 12:54 AM, Tarek Ziadé ziade.tarek@gmail.com wrote:
In my opinion assert should be avoided completely anywhere else than in the tests. If this is a wrong statement, please let me know why :)
I would turn that around. The assert statement should not be used in unit tests; unit tests should use self.assertXyzzy() always.
FWIW this is only true for the unittest module/pkg policy for writing and organising tests. There are other popular test frameworks like nose and pytest which promote using plain asserts within writing unit tests and also allow to write tests in functions. And judging from my tutorials and others places many people appreciate the ease of using asserts as compared to learning tons of new methods. YMMV.
I've also observed that people appreciate using asserts with nose.py and py.test.
They must not appreciate -O. :-)
It might be nice if there were a pragma or module variable to selectively enable asserts for a given test module so that -O would turn-off asserts in the production code but leave them on in a test_suite.
A way to tell Python "if you are going to compile this module/path, don't turn off asserts, no matter what" would be great. Then nose/py.test and whichever tools/apps could fine-tune the handling of asserts. (This would probably be better than marking things inline for those use cases). Then testing with "-O" would work nicely as well which would be appreciated :)
best, holger
Raymond
On Fri, Apr 29, 2011 at 9:55 AM, Holger Krekel holger.krekel@gmail.com wrote:
On Fri, Apr 29, 2011 at 12:31 AM, Raymond Hettinger raymond.hettinger@gmail.com wrote:
On Apr 28, 2011, at 3:07 PM, Guido van Rossum wrote:
On Thu, Apr 28, 2011 at 2:53 PM, Raymond Hettinger raymond.hettinger@gmail.com wrote:
On Apr 28, 2011, at 1:27 PM, Holger Krekel wrote:
On Thu, Apr 28, 2011 at 6:59 PM, Guido van Rossum guido@python.org wrote:
On Thu, Apr 28, 2011 at 12:54 AM, Tarek Ziadé ziade.tarek@gmail.com wrote: > In my opinion assert should be avoided completely anywhere else than > in the tests. If this is a wrong statement, please let me know why :)
I would turn that around. The assert statement should not be used in unit tests; unit tests should use self.assertXyzzy() always.
FWIW this is only true for the unittest module/pkg policy for writing and organising tests. There are other popular test frameworks like nose and pytest which promote using plain asserts within writing unit tests and also allow to write tests in functions. And judging from my tutorials and others places many people appreciate the ease of using asserts as compared to learning tons of new methods. YMMV.
I've also observed that people appreciate using asserts with nose.py and py.test.
They must not appreciate -O. :-)
It might be nice if there were a pragma or module variable to selectively enable asserts for a given test module so that -O would turn-off asserts in the production code but leave them on in a test_suite.
A way to tell Python "if you are going to compile this module/path, don't turn off asserts, no matter what" would be great. Then nose/py.test and whichever tools/apps could fine-tune the handling of asserts. (This would probably be better than marking things inline for those use cases). Then testing with "-O" would work nicely as well which would be appreciated :)
best, holger
Raymond
Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
Any reason why -O behavior regarding removing asserts should not be changed? Or should I go to python-ideas?
On 28 April 2011 23:07, Guido van Rossum guido@python.org wrote:
On Thu, Apr 28, 2011 at 2:53 PM, Raymond Hettinger raymond.hettinger@gmail.com wrote:
On Apr 28, 2011, at 1:27 PM, Holger Krekel wrote:
On Thu, Apr 28, 2011 at 6:59 PM, Guido van Rossum guido@python.org wrote:
On Thu, Apr 28, 2011 at 12:54 AM, Tarek Ziadé ziade.tarek@gmail.com wrote:
In my opinion assert should be avoided completely anywhere else than in the tests. If this is a wrong statement, please let me know why :)
I would turn that around. The assert statement should not be used in unit tests; unit tests should use self.assertXyzzy() always.
FWIW this is only true for the unittest module/pkg policy for writing and organising tests. There are other popular test frameworks like nose and pytest which promote using plain asserts within writing unit tests and also allow to write tests in functions. And judging from my tutorials and others places many people appreciate the ease of using asserts as compared to learning tons of new methods. YMMV.
I've also observed that people appreciate using asserts with nose.py and py.test.
They must not appreciate -O. :-)
Personaly I'd love to get rid of all of -O's meanings apart from setting __debug__ to False. Then you can write a strip tool which could strip all docstrings, just unused docstrings (an improvement over -O), and any "dead" code resulting from setting __debug__ to either True or False. The last thing to do is place assert statements inside a if __debug__ block.
That way you could use the strip tool on the modules under test but not on the test modules.
Regards Floris
PS: I actually wrote some prototype code for such a strip tool last year but never finished it off, so I'm pretty sure most of this is possible.
participants (16)
-
Antoine Pitrou -
Barry Warsaw -
Benjamin Peterson -
Floris Bruynooghe -
Fred Drake -
Glyph Lefkowitz -
Guido van Rossum -
Holger Krekel -
Maciej Fijalkowski -
Michael Foord -
Raymond Hettinger -
Robert Kern -
skip＠pobox.com -
Stephen J. Turnbull -
Tarek Ziadé -
Terry Reedy