Ensuring asserts are always on in a file
Some people (like myself, or the coworkers of [this person](https://mail.python.org/archives/list/python-ideas@python.org/thread/PLXOXKA...)) just like to use asserts as a convenient way to check things. We don't want asserts to ever be turned off. Maybe there could be some kind of compiler directive which means "in this file, even with -O, keep the asserts". Maybe the line `assert __debug__`?
On 5/17/20 5:04 PM, Alex Hall wrote:
Some people (like myself, or the coworkers of [this person](https://mail.python.org/archives/list/python-ideas@python.org/thread/PLXOXKA...)) just like to use asserts as a convenient way to check things. We don't want asserts to ever be turned off. Maybe there could be some kind of compiler directive which means "in this file, even with -O, keep the asserts". Maybe the line `assert __debug__`?
My answer to that is 'Your doing int wrong', that is like asking for an if statement to not bother checking some of its conditionl. the assert operation has, like forever, been a debugging aid to say this condition should ALWAYS/NEVER occur, check for it in Debug Builds, but not for production builds. In some cases (some languages) the compiler might even optimize to code based on the knowledge that the condition, even in code executed before the condition (if it can be sure the assert will be reached). To quote the documentation: Assertions should *not* be used to test for failure cases that can occur because of bad user input or operating system/environment failures, such as a file not being found. Instead, you should raise an exception, or print an error message, or whatever is appropriate. One important reason why assertions should only be used for self-tests of the program is that assertions can be disabled at compile time. Basically the statement assert cond is the same as if __debug__ and not cond: raise AssertionError If you really what what you describe, add the following to your code: if not __debug__: raise AssertionError, "Please don't disable assertions" (This won't enable the assertions, but will let the user know that you need them enabled) -- Richard Damon
On Mon, May 18, 2020 at 8:04 AM Richard Damon <Richard@damon-family.org> wrote:
As a bonus that also asserts that you're running an ancient version of Python.... probably not what you want :) if not __debug__: raise AssertionError("Assertions are required for this module, because it is broken.") ChrisA
On Mon, May 18, 2020 at 12:03 AM Richard Damon <Richard@damon-family.org> wrote:
That bit about other languages is fascinating! And a bit scary...
I understand this, but I still don't understand in what situation people use assertions 'correctly'. To me an assert implies: 1. If the condition is false: that's bad, and the code shouldn't keep running. 2. I'm not 100% sure the condition is always true (a bug in my code is always a possiblity) and I need code to check for me. In that case I want the check there even more in production than in development. It never makes sense to me for such checks to be turned off. I understand the reason to turn them off is performance, but this always seems like a minor optimisation. I don't want code to be significantly slower without -O. For me it makes development and testing slow and painful, for users it pushes them to use a global switch to solve a local problem at the cost of safety. Really slow correctness checking is generally reserved for continuous integration tests. I've never wanted to put something significantly slow in an assert and I've never seen anyone else do that, or write in their documentation "this code is best used with -O". So the tradeoff always seems clear - accept a small performance hit for better peace of mind of correctness. If I was really concerned about speed, Python is already the wrong language.
On 5/18/20 5:25 AM, Alex Hall wrote:
The idea is that during debug, you have added sentinel checking the pre- and post- conditions to routines. Pre-conditions are those requirements that MUST be met to call the function, things that the caller was supposed to make sure were true to begin with (either from other routines post conditions or what it has checked itself. A failure of a per-condition indicate a bug in the caller, and routines have these asserts to help find errors in the caller. The post conditions are the things the routine promises to deliver, a failure of a post condition indicates a bug in this routine. One part of asserts is there presence documents what the routines pre- and post-conditions are (in addition to their debugging help) Many of these tests are trivial (thing like simple tests on values) and for these the removal may not matter that much (but sometimes a 1% gain is important, maybe not so much in Python though). Some of the tests may be very expensive, test like 'tree is balanced' or 'list has no duplicate values' may require more work to verify then the routine itself, and these can be important to remove in production releases. For the 'cheap' asserts, if you want to keep the test in the code because you aren't sure you have debugged enough, you can replace the assert with an if, with the assert statement in the conditional (or just raise the exception yourself, or better use a custom problem report routine). The normal reason to not want to do this is that the asserts output isn't really friendly to the average user, but is designed for the developer. In some cases, it might even reveal information that the end user isn't really supposed to know. -- Richard Damon
On Mon, May 18, 2020 at 11:25:39AM +0200, Alex Hall wrote:
I understand this, but I still don't understand in what situation people use assertions 'correctly'.
https://import-that.dreamwidth.org/676.html
They apply to many other conditional checks apart from assertions. The question should be, what distinguishes an *assertion* from other conditional checks followed by raise? Both trigger on errors; the difference is in the semantics: assert is intended for use during development. If we are confident that there are no bugs (at least not in the areas checked by assertions), then the assertions are safe to disable in production. If we're not confident, then we're running a system which is not ready for production, and we should be honest about that at least to ourselves even if we don't exactly advertise that fact :-) And if removing the asserts could *cause* bugs, rather than reveal their existence, then they certainly shouldn't be asserts.
In that case I want the check there even more in production than in development. It never makes sense to me for such checks to be turned off.
That opinion pretty much goes against the standard practice in every language that I know of that has assertions. "Assertions that you cannot turn off" are just regular error checking. The ability to disable assertions is a feature, not a bug, and is the primary feature that distinguishes asserts from `if... raise`. --------------------------- --------------------------------- assert if...raise --------------------------- --------------------------------- raises if not condition raises if condition can be disabled cannot be disabled limited to AssertionError no limit to the exception type --------------------------- --------------------------------- If you expect the assertions to fail in production, why not fix the error conditions that could lead them to fail? Probably no other language makes assertions a bigger part of the language than Eiffel. They have, by my count, at least five kinds of assertions: * preconditions * postconditions * loop invariants * class invariants * checks and they are so fundamental to Eiffel's programming model that they are part of the syntactic structure of classes and methods. But even Eiffel designed assertions to be disabled: "During development and testing, assertion monitoring should be turned on at the highest possible level. [...] When releasing the final version of a system, it is usually appropriate to turn off assertion monitoring, or bring it down to the `require` level. The exact policy depends on the circumstances; it is a trade off between efficiency considerations, the potential cost of mistakes, and how much the developers and quality assurance team trust the product." https://www.eiffel.org/doc/eiffel/ET-_Design_by_Contract_%28tm%29%2C_Asserti... Compared to Eiffel's fine-grained assertion system, Python's is pretty coarse, we don't have multiple levels of assertion testing, just All On and All Off. But the principle still applies. Assertions are intended to be capable of being disabled.
I understand the reason to turn them off is performance, but this always seems like a minor optimisation.
Certainly in Python the optimizations tend to be small, but not always insignificant. And even small things can add up to make a significant difference.
I don't want code to be significantly slower without -O.
So you want it to be equally slow with -O? *wink* The aim isn't to intentionally slow down the unoptimised code, but to speed up the optimized code by trading off some "just in case" tests which should never fail for additional speed. It is the person running the code, not the developer, who decides whether to make that tradeoff or not. (Unless the developer and the user are the same person, in which case why do you care? Just don't run your code with -O. If you are your application's sole user, then do whatever you like, it's okay.) If you know a test should never fail, then it is safe to turn it off. If it's not safe to turn it off, then it's not an assertion. (Safety, of course, is not an absolute. If it were absolute, we wouldn't bother running the assertions at all, ever, since we would know that they absolutely will never fail.)
For me it makes development and testing slow and painful,
Sorry, are you saying that assertions make development and testing slow and painful? If that's not what you mean, I don't know what this means. In pre-production and debugging, you run your code without -O. If the application is slow and painful, you have to solve that regardless of assert. When you are satisfied that the application is ready for production, you hand it over to your users who may or may not run it with -O, that's not your decision to make.
for users it pushes them to use a global switch to solve a local problem at the cost of safety.
If that is true, that's a sign you are misusing assertions. If the end user sees an AssertionError failure, that is indistinguishable from a bug in the code that causes it to crash. If your asserts trigger in production, that's not "safety", that's a bug.
*shrug* The cost of lots of cheap tests can still add up. If you haven't compared the speed of your application with and without -O then you don't know what the total cost of those assertions are. You might be surprised, or you might be disappointed about how little benefit you get. It's hard to say. -- Steven
On Sun, May 17, 2020 at 09:04:50PM -0000, Alex Hall wrote:
That's not your choice. The choice to turn assertions off belongs to the person running the code, not you. If you are not prepared for the user to turn assertions off, then your code is buggy.
Indeed this feature already exists. It's written as: if condition: raise Exception and you can control when it is enabled or disabled, the end user can't change that. Even better, you can use the correct, meaningful exception instead of using the wrong, misleading exception. `assert` is for assertions, not error checking. If you don't want the semantics of an assertion, then don't use `assert`. -- Steven
The closest you may ever get to something like this is a clean separation of -O flags instead of the current -O/-OO options. That way you can flip on everything *but* assertion removal. But a per-file directive I don't see happening, and even the flag separation has never caught on enough for anyone to put in the effort to get a PoC working since the only people who ask for this are people wanting permanent asserts and that leads the usual "you're doing it wrong" comments. Probably best/only way to motivate it is to open up -O in a way to allow for more optimizations, but once again that's a lot of work. On Sun, May 17, 2020 at 2:08 PM Alex Hall <alex.mojaki@gmail.com> wrote:
On 5/17/20 5:04 PM, Alex Hall wrote:
Some people (like myself, or the coworkers of [this person](https://mail.python.org/archives/list/python-ideas@python.org/thread/PLXOXKA...)) just like to use asserts as a convenient way to check things. We don't want asserts to ever be turned off. Maybe there could be some kind of compiler directive which means "in this file, even with -O, keep the asserts". Maybe the line `assert __debug__`?
My answer to that is 'Your doing int wrong', that is like asking for an if statement to not bother checking some of its conditionl. the assert operation has, like forever, been a debugging aid to say this condition should ALWAYS/NEVER occur, check for it in Debug Builds, but not for production builds. In some cases (some languages) the compiler might even optimize to code based on the knowledge that the condition, even in code executed before the condition (if it can be sure the assert will be reached). To quote the documentation: Assertions should *not* be used to test for failure cases that can occur because of bad user input or operating system/environment failures, such as a file not being found. Instead, you should raise an exception, or print an error message, or whatever is appropriate. One important reason why assertions should only be used for self-tests of the program is that assertions can be disabled at compile time. Basically the statement assert cond is the same as if __debug__ and not cond: raise AssertionError If you really what what you describe, add the following to your code: if not __debug__: raise AssertionError, "Please don't disable assertions" (This won't enable the assertions, but will let the user know that you need them enabled) -- Richard Damon
On Mon, May 18, 2020 at 8:04 AM Richard Damon <Richard@damon-family.org> wrote:
As a bonus that also asserts that you're running an ancient version of Python.... probably not what you want :) if not __debug__: raise AssertionError("Assertions are required for this module, because it is broken.") ChrisA
On Mon, May 18, 2020 at 12:03 AM Richard Damon <Richard@damon-family.org> wrote:
That bit about other languages is fascinating! And a bit scary...
I understand this, but I still don't understand in what situation people use assertions 'correctly'. To me an assert implies: 1. If the condition is false: that's bad, and the code shouldn't keep running. 2. I'm not 100% sure the condition is always true (a bug in my code is always a possiblity) and I need code to check for me. In that case I want the check there even more in production than in development. It never makes sense to me for such checks to be turned off. I understand the reason to turn them off is performance, but this always seems like a minor optimisation. I don't want code to be significantly slower without -O. For me it makes development and testing slow and painful, for users it pushes them to use a global switch to solve a local problem at the cost of safety. Really slow correctness checking is generally reserved for continuous integration tests. I've never wanted to put something significantly slow in an assert and I've never seen anyone else do that, or write in their documentation "this code is best used with -O". So the tradeoff always seems clear - accept a small performance hit for better peace of mind of correctness. If I was really concerned about speed, Python is already the wrong language.
On 5/18/20 5:25 AM, Alex Hall wrote:
The idea is that during debug, you have added sentinel checking the pre- and post- conditions to routines. Pre-conditions are those requirements that MUST be met to call the function, things that the caller was supposed to make sure were true to begin with (either from other routines post conditions or what it has checked itself. A failure of a per-condition indicate a bug in the caller, and routines have these asserts to help find errors in the caller. The post conditions are the things the routine promises to deliver, a failure of a post condition indicates a bug in this routine. One part of asserts is there presence documents what the routines pre- and post-conditions are (in addition to their debugging help) Many of these tests are trivial (thing like simple tests on values) and for these the removal may not matter that much (but sometimes a 1% gain is important, maybe not so much in Python though). Some of the tests may be very expensive, test like 'tree is balanced' or 'list has no duplicate values' may require more work to verify then the routine itself, and these can be important to remove in production releases. For the 'cheap' asserts, if you want to keep the test in the code because you aren't sure you have debugged enough, you can replace the assert with an if, with the assert statement in the conditional (or just raise the exception yourself, or better use a custom problem report routine). The normal reason to not want to do this is that the asserts output isn't really friendly to the average user, but is designed for the developer. In some cases, it might even reveal information that the end user isn't really supposed to know. -- Richard Damon
On Mon, May 18, 2020 at 11:25:39AM +0200, Alex Hall wrote:
I understand this, but I still don't understand in what situation people use assertions 'correctly'.
https://import-that.dreamwidth.org/676.html
They apply to many other conditional checks apart from assertions. The question should be, what distinguishes an *assertion* from other conditional checks followed by raise? Both trigger on errors; the difference is in the semantics: assert is intended for use during development. If we are confident that there are no bugs (at least not in the areas checked by assertions), then the assertions are safe to disable in production. If we're not confident, then we're running a system which is not ready for production, and we should be honest about that at least to ourselves even if we don't exactly advertise that fact :-) And if removing the asserts could *cause* bugs, rather than reveal their existence, then they certainly shouldn't be asserts.
In that case I want the check there even more in production than in development. It never makes sense to me for such checks to be turned off.
That opinion pretty much goes against the standard practice in every language that I know of that has assertions. "Assertions that you cannot turn off" are just regular error checking. The ability to disable assertions is a feature, not a bug, and is the primary feature that distinguishes asserts from `if... raise`. --------------------------- --------------------------------- assert if...raise --------------------------- --------------------------------- raises if not condition raises if condition can be disabled cannot be disabled limited to AssertionError no limit to the exception type --------------------------- --------------------------------- If you expect the assertions to fail in production, why not fix the error conditions that could lead them to fail? Probably no other language makes assertions a bigger part of the language than Eiffel. They have, by my count, at least five kinds of assertions: * preconditions * postconditions * loop invariants * class invariants * checks and they are so fundamental to Eiffel's programming model that they are part of the syntactic structure of classes and methods. But even Eiffel designed assertions to be disabled: "During development and testing, assertion monitoring should be turned on at the highest possible level. [...] When releasing the final version of a system, it is usually appropriate to turn off assertion monitoring, or bring it down to the `require` level. The exact policy depends on the circumstances; it is a trade off between efficiency considerations, the potential cost of mistakes, and how much the developers and quality assurance team trust the product." https://www.eiffel.org/doc/eiffel/ET-_Design_by_Contract_%28tm%29%2C_Asserti... Compared to Eiffel's fine-grained assertion system, Python's is pretty coarse, we don't have multiple levels of assertion testing, just All On and All Off. But the principle still applies. Assertions are intended to be capable of being disabled.
I understand the reason to turn them off is performance, but this always seems like a minor optimisation.
Certainly in Python the optimizations tend to be small, but not always insignificant. And even small things can add up to make a significant difference.
I don't want code to be significantly slower without -O.
So you want it to be equally slow with -O? *wink* The aim isn't to intentionally slow down the unoptimised code, but to speed up the optimized code by trading off some "just in case" tests which should never fail for additional speed. It is the person running the code, not the developer, who decides whether to make that tradeoff or not. (Unless the developer and the user are the same person, in which case why do you care? Just don't run your code with -O. If you are your application's sole user, then do whatever you like, it's okay.) If you know a test should never fail, then it is safe to turn it off. If it's not safe to turn it off, then it's not an assertion. (Safety, of course, is not an absolute. If it were absolute, we wouldn't bother running the assertions at all, ever, since we would know that they absolutely will never fail.)
For me it makes development and testing slow and painful,
Sorry, are you saying that assertions make development and testing slow and painful? If that's not what you mean, I don't know what this means. In pre-production and debugging, you run your code without -O. If the application is slow and painful, you have to solve that regardless of assert. When you are satisfied that the application is ready for production, you hand it over to your users who may or may not run it with -O, that's not your decision to make.
for users it pushes them to use a global switch to solve a local problem at the cost of safety.
If that is true, that's a sign you are misusing assertions. If the end user sees an AssertionError failure, that is indistinguishable from a bug in the code that causes it to crash. If your asserts trigger in production, that's not "safety", that's a bug.
*shrug* The cost of lots of cheap tests can still add up. If you haven't compared the speed of your application with and without -O then you don't know what the total cost of those assertions are. You might be surprised, or you might be disappointed about how little benefit you get. It's hard to say. -- Steven
On Sun, May 17, 2020 at 09:04:50PM -0000, Alex Hall wrote:
That's not your choice. The choice to turn assertions off belongs to the person running the code, not you. If you are not prepared for the user to turn assertions off, then your code is buggy.
Indeed this feature already exists. It's written as: if condition: raise Exception and you can control when it is enabled or disabled, the end user can't change that. Even better, you can use the correct, meaningful exception instead of using the wrong, misleading exception. `assert` is for assertions, not error checking. If you don't want the semantics of an assertion, then don't use `assert`. -- Steven
The closest you may ever get to something like this is a clean separation of -O flags instead of the current -O/-OO options. That way you can flip on everything *but* assertion removal. But a per-file directive I don't see happening, and even the flag separation has never caught on enough for anyone to put in the effort to get a PoC working since the only people who ask for this are people wanting permanent asserts and that leads the usual "you're doing it wrong" comments. Probably best/only way to motivate it is to open up -O in a way to allow for more optimizations, but once again that's a lot of work. On Sun, May 17, 2020 at 2:08 PM Alex Hall <alex.mojaki@gmail.com> wrote:
participants (5)
-
Alex Hall -
Brett Cannon -
Chris Angelico -
Richard Damon -
Steven D'Aprano