Runtime assertion with no overhead when not active
Hi folks, I intend to add a runtime assertion feature in python. Before submitting a PEP, I am sending a draft to this mailing list to discuss whether it would make sense (above this message). I have actually been using it for the last 2 years and it has proven to be robust and has achieved its primary goal. Briefly, the idea is to add a new assert that can be switch on/off depending on some variable/mechanism at runtime. The whole point of this assert is that it should not bring any overhead when off, i.e. by avoiding evaluating the expression enclosed in the runtime assert. It thus relies on Python grammar. Thanks for your feedback. Eloi Abstract This PEP aims at offering a runtime assert functionnality, extending the compiletime assert already available. Rationale There is no runtime assert relying on python grammar available. For diagnostics and/or debugging reasons, it would be valuable to add such a feature. A runtime assert makes sense when extra checks would be needed in a production environment (where non-debug builds are used). By extending the current python grammar, it would be possible to limit the overhead induces by those runtime asserts when running in a non "assertive" mode (-ed). The idea here is to avoid evaluating the expression when runtime assertion is not active. A brief example why avoiding evaluating the expression is needed to avoid any overhead in case the runtime assert should be ignored. :: runtime_assert( 999 in { i:None for i in range( 10000000 ) } ) Usage :: runtime_assert( expr ) #would result in if expr and runtime_assert_active: print RuntimeAssertionError() Implementation details There is already an implementation available, robust enough for production. The implementation is mainly grammar based, and thus the parser and the grammar needs to be updated: * Python/graminit.c * Python/ast.c * Python/symtable.c * Python/compile.c * Python/Python-ast.c * Python/sysmodule.c * Modules/parsermodule.c * Include/Python-ast.h * Include/graminit.h * Include/pydebug.h * Grammar/Grammar * Parser/Python.asdl * Lib/lib2to3/Grammar.txt * Lib/compiler/ast.py * Lib/compiler/pycodegen.py * Lib/compiler/transformer.py * Lib/symbol.py * Modules/parsermodule.c References [1] PEP 306, How to Change Python's Grammar (http://www.python.org/dev/peps/pep-0306) Copyright This document has been placed in the public domain.
05.05.18 11:04, Eloi Gaudry пише:
You should have an overhead for checking if it is switched on/off, isn't? And this overhead is virtually the same as testing the value of the global boolean variable.
How this will be different from if debug and not expr: print(RuntimeAssertionError()) ?
I meant avoiding the overhead of the expression evaluation enclosed in the assert itself, not the active/disabled state (that comes at virtually no cost). ex:
runtime_assert( { i:None for i in range( 10000000 ) } )
By using the syntax you describe ('boolean and not expr'), you loose all the benefit of the Python grammar: - self-contained - able to catch semantic/syntax bugs ex:
-----Original Message----- From: Python-ideas <python-ideas-bounces+eloi.gaudry=fft.be@python.org> On Behalf Of Serhiy Storchaka Sent: Saturday, May 5, 2018 2:05 PM To: python-ideas@python.org Subject: Re: [Python-ideas] Runtime assertion with no overhead when not active 05.05.18 11:04, Eloi Gaudry пише:
You should have an overhead for checking if it is switched on/off, isn't? And this overhead is virtually the same as testing the value of the global boolean variable.
How this will be different from if debug and not expr: print(RuntimeAssertionError()) ? _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On Sat, May 05, 2018 at 08:04:45AM +0000, Eloi Gaudry wrote:
Hi folks, I intend to add a runtime assertion feature in python.
I'm very interested in this idea, but I'm afraid your draft PEP isn't very clear about this feature. Comments below.
Why not this? if DEBUG_ENABLED: run_check(expression) where run_check is an ordinary function. If DEBUG_ENABLED is false, the only runtime overhead is the name lookup and bool test, the expression is not evaluated. What features does your suggestion offer over this simple technique? [...]
You say that you have been using this feature in production for two years. You should give real-life examples, not fake examples like the above. That example would be better written as: runtime_assert(True) since no part of the test actually depends on either the enormous dict or range objects that you build only to throw away.
Surely the test should be the other way around? if runtime_assert_active and expr: print(RuntimeAssertionError()) otherwise the expression is evaluated regardless of whether the runtime assertions are active or not. Please use Python 3 syntax for the PEP, as Python 2 is in maintenance mode and will absolutely not get any new language features. Some more observations: I like the idea of more control over runtime checking, but your PEP should justify why this is a good idea and why "just write more unit tests" isn't the solution. What happens if the caller has defined their own function or variable called runtime_assert? Can the caller customise the assertion error message? If you are changing the grammar, why not make runtime_assert a statement? Can the caller set an application-wide global across all modules, or do they have to set runtime_assert_active = True in every module they want to use runtime_assert? -- Steve
On Sun, 2018-05-06 at 01:30 +1000, Steven D'Aprano wrote:
> On Sat, May 05, 2018 at 08:04:45AM +0000, Eloi Gaudry wrote:
>
> > Hi folks,
> > I intend to add a runtime assertion feature in python.
>
> I'm very interested in this idea, but I'm afraid your draft PEP
> isn't
> very clear about this feature. Comments below.
Thanks, I'll try to clarify this here.
> > Rationale
> > There is no runtime assert relying on python grammar available.
> > For
> > diagnostics and/or debugging reasons, it would be valuable to add
> > such
> > a feature.
>
> Why not this?
>
> if DEBUG_ENABLED:
> run_check(expression)
>
> where run_check is an ordinary function. If DEBUG_ENABLED is false,
> the
> only runtime overhead is the name lookup and bool test, the
> expression
> is not evaluated.
for the sake of clarity across any Python application/extension for
instance, and because having run_check defined as a statement seems
actually the good thing to do.
>
> What features does your suggestion offer over this simple technique?
Code readiness I think (kiss) : simpler and shorter.
The simple technique would still be used for other purpose (assert that
are context based for instance).
>
> [...]
> > A brief example why avoiding evaluating the expression is needed
> > to
> > avoid any overhead in case the runtime assert should be ignored.
> > ::
> > runtime_assert( 999 in { i:None for i in range( 10000000 ) } )
>
> You say that you have been using this feature in production for two
> years. You should give real-life examples, not fake examples like
> the
> above. That example would be better written as:
>
> runtime_assert(True)
>
> since no part of the test actually depends on either the enormous
> dict
> or range objects that you build only to throw away.
Real life examples would be more application-based I think.
In my case, I used this for switching off some heaving computing
expression evaluation at runtime. Those expression would have been
responsible for sorting vectors and communicating them through MPI
using some method from our framework, in order to finally return a
boolean.
> > Usage
> > ::
> >
> > runtime_assert( expr )
> >
> > #would result in
> > if expr and runtime_assert_active:
> > print RuntimeAssertionError()
>
> Surely the test should be the other way around?
>
> if runtime_assert_active and expr:
> print(RuntimeAssertionError())
otherwise the expression is evaluated regardless of whether the
runtime
> assertions are active or not.
You are absolutely right :)
> Please use Python 3 syntax for the PEP, as Python 2 is in
> maintenance
> mode and will absolutely not get any new language features.
Sure
>
> Some more observations:
>
> I like the idea of more control over runtime checking, but your PEP
> should justify why this is a good idea and why "just write more unit
> tests" isn't the solution.
My main reasons is the one I gave in the context description of the
PEP. Assert remains an efficient way for controlling your application
behavior when running in production (as opposed to a debug build).
Writing more unit tests is always a good practice, but :
- it does not necessarily cover all possible paths that are runtime
dependent (an application relies on various app-parameters and
architecture-parameters for instance)
- it does not offer a solution for your extension/application consumers
(that might be using your application in a very specific set of
paths/set of decision/etc.).
> What happens if the caller has defined their own function or
> variable
> called runtime_assert?
making runtime_assert a statement (as assert is already) would forbid
such a definition, thus it would break backward compatibility (in those
specific cases) for sure.
> Can the caller customise the assertion error message?
At this stage, no, but I don't know if this would be something
difficult to add.
> If you are changing the grammar, why not make runtime_assert a
> statement?
I'm not really changing the grammar, just making 'runtime_assert' a
statement as 'assert' already is (and IMHO could have been named as
debug_assert). There is no significant difference between both assert
in term of implementation.
> Can the caller set an application-wide global across all modules, or
> do
> they have to set
>
> runtime_assert_active = True
>
> in every module they want to use runtime_assert?
I would vote in favor of having two different ways to activate or not
the runtime assertion evaluation:
- the first one would consist of controlling a global variable at the
C-level (the current recipe in cPython) and would be used by python
extension. This is the solution I am using personnaly.
- the second would use a set method available in pure python (not
implemented, to be done at C-level too I think) to easily (de)activate
all the runtime asserts.
They are basically the same thing, with one difference being that runtime_assert would be used for extension mainly, and switchable on/off without using -o flag on the command line. Eloi ________________________________ From: Python-ideas <python-ideas-bounces+eloi.gaudry=fft.be@python.org> on behalf of Michel Desmoulin <desmoulinmichel@gmail.com> Sent: Tuesday, June 5, 2018 3:33:30 PM To: python-ideas@python.org Subject: Re: [Python-ideas] Runtime assertion with no overhead when not active Maybe somebody already answered this, but what's the difference between this and the keyword "assert", which basically can be stripped of using "python -o" ? Le 05/05/2018 à 10:04, Eloi Gaudry a écrit :
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
05.05.18 11:04, Eloi Gaudry пише:
You should have an overhead for checking if it is switched on/off, isn't? And this overhead is virtually the same as testing the value of the global boolean variable.
How this will be different from if debug and not expr: print(RuntimeAssertionError()) ?
I meant avoiding the overhead of the expression evaluation enclosed in the assert itself, not the active/disabled state (that comes at virtually no cost). ex:
runtime_assert( { i:None for i in range( 10000000 ) } )
By using the syntax you describe ('boolean and not expr'), you loose all the benefit of the Python grammar: - self-contained - able to catch semantic/syntax bugs ex:
-----Original Message----- From: Python-ideas <python-ideas-bounces+eloi.gaudry=fft.be@python.org> On Behalf Of Serhiy Storchaka Sent: Saturday, May 5, 2018 2:05 PM To: python-ideas@python.org Subject: Re: [Python-ideas] Runtime assertion with no overhead when not active 05.05.18 11:04, Eloi Gaudry пише:
You should have an overhead for checking if it is switched on/off, isn't? And this overhead is virtually the same as testing the value of the global boolean variable.
How this will be different from if debug and not expr: print(RuntimeAssertionError()) ? _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On Sat, May 05, 2018 at 08:04:45AM +0000, Eloi Gaudry wrote:
Hi folks, I intend to add a runtime assertion feature in python.
I'm very interested in this idea, but I'm afraid your draft PEP isn't very clear about this feature. Comments below.
Why not this? if DEBUG_ENABLED: run_check(expression) where run_check is an ordinary function. If DEBUG_ENABLED is false, the only runtime overhead is the name lookup and bool test, the expression is not evaluated. What features does your suggestion offer over this simple technique? [...]
You say that you have been using this feature in production for two years. You should give real-life examples, not fake examples like the above. That example would be better written as: runtime_assert(True) since no part of the test actually depends on either the enormous dict or range objects that you build only to throw away.
Surely the test should be the other way around? if runtime_assert_active and expr: print(RuntimeAssertionError()) otherwise the expression is evaluated regardless of whether the runtime assertions are active or not. Please use Python 3 syntax for the PEP, as Python 2 is in maintenance mode and will absolutely not get any new language features. Some more observations: I like the idea of more control over runtime checking, but your PEP should justify why this is a good idea and why "just write more unit tests" isn't the solution. What happens if the caller has defined their own function or variable called runtime_assert? Can the caller customise the assertion error message? If you are changing the grammar, why not make runtime_assert a statement? Can the caller set an application-wide global across all modules, or do they have to set runtime_assert_active = True in every module they want to use runtime_assert? -- Steve
On Sun, 2018-05-06 at 01:30 +1000, Steven D'Aprano wrote:
> On Sat, May 05, 2018 at 08:04:45AM +0000, Eloi Gaudry wrote:
>
> > Hi folks,
> > I intend to add a runtime assertion feature in python.
>
> I'm very interested in this idea, but I'm afraid your draft PEP
> isn't
> very clear about this feature. Comments below.
Thanks, I'll try to clarify this here.
> > Rationale
> > There is no runtime assert relying on python grammar available.
> > For
> > diagnostics and/or debugging reasons, it would be valuable to add
> > such
> > a feature.
>
> Why not this?
>
> if DEBUG_ENABLED:
> run_check(expression)
>
> where run_check is an ordinary function. If DEBUG_ENABLED is false,
> the
> only runtime overhead is the name lookup and bool test, the
> expression
> is not evaluated.
for the sake of clarity across any Python application/extension for
instance, and because having run_check defined as a statement seems
actually the good thing to do.
>
> What features does your suggestion offer over this simple technique?
Code readiness I think (kiss) : simpler and shorter.
The simple technique would still be used for other purpose (assert that
are context based for instance).
>
> [...]
> > A brief example why avoiding evaluating the expression is needed
> > to
> > avoid any overhead in case the runtime assert should be ignored.
> > ::
> > runtime_assert( 999 in { i:None for i in range( 10000000 ) } )
>
> You say that you have been using this feature in production for two
> years. You should give real-life examples, not fake examples like
> the
> above. That example would be better written as:
>
> runtime_assert(True)
>
> since no part of the test actually depends on either the enormous
> dict
> or range objects that you build only to throw away.
Real life examples would be more application-based I think.
In my case, I used this for switching off some heaving computing
expression evaluation at runtime. Those expression would have been
responsible for sorting vectors and communicating them through MPI
using some method from our framework, in order to finally return a
boolean.
> > Usage
> > ::
> >
> > runtime_assert( expr )
> >
> > #would result in
> > if expr and runtime_assert_active:
> > print RuntimeAssertionError()
>
> Surely the test should be the other way around?
>
> if runtime_assert_active and expr:
> print(RuntimeAssertionError())
otherwise the expression is evaluated regardless of whether the
runtime
> assertions are active or not.
You are absolutely right :)
> Please use Python 3 syntax for the PEP, as Python 2 is in
> maintenance
> mode and will absolutely not get any new language features.
Sure
>
> Some more observations:
>
> I like the idea of more control over runtime checking, but your PEP
> should justify why this is a good idea and why "just write more unit
> tests" isn't the solution.
My main reasons is the one I gave in the context description of the
PEP. Assert remains an efficient way for controlling your application
behavior when running in production (as opposed to a debug build).
Writing more unit tests is always a good practice, but :
- it does not necessarily cover all possible paths that are runtime
dependent (an application relies on various app-parameters and
architecture-parameters for instance)
- it does not offer a solution for your extension/application consumers
(that might be using your application in a very specific set of
paths/set of decision/etc.).
> What happens if the caller has defined their own function or
> variable
> called runtime_assert?
making runtime_assert a statement (as assert is already) would forbid
such a definition, thus it would break backward compatibility (in those
specific cases) for sure.
> Can the caller customise the assertion error message?
At this stage, no, but I don't know if this would be something
difficult to add.
> If you are changing the grammar, why not make runtime_assert a
> statement?
I'm not really changing the grammar, just making 'runtime_assert' a
statement as 'assert' already is (and IMHO could have been named as
debug_assert). There is no significant difference between both assert
in term of implementation.
> Can the caller set an application-wide global across all modules, or
> do
> they have to set
>
> runtime_assert_active = True
>
> in every module they want to use runtime_assert?
I would vote in favor of having two different ways to activate or not
the runtime assertion evaluation:
- the first one would consist of controlling a global variable at the
C-level (the current recipe in cPython) and would be used by python
extension. This is the solution I am using personnaly.
- the second would use a set method available in pure python (not
implemented, to be done at C-level too I think) to easily (de)activate
all the runtime asserts.
They are basically the same thing, with one difference being that runtime_assert would be used for extension mainly, and switchable on/off without using -o flag on the command line. Eloi ________________________________ From: Python-ideas <python-ideas-bounces+eloi.gaudry=fft.be@python.org> on behalf of Michel Desmoulin <desmoulinmichel@gmail.com> Sent: Tuesday, June 5, 2018 3:33:30 PM To: python-ideas@python.org Subject: Re: [Python-ideas] Runtime assertion with no overhead when not active Maybe somebody already answered this, but what's the difference between this and the keyword "assert", which basically can be stripped of using "python -o" ? Le 05/05/2018 à 10:04, Eloi Gaudry a écrit :
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
participants (4)
-
Eloi Gaudry -
Michel Desmoulin -
Serhiy Storchaka -
Steven D'Aprano