Grr, Google Groups gateway messes up reply-to. Apologies to anyone who gets a double-post, please can posters ensure that reply-to is set to the list, and *not* to the Google Groups gateway? Thanks. Paul On 16 January 2018 at 15:54, Paul Moore <p.f.moore@gmail.com> wrote:
Thanks Paul Le mardi 16 janvier 2018 16:56:57 UTC+1, Paul Moore a écrit :
Why does this need to be a statement at all? Unlike assert, it's always executed, so it can be defined as a simple function
Very good point. Actually that's what I already provide in valid8 with the assert_valid function. See all examples here <https://smarie.github.io/python-valid8/#usage-examples-inline>. Let's consider this example where users want to define on-the-fly one of the validation functions, and combine it with another with a 'or': assert_valid('surface', surf, or_(lambda x: (x >= 0) & (x < 10000), is_foo_compliant), help_msg="surface should be 0=<x<10000 or foo compliant") How ugly for something so simple ! I tried to make it slightly more compact by developping a mini lambda syntax but it obviously makes it slower. Anyway it becomes assert_valid('surface', surf, or_((x >= 0) & (x < 10000), is_foo_compliant), help_msg="surface should be between 0 and 10000 or foo compliant") or even (if you pre-convert Is_foo_compliant to mini_lambda) assert_valid('surface', surf, ((x >= 0) & (x < 10000)) | Is_foo_compliant), help_msg="surface should be between 0 and 10000 or foo compliant") There are three reasons why having a 'validate' statement would improve this: * no more parenthesis: more elegant and readable * inline use of python (1): no more use of lambda or mini_lambda, no performance overhead * inline use of python (2): composition would not require custom function composition operators such as 'or_' (above) or mini-lambda composition anymore, it could be built-in in any language element used after <validate> resulting in validate (surf >= 0) & (surf < 10000) or is_foo_compliant(surf), "surface should be 0=<x<10000 or foo compliant" (I removed the variable name alias 'surface' since I don't know if it should remain or not) Elegant, isn't it ?
I don't see how a validate statement avoids having to deal with all of
the complexity you mention here.
It would obviously need to be quite smart :) but it is possible, since I can do it with assert_valid today.
I fixed the reply-to this time, looks like you're still getting messed up by Google Groups. On 16 January 2018 at 16:25, smarie <sylvain.marie@schneider-electric.com> wrote:
Why do you do this? What's the requirement for delaying evaluation of the condition? A validate statement in Python wouldn't be any better able to do that, so it'd be just as ugly with a statement. There's no reason I can see why I'd ever need delayed evaluation, so what's wrong with just assert_valid(0 <= surf < 10000 and is_foo_compliant(surf), help_msg="surface should be 0=<x<10000 or foo compliant")
So how will that work any differently than the function version I gave above? In terms of the language syntax, it would just be validate EXPR, EXPR and the first EXPR will be evaluated as a boolean, and if it's false an exception will be raised with the second expression as the message. There's no delayed evaluation involved, so a function would work exactly the same.
No more so than my function version, but yes far more so than yours... Paul
(trying with direct reply this time)
Why do you do this? What's the requirement for delaying evaluation of the condition?
Thanks for challenging my poorly chosen examples :) The primary requirement is about *catching* unwanted/uncontrolled/heterogenous exceptions happening in the underlying functions that are combined together to provide the validation means, so as to provide a uniform/consistent outcome however diverse the underlying functions are (they can return booleans or raise exceptions, or both). In your proposal, if 'is_foo_compliant' raises an exception, it will not be caught by 'assert_valid', therefore the ValidationError will not be raised. So this is not what I want as an application developer. -- Sylvain -----Message d'origine----- De : Paul Moore [mailto:p.f.moore@gmail.com] Envoyé : mardi 16 janvier 2018 18:01 À : Sylvain MARIE <sylvain.marie@schneider-electric.com> Cc : Python-Ideas <python-ideas@python.org> Objet : Re: [Python-ideas] Repurpose `assert' into a general-purpose check I fixed the reply-to this time, looks like you're still getting messed up by Google Groups. On 16 January 2018 at 16:25, smarie <sylvain.marie@schneider-electric.com> wrote:
Why do you do this? What's the requirement for delaying evaluation of the condition? A validate statement in Python wouldn't be any better able to do that, so it'd be just as ugly with a statement. There's no reason I can see why I'd ever need delayed evaluation, so what's wrong with just assert_valid(0 <= surf < 10000 and is_foo_compliant(surf), help_msg="surface should be 0=<x<10000 or foo compliant")
So how will that work any differently than the function version I gave above? In terms of the language syntax, it would just be validate EXPR, EXPR and the first EXPR will be evaluated as a boolean, and if it's false an exception will be raised with the second expression as the message. There's no delayed evaluation involved, so a function would work exactly the same.
No more so than my function version, but yes far more so than yours... Paul ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. ______________________________________________________________________
On 16 January 2018 at 17:36, Sylvain MARIE <sylvain.marie@schneider-electric.com> wrote:
Ah, OK. But nothing in your proposal for a new statement suggests you wanted that, and assert doesn't work like that, so I hadn't realised that's what you were after. You could of course simply do: def assert_valid(expr, help_msg): # Catch exceptions in expr() as you see fit if not expr(): raise ValidationError(help_msg) assert_valid(lambda: 0 <= surf < 10000 and is_foo_compliant(surf), help_msg="surface should be 0=<x<10000 or foo compliant") No need for a whole expression language :-) Paul
Perhaps the OP can look into Python macro libraries to get the wanted syntax? https://github.com/lihaoyi/macropy On Tue, Jan 16, 2018 at 2:35 PM, Paul Moore <p.f.moore@gmail.com> wrote:
-- Juancarlo *Añez*
I think having a means for such validations separate from assertions would be helpful. However, I agree with Steven that 'validate' would be a bad keyword choice. Besides breaking compatibility with programs that use 'validate', it would break wsgiref.validate <https://docs.python.org/3.6/library/wsgiref.html#module-wsgiref.validate> in the standard library. ᐧ On Tue, Jan 16, 2018 at 2:22 PM, Juancarlo Añez <apalala@gmail.com> wrote:
On Wed, Jan 17, 2018 at 12:19:51PM -0500, Nikolas Vanderhoof wrote:
I think having a means for such validations separate from assertions would be helpful.
What semantics would this "validate" statement have, and how would it be different from what we can write now? if not condition: raise SomeException(message) validate condition, SomeException, message # or some other name Unless it does something better than a simple "if ... raise", there's not much point in adding a keyword just to save a few keystrokes. To justify a keyword, it needs to do something special that a built-in function can't do, like delayed evaluation (without wrapping the expression in a function). -- Steve
On 18 January 2018 at 07:46, Steven D'Aprano <steve@pearwood.info> wrote:
My reaction to these threads for a while has been "We should just add a function for unconditional assertions in expression form", and I finally got around to posting that to the issue tracker rather than leaving it solely in mailing list posts: https://bugs.python.org/issue32590 The gist of the idea is to add a new ensure() builtin along the lines of: class ValidationError(AssertionError): pass _MISSING = object() def ensure(condition, msg=_MISSING, exc_type=ValidationError): if not condition: if msg is _MISSING: msg = condition raise exc_type(msg) There's no need to involve the compiler if you're never going to optimise the code out, and code-rewriters like the one in pytest can be taught to recognise "ensure(condition)" as being comparable to an assert statement. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 17/01/2018 17:19, Nikolas Vanderhoof wrote:
To me it looks like this discussion has basically split into two separate use cases: 1. Using assert in a way that it will not (ever) get turned off. 2. The specific case of ensuring that a variable/parameter is an instance of a specific type. and I would like to suggest two separate possibly syntaxes that might make sense. 1. For asserts that should not be disabled we could have an always qualifier optionally added to assert, either as "assert condition exception always" or "assert always condition exception", that disables the optimisation for that specific exception. This would make it clearer that the developer needs this specific check always. Alternatively, we could consider a scoped flag, say keep_asserts, that sets the same. 2. For the specific, and to me more desirable, use case of ensuring type compliance how about an ensure keyword, (or possibly function), with a syntax of "ensure var type" or "ensure(var, type)" which goes a little further by attempting to convert the type of var to type and only if var cannot be converted raises a type exception. This second syntax could, of course, be implemented as a library function rather than a change to python itself. Either option could have an optional exception to raise, with the default being a type error. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer.
On Thu, Jan 18, 2018 at 4:21 PM, Steve Barnes <gadgetsteve@live.co.uk> wrote:
But if they're never to be compiled out, why do they need special syntax? assert always x >= 0, "x must be positive" can become if x < 0: raise ValueError("x must be positive") I haven't yet seen any justification for syntax here. The nearest I've seen is that this "ensure" action is more like: try: cond = x >= 0 except BaseException: raise AssertionError("x must be positive") else: if not cond: raise AssertionError("x must be positive") Which, IMO, is a bad idea, and I'm not sure anyone was actually advocating it anyway. ChrisA
Has something like this been proposed for Python before? This seems to align more with the intended use of assert that's been pointed out in this thread. In what case though would one want to disable some but not all of these pre, post, or invariant assertions? On Thu, Jan 18, 2018 at 2:12 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
ᐧ
On Thu, Jan 18, 2018 at 7:51 PM, Nikolas Vanderhoof < nikolasrvanderhoof@gmail.com> wrote:
Oh, many times, starting in the late '90s IIRC (Paul Dubois was a big fan). The problems are twofold: (a) it would require a lot of new keywords or ugly syntax; and (b) there would have to be a way to enable each form separately *per module or package*. Eiffel solves that (AFAIC) through separate compilation -- e.g. a stable version of a library might disable invariants and post-conditions but keep pre-conditions, since those could be violated by less mature application code; or a mature application could disable all checks and link with optimized library binaries that also have disabled all checks. I'm sure other scenarios are also viable. But that solution isn't available in Python, where command line flags apply to *all* modules being imported. (Note: even if you have a solution for (b), getting past (a) isn't so easy. So don't get nerd-sniped by the solution for (b) alone.) -- --Guido van Rossum (python.org/~guido)
On Thu, Jan 18, 2018 at 08:00:01PM -0800, Guido van Rossum wrote:
Indeed. I fear that Python's design will never be a good match for Eiffel's Design By Contract. Nevertheless I still have hope that there could be things we can learn from it. After all, DBC is as much a state of mind as it is syntax. Here's a blast from the past: https://www.python.org/doc/essays/metaclasses/ http://legacy.python.org/doc/essays/metaclasses/Eiffel.py This was my first introduction to the idea of software contracts! -- Steve
Indeed, I was the one advocating for it :) Based on all the feedback I received from this discussion, I realized that my implementation was completely flawed by the fact that I had done the class and functions decorators first, and wanted to apply the same pattern to the inline validator, resulting in this assert_valid with overkill delayed evaluation. Resulting in me saying that the only way out would be a new python language element. I tried my best to update valid8 and reached a new stable point with version 3.0.0, providing 2 main utilities for inline validation: - the simple but not so powerful `quick_valid` function - the more verbose (2 lines) but much more generic `wrap_valid` context manager (that's the best I can do today !) The more capable but delayed-evaluation based `assert_valid` is not recommended anymore, or just a tool to replicate what is done in the function and class validation decorators. Like the decorators, it adds the ability to blend two styles of base functions (boolean testers and failure raisers) with boolean operators seamlessly. But the complexity is not worth it for inline validation (it seems to be worth it for decorators). See https://smarie.github.io/python-valid8 for the new updated documentation. I also updated the problem description page at https://smarie.github.io/python-valid8/why_validation/ so as to keep a reference of the problem description and "wishlist" (whether it is implemented by this library or by new language elements in the future). Do not hesitate to contribute or send me your edits (off-list). I would love to get feedback from anyone concerning this library, whether you consider it's useless or "interesting but...". We should probably take this offline though, so as not to pollute the initial thread. Thanks again, a great weekend to all (end of the day here in france ;) ) Kind regards Sylvain
Thanks Paul Le mardi 16 janvier 2018 16:56:57 UTC+1, Paul Moore a écrit :
Why does this need to be a statement at all? Unlike assert, it's always executed, so it can be defined as a simple function
Very good point. Actually that's what I already provide in valid8 with the assert_valid function. See all examples here <https://smarie.github.io/python-valid8/#usage-examples-inline>. Let's consider this example where users want to define on-the-fly one of the validation functions, and combine it with another with a 'or': assert_valid('surface', surf, or_(lambda x: (x >= 0) & (x < 10000), is_foo_compliant), help_msg="surface should be 0=<x<10000 or foo compliant") How ugly for something so simple ! I tried to make it slightly more compact by developping a mini lambda syntax but it obviously makes it slower. Anyway it becomes assert_valid('surface', surf, or_((x >= 0) & (x < 10000), is_foo_compliant), help_msg="surface should be between 0 and 10000 or foo compliant") or even (if you pre-convert Is_foo_compliant to mini_lambda) assert_valid('surface', surf, ((x >= 0) & (x < 10000)) | Is_foo_compliant), help_msg="surface should be between 0 and 10000 or foo compliant") There are three reasons why having a 'validate' statement would improve this: * no more parenthesis: more elegant and readable * inline use of python (1): no more use of lambda or mini_lambda, no performance overhead * inline use of python (2): composition would not require custom function composition operators such as 'or_' (above) or mini-lambda composition anymore, it could be built-in in any language element used after <validate> resulting in validate (surf >= 0) & (surf < 10000) or is_foo_compliant(surf), "surface should be 0=<x<10000 or foo compliant" (I removed the variable name alias 'surface' since I don't know if it should remain or not) Elegant, isn't it ?
I don't see how a validate statement avoids having to deal with all of
the complexity you mention here.
It would obviously need to be quite smart :) but it is possible, since I can do it with assert_valid today.
I fixed the reply-to this time, looks like you're still getting messed up by Google Groups. On 16 January 2018 at 16:25, smarie <sylvain.marie@schneider-electric.com> wrote:
Why do you do this? What's the requirement for delaying evaluation of the condition? A validate statement in Python wouldn't be any better able to do that, so it'd be just as ugly with a statement. There's no reason I can see why I'd ever need delayed evaluation, so what's wrong with just assert_valid(0 <= surf < 10000 and is_foo_compliant(surf), help_msg="surface should be 0=<x<10000 or foo compliant")
So how will that work any differently than the function version I gave above? In terms of the language syntax, it would just be validate EXPR, EXPR and the first EXPR will be evaluated as a boolean, and if it's false an exception will be raised with the second expression as the message. There's no delayed evaluation involved, so a function would work exactly the same.
No more so than my function version, but yes far more so than yours... Paul
(trying with direct reply this time)
Why do you do this? What's the requirement for delaying evaluation of the condition?
Thanks for challenging my poorly chosen examples :) The primary requirement is about *catching* unwanted/uncontrolled/heterogenous exceptions happening in the underlying functions that are combined together to provide the validation means, so as to provide a uniform/consistent outcome however diverse the underlying functions are (they can return booleans or raise exceptions, or both). In your proposal, if 'is_foo_compliant' raises an exception, it will not be caught by 'assert_valid', therefore the ValidationError will not be raised. So this is not what I want as an application developer. -- Sylvain -----Message d'origine----- De : Paul Moore [mailto:p.f.moore@gmail.com] Envoyé : mardi 16 janvier 2018 18:01 À : Sylvain MARIE <sylvain.marie@schneider-electric.com> Cc : Python-Ideas <python-ideas@python.org> Objet : Re: [Python-ideas] Repurpose `assert' into a general-purpose check I fixed the reply-to this time, looks like you're still getting messed up by Google Groups. On 16 January 2018 at 16:25, smarie <sylvain.marie@schneider-electric.com> wrote:
Why do you do this? What's the requirement for delaying evaluation of the condition? A validate statement in Python wouldn't be any better able to do that, so it'd be just as ugly with a statement. There's no reason I can see why I'd ever need delayed evaluation, so what's wrong with just assert_valid(0 <= surf < 10000 and is_foo_compliant(surf), help_msg="surface should be 0=<x<10000 or foo compliant")
So how will that work any differently than the function version I gave above? In terms of the language syntax, it would just be validate EXPR, EXPR and the first EXPR will be evaluated as a boolean, and if it's false an exception will be raised with the second expression as the message. There's no delayed evaluation involved, so a function would work exactly the same.
No more so than my function version, but yes far more so than yours... Paul ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. ______________________________________________________________________
On 16 January 2018 at 17:36, Sylvain MARIE <sylvain.marie@schneider-electric.com> wrote:
Ah, OK. But nothing in your proposal for a new statement suggests you wanted that, and assert doesn't work like that, so I hadn't realised that's what you were after. You could of course simply do: def assert_valid(expr, help_msg): # Catch exceptions in expr() as you see fit if not expr(): raise ValidationError(help_msg) assert_valid(lambda: 0 <= surf < 10000 and is_foo_compliant(surf), help_msg="surface should be 0=<x<10000 or foo compliant") No need for a whole expression language :-) Paul
Perhaps the OP can look into Python macro libraries to get the wanted syntax? https://github.com/lihaoyi/macropy On Tue, Jan 16, 2018 at 2:35 PM, Paul Moore <p.f.moore@gmail.com> wrote:
-- Juancarlo *Añez*
I think having a means for such validations separate from assertions would be helpful. However, I agree with Steven that 'validate' would be a bad keyword choice. Besides breaking compatibility with programs that use 'validate', it would break wsgiref.validate <https://docs.python.org/3.6/library/wsgiref.html#module-wsgiref.validate> in the standard library. ᐧ On Tue, Jan 16, 2018 at 2:22 PM, Juancarlo Añez <apalala@gmail.com> wrote:
On Wed, Jan 17, 2018 at 12:19:51PM -0500, Nikolas Vanderhoof wrote:
I think having a means for such validations separate from assertions would be helpful.
What semantics would this "validate" statement have, and how would it be different from what we can write now? if not condition: raise SomeException(message) validate condition, SomeException, message # or some other name Unless it does something better than a simple "if ... raise", there's not much point in adding a keyword just to save a few keystrokes. To justify a keyword, it needs to do something special that a built-in function can't do, like delayed evaluation (without wrapping the expression in a function). -- Steve
On 18 January 2018 at 07:46, Steven D'Aprano <steve@pearwood.info> wrote:
My reaction to these threads for a while has been "We should just add a function for unconditional assertions in expression form", and I finally got around to posting that to the issue tracker rather than leaving it solely in mailing list posts: https://bugs.python.org/issue32590 The gist of the idea is to add a new ensure() builtin along the lines of: class ValidationError(AssertionError): pass _MISSING = object() def ensure(condition, msg=_MISSING, exc_type=ValidationError): if not condition: if msg is _MISSING: msg = condition raise exc_type(msg) There's no need to involve the compiler if you're never going to optimise the code out, and code-rewriters like the one in pytest can be taught to recognise "ensure(condition)" as being comparable to an assert statement. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 17/01/2018 17:19, Nikolas Vanderhoof wrote:
To me it looks like this discussion has basically split into two separate use cases: 1. Using assert in a way that it will not (ever) get turned off. 2. The specific case of ensuring that a variable/parameter is an instance of a specific type. and I would like to suggest two separate possibly syntaxes that might make sense. 1. For asserts that should not be disabled we could have an always qualifier optionally added to assert, either as "assert condition exception always" or "assert always condition exception", that disables the optimisation for that specific exception. This would make it clearer that the developer needs this specific check always. Alternatively, we could consider a scoped flag, say keep_asserts, that sets the same. 2. For the specific, and to me more desirable, use case of ensuring type compliance how about an ensure keyword, (or possibly function), with a syntax of "ensure var type" or "ensure(var, type)" which goes a little further by attempting to convert the type of var to type and only if var cannot be converted raises a type exception. This second syntax could, of course, be implemented as a library function rather than a change to python itself. Either option could have an optional exception to raise, with the default being a type error. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer.
On Thu, Jan 18, 2018 at 4:21 PM, Steve Barnes <gadgetsteve@live.co.uk> wrote:
But if they're never to be compiled out, why do they need special syntax? assert always x >= 0, "x must be positive" can become if x < 0: raise ValueError("x must be positive") I haven't yet seen any justification for syntax here. The nearest I've seen is that this "ensure" action is more like: try: cond = x >= 0 except BaseException: raise AssertionError("x must be positive") else: if not cond: raise AssertionError("x must be positive") Which, IMO, is a bad idea, and I'm not sure anyone was actually advocating it anyway. ChrisA
Has something like this been proposed for Python before? This seems to align more with the intended use of assert that's been pointed out in this thread. In what case though would one want to disable some but not all of these pre, post, or invariant assertions? On Thu, Jan 18, 2018 at 2:12 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
ᐧ
On Thu, Jan 18, 2018 at 7:51 PM, Nikolas Vanderhoof < nikolasrvanderhoof@gmail.com> wrote:
Oh, many times, starting in the late '90s IIRC (Paul Dubois was a big fan). The problems are twofold: (a) it would require a lot of new keywords or ugly syntax; and (b) there would have to be a way to enable each form separately *per module or package*. Eiffel solves that (AFAIC) through separate compilation -- e.g. a stable version of a library might disable invariants and post-conditions but keep pre-conditions, since those could be violated by less mature application code; or a mature application could disable all checks and link with optimized library binaries that also have disabled all checks. I'm sure other scenarios are also viable. But that solution isn't available in Python, where command line flags apply to *all* modules being imported. (Note: even if you have a solution for (b), getting past (a) isn't so easy. So don't get nerd-sniped by the solution for (b) alone.) -- --Guido van Rossum (python.org/~guido)
On Thu, Jan 18, 2018 at 08:00:01PM -0800, Guido van Rossum wrote:
Indeed. I fear that Python's design will never be a good match for Eiffel's Design By Contract. Nevertheless I still have hope that there could be things we can learn from it. After all, DBC is as much a state of mind as it is syntax. Here's a blast from the past: https://www.python.org/doc/essays/metaclasses/ http://legacy.python.org/doc/essays/metaclasses/Eiffel.py This was my first introduction to the idea of software contracts! -- Steve
Indeed, I was the one advocating for it :) Based on all the feedback I received from this discussion, I realized that my implementation was completely flawed by the fact that I had done the class and functions decorators first, and wanted to apply the same pattern to the inline validator, resulting in this assert_valid with overkill delayed evaluation. Resulting in me saying that the only way out would be a new python language element. I tried my best to update valid8 and reached a new stable point with version 3.0.0, providing 2 main utilities for inline validation: - the simple but not so powerful `quick_valid` function - the more verbose (2 lines) but much more generic `wrap_valid` context manager (that's the best I can do today !) The more capable but delayed-evaluation based `assert_valid` is not recommended anymore, or just a tool to replicate what is done in the function and class validation decorators. Like the decorators, it adds the ability to blend two styles of base functions (boolean testers and failure raisers) with boolean operators seamlessly. But the complexity is not worth it for inline validation (it seems to be worth it for decorators). See https://smarie.github.io/python-valid8 for the new updated documentation. I also updated the problem description page at https://smarie.github.io/python-valid8/why_validation/ so as to keep a reference of the problem description and "wishlist" (whether it is implemented by this library or by new language elements in the future). Do not hesitate to contribute or send me your edits (off-list). I would love to get feedback from anyone concerning this library, whether you consider it's useless or "interesting but...". We should probably take this offline though, so as not to pollute the initial thread. Thanks again, a great weekend to all (end of the day here in france ;) ) Kind regards Sylvain
participants (11)
-
Chris Angelico -
Ethan Furman -
Guido van Rossum -
Juancarlo Añez -
Nick Coghlan -
Nikolas Vanderhoof -
Paul Moore -
smarie -
Steve Barnes -
Steven D'Aprano -
Sylvain MARIE