Add @parametrize decorator to unittest library
When writing some unit tests with the standard Unittest library, I missed being able to create parameterized tests. This functionality exists in PyTest (https://docs.pytest.org/en/6.2.x/parametrize.html) and there is also a library called *parameterized*(https://github.com/wolever/parameterized) which aims to add this functionality. However, I think it would be beneficial to have a decorator in Python's standard Unittest library. Note: If this functionality exists natively in Python, please put some example or documentation link below. Thanks in advance.
Hi Leonardo, On Tue, Sep 07, 2021 at 02:31:26AM -0000, Leonardo Freua wrote:
When writing some unit tests with the standard Unittest library, I missed being able to create parameterized tests.
Could you please explain what you mean by "parameterized tests", and how you would use a decorator for it? What you mean by it may not be what other people understand it to be. -- Steve
I believe he is looking for something like pytest’s parameterize: https://docs.pytest.org/en/6.2.x/example/parametrize.html This is actually pretty basic functionality for writing DRY tests, a key missing feature. Frankly though, unittest is painfully unpythonic and hard to extend. I’ve given up. Just use pytest. -CHB On Mon, Sep 6, 2021 at 7:56 PM Steven D'Aprano <steve@pearwood.info> wrote:
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
I won't propose a syntax, but I think it would be useful if *assert* could raise an exception different from *AssertionError*. This is in the context of "Design by contrast" (DBC) as a useful companion to "Test-driven development" and other forms of external tests. Basically, the proposal is to allow for an exception type and value to be specified in *assert*, instead of the customary: if not assertion: raise ValueError('a message') Perhaps the assertion would not go away with *-OO* when an exception type is specified? From my experience, assertions are useful in most non-trivial cases, and should be there on any complex case. Perhaps we'd use more assertions if they become easier to use and more customizable? The agile trend has been in favor of external tests (like unit tests). I think that we should not forget about the value of software that verifies its state at runtime through assertions. An SO reference to the basic inquiry: https://stackoverflow.com/questions/1569049/making-pythons-assert-throw-an-e... -- Juancarlo *Añez*
On 2021-09-07 11:12:37, Juancarlo Añez wrote:
What's wrong with:
From what I can tell, this is optimised away when optimisations are turned on. Maybe wrap that in a function?
Perhaps the assertion would not go away with *-OO* when an exception type is specified?
Isn't the whole point of assertions that they go away with optimisations? If the code cannot be optimised away, raise the exception at runtime, it's just regular code.
On Tue, Sep 7, 2021 at 8:37 AM Simão Afonso < simao.afonso@powertools-tech.com> wrote:
what are you proposing instead? Maybe: assert assertion, raise ValueError("a message") (currently a syntax error) in what way is that better than your original: if not assertion: raise ValueError("a message") or even: if not assertion: raise ValueError("a message") what is the value in the "assert" statement? So why do we have athe at all? Because they ARE turned off by -OO :-) -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Tue, Sep 07, 2021 at 11:12:37AM -0400, Juancarlo Añez wrote:
I don't see why that would be useful. DBC assertions are assertions. You are *asserting* that a condition is always true. Since it is always true, it should be safe to disable those DBC assertion checks once your software is in production. I could *maybe* see that having fine distinction between pre-condition, post-condition and invariant failures would be useful, but without a system in place to allow those to be globally enabled/disabled separately, what's the point? In any case, in the event of a contract failure, there's really nothing you can do except log the error and exit. Raising errors like TypeError etc will encourage people to abuse assertions and contracts by catching those exceptions, for checking caller-supplied parameters and user data, or for flow control. -- Steve
Steven, The purpose is to make it easier to make software more resilient. The inspiration was this article that reminded me that software *_will always fail_*, and also reminded me about all the discussions around DBC and Eiffel: https://www.youtube.com/watch?v=AaZ_RSt0KP8 IOW, my premise is that we should be using *_lots_* of assertions, *_always_*, and that for that we need to make them easier to write, and easier to handle in the event of the unavoidable failures. Seeking unit-test coverage is not enough because unit tests don't run in production. I will concede that code written under the *"Python culture"* tends to be resilient because the semantics of defaults and border conditions are explicit in the documentation, and implemented thus. Perhaps it's enough to allow for: *assert **cond**, *ExType(args) On Tue, Sep 7, 2021 at 9:28 PM Steven D'Aprano <steve@pearwood.info> wrote:
-- Juancarlo *Añez*
Let me re-*assert* ;-) External tests will not be invoked at runtime, yet failures *_will_* occur at runtime because of a variety of environmental factors (including cosmic rays). Software should assert at least some of its preconditions, postconditions, and invariants. How can we make that easy and syntactically pleasant? On Thu, Sep 9, 2021 at 12:02 PM Juancarlo Añez <apalala@gmail.com> wrote:
-- Juancarlo *Añez*
Ooh, that’s a nice idea. If the message is an exception instance, raise it instead of AssertionError. Unfortunately it’s not 100% backwards compatible. We could address that with the syntax assert cond, raise=ExcType(args) Maybe we could deprecate the case assert cond, ExcType(args) So that eventually the raise= keyword can become optional. —Guido On Thu, Sep 9, 2021 at 09:04 Juancarlo Añez <apalala@gmail.com> wrote:
-- --Guido (mobile)
Well, if the idea makes sense, then I'm certain that we'll have a very long and productive discussion about the best syntax here (re: *:=*). ;-) For backwards compatibility and no surprises: *assert: *ExType, cond, args It doesn't break anything, ExtType defaults to AssertionError, and linters can check that *args* match ExType. A more readable and pythonic syntax would be: *assert *cond: ExtType(args) Forgoing the comma doesn't break anything, linters can check the ExType() typing, and the semantics would be those of: *if* __debug__ *and* *not *cond: *raise* ExType(args) Although I would drop the check for *__debug__* if an explicit ExtType is given. It's similar to the changes introduced for: *except* (OneEx, TwoEx): On Thu, Sep 9, 2021 at 12:16 PM Guido van Rossum <guido@python.org> wrote:
-- Juancarlo *Añez*
On 2021-09-09 22:31, Juancarlo Añez wrote:
Or, perhaps: assert cond: raise ExtType(args) That would make it more like an if and would leave open the possibility of extending it to accept multiple lines: assert cond: # Write some debugging info to a log. ... raise ExtType(args) Just a thought.
assert cond: # Write some debugging info to a log. ... raise ExtType(args) I like the idea of preparing the arguments for the assertion message in a context that gets executed only when the assertion fails. On Thu, Sep 9, 2021 at 7:16 PM MRAB <python@mrabarnett.plus.com> wrote:
-- Juancarlo *Añez*
Juancarlo Añez writes:
The more I see this elaborated, the less I like it. "assert" has a traditional meaning in programming, which is only a slight extension on the English meaning (that you can turn the check off for performance). Although the idea that assert could be the fundamental building block for design-by-contract (DBC), to my mind this has three problems. (1) You're coopting a keyword that has a consistent established meaning in English, in programming languages in general, and in Python, and changing that meaning. (2) I doubt that "assert" alone is sufficient for DBC. In practice, you'll want a more expressive, larger vocabulary. It's not clear that *any* of that vocabulary is best expressed by "assert" at this point. If not, why mess with what works? (3) There's a subtle difference between an assertion and a contract. An assertion is a statement of fact: it's either right or it's wrong. But the whole point of contracts in the real world is that they are constraints on behavior to be enforced. The assumption is that it's possible to be in violation of the contract. The point of *asserting* something is that you believe it's absolutely true, there's no point in going on if it's false. You have to start over. A presumption that violation is possible may not be true of the contracts in DBC, I don't know. But I would prefer not to mix the difficulties of the semantics of "contract" with the simplicity of the semantics of "assert". Steve
Stephen, Assertions have their roots in mathematics, as does most everything in programming languages. I'm happy about dropping the DBC theme and rebooting to make *assert* easier to use so it gets used more. I liked the suggestion of: *assert *cond: # prepare failure information *raise *ExType(args) Because the current: *assert *cond, arg Will always raise AssertionError, and do a str(arg), which means that *arg* must be simple, or rely on a function that creates more complex reporting. It think it would be fine if ExtType->AssertionError is enforced. Then there could be a hierarchy of AssertionError depending on the type of problem or its severity. Of course, none of this will make sense to programmers with a strong belief that assertions must always be turned off in production runs. I, on the other hand, believe that any complex computation or algorithm should assert its expected state and abort if the expectations are not met, even when the reason for the failure is unknown. Think about financial, insurance, transportation, or medical algorithms. By epistemological reasons a failure caused by a bug is almost *_always_* unexpected, and the reasons for the failure are unknown at runtime (a shorter version is that "programmers don't write bugs on purpose, so bugs are always unforseen"). On Sat, Sep 11, 2021 at 12:32 AM Stephen J. Turnbull < stephenjturnbull@gmail.com> wrote:
-- Juancarlo *Añez*
It's fair to note that complex arguments to assertions can be hacked with: *assert* cond, ( f'{conputesomething} {and more}' f''{some more}' ) The exception type can't be changed, though. On Sat, Sep 11, 2021 at 9:17 AM Juancarlo Añez <apalala@gmail.com> wrote:
-- Juancarlo *Añez*
On 11.09.2021 15:17, Juancarlo Añez wrote:
Of course, none of this will make sense to programmers with a strong belief that assertions must always be turned off in production runs.
You seem to be missing that the Python optimize mode turns off all code which is related to debugging (__debug__ is set to False, the compiler doesn't generate code for "if __debug__: ..." statements). assert is just one of the instances where this happens: https://docs.python.org/3/reference/simple_stmts.html#grammar-token-assert-s... asserts are meant to help find bugs in programs, not check for user input errors. They document assumptions a programmer has made when writing the code, which are then tested with a test suite to make sure the assumptions hold as invariants of the application. For anything which can go wrong at production run-time, please use normal if-statement checks and raise appropriate exceptions. Using assert in such cases is dangerous and can render your application broken, while everything appears to be running fine when testing. Unlike regular if-statement checks, asserts are meant to never trigger an exception in production code -- which is why they are removed with -O. This is not about whether or not to use -O in production environments, it's about preventing user input exceptions from going unnoticed when code is run with -O (e.g. the deployment team or a user decided to always use -O in production). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Sep 11 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
Marc-Andre, I must agree that the current state of assertions with "-O" and "-OO" is difficult to impossible to change. Perhaps I'm looking for an: *invariant* cond: etc Regards, On Sat, Sep 11, 2021 at 11:00 AM Marc-Andre Lemburg <mal@egenix.com> wrote:
-- Juancarlo *Añez*
On Sat, Sep 11, 2021 at 02:30:10PM -0400, Juancarlo Añez wrote:
*invariant* cond: etc
A software invariant is still an assertion. In another post, I semi-suggested a new (soft) keyword: unless condition: # block raise Exception But really, it's just an "if not". if not condition: unless condition: assert condition: are all exactly the same amount of typing. It's not clear that learning a new keyword `unless` is more readable than just using `if not`. I think it might work better in languages that attempt to follow natural language (e.g. XTalk languages) than one like Python with a relatively minimal amount of syntax. But it is clear to me that using `assert` for non-assertions is misleading. -- Steve
Steve, I've seen *unless* discussed and rejected before, but it is a good enough syntax for the purpose of asserting invariants, and it should be useful for other purposes (because it has been proposed before). The semantics are clear, and the implementation is simple with the new Python parser shortcutting to an "*if not"*. I do really like this option. On Sun, Sep 12, 2021 at 1:46 AM Steven D'Aprano <steve@pearwood.info> wrote:
-- Juancarlo *Añez*
On Sat, Sep 11, 2021 at 9:20 AM Juancarlo Añez <apalala@gmail.com> wrote:
I'm happy about dropping the DBC theme and rebooting to make *assert* easier to use so it gets used more.
I agree with Steven, Marc-Andé, and others in seeing "using assertions more" as an anti-goal. This isn't to say that a given program should have fewer—nor more—lines that start with `assert`. Rather, assertions should be use to ASSERT conditions that should NEVER be violated. They are NOT to check whether user input is bad. Nor whether a disk is unreliable. Nor whether a computation takes too long. Nor whether a value is outside of a useful range. Nor any of the other things that regular exception handling is well designed for. If for some reason you find yourself completely unable to write "if not (condition_to_check): ..." for mysterious reasons, you could use assert this way (but don't):
But again, that's bad code for the same reason your proposal would be bad code. Assertions have the entire purpose of being possible to disable, and to express conditions a programmer believes are *necessarily true*. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
David, It seems I didn't state clearly enough my original statement, which is that software will *_always_ *fail, even because of faulty hardware components, or cosmic rays. For software to be resilient, it must assert it's expected state. But that doesn't have to be with the *assert* statement, perhaps less so when the Python tradition is that those go away with just a "-O". I won't vouch for an *"invariant"* statement, because that's probably too much to ask for. So I'll just use *"if"* to assert more invariants in the code I write. Cheers, On Sat, Sep 11, 2021 at 2:05 PM David Mertz, Ph.D. <david.mertz@gmail.com> wrote:
-- Juancarlo *Añez*
On Sat, Sep 11, 2021 at 02:37:25PM -0400, Juancarlo Añez wrote:
If you expect your software to be resilient given faulty hardware, using exceptions is not the way to do it. If the hardware is faulty, you can't rely on either the condition test (whether written with assert or not) or the recovery block to behave correctly. Hardware failure resilient software has to be designed from the very lowest levels of machine code. Given faulty hardware or cosmic-ray flipping of bits in memory, you are far more likely to get an interpreter seg fault than a failure you can test for and handle. Or an OS level failure. Machine crashing or locking up is way more likely than anything you can handle in Python. -- Steve
On Thu, Sep 09, 2021 at 09:16:23AM -0700, Guido van Rossum wrote:
assert cond, raise=ExcType(args)
How about this? if not cond: raise ExcType(args) This is 100% backwards compatible, doesn't not abuse `assert`, will not be removed when running under -O, and takes exactly the same number of characters to type. If we are seriously considering DBC for Python, here's is my initial proposal for function and method contracts. I don't have a proposal for class invariants as yet. def function(args): """Docstring""" require: # Preconditions checked before calling the function body. block ensure(result): # Postconditions checked after calling the function body. block # body of function `require` and `ensure` would be soft-keywords, and legal only inside a `def` block, after the docstring and before the body of the function. If present, the `require` block is executed before the body of the function. The precondition has access to the function arguments, but otherwise runs in its own namespace. Since it runs before the body, none of the body's local variables will be defined (apart from the function arguments). If present, the `ensure` block is executed after the body of the function. It also runs in its own namespace, but has access to the function's body namespace. It also receives the function body's return result as a named argument. The parameter name "result" is not mandatory. Both contract blocks are expected to either raise AssertionError, or just return. Raising other exceptions is permitted but discouraged. Catching these exceptions is permitted but discouraged except for the purpose of logging. When running under -O, both condition blocks are ignored. (At some time in the future we may add a fine-grained interface to disable them.) I have chosen the names `require` and `ensure` to match the names used by Eiffel. https://www.eiffel.org/doc/solutions/Design_by_Contract_and_Assertions If the `require` and `ensure` blocks are all assertions, we can treat the blocks as executable *specifications documention*: def function(spam, eggs, cheese, aardvark): require: assert isinstance(spam, Food) assert -1 <= eggs < MAXEGGS assert cheese.colour = 'blue' assert aardvark is not None # function body That should allow third-party checkers and tools to either do static pre- and post-condition checking, or to generate documentation from the conditions. -- Steve
On Fri, Sep 10, 2021 at 7:30 AM Steven D'Aprano <steve@pearwood.info> wrote:
[...] If we are seriously considering DBC for Python, [...]
As I think I wrote in the previous thread about DBC in Python, there are already several DBC libraries on PyPI and we should precisely evaluate both their design and impact (e.g. usage). One of the most recent / best designed ones seems to be Deal by Orsinium: https://github.com/life4/deal The Wikipedia page for DBC lists a few others: "Python, using packages like deal, icontract, PyContracts, Decontractors, dpcontracts, zope.interface, PyDBC or Contracts for Python. A permanent change to Python to support design by contracts was proposed in PEP-316, but deferred." here's is my initial
proposal for function and method contracts. I don't have a proposal for class invariants as yet.
Deal (and similar, older, libraries) addresses similar needs, within the context of the existing Python syntax. So yes, syntax can be improved. But : 1) Do people (at least a significant part) want DBC in their Python code. Is DBC still a popular idea in the software industry ? (My 2 cts: I don't believe so). 2) Is one of the existing approach (i.e. through a library) enough for these persons, or do we really need to change the language ? (I don't know, we need people who actually do DBC with existing libraries to tell us). 3) I think there is more to DBC that simply checking preconditions and postconditions and invariants at runtime, just as typechecking Python is a bit more work than sprinkling your code with "assert isinstance(something, SomeClass)". Just as for type systems, DBC becomes really useful when you have a theorem prover that can prove (or refute) the consistency of your DBC constraints. This is the hard part (IMHO syntax is just the cherry on the cake). Cheers, S. -- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Co-Founder & Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Co-Founder & Board Member, Association Professionnelle Européenne du Logiciel Libre (APELL) - https://www.apell.info/ Co-Founder & Spokesperson, European Cloud Industrial Alliance (EUCLIDIA) - https://www.euclidia.eu/ Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ & http://pydata.fr/
And 4) DBC constraints also become more useful than standard assertions when used in the context of a class hierarchy. I.e. preconditions, postconditions and invariants should be stronger (or sometimes weaker, IIRC, there are subtle rules related to variance and contravariance) for subclasses than for superclasses. This is another subject that should be dealt with if we want the true power of DBC in Python. S. On Fri, Sep 10, 2021 at 9:26 AM Stéfane Fermigier <sf@fermigier.com> wrote:
-- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Co-Founder & Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Co-Founder & Board Member, Association Professionnelle Européenne du Logiciel Libre (APELL) - https://www.apell.info/ Co-Founder & Spokesperson, European Cloud Industrial Alliance (EUCLIDIA) - https://www.euclidia.eu/ Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ & http://pydata.fr/
Take a look at the archives of this list -- there was a large conversation about DBC a while back (a year, two years ??? ) I think if you really want to support DBC, there will need to be more changes than this -- though there are libraries that support it with current Python. Also, let's be clear about hte language re use -- when you say: there should be lots of assertions, do you mean the english were assertion, or the Python keyword assert? If the former, than you are free, or course, to use a ordinary if and raise to make all sort sof assertions about the code at runtime. To me -- the entire point of the assert statement is that it is for testing, and can be turned off. If you want to check a condition always, just use an if and a raise: How is this: if value < 0: raise ValueError("this only works with positive numbers") Any more difficult to read or write than: assert value >= 0, raise ValueError("this only works with positive numbers") -CHB On Thu, Sep 9, 2021 at 9:05 AM Juancarlo Añez <apalala@gmail.com> wrote:
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Thu, Sep 9, 2021 at 12:38 PM Christopher Barker <pythonchb@gmail.com> wrote:
I have never heard of DBC and don't have a clue what is stands for. I am not a pro software developer. But I would read these two lines differently, semantically. The if version feels like someone is saying "I am checking for a thing that could happen". The assert version would feel more like "maybe this could happen and I am checking for it because I am scared not to let's just put it here just in case", or said another way "it is imperative that this not happen". Not sure if that is an argument for or against, and maybe I'm the only one who would see the semantics differently. --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
On Thu, Sep 09, 2021 at 12:53:34PM -0400, Ricky Teachey wrote:
I have never heard of DBC and don't have a clue what is stands for. I am not a pro software developer.
DBC stands for "Design By Contract", it is a methodology for developing software. https://en.wikipedia.org/wiki/Design_by_contract The language Eiffel is especially know for DBC, and it has contract testing built into the syntax. https://www.eiffel.com/values/design-by-contract/introduction/ https://www.eiffel.org/doc/solutions/Design_by_Contract_and_Assertions
Indeed. Assertions are statements about the expected state of the program, not queries about the current state. https://import-that.dreamwidth.org/676.html -- Steve
Christopher, The *if* version has the condition inverted, and it's structure makes it look as part of the algorithm, instead of an assertion over an invariant. Following with the ideas on this thread, it would be possible for -OO to turn off only assertions with exceptions that descend from AssertionError. More on my answer to Steven On Thu, Sep 9, 2021 at 12:35 PM Christopher Barker <pythonchb@gmail.com> wrote:
-- Juancarlo *Añez*
On Thu, Sep 09, 2021 at 12:02:13PM -0400, Juancarlo Añez wrote:
Steven,
The purpose is to make it easier to make software more resilient.
Thank you but I understand the purpose of DBC and how it can be helpful for writing resilient software. If you search the archives, you will see that not only have Eiffel-style contracts been discussed before, but I am a great proponent of them. What I question is that allowing assert to raise non-assertions will lead to *more* resilient software rather than less. I know far too many people who insist on abusing assertions as a lazy way to validate caller- or user-supplied data, which is **not** a contract. Contracts apply only to calls between parts of your own application. Contracts do not apply to: - checking external data, which can be corrupt or invalid; - checking data provided by the end-user, who can never be trusted to keep to the contract; - checking input in library functions that may be called by other libraries or programs that are not a party to the contract; but I see too many people doing exactly that because it saves them a line of code. And because they are abusing assert, that means that their software is *less* resilient, because it will fail to perform those checks if the end-user runs it with assertions disabled. If `assert` can be used to raise *any* sort of exception, not just AssertionError, that will encourage that abuse and lead to less resilient software, not more.
IOW, my premise is that we should be using *_lots_* of assertions, *_always_*, and that for that we need to make them easier to write,
How can assertions be easier to write than they are now? assert condition You don't get any easier than that. That is the bare minimum needed for contract checking.
and easier to handle in the event of the unavoidable failures.
If you believe that contract violations are *unavoidable* then you have misunderstood the purpose of DBC. Contracts apply between parts of your application. A contract violation is a *bug in your application* and so is completely avoidable. (Except in the sense that we are all imperfect and therefore all software contains bugs, nevertheless we can asymptopically approach the status of being free from bugs.) Contracts do not apply between libraries and their callers, or between software and end-users. That would be rude and hostile, and the opposite of resilient. *Data and input validation* errors are unavoidable: there will always be corrupt files, incorrect data, arguments passed in the wrong order, etc. But they are not contracts and should not be tested for using assert because we cannot ever safely disable that validation. The point of assertions and contracts is that you *don't catch them*. Assertions and contracts are hard requirements. If the contract is violated you have a bug in your software and the only safe thing to do is to log the error and gracefully exit. If you can catch and recover from an error, its not a contract violation. It's just using exceptions for flow control. # Exceptions as flow control. # LBYL if condition: do_this() else: do_that() # EAFP try: do_this() except ConditionFailure: do_that()
Seeking unit-test coverage is not enough because unit tests don't run in production.
If you do DBC correctly, contract assertions don't run in production either. It is up to the end-user whether the contracts run or not. (I believe that the Eiffel community tends to recommend that pre- condition testing remain in production, and that post-conditions, invariants and other contracts be disabled, but that's entirely under the control of the end-user.) One simple test for whether something can be a contract or not is this: Could it be safe for me to remove this test? Not necessarily today, but some day, in the future, when I am confident that the code is bug free. If your answer is Yes, then it could be a contract written with assert. But if the answer is No, or I Don't Know, then it cannot be a contract, and it should not be written using assert. -- Steve
On 10.09.2021 05:49, Steven D'Aprano wrote:
I concur. asserts are meant for verifying assumptions the code designer and wants to verify *during development*. In C they lead to a core dump which aids in finding the cause of the problem. In Python, an AssertionError is raised with the same intent. In Python, using assert for anything that is not development related is dangerous, since production code running with -O (optimized mode) will not even run those assertions - for a good reason: they are meant only for checking assumptions in development. For any non-development related error checking, normal if statements should be used, not assert. I very often see code bases, which abuse assert as a quick way to check types, ranges, valid flag combinations, etc. Since all these things can go wrong in production as well, those checks need to be handled with if statements, not assert. If we'd not allow assert to also raise non-AssertionErrors, we'd get even more abuse. In fact, I'd be in favor of deprecating assert altogether, if it were not for pytest using it for testing - which is a valid use case, since those tests are not run in production, but again, most likely leads to programmers thinking that they can use the same logic in the actual production code. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Sep 10 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On Tue, 2021-09-07 at 02:31 +0000, Leonardo Freua wrote:
Hi Leonardo, Please check out subtests, they allow you to achieve what you are looking for :) https://docs.python.org/3/library/unittest.html#distinguishing-test-iteratio... Cheers, Filipe Laíns
unittest.subTest, hypothesis @given + pytest From https://docs.python.org/3/library/unittest.html#distinguishing-test-iteratio... : ``` When there are very small differences among your tests, for instance some parameters, unittest allows you to distinguish them inside the body of a test method using the subTest() context manager. For example, the following test: class NumbersTest(unittest.TestCase): def test_even(self): """ Test that numbers between 0 and 5 are all even. """ for i in range(0, 6): with self.subTest(i=i): self.assertEqual(i % 2, 0) ``` https://hypothesis.readthedocs.io/en/latest/quickstart.html#writing-tests : Think of a normal unit test as being something like the following: 1. Set up some data. 2. Perform some operations on the data. 3. Assert something about the result. Hypothesis lets you write tests which instead look like this: 1. For all data matching some specification. 2. Perform some operations on the data. 3. Assert something about the result. This is often called property-based testing, and was popularised by the Haskell library Quickcheck. It works by generating arbitrary data matching your specification and checking that your guarantee still holds in that case. If it finds an example where it doesn’t, it takes that example and cuts it down to size, simplifying it until it finds a much smaller example that still causes the problem. It then saves that example for later, so that once it has found a problem with your code it will not forget it in the future. Writing tests of this form usually consists of deciding on guarantees that your code should make - properties that should always hold true, regardless of what the world throws at you. Examples of such guarantees might be: https://hypothesis.readthedocs.io/en/latest/details.html#test-statistics : ```bash pytest --hypothesis-show-statistics ``` ``` - during generate phase (0.06 seconds): - Typical runtimes: < 1ms, ~ 47% in data generation - 100 passing examples, 0 failing examples, 0 invalid examples - Stopped because settings.max_examples=100 ``` ```python from hypothesis import given, strategies as st @given(st.integers(), st.integers()) def test_ints_are_commutative(x, y): assert x + y == y + x @given(x=st.integers(), y=st.integers()) def test_ints_cancel(x, y): assert (x + y) - y == x @given(st.lists(st.integers())) def test_reversing_twice_gives_same_list(xs): # This will generate lists of arbitrary length (usually between 0 and # 100 elements) whose elements are integers. ys = list(xs) ys.reverse() ys.reverse() assert xs == ys @given(st.tuples(st.booleans(), st.text())) def test_look_tuples_work_too(t): # A tuple is generated as the one you provided, with the corresponding # types in those positions. assert len(t) == 2 assert isinstance(t[0], bool) assert isinstance(t[1], str) ``` https://hypothesis.readthedocs.io/en/latest/details.html#defining-strategies :
https://hypothesis.readthedocs.io/en/latest/details.html#defining-strategies :
By default, Hypothesis will handle the global random and numpy.random random number generators for you, and you can register others:
https://hypothesis.readthedocs.io/en/latest/supported.html#testing-framework... : ``` In terms of what’s actually known to work: - Hypothesis integrates as smoothly with pytest and unittest as we can make it, and this is verified as part of the CI. - pytest fixtures work in the usual way for tests that have been decorated with @given - just avoid passing a strategy for each argument that will be supplied by a fixture. However, each fixture will run once for the whole function, not once per example. Decorating a fixture function with @given is meaningless. ``` On Tue, Sep 7, 2021, 09:16 Filipe Laíns <lains@archlinux.org> wrote:
07.09.21 05:31, Leonardo Freua пише:
It was discussed before and subTest() was added as more general alternative. Instead of @parametrize("a", [1, 2, 10]) def test(self, a): ... you should write def test(self): for a in [1, 2, 10]: with self.subTest(a=a): ... The advantage is that you can generate parameters at run time. You can add a code before and after the loop. Use several sequential loops. There are also other use cases for subTest(). The disadvantage is that it adds at least two indentation levels (more if you use nested loops) to the test code. A simple decorator would be more convenient in simple cases. It is easy to implement the decorator in terms of subtests, but currently there are some issues with outputting results of subtests: https://bugs.python.org/issue25894 https://bugs.python.org/issue30856
Hi Leonardo, On Tue, Sep 07, 2021 at 02:31:26AM -0000, Leonardo Freua wrote:
When writing some unit tests with the standard Unittest library, I missed being able to create parameterized tests.
Could you please explain what you mean by "parameterized tests", and how you would use a decorator for it? What you mean by it may not be what other people understand it to be. -- Steve
I believe he is looking for something like pytest’s parameterize: https://docs.pytest.org/en/6.2.x/example/parametrize.html This is actually pretty basic functionality for writing DRY tests, a key missing feature. Frankly though, unittest is painfully unpythonic and hard to extend. I’ve given up. Just use pytest. -CHB On Mon, Sep 6, 2021 at 7:56 PM Steven D'Aprano <steve@pearwood.info> wrote:
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
I won't propose a syntax, but I think it would be useful if *assert* could raise an exception different from *AssertionError*. This is in the context of "Design by contrast" (DBC) as a useful companion to "Test-driven development" and other forms of external tests. Basically, the proposal is to allow for an exception type and value to be specified in *assert*, instead of the customary: if not assertion: raise ValueError('a message') Perhaps the assertion would not go away with *-OO* when an exception type is specified? From my experience, assertions are useful in most non-trivial cases, and should be there on any complex case. Perhaps we'd use more assertions if they become easier to use and more customizable? The agile trend has been in favor of external tests (like unit tests). I think that we should not forget about the value of software that verifies its state at runtime through assertions. An SO reference to the basic inquiry: https://stackoverflow.com/questions/1569049/making-pythons-assert-throw-an-e... -- Juancarlo *Añez*
On 2021-09-07 11:12:37, Juancarlo Añez wrote:
What's wrong with:
From what I can tell, this is optimised away when optimisations are turned on. Maybe wrap that in a function?
Perhaps the assertion would not go away with *-OO* when an exception type is specified?
Isn't the whole point of assertions that they go away with optimisations? If the code cannot be optimised away, raise the exception at runtime, it's just regular code.
On Tue, Sep 7, 2021 at 8:37 AM Simão Afonso < simao.afonso@powertools-tech.com> wrote:
what are you proposing instead? Maybe: assert assertion, raise ValueError("a message") (currently a syntax error) in what way is that better than your original: if not assertion: raise ValueError("a message") or even: if not assertion: raise ValueError("a message") what is the value in the "assert" statement? So why do we have athe at all? Because they ARE turned off by -OO :-) -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Tue, Sep 07, 2021 at 11:12:37AM -0400, Juancarlo Añez wrote:
I don't see why that would be useful. DBC assertions are assertions. You are *asserting* that a condition is always true. Since it is always true, it should be safe to disable those DBC assertion checks once your software is in production. I could *maybe* see that having fine distinction between pre-condition, post-condition and invariant failures would be useful, but without a system in place to allow those to be globally enabled/disabled separately, what's the point? In any case, in the event of a contract failure, there's really nothing you can do except log the error and exit. Raising errors like TypeError etc will encourage people to abuse assertions and contracts by catching those exceptions, for checking caller-supplied parameters and user data, or for flow control. -- Steve
Steven, The purpose is to make it easier to make software more resilient. The inspiration was this article that reminded me that software *_will always fail_*, and also reminded me about all the discussions around DBC and Eiffel: https://www.youtube.com/watch?v=AaZ_RSt0KP8 IOW, my premise is that we should be using *_lots_* of assertions, *_always_*, and that for that we need to make them easier to write, and easier to handle in the event of the unavoidable failures. Seeking unit-test coverage is not enough because unit tests don't run in production. I will concede that code written under the *"Python culture"* tends to be resilient because the semantics of defaults and border conditions are explicit in the documentation, and implemented thus. Perhaps it's enough to allow for: *assert **cond**, *ExType(args) On Tue, Sep 7, 2021 at 9:28 PM Steven D'Aprano <steve@pearwood.info> wrote:
-- Juancarlo *Añez*
Let me re-*assert* ;-) External tests will not be invoked at runtime, yet failures *_will_* occur at runtime because of a variety of environmental factors (including cosmic rays). Software should assert at least some of its preconditions, postconditions, and invariants. How can we make that easy and syntactically pleasant? On Thu, Sep 9, 2021 at 12:02 PM Juancarlo Añez <apalala@gmail.com> wrote:
-- Juancarlo *Añez*
Ooh, that’s a nice idea. If the message is an exception instance, raise it instead of AssertionError. Unfortunately it’s not 100% backwards compatible. We could address that with the syntax assert cond, raise=ExcType(args) Maybe we could deprecate the case assert cond, ExcType(args) So that eventually the raise= keyword can become optional. —Guido On Thu, Sep 9, 2021 at 09:04 Juancarlo Añez <apalala@gmail.com> wrote:
-- --Guido (mobile)
Well, if the idea makes sense, then I'm certain that we'll have a very long and productive discussion about the best syntax here (re: *:=*). ;-) For backwards compatibility and no surprises: *assert: *ExType, cond, args It doesn't break anything, ExtType defaults to AssertionError, and linters can check that *args* match ExType. A more readable and pythonic syntax would be: *assert *cond: ExtType(args) Forgoing the comma doesn't break anything, linters can check the ExType() typing, and the semantics would be those of: *if* __debug__ *and* *not *cond: *raise* ExType(args) Although I would drop the check for *__debug__* if an explicit ExtType is given. It's similar to the changes introduced for: *except* (OneEx, TwoEx): On Thu, Sep 9, 2021 at 12:16 PM Guido van Rossum <guido@python.org> wrote:
-- Juancarlo *Añez*
On 2021-09-09 22:31, Juancarlo Añez wrote:
Or, perhaps: assert cond: raise ExtType(args) That would make it more like an if and would leave open the possibility of extending it to accept multiple lines: assert cond: # Write some debugging info to a log. ... raise ExtType(args) Just a thought.
assert cond: # Write some debugging info to a log. ... raise ExtType(args) I like the idea of preparing the arguments for the assertion message in a context that gets executed only when the assertion fails. On Thu, Sep 9, 2021 at 7:16 PM MRAB <python@mrabarnett.plus.com> wrote:
-- Juancarlo *Añez*
Juancarlo Añez writes:
The more I see this elaborated, the less I like it. "assert" has a traditional meaning in programming, which is only a slight extension on the English meaning (that you can turn the check off for performance). Although the idea that assert could be the fundamental building block for design-by-contract (DBC), to my mind this has three problems. (1) You're coopting a keyword that has a consistent established meaning in English, in programming languages in general, and in Python, and changing that meaning. (2) I doubt that "assert" alone is sufficient for DBC. In practice, you'll want a more expressive, larger vocabulary. It's not clear that *any* of that vocabulary is best expressed by "assert" at this point. If not, why mess with what works? (3) There's a subtle difference between an assertion and a contract. An assertion is a statement of fact: it's either right or it's wrong. But the whole point of contracts in the real world is that they are constraints on behavior to be enforced. The assumption is that it's possible to be in violation of the contract. The point of *asserting* something is that you believe it's absolutely true, there's no point in going on if it's false. You have to start over. A presumption that violation is possible may not be true of the contracts in DBC, I don't know. But I would prefer not to mix the difficulties of the semantics of "contract" with the simplicity of the semantics of "assert". Steve
Stephen, Assertions have their roots in mathematics, as does most everything in programming languages. I'm happy about dropping the DBC theme and rebooting to make *assert* easier to use so it gets used more. I liked the suggestion of: *assert *cond: # prepare failure information *raise *ExType(args) Because the current: *assert *cond, arg Will always raise AssertionError, and do a str(arg), which means that *arg* must be simple, or rely on a function that creates more complex reporting. It think it would be fine if ExtType->AssertionError is enforced. Then there could be a hierarchy of AssertionError depending on the type of problem or its severity. Of course, none of this will make sense to programmers with a strong belief that assertions must always be turned off in production runs. I, on the other hand, believe that any complex computation or algorithm should assert its expected state and abort if the expectations are not met, even when the reason for the failure is unknown. Think about financial, insurance, transportation, or medical algorithms. By epistemological reasons a failure caused by a bug is almost *_always_* unexpected, and the reasons for the failure are unknown at runtime (a shorter version is that "programmers don't write bugs on purpose, so bugs are always unforseen"). On Sat, Sep 11, 2021 at 12:32 AM Stephen J. Turnbull < stephenjturnbull@gmail.com> wrote:
-- Juancarlo *Añez*
It's fair to note that complex arguments to assertions can be hacked with: *assert* cond, ( f'{conputesomething} {and more}' f''{some more}' ) The exception type can't be changed, though. On Sat, Sep 11, 2021 at 9:17 AM Juancarlo Añez <apalala@gmail.com> wrote:
-- Juancarlo *Añez*
On 11.09.2021 15:17, Juancarlo Añez wrote:
Of course, none of this will make sense to programmers with a strong belief that assertions must always be turned off in production runs.
You seem to be missing that the Python optimize mode turns off all code which is related to debugging (__debug__ is set to False, the compiler doesn't generate code for "if __debug__: ..." statements). assert is just one of the instances where this happens: https://docs.python.org/3/reference/simple_stmts.html#grammar-token-assert-s... asserts are meant to help find bugs in programs, not check for user input errors. They document assumptions a programmer has made when writing the code, which are then tested with a test suite to make sure the assumptions hold as invariants of the application. For anything which can go wrong at production run-time, please use normal if-statement checks and raise appropriate exceptions. Using assert in such cases is dangerous and can render your application broken, while everything appears to be running fine when testing. Unlike regular if-statement checks, asserts are meant to never trigger an exception in production code -- which is why they are removed with -O. This is not about whether or not to use -O in production environments, it's about preventing user input exceptions from going unnoticed when code is run with -O (e.g. the deployment team or a user decided to always use -O in production). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Sep 11 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
Marc-Andre, I must agree that the current state of assertions with "-O" and "-OO" is difficult to impossible to change. Perhaps I'm looking for an: *invariant* cond: etc Regards, On Sat, Sep 11, 2021 at 11:00 AM Marc-Andre Lemburg <mal@egenix.com> wrote:
-- Juancarlo *Añez*
On Sat, Sep 11, 2021 at 02:30:10PM -0400, Juancarlo Añez wrote:
*invariant* cond: etc
A software invariant is still an assertion. In another post, I semi-suggested a new (soft) keyword: unless condition: # block raise Exception But really, it's just an "if not". if not condition: unless condition: assert condition: are all exactly the same amount of typing. It's not clear that learning a new keyword `unless` is more readable than just using `if not`. I think it might work better in languages that attempt to follow natural language (e.g. XTalk languages) than one like Python with a relatively minimal amount of syntax. But it is clear to me that using `assert` for non-assertions is misleading. -- Steve
Steve, I've seen *unless* discussed and rejected before, but it is a good enough syntax for the purpose of asserting invariants, and it should be useful for other purposes (because it has been proposed before). The semantics are clear, and the implementation is simple with the new Python parser shortcutting to an "*if not"*. I do really like this option. On Sun, Sep 12, 2021 at 1:46 AM Steven D'Aprano <steve@pearwood.info> wrote:
-- Juancarlo *Añez*
On Sat, Sep 11, 2021 at 9:20 AM Juancarlo Añez <apalala@gmail.com> wrote:
I'm happy about dropping the DBC theme and rebooting to make *assert* easier to use so it gets used more.
I agree with Steven, Marc-Andé, and others in seeing "using assertions more" as an anti-goal. This isn't to say that a given program should have fewer—nor more—lines that start with `assert`. Rather, assertions should be use to ASSERT conditions that should NEVER be violated. They are NOT to check whether user input is bad. Nor whether a disk is unreliable. Nor whether a computation takes too long. Nor whether a value is outside of a useful range. Nor any of the other things that regular exception handling is well designed for. If for some reason you find yourself completely unable to write "if not (condition_to_check): ..." for mysterious reasons, you could use assert this way (but don't):
But again, that's bad code for the same reason your proposal would be bad code. Assertions have the entire purpose of being possible to disable, and to express conditions a programmer believes are *necessarily true*. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
David, It seems I didn't state clearly enough my original statement, which is that software will *_always_ *fail, even because of faulty hardware components, or cosmic rays. For software to be resilient, it must assert it's expected state. But that doesn't have to be with the *assert* statement, perhaps less so when the Python tradition is that those go away with just a "-O". I won't vouch for an *"invariant"* statement, because that's probably too much to ask for. So I'll just use *"if"* to assert more invariants in the code I write. Cheers, On Sat, Sep 11, 2021 at 2:05 PM David Mertz, Ph.D. <david.mertz@gmail.com> wrote:
-- Juancarlo *Añez*
On Sat, Sep 11, 2021 at 02:37:25PM -0400, Juancarlo Añez wrote:
If you expect your software to be resilient given faulty hardware, using exceptions is not the way to do it. If the hardware is faulty, you can't rely on either the condition test (whether written with assert or not) or the recovery block to behave correctly. Hardware failure resilient software has to be designed from the very lowest levels of machine code. Given faulty hardware or cosmic-ray flipping of bits in memory, you are far more likely to get an interpreter seg fault than a failure you can test for and handle. Or an OS level failure. Machine crashing or locking up is way more likely than anything you can handle in Python. -- Steve
On Thu, Sep 09, 2021 at 09:16:23AM -0700, Guido van Rossum wrote:
assert cond, raise=ExcType(args)
How about this? if not cond: raise ExcType(args) This is 100% backwards compatible, doesn't not abuse `assert`, will not be removed when running under -O, and takes exactly the same number of characters to type. If we are seriously considering DBC for Python, here's is my initial proposal for function and method contracts. I don't have a proposal for class invariants as yet. def function(args): """Docstring""" require: # Preconditions checked before calling the function body. block ensure(result): # Postconditions checked after calling the function body. block # body of function `require` and `ensure` would be soft-keywords, and legal only inside a `def` block, after the docstring and before the body of the function. If present, the `require` block is executed before the body of the function. The precondition has access to the function arguments, but otherwise runs in its own namespace. Since it runs before the body, none of the body's local variables will be defined (apart from the function arguments). If present, the `ensure` block is executed after the body of the function. It also runs in its own namespace, but has access to the function's body namespace. It also receives the function body's return result as a named argument. The parameter name "result" is not mandatory. Both contract blocks are expected to either raise AssertionError, or just return. Raising other exceptions is permitted but discouraged. Catching these exceptions is permitted but discouraged except for the purpose of logging. When running under -O, both condition blocks are ignored. (At some time in the future we may add a fine-grained interface to disable them.) I have chosen the names `require` and `ensure` to match the names used by Eiffel. https://www.eiffel.org/doc/solutions/Design_by_Contract_and_Assertions If the `require` and `ensure` blocks are all assertions, we can treat the blocks as executable *specifications documention*: def function(spam, eggs, cheese, aardvark): require: assert isinstance(spam, Food) assert -1 <= eggs < MAXEGGS assert cheese.colour = 'blue' assert aardvark is not None # function body That should allow third-party checkers and tools to either do static pre- and post-condition checking, or to generate documentation from the conditions. -- Steve
On Fri, Sep 10, 2021 at 7:30 AM Steven D'Aprano <steve@pearwood.info> wrote:
[...] If we are seriously considering DBC for Python, [...]
As I think I wrote in the previous thread about DBC in Python, there are already several DBC libraries on PyPI and we should precisely evaluate both their design and impact (e.g. usage). One of the most recent / best designed ones seems to be Deal by Orsinium: https://github.com/life4/deal The Wikipedia page for DBC lists a few others: "Python, using packages like deal, icontract, PyContracts, Decontractors, dpcontracts, zope.interface, PyDBC or Contracts for Python. A permanent change to Python to support design by contracts was proposed in PEP-316, but deferred." here's is my initial
proposal for function and method contracts. I don't have a proposal for class invariants as yet.
Deal (and similar, older, libraries) addresses similar needs, within the context of the existing Python syntax. So yes, syntax can be improved. But : 1) Do people (at least a significant part) want DBC in their Python code. Is DBC still a popular idea in the software industry ? (My 2 cts: I don't believe so). 2) Is one of the existing approach (i.e. through a library) enough for these persons, or do we really need to change the language ? (I don't know, we need people who actually do DBC with existing libraries to tell us). 3) I think there is more to DBC that simply checking preconditions and postconditions and invariants at runtime, just as typechecking Python is a bit more work than sprinkling your code with "assert isinstance(something, SomeClass)". Just as for type systems, DBC becomes really useful when you have a theorem prover that can prove (or refute) the consistency of your DBC constraints. This is the hard part (IMHO syntax is just the cherry on the cake). Cheers, S. -- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Co-Founder & Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Co-Founder & Board Member, Association Professionnelle Européenne du Logiciel Libre (APELL) - https://www.apell.info/ Co-Founder & Spokesperson, European Cloud Industrial Alliance (EUCLIDIA) - https://www.euclidia.eu/ Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ & http://pydata.fr/
And 4) DBC constraints also become more useful than standard assertions when used in the context of a class hierarchy. I.e. preconditions, postconditions and invariants should be stronger (or sometimes weaker, IIRC, there are subtle rules related to variance and contravariance) for subclasses than for superclasses. This is another subject that should be dealt with if we want the true power of DBC in Python. S. On Fri, Sep 10, 2021 at 9:26 AM Stéfane Fermigier <sf@fermigier.com> wrote:
-- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Co-Founder & Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Co-Founder & Board Member, Association Professionnelle Européenne du Logiciel Libre (APELL) - https://www.apell.info/ Co-Founder & Spokesperson, European Cloud Industrial Alliance (EUCLIDIA) - https://www.euclidia.eu/ Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ & http://pydata.fr/
Take a look at the archives of this list -- there was a large conversation about DBC a while back (a year, two years ??? ) I think if you really want to support DBC, there will need to be more changes than this -- though there are libraries that support it with current Python. Also, let's be clear about hte language re use -- when you say: there should be lots of assertions, do you mean the english were assertion, or the Python keyword assert? If the former, than you are free, or course, to use a ordinary if and raise to make all sort sof assertions about the code at runtime. To me -- the entire point of the assert statement is that it is for testing, and can be turned off. If you want to check a condition always, just use an if and a raise: How is this: if value < 0: raise ValueError("this only works with positive numbers") Any more difficult to read or write than: assert value >= 0, raise ValueError("this only works with positive numbers") -CHB On Thu, Sep 9, 2021 at 9:05 AM Juancarlo Añez <apalala@gmail.com> wrote:
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Thu, Sep 9, 2021 at 12:38 PM Christopher Barker <pythonchb@gmail.com> wrote:
I have never heard of DBC and don't have a clue what is stands for. I am not a pro software developer. But I would read these two lines differently, semantically. The if version feels like someone is saying "I am checking for a thing that could happen". The assert version would feel more like "maybe this could happen and I am checking for it because I am scared not to let's just put it here just in case", or said another way "it is imperative that this not happen". Not sure if that is an argument for or against, and maybe I'm the only one who would see the semantics differently. --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
On Thu, Sep 09, 2021 at 12:53:34PM -0400, Ricky Teachey wrote:
I have never heard of DBC and don't have a clue what is stands for. I am not a pro software developer.
DBC stands for "Design By Contract", it is a methodology for developing software. https://en.wikipedia.org/wiki/Design_by_contract The language Eiffel is especially know for DBC, and it has contract testing built into the syntax. https://www.eiffel.com/values/design-by-contract/introduction/ https://www.eiffel.org/doc/solutions/Design_by_Contract_and_Assertions
Indeed. Assertions are statements about the expected state of the program, not queries about the current state. https://import-that.dreamwidth.org/676.html -- Steve
Christopher, The *if* version has the condition inverted, and it's structure makes it look as part of the algorithm, instead of an assertion over an invariant. Following with the ideas on this thread, it would be possible for -OO to turn off only assertions with exceptions that descend from AssertionError. More on my answer to Steven On Thu, Sep 9, 2021 at 12:35 PM Christopher Barker <pythonchb@gmail.com> wrote:
-- Juancarlo *Añez*
On Thu, Sep 09, 2021 at 12:02:13PM -0400, Juancarlo Añez wrote:
Steven,
The purpose is to make it easier to make software more resilient.
Thank you but I understand the purpose of DBC and how it can be helpful for writing resilient software. If you search the archives, you will see that not only have Eiffel-style contracts been discussed before, but I am a great proponent of them. What I question is that allowing assert to raise non-assertions will lead to *more* resilient software rather than less. I know far too many people who insist on abusing assertions as a lazy way to validate caller- or user-supplied data, which is **not** a contract. Contracts apply only to calls between parts of your own application. Contracts do not apply to: - checking external data, which can be corrupt or invalid; - checking data provided by the end-user, who can never be trusted to keep to the contract; - checking input in library functions that may be called by other libraries or programs that are not a party to the contract; but I see too many people doing exactly that because it saves them a line of code. And because they are abusing assert, that means that their software is *less* resilient, because it will fail to perform those checks if the end-user runs it with assertions disabled. If `assert` can be used to raise *any* sort of exception, not just AssertionError, that will encourage that abuse and lead to less resilient software, not more.
IOW, my premise is that we should be using *_lots_* of assertions, *_always_*, and that for that we need to make them easier to write,
How can assertions be easier to write than they are now? assert condition You don't get any easier than that. That is the bare minimum needed for contract checking.
and easier to handle in the event of the unavoidable failures.
If you believe that contract violations are *unavoidable* then you have misunderstood the purpose of DBC. Contracts apply between parts of your application. A contract violation is a *bug in your application* and so is completely avoidable. (Except in the sense that we are all imperfect and therefore all software contains bugs, nevertheless we can asymptopically approach the status of being free from bugs.) Contracts do not apply between libraries and their callers, or between software and end-users. That would be rude and hostile, and the opposite of resilient. *Data and input validation* errors are unavoidable: there will always be corrupt files, incorrect data, arguments passed in the wrong order, etc. But they are not contracts and should not be tested for using assert because we cannot ever safely disable that validation. The point of assertions and contracts is that you *don't catch them*. Assertions and contracts are hard requirements. If the contract is violated you have a bug in your software and the only safe thing to do is to log the error and gracefully exit. If you can catch and recover from an error, its not a contract violation. It's just using exceptions for flow control. # Exceptions as flow control. # LBYL if condition: do_this() else: do_that() # EAFP try: do_this() except ConditionFailure: do_that()
Seeking unit-test coverage is not enough because unit tests don't run in production.
If you do DBC correctly, contract assertions don't run in production either. It is up to the end-user whether the contracts run or not. (I believe that the Eiffel community tends to recommend that pre- condition testing remain in production, and that post-conditions, invariants and other contracts be disabled, but that's entirely under the control of the end-user.) One simple test for whether something can be a contract or not is this: Could it be safe for me to remove this test? Not necessarily today, but some day, in the future, when I am confident that the code is bug free. If your answer is Yes, then it could be a contract written with assert. But if the answer is No, or I Don't Know, then it cannot be a contract, and it should not be written using assert. -- Steve
On 10.09.2021 05:49, Steven D'Aprano wrote:
I concur. asserts are meant for verifying assumptions the code designer and wants to verify *during development*. In C they lead to a core dump which aids in finding the cause of the problem. In Python, an AssertionError is raised with the same intent. In Python, using assert for anything that is not development related is dangerous, since production code running with -O (optimized mode) will not even run those assertions - for a good reason: they are meant only for checking assumptions in development. For any non-development related error checking, normal if statements should be used, not assert. I very often see code bases, which abuse assert as a quick way to check types, ranges, valid flag combinations, etc. Since all these things can go wrong in production as well, those checks need to be handled with if statements, not assert. If we'd not allow assert to also raise non-AssertionErrors, we'd get even more abuse. In fact, I'd be in favor of deprecating assert altogether, if it were not for pytest using it for testing - which is a valid use case, since those tests are not run in production, but again, most likely leads to programmers thinking that they can use the same logic in the actual production code. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Sep 10 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On Tue, 2021-09-07 at 02:31 +0000, Leonardo Freua wrote:
Hi Leonardo, Please check out subtests, they allow you to achieve what you are looking for :) https://docs.python.org/3/library/unittest.html#distinguishing-test-iteratio... Cheers, Filipe Laíns
unittest.subTest, hypothesis @given + pytest From https://docs.python.org/3/library/unittest.html#distinguishing-test-iteratio... : ``` When there are very small differences among your tests, for instance some parameters, unittest allows you to distinguish them inside the body of a test method using the subTest() context manager. For example, the following test: class NumbersTest(unittest.TestCase): def test_even(self): """ Test that numbers between 0 and 5 are all even. """ for i in range(0, 6): with self.subTest(i=i): self.assertEqual(i % 2, 0) ``` https://hypothesis.readthedocs.io/en/latest/quickstart.html#writing-tests : Think of a normal unit test as being something like the following: 1. Set up some data. 2. Perform some operations on the data. 3. Assert something about the result. Hypothesis lets you write tests which instead look like this: 1. For all data matching some specification. 2. Perform some operations on the data. 3. Assert something about the result. This is often called property-based testing, and was popularised by the Haskell library Quickcheck. It works by generating arbitrary data matching your specification and checking that your guarantee still holds in that case. If it finds an example where it doesn’t, it takes that example and cuts it down to size, simplifying it until it finds a much smaller example that still causes the problem. It then saves that example for later, so that once it has found a problem with your code it will not forget it in the future. Writing tests of this form usually consists of deciding on guarantees that your code should make - properties that should always hold true, regardless of what the world throws at you. Examples of such guarantees might be: https://hypothesis.readthedocs.io/en/latest/details.html#test-statistics : ```bash pytest --hypothesis-show-statistics ``` ``` - during generate phase (0.06 seconds): - Typical runtimes: < 1ms, ~ 47% in data generation - 100 passing examples, 0 failing examples, 0 invalid examples - Stopped because settings.max_examples=100 ``` ```python from hypothesis import given, strategies as st @given(st.integers(), st.integers()) def test_ints_are_commutative(x, y): assert x + y == y + x @given(x=st.integers(), y=st.integers()) def test_ints_cancel(x, y): assert (x + y) - y == x @given(st.lists(st.integers())) def test_reversing_twice_gives_same_list(xs): # This will generate lists of arbitrary length (usually between 0 and # 100 elements) whose elements are integers. ys = list(xs) ys.reverse() ys.reverse() assert xs == ys @given(st.tuples(st.booleans(), st.text())) def test_look_tuples_work_too(t): # A tuple is generated as the one you provided, with the corresponding # types in those positions. assert len(t) == 2 assert isinstance(t[0], bool) assert isinstance(t[1], str) ``` https://hypothesis.readthedocs.io/en/latest/details.html#defining-strategies :
https://hypothesis.readthedocs.io/en/latest/details.html#defining-strategies :
By default, Hypothesis will handle the global random and numpy.random random number generators for you, and you can register others:
https://hypothesis.readthedocs.io/en/latest/supported.html#testing-framework... : ``` In terms of what’s actually known to work: - Hypothesis integrates as smoothly with pytest and unittest as we can make it, and this is verified as part of the CI. - pytest fixtures work in the usual way for tests that have been decorated with @given - just avoid passing a strategy for each argument that will be supplied by a fixture. However, each fixture will run once for the whole function, not once per example. Decorating a fixture function with @given is meaningless. ``` On Tue, Sep 7, 2021, 09:16 Filipe Laíns <lains@archlinux.org> wrote:
07.09.21 05:31, Leonardo Freua пише:
It was discussed before and subTest() was added as more general alternative. Instead of @parametrize("a", [1, 2, 10]) def test(self, a): ... you should write def test(self): for a in [1, 2, 10]: with self.subTest(a=a): ... The advantage is that you can generate parameters at run time. You can add a code before and after the loop. Use several sequential loops. There are also other use cases for subTest(). The disadvantage is that it adds at least two indentation levels (more if you use nested loops) to the test code. A simple decorator would be more convenient in simple cases. It is easy to implement the decorator in terms of subtests, but currently there are some issues with outputting results of subtests: https://bugs.python.org/issue25894 https://bugs.python.org/issue30856
participants (16)
-
Christopher Barker -
David Mertz, Ph.D. -
Filipe Laíns -
Guido van Rossum -
Juancarlo Añez -
Leonardo Freua -
Marc-Andre Lemburg -
MRAB -
Ricky Teachey -
Rob Cliffe -
Serhiy Storchaka -
Simão Afonso -
Stephen J. Turnbull -
Steven D'Aprano -
Stéfane Fermigier -
Wes Turner