
Hello all, The next 'big' change to unittest will (may?) be the introduction of class and module level setUp and tearDown. This was discussed on Python-ideas and Guido supported them. They can be useful but are also very easy to abuse (too much shared state, monolithic test classes and modules). Several authors of other Python testing frameworks spoke up *against* them, but several *users* of test frameworks spoke up in favour of them. ;-) I'm pretty sure I can introduce setUpClass and setUpModule without breaking compatibility with existing unittest extensions or backwards compatibility issues - with the possible exception of test sorting. Where you have a class level setUp (for example creating a database connection) you don't want the tearDown executed a *long* time after the setUp. In the presence of class or module level setUp /tearDown (but only if they are used) I would expect test sorting to only sort within the class or module [1]. I will introduce the setUp and tearDown as new 'tests' - so failures are reported separately, and all tests in the class / module will have an explicit skip in the event of a setUp failure. A *better* (more general) solution for sharing and managing resources between tests is to use something like TestResources by Robert Collins. http://pypi.python.org/pypi/testresources/ A minimal example of using test resources shows very little boilerplate overhead from what setUpClass (etc) would need, and with the addition of some helper functions could be almost no overhead. I've challenged Robert that if he can provide examples of using Test Resources to meet the class and module level use-cases then I would support bringing Test Resources into the standard library as part of unittest (modulo licensing issues which he is happy to work on). I'm not sure what response I expect from this email, and neither option will be implemented without further discussion - possibly at the PyCon sprints - but I thought I would make it clear what the possible directions are. All the best, Michael Foord [1] I *could* allow sorting of all tests within a module, inserting the setUpClass / tearDownClass in the right place after the sort. It would probably be better to group tests per class anyway and in fact the existing suite sorting support may do this already (in which case it isn't an issue) - I haven't looked into the implementation. -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

Le Tue, 09 Feb 2010 16:42:50 +0000, Michael Foord a écrit :
The next 'big' change to unittest will (may?) be the introduction of class and module level setUp and tearDown. This was discussed on Python-ideas and Guido supported them. They can be useful but are also very easy to abuse (too much shared state, monolithic test classes and modules). Several authors of other Python testing frameworks spoke up *against* them, but several *users* of test frameworks spoke up in favour of them. ;-)
One problem is that it is not obvious what happens with inheritance. If I have a class-level setUp for class B, and class C inherits from B, will there be a separate invocation of setUp for C, or not? (I guess both possibilities have use cases) Antoine.

On 09/02/2010 17:57, Antoine Pitrou wrote:
Le Tue, 09 Feb 2010 16:42:50 +0000, Michael Foord a écrit :
The next 'big' change to unittest will (may?) be the introduction of class and module level setUp and tearDown. This was discussed on Python-ideas and Guido supported them. They can be useful but are also very easy to abuse (too much shared state, monolithic test classes and modules). Several authors of other Python testing frameworks spoke up *against* them, but several *users* of test frameworks spoke up in favour of them. ;-)
One problem is that it is not obvious what happens with inheritance. If I have a class-level setUp for class B, and class C inherits from B, will there be a separate invocation of setUp for C, or not? (I guess both possibilities have use cases)
Well, what I would expect (others may disagree) is that you only have class level setup invoked for classes that have tests (so not for base classes) and that the base-class setUpClass is only called if invoked by the subclass. I haven't thought about *where* the code to do this should go. It *could* go in TestSuite, but that feels like the wrong place. Michael
Antoine.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On Tue, Feb 9, 2010 at 1:44 PM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
On 09/02/2010 17:57, Antoine Pitrou wrote:
Le Tue, 09 Feb 2010 16:42:50 +0000, Michael Foord a écrit :
The next 'big' change to unittest will (may?) be the introduction of class and module level setUp and tearDown. This was discussed on Python-ideas and Guido supported them. They can be useful but are also very easy to abuse (too much shared state, monolithic test classes and modules). Several authors of other Python testing frameworks spoke up *against* them, but several *users* of test frameworks spoke up in favour of them. ;-)
One problem is that it is not obvious what happens with inheritance. If I have a class-level setUp for class B, and class C inherits from B, will there be a separate invocation of setUp for C, or not? (I guess both possibilities have use cases)
Well, what I would expect (others may disagree) is that you only have class level setup invoked for classes that have tests (so not for base classes) and that the base-class setUpClass is only called if invoked by the subclass.
I haven't thought about *where* the code to do this should go. It *could* go in TestSuite, but that feels like the wrong place.
When I implemented this in `dutest` I did it as follows : - Changed suiteClass (since it was an extension ;o) - I had to override the suite's `run` method PS: Probably it's not the right place, but AFAIK it's the only place «we» have to do such things ;o) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Embedding pages? - Trac Users | Google Groups - http://feedproxy.google.com/~r/TracGViz-full/~3/-XtS7h-wjcI/e4cf16474aa3cb87

On Tue, Feb 9, 2010 at 12:57 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Le Tue, 09 Feb 2010 16:42:50 +0000, Michael Foord a écrit :
The next 'big' change to unittest will (may?) be the introduction of class and module level setUp and tearDown. This was discussed on Python-ideas and Guido supported them. They can be useful but are also very easy to abuse (too much shared state, monolithic test classes and modules). Several authors of other Python testing frameworks spoke up *against* them, but several *users* of test frameworks spoke up in favour of them. ;-)
One problem is that it is not obvious what happens with inheritance. If I have a class-level setUp for class B, and class C inherits from B, will there be a separate invocation of setUp for C, or not? (I guess both possibilities have use cases)
Considering JUnit : - The @BeforeClass methods of superclasses will be run before those the current class. - The @AfterClass methods declared in superclasses will be run after those of the current class. However considering that PyUnit is not based on annotations, isn't it possible to specify that explicitly (and assume super-class method not called by default) ? -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: gmane.comp.version-control.subversion.trac.general - http://feedproxy.google.com/~r/TracGViz-full/~3/SLY6s0RazcA/28067

On Tue, Feb 9, 2010 at 11:42 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
Hello all,
Several authors of other Python testing frameworks spoke up *against* them, but several *users* of test frameworks spoke up in favour of them. ;-)
+1 for having something like that included in unittest
I'm pretty sure I can introduce setUpClass and setUpModule without breaking compatibility with existing unittest extensions or backwards compatibility issues
Is it possible to use the names `BeforeClass` and `AfterClass` (just to be make it look similar to JUnit naming conventions ;o) ?
- with the possible exception of test sorting. Where you have a class level setUp (for example creating a database connection) you don't want the tearDown executed a *long* time after the setUp. In the presence of class or module level setUp /tearDown (but only if they are used) I would expect test sorting to only sort within the class or module [1]. I will introduce the setUp and tearDown as new 'tests' - so failures are reported separately,
Perhaps I am missing something, but could you please mention what will happen if a failure is raised inside class-level `tearDown` ?
and all tests in the class / module will have an explicit skip in the event of a setUp failure.
+1
A *better* (more general) solution for sharing and managing resources between tests is to use something like TestResources by Robert Collins. http://pypi.python.org/pypi/testresources/
A minimal example of using test resources shows very little boilerplate overhead from what setUpClass (etc) would need, and with the addition of some helper functions could be almost no overhead. I've challenged Robert that if he can provide examples of using Test Resources to meet the class and module level use-cases then I would support bringing Test Resources into the standard library as part of unittest (modulo licensing issues which he is happy to work on).
I am not really sure about whether unittest API should grow, and grow, and grow, and ... but if that means that TestResources will not be even imported if testers don't do it explicitly in the code (which is not the case of something like class level setUp/tearDown) then +1, otherwise -0.5 -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: TracGViz plugin downloaded more than 1000 times (> 300 from PyPI) - http://feedproxy.google.com/~r/simelo-en/~3/06Exn-JPLIA/tracgviz-plugin-down...

On Tue, Feb 9, 2010 at 1:29 PM, Olemis Lang <olemis@gmail.com> wrote:
On Tue, Feb 9, 2010 at 11:42 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
Hello all,
Several authors of other Python testing frameworks spoke up *against* them, but several *users* of test frameworks spoke up in favour of them. ;-)
+1 for having something like that included in unittest
I'm pretty sure I can introduce setUpClass and setUpModule without breaking compatibility with existing unittest extensions or backwards compatibility issues
Is it possible to use the names `BeforeClass` and `AfterClass` (just to be make it look similar to JUnit naming conventions ;o) ?
Another Q: - class setup method will be a `classmethod` isn't it ? It should not be a regular instance method because IMO it is not bound to a particular `TestCase` instance. -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Embedding pages? - Trac Users | Google Groups - http://feedproxy.google.com/~r/TracGViz-full/~3/-XtS7h-wjcI/e4cf16474aa3cb87

Sorry. I had not finished the previous message On Tue, Feb 9, 2010 at 1:55 PM, Olemis Lang <olemis@gmail.com> wrote:
On Tue, Feb 9, 2010 at 1:29 PM, Olemis Lang <olemis@gmail.com> wrote:
On Tue, Feb 9, 2010 at 11:42 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
Hello all,
Several authors of other Python testing frameworks spoke up *against* them, but several *users* of test frameworks spoke up in favour of them. ;-)
+1 for having something like that included in unittest
I'm pretty sure I can introduce setUpClass and setUpModule without breaking compatibility with existing unittest extensions or backwards compatibility issues
Is it possible to use the names `BeforeClass` and `AfterClass` (just to be make it look similar to JUnit naming conventions ;o) ?
Another Q:
- class setup method will be a `classmethod` isn't it ? It should not be a regular instance method because IMO it is not bound to a particular `TestCase` instance.
- Is it possible to rely on the fact that all class-level tear down methods will be guaranteed to run even if class-level setup method throws an exception ? -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: PEP 391 - Please Vote! - http://feedproxy.google.com/~r/TracGViz-full/~3/hY2h6ZSAFRE/110617

On 09/02/2010 19:00, Olemis Lang wrote:
Sorry. I had not finished the previous message
On Tue, Feb 9, 2010 at 1:55 PM, Olemis Lang<olemis@gmail.com> wrote:
On Tue, Feb 9, 2010 at 1:29 PM, Olemis Lang<olemis@gmail.com> wrote:
On Tue, Feb 9, 2010 at 11:42 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
Hello all,
Several authors of other Python testing frameworks spoke up *against* them, but several *users* of test frameworks spoke up in favour of them. ;-)
+1 for having something like that included in unittest
I'm pretty sure I can introduce setUpClass and setUpModule without breaking compatibility with existing unittest extensions or backwards compatibility issues
Is it possible to use the names `BeforeClass` and `AfterClass` (just to be make it look similar to JUnit naming conventions ;o) ?
Another Q:
- class setup method will be a `classmethod` isn't it ? It should not be a regular instance method because IMO it is not bound to a particular `TestCase` instance.
- Is it possible to rely on the fact that all class-level tear down methods will be guaranteed to run even if class-level setup method throws an exception ?
Yes it will be a classmethod rather than an instance method. I would expect that in common with instance setUp the tearDown would *not* be run if setUp fails. What would be nice would be an extension of addCleanUp so that it can be used by class and module level setUp. Clean-ups largely obsolete the need for tearDown anyway. Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On Tue, Feb 9, 2010 at 2:04 PM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
On 09/02/2010 19:00, Olemis Lang wrote:
Sorry. I had not finished the previous message
On Tue, Feb 9, 2010 at 1:55 PM, Olemis Lang<olemis@gmail.com> wrote:
On Tue, Feb 9, 2010 at 1:29 PM, Olemis Lang<olemis@gmail.com> wrote:
On Tue, Feb 9, 2010 at 11:42 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
Hello all,
Several authors of other Python testing frameworks spoke up *against* them, but several *users* of test frameworks spoke up in favour of them. ;-)
+1 for having something like that included in unittest
I'm pretty sure I can introduce setUpClass and setUpModule without breaking compatibility with existing unittest extensions or backwards compatibility issues
Is it possible to use the names `BeforeClass` and `AfterClass` (just to be make it look similar to JUnit naming conventions ;o) ?
Another Q:
- class setup method will be a `classmethod` isn't it ? It should not be a regular instance method because IMO it is not bound to a particular `TestCase` instance.
- Is it possible to rely on the fact that all class-level tear down methods will be guaranteed to run even if class-level setup method throws an exception ?
Yes it will be a classmethod rather than an instance method.
+1
I would expect that in common with instance setUp the tearDown would *not* be run if setUp fails. What would be nice would be an extension of addCleanUp so that it can be used by class and module level setUp. Clean-ups largely obsolete the need for tearDown anyway.
I really disagree. IMO I am -1 for having `addCleanUp` and so on added to the core API (i.e. `TestCase` class). The same goes for test resources (especially if that means to merge it with the API rather than including it as a separate independent module). The use cases for that feature are not, in general, basic use cases ('cause if they were *simple*, setUp/tearDown would be a *simple* alternative to do the same thing ;o). I repeat that my opinion is that I am -1 for including each and every feature needed for testing purposes jus because it's very (super) useful to solve even many use cases (e.g. context managers, by themselves, are an empty and abstract construct that solve a set of problems *once they are implemented*, but the top-level abstractions are not directly useful by themselves). It's an API. In JUnit there are a lot of useful extensions implemented in `junit-ext` package & Co. (and AFAICR that also includes integration testing & test resources) and besides there are some other important features in JUnit>=4.7 itself, *but not hard coded in TestCase* (e.g. `org.junit.rules.ExternalResource`, ...) and also allowing extensions (e.g. `org.junit.rules.TemporaryFolder`) using well established mechanisms (e.g. inheritance) . This also has the benefit that the responsibilities are distributed to a set of relevant objects following well-known interaction patterns, rather than cluttering a class with all sort of functionalities . PS: I say this and I know that it's quite unlikely that you will reconsider it in order to revert what's being done there . But, if we take a look to JUnit>=4.7, just notice that resource management is not an integral part of `TestCase` at all, and is performed in a more structured way, consistent with the «standard» or officially supported mechanism used to add any other extension to JUnit . Honestly I can obviously see the differences with respect to `addCleanUp` implementation as we know it today. -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: TracGViz plugin downloaded more than 1000 times (> 300 from PyPI) - http://feedproxy.google.com/~r/simelo-en/~3/06Exn-JPLIA/tracgviz-plugin-down...

On Tue, Feb 9, 2010 at 7:29 PM, Olemis Lang <olemis@gmail.com> wrote:
On Tue, Feb 9, 2010 at 11:42 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
Hello all,
Several authors of other Python testing frameworks spoke up *against* them, but several *users* of test frameworks spoke up in favour of them. ;-)
+1 for having something like that included in unittest
hey Olemis, aren't you a test tool author as well? :)
I'm pretty sure I can introduce setUpClass and setUpModule without breaking compatibility with existing unittest extensions or backwards compatibility issues
Is it possible to use the names `BeforeClass` and `AfterClass` (just to be make it look similar to JUnit naming conventions ;o) ?
- with the possible exception of test sorting. Where you have a class level setUp (for example creating a database connection) you don't want the tearDown executed a *long* time after the setUp. In the presence of class or module level setUp /tearDown (but only if they are used) I would expect test sorting to only sort within the class or module [1]. I will introduce the setUp and tearDown as new 'tests' - so failures are reported separately,
Perhaps I am missing something, but could you please mention what will happen if a failure is raised inside class-level `tearDown` ?
and all tests in the class / module will have an explicit skip in the event of a setUp failure.
I think reporting tests as skipped when the setup failed is a bad idea. Out of several years of practise with skips and large test suites (and talking/experiencing many users :) i recommend to reserve skips for platform/dependency/environment mismatches. A Setup Error should just error or fail all the tests in its scope. cheers, holger

On 09/02/2010 19:14, Holger Krekel wrote:
and all tests in the class / module will have an explicit skip in the event of a setUp failure.
I think reporting tests as skipped when the setup failed is a bad idea. Out of several years of practise with skips and large test suites (and talking/experiencing many users :) i recommend to reserve skips for
[snip...] platform/dependency/environment mismatches. A Setup Error should just error or fail all the tests in its scope.
A SetupError instead of a skip sounds good to me. Thanks (although technically the test has been 'skipped' but that's playing with semantics...) Michael
cheers, holger
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On Tue, Feb 9, 2010 at 12:29, Olemis Lang <olemis@gmail.com> wrote:
On Tue, Feb 9, 2010 at 11:42 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
I'm pretty sure I can introduce setUpClass and setUpModule without breaking compatibility with existing unittest extensions or backwards compatibility issues
Is it possible to use the names `BeforeClass` and `AfterClass` (just to be make it look similar to JUnit naming conventions ;o) ? -- Regards,
Olemis.
-1. setUp/tearDown is already well established here so I think it should follow the same convention.

On Tue, Feb 9, 2010 at 2:16 PM, Brian Curtin <brian.curtin@gmail.com> wrote:
On Tue, Feb 9, 2010 at 12:29, Olemis Lang <olemis@gmail.com> wrote:
On Tue, Feb 9, 2010 at 11:42 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
I'm pretty sure I can introduce setUpClass and setUpModule without breaking compatibility with existing unittest extensions or backwards compatibility issues
Is it possible to use the names `BeforeClass` and `AfterClass` (just to be make it look similar to JUnit naming conventions ;o) ? -- Regards,
Olemis.
-1. setUp/tearDown is already well established here so I think it should follow the same convention.
ok no big deal ;o) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Nabble - Trac Users - Embedding pages? - http://feedproxy.google.com/~r/TracGViz-full/~3/MWT7MJBi08w/Embedding-pages-...

Michael Foord <fuzzyman@voidspace.org.uk> writes:
The next 'big' change to unittest will (may?) be the introduction of class and module level setUp and tearDown. This was discussed on Python-ideas and Guido supported them. They can be useful but are also very easy to abuse (too much shared state, monolithic test classes and modules). Several authors of other Python testing frameworks spoke up *against* them, but several *users* of test frameworks spoke up in favour of them. ;-)
I think the perceived need for these is from people trying to use the ‘unittest’ API for test that are *not* unit tests. That is, people have a need for integration tests (test this module's interaction with some other module) or system tests (test the behaviour of the whole running system). They then try to crowbar those tests into ‘unittest’ and finding it lacking, since ‘unittest’ is designed for tests of function-level units, without persistent state between those test cases. Is there a better third-party framework for use in these cases? As Olemis points out later in this thread, I don't think it's good for the ‘unittest’ module to keep growing for uses that aren't focussed on unit tests (as contrasted with other kinds of tests). -- \ “The industrial system is profoundly dependent on commercial | `\ television and could not exist in its present form without it.” | _o__) —John Kenneth Galbraith, _The New Industrial State_, 1967 | Ben Finney

On Tue, Feb 9, 2010 at 4:57 PM, Ben Finney <ben+python@benfinney.id.au> wrote:
Michael Foord <fuzzyman@voidspace.org.uk> writes:
The next 'big' change to unittest will (may?) be the introduction of class and module level setUp and tearDown. This was discussed on Python-ideas and Guido supported them. They can be useful but are also very easy to abuse (too much shared state, monolithic test classes and modules). Several authors of other Python testing frameworks spoke up *against* them, but several *users* of test frameworks spoke up in favour of them. ;-)
I think the perceived need for these is from people trying to use the ‘unittest’ API for test that are *not* unit tests.
I dont't think so. I'll try to explain what I consider is a real use case tomorrow ... -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Free milestone ranch Download - mac software - http://feedproxy.google.com/~r/TracGViz-full/~3/rX6_RmRWThE/

On Tue, Feb 9, 2010 at 10:57 PM, Ben Finney <ben+python@benfinney.id.au> wrote:
Michael Foord <fuzzyman@voidspace.org.uk> writes:
The next 'big' change to unittest will (may?) be the introduction of class and module level setUp and tearDown. This was discussed on Python-ideas and Guido supported them. They can be useful but are also very easy to abuse (too much shared state, monolithic test classes and modules). Several authors of other Python testing frameworks spoke up *against* them, but several *users* of test frameworks spoke up in favour of them. ;-)
I think the perceived need for these is from people trying to use the ‘unittest’ API for test that are *not* unit tests.
That is, people have a need for integration tests (test this module's interaction with some other module) or system tests (test the behaviour of the whole running system). They then try to crowbar those tests into ‘unittest’ and finding it lacking, since ‘unittest’ is designed for tests of function-level units, without persistent state between those test cases.
Is there a better third-party framework for use in these cases? As Olemis points out later in this thread, I don't think it's good for the ‘unittest’ module to keep growing for uses that aren't focussed on unit tests (as contrasted with other kinds of tests).
My general view these days is that for unit tests there is practically not much of a a difference in using unittest, nose or py.test (give or take reporting niceness and flexibility). However, Functional and integration tests involve more complex fixture management and i came to find the setup/teardown on classes and modules lacking. Which is why there is testresources from Rob and funcargs in py.test. The latter allow to setup and teardown resources from a fixture factory which can determine the setup/teardown scope and perform whole-session caching without changing test code. In my Pycon testing tutorial (http://tinyurl.com/ya6b3vr ) i am going to exercise it in depth with beginners and here are docs: http://pytest.org/funcargs.html One nice bit is that you can for a given test module issue "py.test --funcargs" and get a list of resources you can use in your test function - by simply specifying them in the test function. In principle it's possible to port this approach to the stdlib - actually i consider to do it for the std-unittest- running part of py.test because people asked for it - if that proves useful i can imagine to refine it and offer it for inclusion. cheers, holger

On Tue, Feb 9, 2010 at 5:34 PM, Holger Krekel <holger.krekel@gmail.com> wrote:
On Tue, Feb 9, 2010 at 10:57 PM, Ben Finney <ben+python@benfinney.id.au> wrote:
Michael Foord <fuzzyman@voidspace.org.uk> writes:
The next 'big' change to unittest will (may?) be the introduction of class and module level setUp and tearDown. This was discussed on Python-ideas and Guido supported them. They can be useful but are also very easy to abuse (too much shared state, monolithic test classes and modules). Several authors of other Python testing frameworks spoke up *against* them, but several *users* of test frameworks spoke up in favour of them. ;-)
I think the perceived need for these is from people trying to use the ‘unittest’ API for test that are *not* unit tests.
Well the example I was talking about before is when some (critical) resource needed for unittesting requires a very, very heavy initialization process. I'll employ the most recent example (hope it doesn't look like too much biased, it's just to illustrate the whole picture ;o) which is unittests for a framework like Trac . In that case it is critical to have a Trac environment, a ready-to-use DB and backend, initialize the plugins cache by loading relevant plugins, run the actions specified by each IEnvironmentSetup participant, sometimes a ready to use repository (if testing code depending on Trac VCS API) and more ... Just considering these cases someone could : - Create a fake environment used as a stub - But having a single global environment is not a good idea because it would be very difficult to run multiple (independent) tests concurrently (e.g. test multiple Trac plugins concurrently in a dedicated CI environment). So an environment has to be started for every test run and be as isolated as possible from other similar stub environments - The DB and backend can be replaced by using in-memory SQLite connection - Plugins cache and loading is mandatory as well running the actions specified by each IEnvironmentSetup participant - VCS can be mocked, but if it's needed it has to be initialized as well And all this is needed to run *ANY* test of *ANY* kind (that includes unittests ;o) . I hope that, up to this point, you all are convinced of the fact that all this cannot be done for each TestCase instance. That's why something like class-level setup | teardown might be useful to get all this done just once ... but it's not enough Something I consider a limitation of that approach is that it is a little hard to control the scope of setup and teardown. For instance, if I was trying to run Trac test suite I'd like to create the environment stub just once, and not once for every (module | class) containing tests. The current approach does not fit very well scenarios like this (i.e. setup | teardown actions span even beyond single modules ;o) So that's why it seems that the approach included in Trac testing code (i.e. a global shared fixture ) will still be needed, but AFAICR it breaks a little the interface of TC class and setup and tear down has to be performed from the outside. OTOH another minimalistic framework I've been building on top of `dutest` to cope with such scenarios (aka TracDuTest but not oficially released yet :-/ ) seems to handle all those features well enough by using doctest extraglobs or by modifying the global namespace at any given time inside setUp and tearDown (thus hiding all this code from doctests ;o).
One nice bit is that you can for a given test module issue "py.test --funcargs" and get a list of resources you can use in your test function - by simply specifying them in the test function.
In principle it's possible to port this approach to the stdlib - actually i consider to do it for the std-unittest- running part of py.test because people asked for it - if that proves useful i can imagine to refine it and offer it for inclusion.
Considering part of what I've mentioned above: Q: - How could py.test help in cases like this ? - Considering the similitudes with unittest style (at least IMO) I think I'd prefer something like PeckCheck to generate and run parameterized TCs. What d'u think ? (I confess that I don't use py.test , nose ... because I see they use too much magic & ..., but that's just my *VERY* biased opinion, so I won't start a war or alike ;o) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Nabble - Trac Users - Embedding pages? - http://feedproxy.google.com/~r/TracGViz-full/~3/MWT7MJBi08w/Embedding-pages-...

On Wed, 10 Feb 2010 09:45:41 -0500, Olemis Lang <olemis@gmail.com> wrote:
On Tue, Feb 9, 2010 at 5:34 PM, Holger Krekel <holger.krekel@gmail.com> wrote:
On Tue, Feb 9, 2010 at 10:57 PM, Ben Finney <ben+python@benfinney.id.au> wrote:
Michael Foord <fuzzyman@voidspace.org.uk> writes:
The next 'big' change to unittest will (may?) be the introduction of class and module level setUp and tearDown. This was discussed on Python-ideas and Guido supported them. They can be useful but are also very easy to abuse (too much shared state, monolithic test classes and modules). Several authors of other Python testing frameworks spoke up *against* them, but several *users* of test frameworks spoke up in favour of them. ;-)
I think the perceived need for these is from people trying to use the unittest API for test that are *not* unit tests.
Well the example I was talking about before is when some (critical) resource needed for unittesting requires a very, very heavy initialization process. I'll employ the most recent example (hope it doesn't look like too much biased, it's just to illustrate the whole picture ;o) which is unittests for a framework like Trac . In that case it is critical to have a Trac environment, a ready-to-use DB and backend, initialize the plugins cache by loading relevant plugins, run the actions specified by each IEnvironmentSetup participant, sometimes a ready to use repository (if testing code depending on Trac VCS API) and more ... Just considering these cases someone could :
- Create a fake environment used as a stub - But having a single global environment is not a good idea because it would be very difficult to run multiple (independent) tests concurrently (e.g. test multiple Trac plugins concurrently in a dedica= ted CI environment). So an environment has to be started for every test run and be as isolated as possible from other similar stub environments - The DB and backend can be replaced by using in-memory SQLite connection - Plugins cache and loading is mandatory as well running the actions specified by each IEnvironmentSetup participant - VCS can be mocked, but if it's needed it has to be initialized as well
And all this is needed to run *ANY* test of *ANY* kind (that includes unittests ;o) . I hope that, up to this point, you all are convinced
This doesn't sound very unit-testy, really. It sounds like you are operating at a rather high level (closer to integration testing). As someone else said, I don't see anything wrong with using unittest as a basis for doing that myself, but I don't think your example is a clear example of wanting a setup and teardown that is executed once per TestCase for tests that are more obviously what everyone would consider "unit" tests. I do have an example of that, though. I have an external database containing test data. My unittests are generated on the fly so that each generated test method pulls one set of test data from the database and runs the appropriate checks that the package processes the data correctly. (If you are curious, I'm testing email header parsing, and there are a lot of different possible quirky headers that the parser needs to be checked against) Putting the test data in a database makes managing the test data easier, and makes it available to other test frameworks to reuse the data. So, having the connection to the database set up once at TestCase start, and closed at TestCase end, would make the most sense. Currently there's no way I know of to do that, so I open and close the database for every unittest. Fortunately it's an sqlite database, so the run time penalty for doing that isn't prohibitive. -- R. David Murray www.bitdance.com

On Wed, Feb 10, 2010 at 3:56 PM, R. David Murray <rdmurray@bitdance.com> wrote:
On Wed, 10 Feb 2010 09:45:41 -0500, Olemis Lang <olemis@gmail.com> wrote:
On Tue, Feb 9, 2010 at 5:34 PM, Holger Krekel <holger.krekel@gmail.com> wrote:
On Tue, Feb 9, 2010 at 10:57 PM, Ben Finney <ben+python@benfinney.id.au> wrote:
Michael Foord <fuzzyman@voidspace.org.uk> writes:
The next 'big' change to unittest will (may?) be the introduction of class and module level setUp and tearDown. This was discussed on Python-ideas and Guido supported them. They can be useful but are also very easy to abuse (too much shared state, monolithic test classes and modules). Several authors of other Python testing frameworks spoke up *against* them, but several *users* of test frameworks spoke up in favour of them. ;-)
I think the perceived need for these is from people trying to use the unittest API for test that are *not* unit tests.
Well the example I was talking about before is when some (critical) resource needed for unittesting requires a very, very heavy initialization process. I'll employ the most recent example (hope it doesn't look like too much biased, it's just to illustrate the whole picture ;o) which is unittests for a framework like Trac . In that case it is critical to have a Trac environment, a ready-to-use DB and backend, initialize the plugins cache by loading relevant plugins, run the actions specified by each IEnvironmentSetup participant, sometimes a ready to use repository (if testing code depending on Trac VCS API) and more ... Just considering these cases someone could :
- Create a fake environment used as a stub - But having a single global environment is not a good idea because it would be very difficult to run multiple (independent) tests concurrently (e.g. test multiple Trac plugins concurrently in a dedica= ted CI environment). So an environment has to be started for every test run and be as isolated as possible from other similar stub environments - The DB and backend can be replaced by using in-memory SQLite connection - Plugins cache and loading is mandatory as well running the actions specified by each IEnvironmentSetup participant - VCS can be mocked, but if it's needed it has to be initialized as well
And all this is needed to run *ANY* test of *ANY* kind (that includes unittests ;o) . I hope that, up to this point, you all are convinced
This doesn't sound very unit-testy, really. It sounds like you are operating at a rather high level (closer to integration testing). As someone else said, I don't see anything wrong with using unittest as a basis for doing that myself, but I don't think your example is a clear example of wanting a setup and teardown that is executed once per TestCase for tests that are more obviously what everyone would consider "unit" tests.
Well, probably this is OT here but I follow in order to clarify what I am saying. I am not integrating talking about integration tests, but in general, yes they are unittests, but for Trac plugins (i.e. it is possible that others tests won't need all this ;o) . For example let's consider TracRpc plugin. Let's say you are gonna implement an RPC handler that retrieves the ticket summary provided it's ID (pretty simple method indeed) . In that case you need - Implement IRPCHandler interface (in order to extend RPC system ;o) - Query ticket data Let's say you will only test that second part (which is the functional part without any objections ;o). In that case you'll still need a Trac environment, you'll need to setup the DB connection inside of it , and all that just to perform the query . In general, in such cases (e.g. DB access, but there are others ;o), almost everything needs a Trac environment and therefore, at least part of what I mentioned before ;o)
So, having the connection to the database set up once at TestCase start, and closed at TestCase end, would make the most sense. Currently there's no way I know of to do that, so I open and close the database for every unittest. Fortunately it's an sqlite database, so the run time penalty for doing that isn't prohibitive.
I really cannot see the difference between this and what I mentioned before since one of the things that's needed is to create a connexion just once for each test run, but (guess-what !) the connection needs to be set for the environment itself (i.e. trac.env.db ) so first the chicken, then the egg ;o) PS: BTW, The situation you mention is almost the classic example ;o) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: PEP 391 - Please Vote! - http://feedproxy.google.com/~r/TracGViz-full/~3/hY2h6ZSAFRE/110617

On 2010-02-09 15:57 PM, Ben Finney wrote:
Is there a better third-party framework for use in these cases? As Olemis points out later in this thread, I don't think it's good for the ‘unittest’ module to keep growing for uses that aren't focussed on unit tests (as contrasted with other kinds of tests).
nosetests allows you to write such module-level and class-level setup and teardown functions. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

On 09/02/2010 21:57, Ben Finney wrote:
Michael Foord<fuzzyman@voidspace.org.uk> writes:
The next 'big' change to unittest will (may?) be the introduction of class and module level setUp and tearDown. This was discussed on Python-ideas and Guido supported them. They can be useful but are also very easy to abuse (too much shared state, monolithic test classes and modules). Several authors of other Python testing frameworks spoke up *against* them, but several *users* of test frameworks spoke up in favour of them. ;-)
I think the perceived need for these is from people trying to use the ‘unittest’ API for test that are *not* unit tests.
That is, people have a need for integration tests (test this module's interaction with some other module) or system tests (test the behaviour of the whole running system). They then try to crowbar those tests into ‘unittest’ and finding it lacking, since ‘unittest’ is designed for tests of function-level units, without persistent state between those test cases.
I've used unittest for long running functional and integration tests (in both desktop and web applications). The infrastructure it provides is great for this. Don't get hung up on the fact that it is called unittest. In fact for many users the biggest reason it isn't suitable for tests like these is the lack of shared fixture support - which is why the other Python test frameworks provide them and we are going to bring it into unittest. Michael
Is there a better third-party framework for use in these cases? As Olemis points out later in this thread, I don't think it's good for the ‘unittest’ module to keep growing for uses that aren't focussed on unit tests (as contrasted with other kinds of tests).
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On 10:42 pm, fuzzyman@voidspace.org.uk wrote:
On 09/02/2010 21:57, Ben Finney wrote:
Michael Foord<fuzzyman@voidspace.org.uk> writes:
The next 'big' change to unittest will (may?) be the introduction of class and module level setUp and tearDown. This was discussed on Python-ideas and Guido supported them. They can be useful but are also very easy to abuse (too much shared state, monolithic test classes and modules). Several authors of other Python testing frameworks spoke up *against* them, but several *users* of test frameworks spoke up in favour of them. ;-) I think the perceived need for these is from people trying to use the 18unittest 19 API for test that are *not* unit tests.
That is, people have a need for integration tests (test this module's interaction with some other module) or system tests (test the behaviour of the whole running system). They then try to crowbar those tests into 18unittest 19 and finding it lacking, since 18unittest 19 is designed for tests of function-level units, without persistent state between those test cases.
I've used unittest for long running functional and integration tests (in both desktop and web applications). The infrastructure it provides is great for this. Don't get hung up on the fact that it is called unittest. In fact for many users the biggest reason it isn't suitable for tests like these is the lack of shared fixture support - which is why the other Python test frameworks provide them and we are going to bring it into unittest.
For what it's worth, we just finished *removing* support for setUpClass and tearDownClass from Trial. Jean-Paul

On Tue, Feb 9, 2010 at 6:15 PM, <exarkun@twistedmatrix.com> wrote:
On 10:42 pm, fuzzyman@voidspace.org.uk wrote:
On 09/02/2010 21:57, Ben Finney wrote:
Michael Foord<fuzzyman@voidspace.org.uk> writes:
The next 'big' change to unittest will (may?) be the introduction of class and module level setUp and tearDown. This was discussed on Python-ideas and Guido supported them. They can be useful but are also very easy to abuse (too much shared state, monolithic test classes and modules). Several authors of other Python testing frameworks spoke up *against* them, but several *users* of test frameworks spoke up in favour of them. ;-)
I think the perceived need for these is from people trying to use the 18unittest 19 API for test that are *not* unit tests.
That is, people have a need for integration tests (test this module's interaction with some other module) or system tests (test the behaviour of the whole running system). They then try to crowbar those tests into 18unittest 19 and finding it lacking, since 18unittest 19 is designed for tests of function-level units, without persistent state between those test cases.
I've used unittest for long running functional and integration tests (in both desktop and web applications). The infrastructure it provides is great for this. Don't get hung up on the fact that it is called unittest. In fact for many users the biggest reason it isn't suitable for tests like these is the lack of shared fixture support - which is why the other Python test frameworks provide them and we are going to bring it into unittest.
For what it's worth, we just finished *removing* support for setUpClass and tearDownClass from Trial.
Ok ... but why ? Are they considered dangerous for modern societies ? -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: PEP 391 - Please Vote! - http://feedproxy.google.com/~r/TracGViz-full/~3/hY2h6ZSAFRE/110617

On 10 Feb, 02:47 pm, olemis@gmail.com wrote:
On Tue, Feb 9, 2010 at 6:15 PM, <exarkun@twistedmatrix.com> wrote:
For what it's worth, we just finished *removing* support for setUpClass and tearDownClass from Trial.
Ok ... but why ? Are they considered dangerous for modern societies ?
Several reasons: - Over the many years the feature was available, we never found anyone actually benefiting significantly from it. It was mostly used where setUp/tearDown would have worked just as well. - There are many confusing corner cases related to ordering and error handling (particularly in the face of inheritance). Different users invariably have different expectations about how these things work, and there's no way to satisfy them all. One might say that this could apply to any feature, but... - People are exploring other solutions (such as testresources) which may provide better functionality more simply and don't need support deep in the loader/runner/reporter implementations. Jean-Paul

Michael Foord <fuzzyman@voidspace.org.uk> writes:
I've used unittest for long running functional and integration tests (in both desktop and web applications). The infrastructure it provides is great for this. Don't get hung up on the fact that it is called unittest. In fact for many users the biggest reason it isn't suitable for tests like these is the lack of shared fixture support - which is why the other Python test frameworks provide them and we are going to bring it into unittest.
I would argue that one of the things that makes ‘unittest’ good is that it makes it difficult to do the wrong thing — or at least *this* wrong thing. Fixtures persist for the lifetime of a single test case, and no more; that's the way unit tests should work. Making the distinction clearer by using a different API (and *not* extending the ‘unittest’ API) seems to be the right way to go. -- \ “I object to doing things that computers can do.” —Olin Shivers | `\ | _o__) | Ben Finney

On Tue, Feb 9, 2010 at 8:10 PM, Ben Finney <ben+python@benfinney.id.au> wrote:
Michael Foord <fuzzyman@voidspace.org.uk> writes:
I've used unittest for long running functional and integration tests (in both desktop and web applications). The infrastructure it provides is great for this. Don't get hung up on the fact that it is called unittest. In fact for many users the biggest reason it isn't suitable for tests like these is the lack of shared fixture support - which is why the other Python test frameworks provide them and we are going to bring it into unittest.
I would argue that one of the things that makes ‘unittest’ good is that it makes it difficult to do the wrong thing — or at least *this* wrong thing. Fixtures persist for the lifetime of a single test case, and no more; that's the way unit tests should work.
Making the distinction clearer by using a different API (and *not* extending the ‘unittest’ API) seems to be the right way to go.
If that means that development should be focused on including mechanisms to make unittest more extensible instead of complicating the current «relatively simple» API , then I agree . I think about unittest as a framework for writing test cases; but OTOH as a meta-framework to be used as the basic building blocks to build or integrate third-party testing infrastructures (and that includes third-party packages ;o) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Free milestone ranch Download - mac software - http://feedproxy.google.com/~r/TracGViz-full/~3/rX6_RmRWThE/

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Olemis Lang wrote:
On Tue, Feb 9, 2010 at 8:10 PM, Ben Finney <ben+python@benfinney.id.au> wrote:
Michael Foord <fuzzyman@voidspace.org.uk> writes:
I've used unittest for long running functional and integration tests (in both desktop and web applications). The infrastructure it provides is great for this. Don't get hung up on the fact that it is called unittest. In fact for many users the biggest reason it isn't suitable for tests like these is the lack of shared fixture support - which is why the other Python test frameworks provide them and we are going to bring it into unittest. I would argue that one of the things that makes ‘unittest’ good is that it makes it difficult to do the wrong thing — or at least *this* wrong thing. Fixtures persist for the lifetime of a single test case, and no more; that's the way unit tests should work.
Making the distinction clearer by using a different API (and *not* extending the ‘unittest’ API) seems to be the right way to go.
If that means that development should be focused on including mechanisms to make unittest more extensible instead of complicating the current «relatively simple» API , then I agree . I think about unittest as a framework for writing test cases; but OTOH as a meta-framework to be used as the basic building blocks to build or integrate third-party testing infrastructures (and that includes third-party packages ;o)
Just as a point of reference: zope.testing[1] has a "layer" feature which is used to support this usecase: a layer is a class namedd as an attribute of a testcase, e.g.: class FunctionalLayer: @classmethod def setUp(klass): """ Do some expesnive shared setup. """ @classmethod def tearDown(klass): """ Undo the expensive setup. """ class MyTest(unittest.TestCase): layer = FunctionalLayer The zope.testing testrunner groups testcase classes together by layer: each layer's setUp is called, then the testcases for that layer are run, then the layer's tearDown is called. Other features: - - Layer classes can define per-testcase-method 'testSetUp' and 'testTearDown' methods. - - Layers can be composed via inheritance, and don't need to call base layers' methods directly: the testrunner does that for them. These features has been in heavy use for about 3 1/2 years with a lot of success. [1] http://pypi.python.org/pypi/zope.testing/ Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkt0LeYACgkQ+gerLs4ltQ57WgCdFTzc1OHocXj/WTLShP62Q1bx vSAAnAqE/9+o1tZAaSLzlXfxaoRGTiuf =O/b2 -----END PGP SIGNATURE-----

On 04:18 pm, tseaver@palladion.com wrote:
Just as a point of reference: zope.testing[1] has a "layer" feature which is used to support this usecase: a layer is a class namedd as an attribute of a testcase, e.g.:
class FunctionalLayer: @classmethod def setUp(klass): """ Do some expesnive shared setup. """ @classmethod def tearDown(klass): """ Undo the expensive setup. """
class MyTest(unittest.TestCase): layer = FunctionalLayer
The zope.testing testrunner groups testcase classes together by layer: each layer's setUp is called, then the testcases for that layer are run, then the layer's tearDown is called.
Other features:
- - Layer classes can define per-testcase-method 'testSetUp' and 'testTearDown' methods.
- - Layers can be composed via inheritance, and don't need to call base layers' methods directly: the testrunner does that for them.
These features has been in heavy use for about 3 1/2 years with a lot of success.
On the other hand: http://code.mumak.net/2009/09/layers-are-terrible.html I've never used layers myself, so I won't personally weigh in for or against. Jean-Paul

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 exarkun@twistedmatrix.com wrote:
On 04:18 pm, tseaver@palladion.com wrote:
Just as a point of reference: zope.testing[1] has a "layer" feature which is used to support this usecase: a layer is a class namedd as an attribute of a testcase, e.g.:
class FunctionalLayer: @classmethod def setUp(klass): """ Do some expesnive shared setup. """ @classmethod def tearDown(klass): """ Undo the expensive setup. """
class MyTest(unittest.TestCase): layer = FunctionalLayer
The zope.testing testrunner groups testcase classes together by layer: each layer's setUp is called, then the testcases for that layer are run, then the layer's tearDown is called.
Other features:
- - Layer classes can define per-testcase-method 'testSetUp' and 'testTearDown' methods.
- - Layers can be composed via inheritance, and don't need to call base layers' methods directly: the testrunner does that for them.
These features has been in heavy use for about 3 1/2 years with a lot of success.
On the other hand:
http://code.mumak.net/2009/09/layers-are-terrible.html
I've never used layers myself, so I won't personally weigh in for or against.
I don't know the author of that post as a core Zope developer: the fact is that using inheritance to manage the layers works just fine for Zope's thousands of functional tests. As for his objections: if you don't want the superclass methods called, then don't make your layer inherit from it (why else would you?). Sharing setup across test methods is the whole point of layers, or of the other mechanisms being discussed here: while I agree that such tests aren't "unit tests" in the classic sense, they do have their place. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkt0OoAACgkQ+gerLs4ltQ5YSACeLzR+LfkafGB3GLWMgMPvdiPc 8nEAoKuudwJMznZiyrmJD1SHcOkYw3cr =6VG8 -----END PGP SIGNATURE-----

On Thu, Feb 11, 2010 at 11:18 AM, Tres Seaver <tseaver@palladion.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Olemis Lang wrote:
On Tue, Feb 9, 2010 at 8:10 PM, Ben Finney <ben+python@benfinney.id.au> wrote:
Michael Foord <fuzzyman@voidspace.org.uk> writes:
I've used unittest for long running functional and integration tests (in both desktop and web applications). The infrastructure it provides is great for this. Don't get hung up on the fact that it is called unittest. In fact for many users the biggest reason it isn't suitable for tests like these is the lack of shared fixture support - which is why the other Python test frameworks provide them and we are going to bring it into unittest. I would argue that one of the things that makes ‘unittest’ good is that it makes it difficult to do the wrong thing — or at least *this* wrong thing. Fixtures persist for the lifetime of a single test case, and no more; that's the way unit tests should work.
Making the distinction clearer by using a different API (and *not* extending the ‘unittest’ API) seems to be the right way to go.
If that means that development should be focused on including mechanisms to make unittest more extensible instead of complicating the current «relatively simple» API , then I agree . I think about unittest as a framework for writing test cases; but OTOH as a meta-framework to be used as the basic building blocks to build or integrate third-party testing infrastructures (and that includes third-party packages ;o)
Just as a point of reference: zope.testing[1] has a "layer" feature which is used to support this usecase: a layer is a class namedd as an attribute of a testcase, e.g.:
class FunctionalLayer: @classmethod def setUp(klass): """ Do some expesnive shared setup. """ @classmethod def tearDown(klass): """ Undo the expensive setup. """
class MyTest(unittest.TestCase): layer = FunctionalLayer
The zope.testing testrunner groups testcase classes together by layer: each layer's setUp is called, then the testcases for that layer are run, then the layer's tearDown is called.
Other features:
- - Layer classes can define per-testcase-method 'testSetUp' and 'testTearDown' methods.
- - Layers can be composed via inheritance, and don't need to call base layers' methods directly: the testrunner does that for them.
These features has been in heavy use for about 3 1/2 years with a lot of success.
I really like the style and the possibility to control the scope of ( setUp | tearDown ) . That's something I'd really consider to be included in the API ... and if it was accompanied or integrated to something like the @Rule in the backend to make it look like an extension and thus provide «standar mechanism(s)» to get other similar features done outside stdlib too, well, much better ;o) I have to start using Zope ! Damn, I'm wasting my few most happy years ! PS: I confess that I didn't follow the thread @ Py-Ideas. I associated Nick comment to the @Rule because, in JUnit, this is implemented using something similar to Aspect Oriented Programming (something like before and after hooks ;o), and in that case the Pythonic (and IMHO more «explicit») translation could be context managers . Perhaps I misunderstood something in previous messages . -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: PEP 391 - Please Vote! - http://feedproxy.google.com/~r/TracGViz-full/~3/hY2h6ZSAFRE/110617

Michael Foord wrote:
I'm not sure what response I expect from this email, and neither option will be implemented without further discussion - possibly at the PyCon sprints - but I thought I would make it clear what the possible directions are.
I'll repeat what I said in the python-ideas thread [1]: with the advent of PEP 343 and context managers, I see any further extension of the JUnit inspired setUp/tearDown nomenclature as an undesirable direction for Python to take. Instead, I believe unittest should be adjusted to allow appropriate definition of context managers that take effect at the level of the test module, test class and each individual test. For example, given the following method definitions in unittest.TestCase for backwards compatibility: def __enter__(self): self.setUp() def __exit__(self, *args): self.tearDown() The test framework might promise to do the following for each test: with get_module_cm(test_instance): # However identified with get_class_cm(test_instance): # However identified with test_instance: # ** test_instance.test_method() It would then be up to the design of the module and class context manager instances to cache any desired common state. Further design work would also be needed on the underlying API for identifying the module and class context managers given only the test instance to work with. The get_*_cm mechanisms would return a no-op CM if there was no specific CM defined for the supplied TestCase. Cheers, Nick. [1] http://mail.python.org/pipermail/python-ideas/2010-January/006758.html -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

On 11/02/2010 12:30, Nick Coghlan wrote:
Michael Foord wrote:
I'm not sure what response I expect from this email, and neither option will be implemented without further discussion - possibly at the PyCon sprints - but I thought I would make it clear what the possible directions are.
I'll repeat what I said in the python-ideas thread [1]: with the advent of PEP 343 and context managers, I see any further extension of the JUnit inspired setUp/tearDown nomenclature as an undesirable direction for Python to take.
Instead, I believe unittest should be adjusted to allow appropriate definition of context managers that take effect at the level of the test module, test class and each individual test.
For example, given the following method definitions in unittest.TestCase for backwards compatibility:
def __enter__(self): self.setUp()
def __exit__(self, *args): self.tearDown()
The test framework might promise to do the following for each test:
with get_module_cm(test_instance): # However identified with get_class_cm(test_instance): # However identified with test_instance: # ** test_instance.test_method()
Well that is *effectively* how they would work (the semantics) but I don't see how that would fit with the design of unittest to make them work *specifically* like that - especially not if we are to remain compatible with existing unittest extensions. If you can come up with a concrete proposal of how to do this then I'm happy to listen. I'm not saying it is impossible, but it isn't immediately obvious. I don't see any advantage of just using context managers for the sake of it and definitely not at the cost of backwards incompatibility. Michael
It would then be up to the design of the module and class context manager instances to cache any desired common state. Further design work would also be needed on the underlying API for identifying the module and class context managers given only the test instance to work with.
The get_*_cm mechanisms would return a no-op CM if there was no specific CM defined for the supplied TestCase.
Cheers, Nick.
[1] http://mail.python.org/pipermail/python-ideas/2010-January/006758.html
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On Thu, Feb 11, 2010 at 7:41 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
On 11/02/2010 12:30, Nick Coghlan wrote:
Michael Foord wrote:
I'm not sure what response I expect from this email, and neither option will be implemented without further discussion - possibly at the PyCon sprints - but I thought I would make it clear what the possible directions are.
I'll repeat what I said in the python-ideas thread [1]: with the advent of PEP 343 and context managers, I see any further extension of the JUnit inspired setUp/tearDown nomenclature as an undesirable direction for Python to take.
Instead, I believe unittest should be adjusted to allow appropriate definition of context managers that take effect at the level of the test module, test class and each individual test.
For example, given the following method definitions in unittest.TestCase for backwards compatibility:
def __enter__(self): self.setUp()
def __exit__(self, *args): self.tearDown()
The test framework might promise to do the following for each test:
with get_module_cm(test_instance): # However identified with get_class_cm(test_instance): # However identified with test_instance: # ** test_instance.test_method()
What Nick pointed out is the right direction (IMHO), and the one I had in mind since I realized that unittest extensibility is the key feature that needs to be implemented . I even wanted to start a project using this particular architecture to make PyUnit extensible. It's too bad (for me) that I don't have time at all, to move forward an just do it . :( I need days with 38 hrs !!! (at least) :$
Well that is *effectively* how they would work (the semantics) but I don't see how that would fit with the design of unittest to make them work *specifically* like that - especially not if we are to remain compatible with existing unittest extensions.
AFAICS (so not sure especially since there's nothing done to criticize ;o) is that backwards compatibility is not the main stopper ...
If you can come up with a concrete proposal of how to do this then I'm happy to listen. I'm not saying it is impossible, but it isn't immediately obvious. I don't see any advantage of just using context managers for the sake of it and definitely not at the cost of backwards incompatibility.
... but since I have nothing I can show you , everything is still in my mind ... -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Free milestone ranch Download - mac software - http://feedproxy.google.com/~r/TracGViz-full/~3/rX6_RmRWThE/

On 02:41 pm, olemis@gmail.com wrote:
On Thu, Feb 11, 2010 at 7:41 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
On 11/02/2010 12:30, Nick Coghlan wrote:
Michael Foord wrote:
I'm not sure what response I expect from this email, and neither option will be implemented without further discussion - possibly at the PyCon sprints - but I thought I would make it clear what the possible directions are.
I'll repeat what I said in the python-ideas thread [1]: with the advent of PEP 343 and context managers, I see any further extension of the JUnit inspired setUp/tearDown nomenclature as an undesirable direction for Python to take.
Instead, I believe unittest should be adjusted to allow appropriate definition of context managers that take effect at the level of the test module, test class and each individual test.
For example, given the following method definitions in unittest.TestCase for backwards compatibility:
� def __enter__(self): � � self.setUp()
� def __exit__(self, *args): � � self.tearDown()
The test framework might promise to do the following for each test:
� with get_module_cm(test_instance): # However identified � � with get_class_cm(test_instance): # However identified � � � with test_instance: # ** � � � � test_instance.test_method()
What Nick pointed out is the right direction (IMHO), and the one I had
Why? Change for the sake of change is not a good thing. What are the advantages of switching to context managers for this? Perhaps the idea was more strongly justified in the python-ideas thread. Anyone have a link to that?
in mind since I realized that unittest extensibility is the key feature that needs to be implemented . I even wanted to start a project using this particular architecture to make PyUnit extensible.
What makes you think it isn't extensible now? Lots of people are extending it in lots of ways. Jean-Paul

On Thu, Feb 11, 2010 at 10:10 AM, <exarkun@twistedmatrix.com> wrote:
On 02:41 pm, olemis@gmail.com wrote:
On Thu, Feb 11, 2010 at 7:41 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
On 11/02/2010 12:30, Nick Coghlan wrote:
Michael Foord wrote:
I'm not sure what response I expect from this email, and neither option will be implemented without further discussion - possibly at the PyCon sprints - but I thought I would make it clear what the possible directions are.
I'll repeat what I said in the python-ideas thread [1]: with the advent of PEP 343 and context managers, I see any further extension of the JUnit inspired setUp/tearDown nomenclature as an undesirable direction for Python to take.
Instead, I believe unittest should be adjusted to allow appropriate definition of context managers that take effect at the level of the test module, test class and each individual test.
For example, given the following method definitions in unittest.TestCase for backwards compatibility:
def __enter__(self): self.setUp()
def __exit__(self, *args): self.tearDown()
The test framework might promise to do the following for each test:
with get_module_cm(test_instance): # However identified with get_class_cm(test_instance): # However identified with test_instance: # ** test_instance.test_method()
What Nick pointed out is the right direction (IMHO), and the one I had
Why? Change for the sake of change is not a good thing. What are the advantages of switching to context managers for this?
Perhaps the idea was more strongly justified in the python-ideas thread. Anyone have a link to that?
in mind since I realized that unittest extensibility is the key feature that needs to be implemented . I even wanted to start a project using this particular architecture to make PyUnit extensible.
What makes you think it isn't extensible now? Lots of people are extending it in lots of ways.
Nothing I want to spend my time on. Just consider what the authors of JUnit (and XUnit too) thought about JUnit<4.7, what they did in JUnit 4.7, and you'll save me a lot of time I don't have to explain it to you (/me not being rude /me have no time :-/ ) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Nabble - Trac Users - Embedding pages? - http://feedproxy.google.com/~r/TracGViz-full/~3/MWT7MJBi08w/Embedding-pages-...

On Thu, Feb 11, 2010 at 9:41 AM, Olemis Lang <olemis@gmail.com> wrote:
On Thu, Feb 11, 2010 at 7:41 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
On 11/02/2010 12:30, Nick Coghlan wrote:
Michael Foord wrote:
I'm not sure what response I expect from this email, and neither option will be implemented without further discussion - possibly at the PyCon sprints - but I thought I would make it clear what the possible directions are.
I'll repeat what I said in the python-ideas thread [1]: with the advent of PEP 343 and context managers, I see any further extension of the JUnit inspired setUp/tearDown nomenclature as an undesirable direction for Python to take.
Instead, I believe unittest should be adjusted to allow appropriate definition of context managers that take effect at the level of the test module, test class and each individual test.
For example, given the following method definitions in unittest.TestCase for backwards compatibility:
def __enter__(self): self.setUp()
def __exit__(self, *args): self.tearDown()
The test framework might promise to do the following for each test:
with get_module_cm(test_instance): # However identified with get_class_cm(test_instance): # However identified with test_instance: # ** test_instance.test_method()
What Nick pointed out is the right direction (IMHO), and the one I had in mind since I realized that unittest extensibility is the key feature that needs to be implemented . I even wanted to start a project using this particular architecture to make PyUnit extensible. It's too bad (for me) that I don't have time at all, to move forward an just do it .
:(
I need days with 38 hrs !!! (at least)
:$
Well that is *effectively* how they would work (the semantics) but I don't see how that would fit with the design of unittest to make them work *specifically* like that - especially not if we are to remain compatible with existing unittest extensions.
AFAICS (so not sure especially since there's nothing done to criticize ;o) is that backwards compatibility is not the main stopper ...
If you can come up with a concrete proposal of how to do this then I'm happy to listen. I'm not saying it is impossible, but it isn't immediately obvious. I don't see any advantage of just using context managers for the sake of it and definitely not at the cost of backwards incompatibility.
... but since I have nothing I can show you , everything is still in my mind ...
The idea (at least the one in my head ;o) is based on the features recently introduced in JUnit 4.7, especially the @Rule ;o) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Free milestone ranch Download - mac software - http://feedproxy.google.com/~r/TracGViz-full/~3/rX6_RmRWThE/

On Thu, Feb 11, 2010 at 10:11 AM, Olemis Lang <olemis@gmail.com> wrote:
On Thu, Feb 11, 2010 at 9:41 AM, Olemis Lang <olemis@gmail.com> wrote:
On Thu, Feb 11, 2010 at 7:41 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
On 11/02/2010 12:30, Nick Coghlan wrote:
Michael Foord wrote:
I'm not sure what response I expect from this email, and neither option will be implemented without further discussion - possibly at the PyCon sprints - but I thought I would make it clear what the possible directions are.
I'll repeat what I said in the python-ideas thread [1]: with the advent of PEP 343 and context managers, I see any further extension of the JUnit inspired setUp/tearDown nomenclature as an undesirable direction for Python to take.
Instead, I believe unittest should be adjusted to allow appropriate definition of context managers that take effect at the level of the test module, test class and each individual test.
For example, given the following method definitions in unittest.TestCase for backwards compatibility:
def __enter__(self): self.setUp()
def __exit__(self, *args): self.tearDown()
The test framework might promise to do the following for each test:
with get_module_cm(test_instance): # However identified with get_class_cm(test_instance): # However identified with test_instance: # ** test_instance.test_method()
What Nick pointed out is the right direction (IMHO), and the one I had in mind since I realized that unittest extensibility is the key feature that needs to be implemented . I even wanted to start a project using this particular architecture to make PyUnit extensible. It's too bad (for me) that I don't have time at all, to move forward an just do it .
:(
I need days with 38 hrs !!! (at least)
:$
Well that is *effectively* how they would work (the semantics) but I don't see how that would fit with the design of unittest to make them work *specifically* like that - especially not if we are to remain compatible with existing unittest extensions.
AFAICS (so not sure especially since there's nothing done to criticize ;o) is that backwards compatibility is not the main stopper ...
If you can come up with a concrete proposal of how to do this then I'm happy to listen. I'm not saying it is impossible, but it isn't immediately obvious. I don't see any advantage of just using context managers for the sake of it and definitely not at the cost of backwards incompatibility.
... but since I have nothing I can show you , everything is still in my mind ...
The idea (at least the one in my head ;o) is based on the features recently introduced in JUnit 4.7, especially the @Rule
;o)
.. [1] Writing your own JUnit extensions using @Rule | JUnit.org (http://www.junit.org/node/580) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: setUpClass and setUpModule in unittest | Python | Dev - http://feedproxy.google.com/~r/TracGViz-full/~3/x18-60vceqg/806136

On Thu, 11 Feb 2010 12:41:37 +0000, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
On 11/02/2010 12:30, Nick Coghlan wrote:
The test framework might promise to do the following for each test:
with get_module_cm(test_instance): # However identified with get_class_cm(test_instance): # However identified with test_instance: # ** test_instance.test_method()
Well that is *effectively* how they would work (the semantics) but I don't see how that would fit with the design of unittest to make them work *specifically* like that - especially not if we are to remain compatible with existing unittest extensions.
If you can come up with a concrete proposal of how to do this then I'm happy to listen. I'm not saying it is impossible, but it isn't immediately obvious. I don't see any advantage of just using context managers for the sake of it and definitely not at the cost of backwards incompatibility.
I suspect that Nick is saying that it is worth doing for the sake of it, as being more "Pythonic" in some sense. That is, it seems to me that in a modern Python writing something like: @contextlib.contextmanager def foo_cm(testcase): testcase.bar = some_costly_setup_function() yield testcase.bar.close() @contextlib.contextmanager def foo_test_cm(testcase): testcase.baz = Mock(testcase.bar) yield @unittest.case_context(foo_cm) @unittest.test_context(foo_test_cm) class TestFoo(unittest.TestCase): def test_bar: foo = Foo(self.baz, testing=True) self.assertTrue("Context managers are cool") would be easier to write, be more maintainable, and be easier to understand when reading the code than the equivalent setUp and tearDown methods would be. I'm not saying it would be easy to implement, and as you say backward compatibility is a key concern. -- R. David Murray www.bitdance.com

On 11/02/2010 15:56, R. David Murray wrote:
On Thu, 11 Feb 2010 12:41:37 +0000, Michael Foord<fuzzyman@voidspace.org.uk> wrote:
On 11/02/2010 12:30, Nick Coghlan wrote:
The test framework might promise to do the following for each test:
with get_module_cm(test_instance): # However identified with get_class_cm(test_instance): # However identified with test_instance: # ** test_instance.test_method()
Well that is *effectively* how they would work (the semantics) but I don't see how that would fit with the design of unittest to make them work *specifically* like that - especially not if we are to remain compatible with existing unittest extensions.
If you can come up with a concrete proposal of how to do this then I'm happy to listen. I'm not saying it is impossible, but it isn't immediately obvious. I don't see any advantage of just using context managers for the sake of it and definitely not at the cost of backwards incompatibility.
I suspect that Nick is saying that it is worth doing for the sake of it, as being more "Pythonic" in some sense.
That is, it seems to me that in a modern Python writing something like:
@contextlib.contextmanager def foo_cm(testcase): testcase.bar = some_costly_setup_function() yield testcase.bar.close()
@contextlib.contextmanager def foo_test_cm(testcase): testcase.baz = Mock(testcase.bar) yield
@unittest.case_context(foo_cm) @unittest.test_context(foo_test_cm) class TestFoo(unittest.TestCase):
def test_bar: foo = Foo(self.baz, testing=True) self.assertTrue("Context managers are cool")
would be easier to write, be more maintainable, and be easier to understand when reading the code than the equivalent setUp and tearDown methods would be.
I'm not saying it would be easy to implement, and as you say backward compatibility is a key concern.
This is quite different to what Nick *specifically* suggested. It also doesn't suggest a general approach that would easily allow for setUpModule as well. *However*, I am *hoping* to be able to incorporate some or all of Test Resources as a general solution (with simple recipes for the setUpClass and setUpModule cases) - at which point this particular discussion will become moot. All the best, Michael Foord
-- R. David Murray www.bitdance.com
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On Thu, 11 Feb 2010 16:08:54 +0000, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
On 11/02/2010 15:56, R. David Murray wrote:
On Thu, 11 Feb 2010 12:41:37 +0000, Michael Foord<fuzzyman@voidspace.org.uk> wrote:
On 11/02/2010 12:30, Nick Coghlan wrote:
The test framework might promise to do the following for each test:
with get_module_cm(test_instance): # However identified with get_class_cm(test_instance): # However identified with test_instance: # ** test_instance.test_method()
@contextlib.contextmanager def foo_cm(testcase): testcase.bar = some_costly_setup_function() yield testcase.bar.close()
@contextlib.contextmanager def foo_test_cm(testcase): testcase.baz = Mock(testcase.bar) yield
@unittest.case_context(foo_cm) @unittest.test_context(foo_test_cm) class TestFoo(unittest.TestCase):
def test_bar: foo = Foo(self.baz, testing=True) self.assertTrue("Context managers are cool")
This is quite different to what Nick *specifically* suggested. It also doesn't suggest a general approach that would easily allow for setUpModule as well.
I'm not sure how it is different. I thought I was indicating how to do the context manager "discovery" that Nick punted on. (Except for module level, which I didn't have a good idea for).
*However*, I am *hoping* to be able to incorporate some or all of Test Resources as a general solution (with simple recipes for the setUpClass and setUpModule cases) - at which point this particular discussion will become moot.
Which pretty much makes my noddling above moot right now, because having taken a quick look at testresources I think that's a much closer fit for my use cases than class level setup/teardown. So I'm +1 for going the testresources route rather than the setup/teardown route. -- R. David Murray www.bitdance.com

Le Thu, 11 Feb 2010 10:56:32 -0500, R. David Murray a écrit :
@unittest.case_context(foo_cm) @unittest.test_context(foo_test_cm) class TestFoo(unittest.TestCase):
def test_bar: foo = Foo(self.baz, testing=True) self.assertTrue("Context managers are cool")
would be easier to write, be more maintainable, and be easier to understand when reading the code than the equivalent setUp and tearDown methods would be.
I don't think it would be seriously easier to write, more maintainable or easier to understand. There's nothing complicated or obscure in setUp and tearDown methods (the only annoying thing being PEP8 non-compliance). As a matter of fact, nose has a "with_setup()" decorator which allows to avoid writing setUp/tearDown methods. But in my experience it's more annoying to use because: - you have to add the decorator explicitly (setUp/tearDown is always invoked) - you have to create your own recipient for local state (setUp/tearDown can simply use the TestCase instance), or use global variables which is ugly. Regards Antoine.

R. David Murray wrote:
would be easier to write, be more maintainable, and be easier to understand when reading the code than the equivalent setUp and tearDown methods would be.
I'm not saying it would be easy to implement, and as you say backward compatibility is a key concern.
That's the gist of my thinking, yeah. However, a couple of problems of note that occurred to me after an extra day or so of musing on the topic: - the semantics I listed in my original post are broken, since a naive context manager couldn't be used (it would setup and tear down the resources for each test instance, which is what we're trying to avoid). Supporting naive context managers would require iterating over the test classes within the module CM and iterating over the instances within the class CM. - context managers fit best in more procedural code. They're tricky to invoke correctly from code that is split across several methods in different classes (as I believe unittest is), since you can't use a with statement directly to do the invocation for you So I think new setup*/teardown* methods and functions are likely to be a better fit for the unittest architecture. At a later date, it may be worth adding some mixins or other mechanisms that adapt from the unittest setup/teardown model to a CM based model, but to be honest, if I want to use a CM when testing, I'll generally create a more complex test method that iterates through a bunch of test inputs itself. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

R. David Murray wrote:
On Thu, 11 Feb 2010 12:41:37 +0000, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
On 11/02/2010 12:30, Nick Coghlan wrote:
The test framework might promise to do the following for each test:
with get_module_cm(test_instance): # However identified with get_class_cm(test_instance): # However identified with test_instance: # ** test_instance.test_method() Well that is *effectively* how they would work (the semantics) but I don't see how that would fit with the design of unittest to make them work *specifically* like that - especially not if we are to remain compatible with existing unittest extensions.
If you can come up with a concrete proposal of how to do this then I'm happy to listen. I'm not saying it is impossible, but it isn't immediately obvious. I don't see any advantage of just using context managers for the sake of it and definitely not at the cost of backwards incompatibility.
I suspect that Nick is saying that it is worth doing for the sake of it, as being more "Pythonic" in some sense.
That is, it seems to me that in a modern Python writing something like:
@contextlib.contextmanager def foo_cm(testcase): testcase.bar = some_costly_setup_function() yield testcase.bar.close()
@contextlib.contextmanager def foo_test_cm(testcase): testcase.baz = Mock(testcase.bar) yield
@unittest.case_context(foo_cm) @unittest.test_context(foo_test_cm) class TestFoo(unittest.TestCase):
def test_bar: foo = Foo(self.baz, testing=True) self.assertTrue("Context managers are cool")
This reminds me of the decorators I have available in testfixtures: http://packages.python.org/testfixtures/mocking.html http://packages.python.org/testfixtures/logging.html http://packages.python.org/testfixtures/files.html (the last of which is a lot prettier in svn, not had a chance to release :-S) Anyway, these I've ended up making available as context managers as well as decorators... But yes, something similar for sharing state between tests and/or doing setup for each test would be nice :-) cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

On Tue, Feb 9, 2010 at 8:42 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
The next 'big' change to unittest will (may?) be the introduction of class and module level setUp and tearDown. This was discussed on Python-ideas and Guido supported them. They can be useful but are also very easy to abuse (too much shared state, monolithic test classes and modules). Several authors of other Python testing frameworks spoke up *against* them, but several *users* of test frameworks spoke up in favour of them. ;-)
Hi Michael, I have skimmed this thread (hence this reply to the first rather than the last message), but in general I am baffled by the hostility of testing framework developers towards their users. The arguments against class- and module-level seUp/tearDown functions seems to be inspired by religion or ideology more than by the zen of Python. What happened to Practicality Beats Purity? The potential for abuse in and of itself should not be an argument against a feature; it must always be weighed against the advantages. The argument that a unittest framework shouldn't be "abused" for regression tests (or integration tests, or whatever) is also bizarre to my mind. Surely if a testing framework applies to multiple kinds of testing that's a good thing, not something to be frowned upon? There are several alternative testing frameworks available outside the standard library. The provide useful competition with the stlib's unittest and doctest modules, and useful inspiration for potential new features. They also, by and large, evolve much faster than a stdlib module ever could, and including anyone of these in the stdlib might well be the death of it (just as unittest has evolved much slower since it was included). But unittest *is* still evolving, and there is no reason not to keep adding features along the lines of your module/class setUp/tearDown proposal (or extra assertions like assertListEqual, which I am happy to see has been added). On the other hand, I think we should be careful to extend unittest in a consistent way. I shuddered at earlier proposals (on python-ideas) to name the new functions (variations of) set_up and tear_down "to conform with PEP 8" (this would actually have violated that PEP, which explicitly prefers local consistency over global consistency). I also think that using a with-statement or a decorator to indicate the scope of setUp/tearDown operations flies in the face of the existing "style" of the unittest module (um, package, I know :-), which is based on defining setUp and tearDown methods with specific semantics. Regarding the objection that setUp/tearDown for classes would run into issues with subclassing, I propose to let the standard semantics of subclasses do their job. Thus a subclass that overrides setUpClass or tearDownClass is responsible for calling the base class's setUpClass and tearDownClass (and the TestCase base class should provide empty versions of both). The testrunner should only call setUpClass and tearDownClass for classes that have at least one test that is selected. Yes, this would mean that if a base class has a test method and a setUpClass (and tearDownClass) method and a subclass also has a test method and overrides setUpClass (and/or tearDown), the base class's setUpClass and tearDown may be called twice. What's the big deal? If setUpClass and tearDownClass are written properly they should support this. If this behavior is undesired in a particular case, maybe what was really meant were module-level setUp and tearDown, or the class structure should be rearranged. Anyway, Michael, thanks for getting this started -- I support your attempts to improve the unittest package and am writing in the hope that the discussion will soon converge and patches whipped up. -- --Guido van Rossum (python.org/~guido)

On Thu, Feb 11, 2010 at 1:11 PM, Guido van Rossum <guido@python.org> wrote:
On Tue, Feb 9, 2010 at 8:42 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
The next 'big' change to unittest will (may?) be the introduction of class and module level setUp and tearDown. This was discussed on Python-ideas and Guido supported them. They can be useful but are also very easy to abuse (too much shared state, monolithic test classes and modules). Several authors of other Python testing frameworks spoke up *against* them, but several *users* of test frameworks spoke up in favour of them. ;-)
But unittest *is* still evolving,
as well as the XUnit paradigm as a whole, especially considering the recent work committed to and released by JUnit ;o) .
On the other hand, I think we should be careful to extend unittest in a consistent way.
+1 . IMO that's a key indicator of the success of anything related to its evolution .
Regarding the objection that setUp/tearDown for classes would run into issues with subclassing, I propose to let the standard semantics of subclasses do their job. Thus a subclass that overrides setUpClass or tearDownClass is responsible for calling the base class's setUpClass and tearDownClass (and the TestCase base class should provide empty versions of both). The testrunner should only call setUpClass and tearDownClass for classes that have at least one test that is selected.
+1 Considering zope.testing layers proposal, it seems that subclassing of layers works different, isn't it ? -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Nabble - Trac Users - Embedding pages? - http://feedproxy.google.com/~r/TracGViz-full/~3/MWT7MJBi08w/Embedding-pages-...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Olemis Lang wrote:
On Thu, Feb 11, 2010 at 1:11 PM, Guido van Rossum <guido@python.org> wrote:
Regarding the objection that setUp/tearDown for classes would run into issues with subclassing, I propose to let the standard semantics of subclasses do their job. Thus a subclass that overrides setUpClass or tearDownClass is responsible for calling the base class's setUpClass and tearDownClass (and the TestCase base class should provide empty versions of both). The testrunner should only call setUpClass and tearDownClass for classes that have at least one test that is selected.
+1
Considering zope.testing layers proposal, it seems that subclassing of layers works different, isn't it ?
Hmm, I wasn't making a proposal that the unittest module adopt zope.testing's layers model: I was just trying to point out that another model did exist and was being used successfully. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkt0WcUACgkQ+gerLs4ltQ6CgACfb9kQ6vpu6BwOJLBOLDnHnTil dZMAnjdkdT/5RQXGIWFXGuUgnV8rQSuI =ExUu -----END PGP SIGNATURE-----

Hi Guido, On Thu, Feb 11, 2010 at 7:11 PM, Guido van Rossum <guido@python.org> wrote:
On Tue, Feb 9, 2010 at 8:42 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
The next 'big' change to unittest will (may?) be the introduction of class and module level setUp and tearDown. This was discussed on Python-ideas and Guido supported them. They can be useful but are also very easy to abuse (too much shared state, monolithic test classes and modules). Several authors of other Python testing frameworks spoke up *against* them, but several *users* of test frameworks spoke up in favour of them. ;-)
Hi Michael,
I have skimmed this thread (hence this reply to the first rather than the last message), but in general I am baffled by the hostility of testing framework developers towards their users. The arguments against class- and module-level seUp/tearDown functions seems to be inspired by religion or ideology more than by the zen of Python. What happened to Practicality Beats Purity?
Hostility against users? I have not heart that feedback from my users yet - or am i missing some meaning of your words?
The potential for abuse in and of itself should not be an argument against a feature; it must always be weighed against the advantages.
sure.
The argument that a unittest framework shouldn't be "abused" for regression tests (or integration tests, or whatever) is also bizarre to my mind. Surely if a testing framework applies to multiple kinds of testing that's a good thing, not something to be frowned upon?
If an approach has known limitations it's also good to point them out. Also ok to disregard them and still consider something useful enough.
There are several alternative testing frameworks available outside the standard library. The provide useful competition with the stlib's unittest and doctest modules, and useful inspiration for potential new features. They also, by and large, evolve much faster than a stdlib module ever could, and including anyone of these in the stdlib might well be the death of it (just as unittest has evolved much slower since it was included).
Fully agreed :)
But unittest *is* still evolving, and there is no reason not to keep adding features along the lines of your module/class setUp/tearDown proposal (or extra assertions like assertListEqual, which I am happy to see has been added).
On the other hand, I think we should be careful to extend unittest in a consistent way. I shuddered at earlier proposals (on python-ideas) to name the new functions (variations of) set_up and tear_down "to conform with PEP 8" (this would actually have violated that PEP, which explicitly prefers local consistency over global consistency).
If that was me you refer to - i followed PEP8 5 years ago when introducing setup_class/module and i still stand by it, it was supposed to be a more pythonic alternative and i consider PEP8 as part of that. But i agree - introducing it to std-unittest now makes not much sense due to local consistency reasons. I appreciate Michael's effort to help advance testing - we have a good private discussion currently btw - and i am happy to collaborate with him on future issues, setupClass or not :) cheers, holger

On Thu, Feb 11, 2010 at 11:26 AM, Holger Krekel <holger.krekel@gmail.com> wrote:
Hi Guido,
On Thu, Feb 11, 2010 at 7:11 PM, Guido van Rossum <guido@python.org> wrote:
On Tue, Feb 9, 2010 at 8:42 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
The next 'big' change to unittest will (may?) be the introduction of class and module level setUp and tearDown. This was discussed on Python-ideas and Guido supported them. They can be useful but are also very easy to abuse (too much shared state, monolithic test classes and modules). Several authors of other Python testing frameworks spoke up *against* them, but several *users* of test frameworks spoke up in favour of them. ;-)
Hi Michael,
I have skimmed this thread (hence this reply to the first rather than the last message), but in general I am baffled by the hostility of testing framework developers towards their users. The arguments against class- and module-level seUp/tearDown functions seems to be inspired by religion or ideology more than by the zen of Python. What happened to Practicality Beats Purity?
Hostility against users? I have not heart that feedback from my users yet - or am i missing some meaning of your words?
Sorry for the sweeping generality. I was referring to one or more posts (I don't recall by whom) arguing against including class/module setup/teardown functionality based on it being against the notion of unittesting or something like that. I'm sorry, but the thread is too long for me to find the specific post. But I'm pretty sure I saw something like that.
The potential for abuse in and of itself should not be an argument against a feature; it must always be weighed against the advantages.
sure.
The argument that a unittest framework shouldn't be "abused" for regression tests (or integration tests, or whatever) is also bizarre to my mind. Surely if a testing framework applies to multiple kinds of testing that's a good thing, not something to be frowned upon?
If an approach has known limitations it's also good to point them out. Also ok to disregard them and still consider something useful enough.
There are several alternative testing frameworks available outside the standard library. The provide useful competition with the stlib's unittest and doctest modules, and useful inspiration for potential new features. They also, by and large, evolve much faster than a stdlib module ever could, and including anyone of these in the stdlib might well be the death of it (just as unittest has evolved much slower since it was included).
Fully agreed :)
But unittest *is* still evolving, and there is no reason not to keep adding features along the lines of your module/class setUp/tearDown proposal (or extra assertions like assertListEqual, which I am happy to see has been added).
On the other hand, I think we should be careful to extend unittest in a consistent way. I shuddered at earlier proposals (on python-ideas) to name the new functions (variations of) set_up and tear_down "to conform with PEP 8" (this would actually have violated that PEP, which explicitly prefers local consistency over global consistency).
If that was me you refer to - i followed PEP8 5 years ago when introducing setup_class/module and i still stand by it, it was supposed to be a more pythonic alternative and i consider PEP8 as part of that. But i agree - introducing it to std-unittest now makes not much sense due to local consistency reasons.
Ok let's drop it then.
I appreciate Michael's effort to help advance testing - we have a good private discussion currently btw - and i am happy to collaborate with him on future issues, setupClass or not :)
cheers, holger
-- --Guido van Rossum (python.org/~guido)

Guido van Rossum <guido@python.org> writes:
The potential for abuse in and of itself should not be an argument against a feature; it must always be weighed against the advantages.
It's both, surely? The potential for abuse of something is an argument against it; *and* that argument should be weighed against other arguments. Or, in other words: the potential for abuse of a feature is an argument that should not be discarded solely because there are advantages to that feature.
The argument that a unittest framework shouldn't be "abused" for regression tests (or integration tests, or whatever) is also bizarre to my mind. Surely if a testing framework applies to multiple kinds of testing that's a good thing, not something to be frowned upon?
To my mind, an API should take a stand on the “right” way to use it, rather than being a kitchen-sink of whatever ideas had any support. Doing something the right way should be easy, and doing something the wrong way should be awkward. This must be balanced, of course, with the principle that easy things should be easy and difficult things should be possible. But it doesn't necessarily conflict; we just need to take care that the easy and the right align well :-)
There are several alternative testing frameworks available outside the standard library. The provide useful competition with the stlib's unittest and doctest modules, and useful inspiration for potential new features. They also, by and large, evolve much faster than a stdlib module ever could, and including anyone of these in the stdlib might well be the death of it (just as unittest has evolved much slower since it was included).
Right. This is an argument in favour of being assertive and parsimonious in the design of the standard-library ‘unittest’ API: this is the clear and obvious way to use this API, and if someone wants to do it a different way there are alternatives available.
But unittest *is* still evolving, and there is no reason not to keep adding features along the lines of your module/class setUp/tearDown proposal (or extra assertions like assertListEqual, which I am happy to see has been added).
That's a dismissal of the reasons that have been presented, without actually countering those reasons.
Anyway, Michael, thanks for getting this started -- I support your attempts to improve the unittest package and am writing in the hope that the discussion will soon converge and patches whipped up.
Ditto. -- \ “I have had a perfectly wonderful evening, but this wasn't it.” | `\ —Groucho Marx | _o__) | Ben Finney

On 2010-02-11 16:20 PM, Ben Finney wrote:
Guido van Rossum<guido@python.org> writes:
The argument that a unittest framework shouldn't be "abused" for regression tests (or integration tests, or whatever) is also bizarre to my mind. Surely if a testing framework applies to multiple kinds of testing that's a good thing, not something to be frowned upon?
To my mind, an API should take a stand on the “right” way to use it, rather than being a kitchen-sink of whatever ideas had any support. Doing something the right way should be easy, and doing something the wrong way should be awkward.
setUpClass and setUpModule are the "right" way to do many types of integration and functional tests. Integration and functional tests are vital tasks to perform, and unittest provides a good framework otherwise for implementing such tests.
There are several alternative testing frameworks available outside the standard library. The provide useful competition with the stlib's unittest and doctest modules, and useful inspiration for potential new features. They also, by and large, evolve much faster than a stdlib module ever could, and including anyone of these in the stdlib might well be the death of it (just as unittest has evolved much slower since it was included).
Right. This is an argument in favour of being assertive and parsimonious in the design of the standard-library ‘unittest’ API: this is the clear and obvious way to use this API, and if someone wants to do it a different way there are alternatives available.
I would agree if the requirements for unit testing and integration/functional tests were so different. However, unittest provides most of the necessary infrastructure that is common to all of those kinds of testing. It's just that the latter kinds of tests also could use setUpClass/setUpModule. It would be a waste (and honestly kind of ridiculous) to force people to use a whole new framework (which would duplicate unittest in almost its entirety) for want of those two methods. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

On Fri, Feb 12, 2010 at 12:00 AM, Robert Kern <robert.kern@gmail.com> wrote:
On 2010-02-11 16:20 PM, Ben Finney wrote:
Guido van Rossum<guido@python.org> writes:
The argument that a unittest framework shouldn't be "abused" for regression tests (or integration tests, or whatever) is also bizarre to my mind. Surely if a testing framework applies to multiple kinds of testing that's a good thing, not something to be frowned upon?
To my mind, an API should take a stand on the “right” way to use it, rather than being a kitchen-sink of whatever ideas had any support. Doing something the right way should be easy, and doing something the wrong way should be awkward.
setUpClass and setUpModule are the "right" way to do many types of integration and functional tests. Integration and functional tests are vital tasks to perform, and unittest provides a good framework otherwise for implementing such tests.
Ben just expressed his opinion about API design and you claim some truth about testing in general. In my experience, integration and functional testing is a complex and evolving topic, usually requiring more from the tool or framework than classic unit-testing. To name a few issues: * creating tempdirs and files * setting up base environments * starting and stopping servers * mocking components * replaying individual tests * reacting to timeouts * handling asynchronicity * web-javascript integration support * configuring fixtures from config files * CI tool integration and multi-platform deployment * running variations of the same tests across different base configs * ... much much more It's true that you can go and extend unittest for that but a) unittest is just a tiny bit of what is involved for satisfying the needs b) what you are doing then is mostly using the fact that a setup function (or chain) is invoked and a test function is invoked and that python has some builtin modules for handling the above issues. And you are using Python - and Python is nice and (potentially) concise for writing tests, sure. That's not wholly the fault of the unittest module, though :) So. Doing fixtures via static encoding in class and module setup functions is a way to provide a generic framing for writing tests. The "right" way? In many cases and for the about 6 different people i interacted with (and on actual RL code) in the last 2 weeks it does not help incredibly much. There is experiences from other test tool authors indicating similar experiences. I will say that module/class can be helpful and you can do some useful things with it and thus it makes some sense to add it for std-unittest but claiming this is great and most of what you need for "many types" of functional testing is misleading and plays down the many good things you can do with Testing and Python. best, holger

On 2010-02-11 17:57 PM, Holger Krekel wrote:
On Fri, Feb 12, 2010 at 12:00 AM, Robert Kern<robert.kern@gmail.com> wrote:
On 2010-02-11 16:20 PM, Ben Finney wrote:
Guido van Rossum<guido@python.org> writes:
The argument that a unittest framework shouldn't be "abused" for regression tests (or integration tests, or whatever) is also bizarre to my mind. Surely if a testing framework applies to multiple kinds of testing that's a good thing, not something to be frowned upon?
To my mind, an API should take a stand on the “right” way to use it, rather than being a kitchen-sink of whatever ideas had any support. Doing something the right way should be easy, and doing something the wrong way should be awkward.
setUpClass and setUpModule are the "right" way to do many types of integration and functional tests. Integration and functional tests are vital tasks to perform, and unittest provides a good framework otherwise for implementing such tests.
Ben just expressed his opinion about API design and you claim some truth about testing in general.
My first sentence was about API design. My second was justification that the use case is worth designing and API for. You can add implicit "in my opinion"s to just about anything I say, if you wish. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

Holger Krekel wrote:
In my experience, integration and functional testing is a complex and evolving topic, usually requiring more from the tool or framework than classic unit-testing.
Assignment for the reader: compare and contrast unittest and test.regrtest (including test.support and friends) :) Yes, you need extra stuff to do higher level testing, but in many instances unittest still makes a nice framework to hang it on. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

On 11/02/2010 18:11, Guido van Rossum wrote:
On Tue, Feb 9, 2010 at 8:42 AM, Michael Foord<fuzzyman@voidspace.org.uk> wrote:
The next 'big' change to unittest will (may?) be the introduction of class and module level setUp and tearDown. This was discussed on Python-ideas and Guido supported them. They can be useful but are also very easy to abuse (too much shared state, monolithic test classes and modules). [snip...]
The potential for abuse in and of itself should not be an argument against a feature; it must always be weighed against the advantages.
The advantage of setUpClass and setUpModule is that they allow you to have shared fixtures shared between tests, essential for certain kinds of testing. A practical difficulty with setUpClass and setUpModule is that because the scope of the shared fixtures is fixed it makes it much harder to later refactor your tests - either into several classes or into several modules - when the tests grow. My *hope* is that we provide a general solution, possibly based on all or part of Test Resources, with an easy mechanism for the setUpClass and setUpModule but also solves the more general case of sharing fixtures between tests. If that doesn't turn out to be possible then we'll go for a straight implementation of setUpClass / setUpModule. I'm hoping I can get this together in time for the PyCon sprints... Here's a current minimal example of using Test Resources. It could be simplified further with helper functions and by some of the functionality moving into unittest itself. OptimisingTestSuite here ensures that the resource is created before first use (MyTempDir.make is called) and disposed of when finished with (MyTempDir.clean is called). import shutil import tempfile import testresources def load_tests(loader, tests, pattern): # this step could be built into the standard loader return testresources.OptimisingTestSuite(tests) class MyTempDir(testresources.TestResource): def make(self, dependency_resources): return tempfile.mkdtemp() def clean(self, resource): shutil.rmtree(resource) class MyTest(testresources.ResourcedTestCase): resources = [('workdir', MyTempDir())] def test_foo(self): print self.workdir def test_bar(self): print self.workdir Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On Fri, Feb 12, 2010 at 7:49 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
My *hope* is that we provide a general solution, possibly based on all or part of Test Resources, with an easy mechanism for the setUpClass and setUpModule but also solves the more general case of sharing fixtures between tests. If that doesn't turn out to be possible then we'll go for a straight implementation of setUpClass / setUpModule. I'm hoping I can get this together in time for the PyCon sprints...
Do you have a reference for Test Resources?
Here's a current minimal example of using Test Resources. It could be simplified further with helper functions and by some of the functionality moving into unittest itself. OptimisingTestSuite here ensures that the resource is created before first use (MyTempDir.make is called) and disposed of when finished with (MyTempDir.clean is called).
import shutil import tempfile import testresources
def load_tests(loader, tests, pattern): # this step could be built into the standard loader return testresources.OptimisingTestSuite(tests)
class MyTempDir(testresources.TestResource): def make(self, dependency_resources): return tempfile.mkdtemp()
def clean(self, resource): shutil.rmtree(resource)
class MyTest(testresources.ResourcedTestCase): resources = [('workdir', MyTempDir())] def test_foo(self): print self.workdir def test_bar(self): print self.workdir
This came out with all leading indentation removed, but I think I can guess what you meant to write. However from this example I *cannot* guess whether those resources are set up and torn down per test or per test class. Also the notation resources = [('workdir', MyTempDir())] looks pretty ugly -- if 'workdir' ends up being an instance attribute, why not make it a dict instead of a list of tuples? Or even better, a could each resource become a class variable? -- --Guido van Rossum (python.org/~guido)

On 07:48 pm, guido@python.org wrote:
On Fri, Feb 12, 2010 at 7:49 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
My *hope* is that we provide a general solution, possibly based on all or part of Test Resources, with an easy mechanism for the setUpClass and setUpModule but also solves the more general case of sharing fixtures between tests. If that doesn't turn out to be possible then we'll go for a straight implementation of setUpClass / setUpModule. I'm hoping I can get this together in time for the PyCon sprints...
Do you have a reference for Test Resources?
http://pypi.python.org/pypi/testresources/0.2.2
[snip]
However from this example I *cannot* guess whether those resources are set up and torn down per test or per test class. Also the notation
The idea is that you're declaring what the tests need in order to work. You're not explicitly defining the order in which things are set up and torn down. That is left up to another part of the library to determine. One such other part, OptimisingTestSuite, will look at *all* of your tests and find an order which involves the least redundant effort. You might have something else that breaks up the test run across multiple processes and uses the resource declarations to run all tests requiring one thing in one process and all tests requiring another thing somewhere else. You might have still something else that wants to completely randomize the order of tests, and sets up all the resources at the beginning and tears them down at the end. Or you might need to be more memory/whatever conscious than that, and do each set up and tear down around each test. The really nice thing here is that you're not constrained in how you group your tests into classes and modules by what resources you want to use in them. You're free to group them by what they're logically testing, or in whatever other way you wish. Jean-Paul

On Fri, Feb 12, 2010 at 12:20 PM, <exarkun@twistedmatrix.com> wrote:
The idea is that you're declaring what the tests need in order to work. You're not explicitly defining the order in which things are set up and torn down. That is left up to another part of the library to determine.
One such other part, OptimisingTestSuite, will look at *all* of your tests and find an order which involves the least redundant effort.
So is there a way to associate a "cost" with a resource? I may have one resource which is simply a /tmp subdirectory (very cheap) and another that requires starting a database service (very expensive).
You might have something else that breaks up the test run across multiple processes and uses the resource declarations to run all tests requiring one thing in one process and all tests requiring another thing somewhere else.
I admire the approach, though I am skeptical. We have a thing to split up tests at Google which looks at past running times for tests to make an informed opinion. Have you thought of that?
You might have still something else that wants to completely randomize the order of tests, and sets up all the resources at the beginning and tears them down at the end. Or you might need to be more memory/whatever conscious than that, and do each set up and tear down around each test.
How does your code know the constraints?
The really nice thing here is that you're not constrained in how you group your tests into classes and modules by what resources you want to use in them. You're free to group them by what they're logically testing, or in whatever other way you wish.
I guess this requires some trust in the system. :-) -- --Guido van Rossum (python.org/~guido)

On 08:27 pm, guido@python.org wrote:
On Fri, Feb 12, 2010 at 12:20 PM, <exarkun@twistedmatrix.com> wrote:
The idea is that you're declaring what the tests need in order to work. You're not explicitly defining the order in which things are set up and torn down. �That is left up to another part of the library to determine.
One such other part, OptimisingTestSuite, will look at *all* of your tests and find an order which involves the least redundant effort.
So is there a way to associate a "cost" with a resource? I may have one resource which is simply a /tmp subdirectory (very cheap) and another that requires starting a database service (very expensive).
I don't think so. From the docs, "This TestSuite will introspect all the test cases it holds directly and if they declare needed resources, will run the tests in an order that attempts to minimise the number of setup and tear downs required.".
You might have something else that breaks up the test run across multiple processes and uses the resource declarations to run all tests requiring one thing in one process and all tests requiring another thing somewhere else.
I admire the approach, though I am skeptical. We have a thing to split up tests at Google which looks at past running times for tests to make an informed opinion. Have you thought of that?
You might have still something else that wants to completely randomize the order of tests, and sets up all the resources at the beginning and tears them down at the end. Or you might need to be more memory/whatever conscious than that, and do each set up and tear down around each test.
How does your code know the constraints?
To be clear, aside from OptimisingTestSuite, I don't think testresources implements any of the features I talked about. They're just things one might want to and be able to implement, given a test suite which uses testresources.
The really nice thing here is that you're not constrained in how you group your tests into classes and modules by what resources you want to use in them. You're free to group them by what they're logically testing, or in whatever other way you wish.
I guess this requires some trust in the system. :-)
Jean-Paul

On Fri, 2010-02-12 at 12:27 -0800, Guido van Rossum wrote:
On Fri, Feb 12, 2010 at 12:20 PM, <exarkun@twistedmatrix.com> wrote:
The idea is that you're declaring what the tests need in order to work. You're not explicitly defining the order in which things are set up and torn down. That is left up to another part of the library to determine.
One such other part, OptimisingTestSuite, will look at *all* of your tests and find an order which involves the least redundant effort.
So is there a way to associate a "cost" with a resource? I may have one resource which is simply a /tmp subdirectory (very cheap) and another that requires starting a database service (very expensive).
From the pydoc: :ivar setUpCost: The relative cost to construct a resource of this type. One good approach is to set this to the number of seconds it normally takes to set up the resource. :ivar tearDownCost: The relative cost to tear down a resource of this type. One good approach is to set this to the number of seconds it normally takes to tear down the resource.
You might have something else that breaks up the test run across multiple processes and uses the resource declarations to run all tests requiring one thing in one process and all tests requiring another thing somewhere else.
I admire the approach, though I am skeptical. We have a thing to split up tests at Google which looks at past running times for tests to make an informed opinion. Have you thought of that?
I think thats a great way to do it; in fact doing the same thing to assign setup and teardown costs would be lovely; I should write some glue to do that automatically for people. -Rob

On 12/02/2010 19:48, Guido van Rossum wrote:
[snip...]
Here's a current minimal example of using Test Resources. It could be simplified further with helper functions and by some of the functionality moving into unittest itself. OptimisingTestSuite here ensures that the resource is created before first use (MyTempDir.make is called) and disposed of when finished with (MyTempDir.clean is called).
import shutil import tempfile import testresources
def load_tests(loader, tests, pattern): # this step could be built into the standard loader return testresources.OptimisingTestSuite(tests)
class MyTempDir(testresources.TestResource): def make(self, dependency_resources): return tempfile.mkdtemp()
def clean(self, resource): shutil.rmtree(resource)
class MyTest(testresources.ResourcedTestCase): resources = [('workdir', MyTempDir())] def test_foo(self): print self.workdir def test_bar(self): print self.workdir
This came out with all leading indentation removed, but I think I can guess what you meant to write.
However from this example I *cannot* guess whether those resources are set up and torn down per test or per test class. This particular example is the equivalent of setUpClass - so by declaring the resource as a class attribute it will created before the first test for the class is run and disposed of after the last test for
For goodness sake. Sorry about that. the class. You could *also* create a single resource and share it between several test classes, or even across classes in several modules, and have it created and disposed of at the right point. I've copied Rob Collins in on this email in case I've misunderstood.
Also the notation
resources = [('workdir', MyTempDir())]
looks pretty ugly -- if 'workdir' ends up being an instance attribute, why not make it a dict instead of a list of tuples? Or even better, a could each resource become a class variable?
I guess we could introspect the class for every attribute that is a resource, but I prefer some way of explicitly declaring which resources a TestCase is using. Michael Foord -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On Sat, 2010-02-13 at 01:04 +0000, Michael Foord wrote:
However from this example I *cannot* guess whether those resources are set up and torn down per test or per test class. This particular example is the equivalent of setUpClass - so by declaring the resource as a class attribute it will created before the first test for the class is run and disposed of after the last test for the class.
You could *also* create a single resource and share it between several test classes, or even across classes in several modules, and have it created and disposed of at the right point. I've copied Rob Collins in on this email in case I've misunderstood.
Yes, precisely.
Also the notation
resources = [('workdir', MyTempDir())]
looks pretty ugly -- if 'workdir' ends up being an instance attribute, why not make it a dict instead of a list of tuples? Or even better, a could each resource become a class variable?
I have two key 'todos' planned in the notation for testresources: - I want to make a decorator to let individual tests (rather than /just/ class scope as currently happens) get resources easily. Something like @resource(workdir=MyTempDir()) def test_foo(self): pass Secondly, I want to make the class level 'resources' list into a dict. It was a list initially, for misguided reasons that no longer apply, and it would be much nicer and clearer as a dict.
I guess we could introspect the class for every attribute that is a resource, but I prefer some way of explicitly declaring which resources a TestCase is using.
I could see doing the below as an alternative: @resource(workdir=MyTempDir()) class TestFoo(TestCase): ... I'm not personally very keen on inspecting everything in self.__dict__, I suspect it would tickle bugs in other unittest extensions. However I'm not really /against/ it - I don't think it will result in bad test behaviour or isolation issues. So if users would like it, lets do it. -Rob

Robert Collins <robertc <at> robertcollins.net> writes:
I'm not personally very keen on inspecting everything in self.__dict__, I suspect it would tickle bugs in other unittest extensions. However I'm not really /against/ it - I don't think it will result in bad test behaviour or isolation issues. So if users would like it, lets do it.
Why not take all resource_XXX attributes? By the way, how does a given test access the allocated resource? Say, the DB connection. Does it become an attribute of the test case instance? Thank you Antoine.

On Sat, 2010-02-13 at 10:42 +0000, Antoine Pitrou wrote:
Robert Collins <robertc <at> robertcollins.net> writes:
I'm not personally very keen on inspecting everything in self.__dict__, I suspect it would tickle bugs in other unittest extensions. However I'm not really /against/ it - I don't think it will result in bad test behaviour or isolation issues. So if users would like it, lets do it.
Why not take all resource_XXX attributes?
Sure, though if we're just introspecting I don't see much risk difference between resource_XXX and all XXX. As I say above I'm happy to do it if folk think it will be nice.
By the way, how does a given test access the allocated resource? Say, the DB connection. Does it become an attribute of the test case instance?
yes. Given class Foo(TestCase): resources = {'thing', MyResourceManager()} def test_foo(self): self.thing self.thing will access the resource returned by MyResourceManager.make() -Rob

On 13/02/2010 11:00, Robert Collins wrote:
On Sat, 2010-02-13 at 10:42 +0000, Antoine Pitrou wrote:
Robert Collins<robertc<at> robertcollins.net> writes:
I'm not personally very keen on inspecting everything in self.__dict__, I suspect it would tickle bugs in other unittest extensions. However I'm not really /against/ it - I don't think it will result in bad test behaviour or isolation issues. So if users would like it, lets do it.
Why not take all resource_XXX attributes?
Sure, though if we're just introspecting I don't see much risk difference between resource_XXX and all XXX. As I say above I'm happy to do it if folk think it will be nice.
We could introspect all class attributes and find all the resources. We should use dir(...) rather than looking in the class __dict__ so that resources can be inherited. However, it sounds like Guido isn't a fan of Test Resources *instead* of setUpClass / module because it doesn't offer a simple solution for the setUpModule use case and the API still needs to mature. All the best, Michael
By the way, how does a given test access the allocated resource? Say, the DB connection. Does it become an attribute of the test case instance?
yes. Given
class Foo(TestCase): resources = {'thing', MyResourceManager()} def test_foo(self): self.thing
self.thing will access the resource returned by MyResourceManager.make()
-Rob
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

Michael Foord <fuzzyman@voidspace.org.uk> writes:
The advantage of setUpClass and setUpModule is that they allow you to have shared fixtures shared between tests, essential for certain kinds of testing. […]
Yes, this would be very useful for non-unit tests.
My *hope* is that we provide a general solution, possibly based on all or part of Test Resources, with an easy mechanism for the setUpClass and setUpModule but also solves the more general case of sharing fixtures between tests.
+1 for having these in a more general “testing API”, with the ‘unittest’ API a special case that does *not* share fixtures between tests. -- \ “Following fashion and the status quo is easy. Thinking about | `\ your users' lives and creating something practical is much | _o__) harder.” —Ryan Singer, 2008-07-09 | Ben Finney

On Feb 11, 2010, at 1:11 PM, Guido van Rossum wrote:
I have skimmed this thread (hence this reply to the first rather than the last message), but in general I am baffled by the hostility of testing framework developers towards their users. The arguments against class- and module-level seUp/tearDown functions seems to be inspired by religion or ideology more than by the zen of Python. What happened to Practicality Beats Purity?
My sentiments tend to echo Jean-Paul Calderone's in this regard, but I think what he's saying bears a lot of repeating. We really screwed up this feature in Twisted and I'd like to make sure that the stdlib doesn't repeat the mistake. (Granted, we screwed it up extra bad <http://twistedmatrix.com/trac/ticket/2303>, but I do think many of the problems we encountered are inherent.) The issue is not that we test-framework developers don't like our users, or want to protect them from themselves. It is that our users - ourselves chief among them - desire features like "I want my tests to be transparently optimized across N cores and N disks". I can understand how resistance to setUp/tearDown*Class/Module comes across as user-hostility, but I can assure you this is not the case. It's subtle and difficult to explain how incompatible with these advanced features the *apparently* straightforward semantics of setting up and tearing down classes and modules. Most questions of semantics can be resolved with a simple decision, and it's not clear how that would interfere with other features. In Twisted's implementation of setUpClass and tearDownClass, everything seemed like it worked right up until the point where it didn't. The test writer thinks that they're writing "simple" setUpClass and tearDownClass methods to optimize things, except almost by definition a setUpClass method needs to manipulate global state, shared across tests. Which means that said state starts getting confused when it is set up and torn down concurrently across multiple processes. These methods seem simple, but do they touch the filesystem? Do they touch a shared database, even a little? How do they determine a unique location to do that? Without generally available tools to allow test writers to mess with the order and execution environment of their tests, one tends to write tests that rely on these implementation and ordering accidents, which means that when such a tool does arrive, things start breaking in unpredictable ways.
The argument that a unittest framework shouldn't be "abused" for regression tests (or integration tests, or whatever) is also bizarre to my mind. Surely if a testing framework applies to multiple kinds of testing that's a good thing, not something to be frowned upon?
For what it's worth, I am a big fan of abusing test frameworks in generally, and pyunit specifically, to perform every possible kind of testing. In fact, I find setUpClass more hostile to *other* kinds of testing, because this convenience for simple integration tests makes more involved, performance-intensive integration tests harder to write and manage.
On the other hand, I think we should be careful to extend unittest in a consistent way. I shuddered at earlier proposals (on python-ideas) to name the new functions (variations of) set_up and tear_down "to conform with PEP 8" (this would actually have violated that PEP, which explicitly prefers local consistency over global consistency).
This is a very important point. But, it's important not only to extend unittest itself in a consistent way, but to clearly describe the points of extensibility so that third-party things can continue to extend unittest themselves, and cooperate with each other using some defined protocol so that you can combine those tools. I tried to write about this problem a while ago <http://glyf.livejournal.com/72505.html> - the current extensibility API (which is mostly just composing "run()") is sub-optimal in many ways, but it's important not to break it. And setUpClass does inevitably start to break those integration points down, because it implies certain things, like the fact that classes and modules are suites, or are otherwise grouped together in test ordering. This makes it difficult to create custom suites, to do custom ordering, custom per-test behavior (like metrics collection before and after run(), or gc.collect() after each test, or looking for newly-opened-but-not-cleaned-up external resources like file descriptors after each tearDown). Again: these are all concrete features that *users* of test frameworks want, not just idle architectural fantasy of us framework hackers. I haven't had the opportunity to read the entire thread, so I don't know if this discussion has come to fruition, but I can see that some attention has been paid to these difficulties. I have no problem with setUpClass or tearDownClass hooks *per se*, as long as they can be implemented in a way which explicitly preserves extensibility.
Regarding the objection that setUp/tearDown for classes would run into issues with subclassing, I propose to let the standard semantics of subclasses do their job. Thus a subclass that overrides setUpClass or tearDownClass is responsible for calling the base class's setUpClass and tearDownClass (and the TestCase base class should provide empty versions of both). The testrunner should only call setUpClass and tearDownClass for classes that have at least one test that is selected.
Yes, this would mean that if a base class has a test method and a setUpClass (and tearDownClass) method and a subclass also has a test method and overrides setUpClass (and/or tearDown), the base class's setUpClass and tearDown may be called twice. What's the big deal? If setUpClass and tearDownClass are written properly they should support this.
Just to be clear: by "written properly" you mean, written as classmethods, storing their data only on 'cls', right?
If this behavior is undesired in a particular case, maybe what was really meant were module-level setUp and tearDown, or the class structure should be rearranged.
There's also a bit of an open question here for me: if subclassing is allowed, and module-level setup and teardown are allowed, then what if I define a test class with test methods in module 'a', as well as module setup and teardown, then subclass it in 'b' which *doesn't* have setup and teardown... is the subclass in 'b' always assumed to depend on the module-level setup in 'a'? Is there a way that it could be made not to if it weren't necessary? What if it stubs out all of its test methods? In the case of classes you've got the 'cls' variable to describe the dependency and the shared state, but in the case of modules, inheritance doesn't create an additional module object to hold on to. testresources very neatly sidesteps this problem by just providing an API to say "this test case depends on that test resource", without relying on the grouping of tests within classes, modules, or packages. Of course you can just define a class-level or module-level resource and then have all your tests depend on it, which gives you the behavior of setUpClass and setUpModule in a more general way. -glyph

On Fri, Feb 12, 2010 at 8:01 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
On Feb 11, 2010, at 1:11 PM, Guido van Rossum wrote:
I have skimmed this thread (hence this reply to the first rather than the last message), but in general I am baffled by the hostility of testing framework developers towards their users. The arguments against class- and module-level seUp/tearDown functions seems to be inspired by religion or ideology more than by the zen of Python. What happened to Practicality Beats Purity?
My sentiments tend to echo Jean-Paul Calderone's in this regard, but I think what he's saying bears a lot of repeating. We really screwed up this feature in Twisted and I'd like to make sure that the stdlib doesn't repeat the mistake. (Granted, we screwed it up extra bad <http://twistedmatrix.com/trac/ticket/2303>, but I do think many of the problems we encountered are inherent.)
Especially since you screwed up extra bad, the danger exists that you're overreacting.
The issue is not that we test-framework developers don't like our users, or want to protect them from themselves. It is that our users - ourselves chief among them - desire features like "I want my tests to be transparently optimized across N cores and N disks".
Yeah, users ask for impossible features all the time. ;-) Seriously, we do this at Google on a massive scale, for many languages including Python. It's works well but takes getting used to: while time is saved waiting for tests, some time is wasted debugging tests that run fine on the developer's workstation but not in the test cluster. We've developed quite a few practices around this, which include ways to override and control the test distribution, as well as reports showing the historical "flakiness" for each tests.
I can understand how resistance to setUp/tearDown*Class/Module comes across as user-hostility, but I can assure you this is not the case. It's subtle and difficult to explain how incompatible with these advanced features the *apparently* straightforward semantics of setting up and tearing down classes and modules. Most questions of semantics can be resolved with a simple decision, and it's not clear how that would interfere with other features.
In Twisted's implementation of setUpClass and tearDownClass, everything seemed like it worked right up until the point where it didn't. The test writer thinks that they're writing "simple" setUpClass and tearDownClass methods to optimize things, except almost by definition a setUpClass method needs to manipulate global state, shared across tests. Which means that said state starts getting confused when it is set up and torn down concurrently across multiple processes. These methods seem simple, but do they touch the filesystem? Do they touch a shared database, even a little? How do they determine a unique location to do that? Without generally available tools to allow test writers to mess with the order and execution environment of their tests, one tends to write tests that rely on these implementation and ordering accidents, which means that when such a tool does arrive, things start breaking in unpredictable ways.
Been there, done that. The guideline should be that setUpClass and friends save time but should still isolate themselves from other copies that might run concurrently. E.g. if you have to copy a ton of stuff into the filesystem, you should still put it in a temp dir with a randomized name, and store that name as a class variable. When there's a global resource (such as a database) that really can't be shared, well, you have to come up with a way to lock it -- that's probably necessary even if your tests ran completely serialized, unless there's only one developer and she never multitasks. :-)
The argument that a unittest framework shouldn't be "abused" for regression tests (or integration tests, or whatever) is also bizarre to my mind. Surely if a testing framework applies to multiple kinds of testing that's a good thing, not something to be frowned upon?
For what it's worth, I am a big fan of abusing test frameworks in generally, and pyunit specifically, to perform every possible kind of testing. In fact, I find setUpClass more hostile to *other* kinds of testing, because this convenience for simple integration tests makes more involved, performance-intensive integration tests harder to write and manage.
That sounds odd, as if the presence of this convenience would prohibit you from also implement other features.
On the other hand, I think we should be careful to extend unittest in a consistent way. I shuddered at earlier proposals (on python-ideas) to name the new functions (variations of) set_up and tear_down "to conform with PEP 8" (this would actually have violated that PEP, which explicitly prefers local consistency over global consistency).
This is a very important point. But, it's important not only to extend unittest itself in a consistent way, but to clearly describe the points of extensibility so that third-party things can continue to extend unittest themselves, and cooperate with each other using some defined protocol so that you can combine those tools.
Yeah, and I suspect that the original pyunit (now unittest) wasn't always clear on this point.
I tried to write about this problem a while ago <http://glyf.livejournal.com/72505.html> - the current extensibility API (which is mostly just composing "run()") is sub-optimal in many ways, but it's important not to break it.
I expect that *eventually* something will come along that is so much better than unittest that, once matured, we'll want it in the stdlib. (Or, alternatively, eventually stdlib inclusion won't be such a big deal any more since distros mix and match. But then inclusion in a distro would become every package developer's goal -- and then the circle would be round, since distros hardly move faster than Python releases...) But in the mean time I believe evolving unittest is the right thing to do. Adding new methods is relatively easy. Adding whole new paradigms (like testresources) is a lot harder, eventually in the light of the latter's relative immaturity.
And setUpClass does inevitably start to break those integration points down, because it implies certain things, like the fact that classes and modules are suites, or are otherwise grouped together in test ordering.
I expect that is what the majority of unittest users already believe.
This makes it difficult to create custom suites, to do custom ordering, custom per-test behavior (like metrics collection before and after run(), or gc.collect() after each test, or looking for newly-opened-but-not-cleaned-up external resources like file descriptors after each tearDown).
True, the list never ends.
Again: these are all concrete features that *users* of test frameworks want, not just idle architectural fantasy of us framework hackers.
I expect that most bleeding edge users will end up writing a custom framework, or at least work with a bleeding edge framework that change change rapidly to meet their urgent needs.
I haven't had the opportunity to read the entire thread, so I don't know if this discussion has come to fruition, but I can see that some attention has been paid to these difficulties. I have no problem with setUpClass or tearDownClass hooks *per se*, as long as they can be implemented in a way which explicitly preserves extensibility.
That's good to know. I have no doubt they (and setUpModule c.s.) can be done in a clean, extensible way. And that doesn't mean we couldn't also add other features -- after all, not all users have the same needs. (If you read the Zen of Python, you'll see that TOOWTDI has several qualifications. :-)
Regarding the objection that setUp/tearDown for classes would run into issues with subclassing, I propose to let the standard semantics of subclasses do their job. Thus a subclass that overrides setUpClass or tearDownClass is responsible for calling the base class's setUpClass and tearDownClass (and the TestCase base class should provide empty versions of both). The testrunner should only call setUpClass and tearDownClass for classes that have at least one test that is selected.
Yes, this would mean that if a base class has a test method and a setUpClass (and tearDownClass) method and a subclass also has a test method and overrides setUpClass (and/or tearDown), the base class's setUpClass and tearDown may be called twice. What's the big deal? If setUpClass and tearDownClass are written properly they should support this.
Just to be clear: by "written properly" you mean, written as classmethods, storing their data only on 'cls', right?
Yes. And avoiding referencing unique global resources (both within and outside the current process).
If this behavior is undesired in a particular case, maybe what was really meant were module-level setUp and tearDown, or the class structure should be rearranged.
There's also a bit of an open question here for me: if subclassing is allowed, and module-level setup and teardown are allowed, then what if I define a test class with test methods in module 'a', as well as module setup and teardown, then subclass it in 'b' which *doesn't* have setup and teardown... is the subclass in 'b' always assumed to depend on the module-level setup in 'a'?
You shouldn't be doing that kind of thing, but for definiteness, the answer is "no". If you use class setup/teardown instead you can control this via inheritance. At the module level, if you really want to do this, b's module setup would have to explicitly call a's module setup.
Is there a way that it could be made not to if it weren't necessary? What if it stubs out all of its test methods? In the case of classes you've got the 'cls' variable to describe the dependency and the shared state, but in the case of modules, inheritance doesn't create an additional module object to hold on to.
It should be "no" so that you can explicitly code up "yes" if you want to. The other way around would be much messier, as you describe.
testresources very neatly sidesteps this problem by just providing an API to say "this test case depends on that test resource", without relying on the grouping of tests within classes, modules, or packages. Of course you can just define a class-level or module-level resource and then have all your tests depend on it, which gives you the behavior of setUpClass and setUpModule in a more general way.
I wish it was always a matter of "resources". I've seen use cases for module-level setup that were much messier than that (e.g. fixing import paths). I expect it will be a while before the testresources design has been shaken out sufficiently for it to be included in the stdlib. -- --Guido van Rossum (python.org/~guido)

On Feb 13, 2010, at 12:46 PM, Guido van Rossum wrote:
On Fri, Feb 12, 2010 at 8:01 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
On Feb 11, 2010, at 1:11 PM, Guido van Rossum wrote:
For what it's worth, I am a big fan of abusing test frameworks in generally, and pyunit specifically, to perform every possible kind of testing. In fact, I find setUpClass more hostile to *other* kinds of testing, because this convenience for simple integration tests makes more involved, performance-intensive integration tests harder to write and manage.
That sounds odd, as if the presence of this convenience would prohibit you from also implement other features.
Well, that is the main point I'm trying to make. There are ways to implement setUpClass that *do* make the implementation of other features effectively impossible, by breaking the integration mechanisms between tests and framework, and between multiple testing frameworks. And I am pretty sure this is not just my over-reaction; Michael still appears to be wrestling with the problems I'm describing. In a recent message he was talking about either breaking compatibility with TestSuite implementations that override run(), or test-reordering - both of which I consider important, core features of the unittest module.
I tried to write about this problem a while ago <http://glyf.livejournal.com/72505.html> - the current extensibility API (which is mostly just composing "run()") is sub-optimal in many ways, but it's important not to break it.
I expect that *eventually* something will come along that is so much better than unittest that, once matured, we'll want it in the stdlib.
I'm not sure what point you're trying to make here. I was saying "it's not perfect, but we should be careful not to break it, because it's all we've got". Are you saying that we shouldn't worry about unittest's composition API, because it's just a stopgap until something better comes along?
(Or, alternatively, eventually stdlib inclusion won't be such a big deal any more since distros mix and match. But then inclusion in a distro would become every package developer's goal -- and then the circle would be round, since distros hardly move faster than Python releases...)
But in the mean time I believe evolving unittest is the right thing to do. Adding new methods is relatively easy. Adding whole new paradigms (like testresources) is a lot harder, eventually in the light of the latter's relative immaturity.
I disagree with your classification of the solutions. First and foremost: setUpClass is not a "new method", it's a pile of new code to call that method, to deal with ordering that method, etc. Code which has not yet been written or tested or tried in the real world. It is beyond simply immature, it's hypothetical. We do have an implementation of this code in Twisted, but as I have said, it's an albatross we are struggling to divest ourselves of, not something we'd like to propose for inclusion in the standard library. (Nose has this feature as well, but I doubt their implementation would be usable, since their idea of a 'test' isn't really TestCase based.) testresources, by contrast, is a tested, existing package, which people are already using, using a long-standing integration mechanism that has been part of unittest since its first implementation. Granted, I would not contest that it is "immature"; it is still fairly new, and doesn't have a huge number of uses, but it's odd to criticize it on grounds of maturity when it's so much *more* mature than the alternative. While superficially the programming interface to testresources is slightly more unusual, this is only because programmers don't think to hard about what unittest actually does with your code, and testresources requires a little more familiarity with that.
And setUpClass does inevitably start to break those integration points down, because it implies certain things, like the fact that classes and modules are suites, or are otherwise grouped together in test ordering.
I expect that is what the majority of unittest users already believe.
Yes, but they're wrong, and enforcing this misconception doesn't help anyone. There are all kinds of assumptions that most python developers have about how Python works which are vaguely incorrect abstractions over the actual behavior.
This makes it difficult to create custom suites, to do custom ordering, custom per-test behavior (like metrics collection before and after run(), or gc.collect() after each test, or looking for newly-opened-but-not-cleaned-up external resources like file descriptors after each tearDown).
True, the list never ends.
Again: these are all concrete features that *users* of test frameworks want, not just idle architectural fantasy of us framework hackers.
I expect that most bleeding edge users will end up writing a custom framework, or at least work with a bleeding edge framework that change change rapidly to meet their urgent needs.
Yes, exactly. Users who want an esoteric feature like setUpClass should use a custom framework :). The standard library's job, in my view, is to provide an integration point for those more advanced framework so that different "bleeding edge" frameworks have a mechanism to communicate, and users have some level of choice about what tools they use to run their tests. I have not always thought this way. Originally, my intention was for twisted's test framework to be a complete departure from the standard library and just do its own thing. But, both users of the framework and more savvy test developers have gradually convinced me that it's really useful to be able to load stdlib tests with a variety of tools. This doesn't mean that I think the stdlib should stop changing completely, but I do think changes which potentially break compatibility with _basic_ test framework things like re-ordering tests or overriding a core method like 'run' need to be done extremely carefully.
I haven't had the opportunity to read the entire thread, so I don't know if this discussion has come to fruition, but I can see that some attention has been paid to these difficulties. I have no problem with setUpClass or tearDownClass hooks *per se*, as long as they can be implemented in a way which explicitly preserves extensibility.
That's good to know. I have no doubt they (and setUpModule c.s.) can be done in a clean, extensible way. And that doesn't mean we couldn't also add other features -- after all, not all users have the same needs. (If you read the Zen of Python, you'll see that TOOWTDI has several qualifications. :-)
Recent messages indicate that this is still a problem. But perhaps it will be solved! I can yell at Michael in person in a few days, anyway :). <snip a bunch of questions about behavior of the feature> All those answers were pretty reasonable, so I will avoid retreading them.
testresources very neatly sidesteps this problem by just providing an API to say "this test case depends on that test resource", without relying on the grouping of tests within classes, modules, or packages. Of course you can just define a class-level or module-level resource and then have all your tests depend on it, which gives you the behavior of setUpClass and setUpModule in a more general way.
I wish it was always a matter of "resources". I've seen use cases for module-level setup that were much messier than that (e.g. fixing import paths). I expect it will be a while before the testresources design has been shaken out sufficiently for it to be included in the stdlib.
Why isn't loadable code a "resource" like anything else? I haven't used testresources specifically for this, but I've definitely written fixture setup and teardown code that dealt with sys.path.

On 15/02/2010 20:27, Glyph Lefkowitz wrote:
On Feb 13, 2010, at 12:46 PM, Guido van Rossum wrote:
On Fri, Feb 12, 2010 at 8:01 PM, Glyph Lefkowitz <glyph@twistedmatrix.com <mailto:glyph@twistedmatrix.com>> wrote:
On Feb 11, 2010, at 1:11 PM, Guido van Rossum wrote:
For what it's worth, I am a big fan of abusing test frameworks in generally, and pyunit specifically, to perform every possible kind of testing. In fact, I find setUpClass more hostile to *other* kinds of testing, because this convenience for simple integration tests makes more involved, performance-intensive integration tests harder to write and manage.
That sounds odd, as if the presence of this convenience would prohibit you from also implement other features.
Well, that is the main point I'm trying to make. There are ways to implement setUpClass that *do* make the implementation of other features effectively impossible, by breaking the integration mechanisms between tests and framework, and between multiple testing frameworks.
And I am pretty sure this is not just my over-reaction; Michael still appears to be wrestling with the problems I'm describing.
And I appreciate your input.
In a recent message he was talking about either breaking compatibility with TestSuite implementations that override run(), or test-reordering - both of which I consider important, core features of the unittest module.
Well, by "breaking compatibility with custom TestSuite implementations that override run" I mean that is one possible place to put the functionality. Code that does override it will *not* stop working, it just won't support the new features. If we chose this implementation strategy there would be no compatibility issues for existing tests / frameworks that don't use the new features. If tests do want to use the new features then the framework authors will need to ensure they are compatible with them. This seems like a reasonable trade-off to me. We can ensure that it is easy to write custom TestSuite objects that work with earlier versions of unittest but are also compatible with setUpClass in 2.7 (and document the recipe - although I expect it will just mean that TestSuite.run should call a single method if it exists). Perhaps a better idea might be to also add startTest and stopTest methods to TestSuite so that frameworks can build in features like timing tests (etc) without having to override run itself. This is already possible in the TestResult of course, which is a more common extensibility point in *my* experience. All the best, Michael
I tried to write about this problem a while ago <http://glyf.livejournal.com/72505.html> - the current extensibility API (which is mostly just composing "run()") is sub-optimal in many ways, but it's important not to break it.
I expect that *eventually* something will come along that is so much better than unittest that, once matured, we'll want it in the stdlib.
I'm not sure what point you're trying to make here. I was saying "it's not perfect, but we should be careful not to break it, because it's all we've got". Are you saying that we shouldn't worry about unittest's composition API, because it's just a stopgap until something better comes along?
(Or, alternatively, eventually stdlib inclusion won't be such a big deal any more since distros mix and match. But then inclusion in a distro would become every package developer's goal -- and then the circle would be round, since distros hardly move faster than Python releases...)
But in the mean time I believe evolving unittest is the right thing to do. Adding new methods is relatively easy. Adding whole new paradigms (like testresources) is a lot harder, eventually in the light of the latter's relative immaturity.
I disagree with your classification of the solutions.
First and foremost: setUpClass is not a "new method", it's a pile of new code to call that method, to deal with ordering that method, etc. Code which has not yet been written or tested or tried in the real world. It is beyond simply immature, it's hypothetical. We do have an implementation of this code in Twisted, but as I have said, it's an albatross we are struggling to divest ourselves of, not something we'd like to propose for inclusion in the standard library. (Nose has this feature as well, but I doubt their implementation would be usable, since their idea of a 'test' isn't really TestCase based.)
testresources, by contrast, is a tested, existing package, which people are already using, using a long-standing integration mechanism that has been part of unittest since its first implementation. Granted, I would not contest that it is "immature"; it is still fairly new, and doesn't have a huge number of uses, but it's odd to criticize it on grounds of maturity when it's so much *more* mature than the alternative.
While superficially the programming interface to testresources is slightly more unusual, this is only because programmers don't think to hard about what unittest actually does with your code, and testresources requires a little more familiarity with that.
And setUpClass does inevitably start to break those integration points down, because it implies certain things, like the fact that classes and modules are suites, or are otherwise grouped together in test ordering.
I expect that is what the majority of unittest users already believe.
Yes, but they're wrong, and enforcing this misconception doesn't help anyone. There are all kinds of assumptions that most python developers have about how Python works which are vaguely incorrect abstractions over the actual behavior.
This makes it difficult to create custom suites, to do custom ordering, custom per-test behavior (like metrics collection before and after run(), or gc.collect() after each test, or looking for newly-opened-but-not-cleaned-up external resources like file descriptors after each tearDown).
True, the list never ends.
Again: these are all concrete features that *users* of test frameworks want, not just idle architectural fantasy of us framework hackers.
I expect that most bleeding edge users will end up writing a custom framework, or at least work with a bleeding edge framework that change change rapidly to meet their urgent needs.
Yes, exactly. Users who want an esoteric feature like setUpClass should use a custom framework :). The standard library's job, in my view, is to provide an integration point for those more advanced framework so that different "bleeding edge" frameworks have a mechanism to communicate, and users have some level of choice about what tools they use to run their tests.
I have not always thought this way. Originally, my intention was for twisted's test framework to be a complete departure from the standard library and just do its own thing. But, both users of the framework and more savvy test developers have gradually convinced me that it's really useful to be able to load stdlib tests with a variety of tools.
This doesn't mean that I think the stdlib should stop changing completely, but I do think changes which potentially break compatibility with _basic_ test framework things like re-ordering tests or overriding a core method like 'run' need to be done extremely carefully.
I haven't had the opportunity to read the entire thread, so I don't know if this discussion has come to fruition, but I can see that some attention has been paid to these difficulties. I have no problem with setUpClass or tearDownClass hooks *per se*, as long as they can be implemented in a way which explicitly preserves extensibility.
That's good to know. I have no doubt they (and setUpModule c.s.) can be done in a clean, extensible way. And that doesn't mean we couldn't also add other features -- after all, not all users have the same needs. (If you read the Zen of Python, you'll see that TOOWTDI has several qualifications. :-)
Recent messages indicate that this is still a problem. But perhaps it will be solved! I can yell at Michael in person in a few days, anyway :).
<snip a bunch of questions about behavior of the feature>
All those answers were pretty reasonable, so I will avoid retreading them.
testresources very neatly sidesteps this problem by just providing an API to say "this test case depends on that test resource", without relying on the grouping of tests within classes, modules, or packages. Of course you can just define a class-level or module-level resource and then have all your tests depend on it, which gives you the behavior of setUpClass and setUpModule in a more general way.
I wish it was always a matter of "resources". I've seen use cases for module-level setup that were much messier than that (e.g. fixing import paths). I expect it will be a while before the testresources design has been shaken out sufficiently for it to be included in the stdlib.
Why isn't loadable code a "resource" like anything else? I haven't used testresources specifically for this, but I've definitely written fixture setup and teardown code that dealt with sys.path.
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On Feb 15, 2010, at 3:50 PM, Michael Foord wrote:
On 15/02/2010 20:27, Glyph Lefkowitz wrote:
On Feb 13, 2010, at 12:46 PM, Guido van Rossum wrote:
On Fri, Feb 12, 2010 at 8:01 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
I find setUpClass more hostile to *other* kinds of testing, because this convenience for simple integration tests makes more involved, performance-intensive integration tests harder to write and manage.
That sounds odd, as if the presence of this convenience would prohibit you from also implement other features.
And I am pretty sure this is not just my over-reaction; Michael still appears to be wrestling with the problems I'm describing.
And I appreciate your input.
Thanks :).
In a recent message he was talking about either breaking compatibility with TestSuite implementations that override run(), or test-reordering - both of which I consider important, core features of the unittest module.
Well, by "breaking compatibility with custom TestSuite implementations that override run" I mean that is one possible place to put the functionality. Code that does override it will *not* stop working, it just won't support the new features.
Ah, I see. This doesn't sound *too* bad, but I'd personally prefer it if the distinction were a bit more clearly drawn. I'd like frameworks to be able to implement extension functionality without having to first stub out functionality. In other words, if I want a test suite without setUpClass, I'd prefer to avoid having an abstraction inversion. Practically speaking this could be implemented by having a very spare, basic TestSuite base class and ClassSuite/ModuleSuite subclasses which implement the setUpXXX functionality.
If we chose this implementation strategy there would be no compatibility issues for existing tests / frameworks that don't use the new features.
That's very good to hear.
If tests do want to use the new features then the framework authors will need to ensure they are compatible with them. This seems like a reasonable trade-off to me. We can ensure that it is easy to write custom TestSuite objects that work with earlier versions of unittest but are also compatible with setUpClass in 2.7 (and document the recipe - although I expect it will just mean that TestSuite.run should call a single method if it exists).
This is something that I hope Jonathan Lange or Robert Collins will chime in to comment on: expanding the protocol between suite and test is an area which is fraught with peril, but it seems like it's something that test framework authors always want to do. (Personally, *I* really want to do it because I want to be able to run things asynchronously, so the semantics of 'run()' need to change pretty dramatically to support that...) It might be good to eventually develop a general mechanism for this, rather than building up an ad-hoc list of test-feature compatibility recipes which involve a list of if hasattr(...): foo(); checks in every suite implementation.
Perhaps a better idea might be to also add startTest and stopTest methods to TestSuite so that frameworks can build in features like timing tests (etc) without having to override run itself. This is already possible in the TestResult of course, which is a more common extensibility point in *my* experience.
I think timing and monitoring tests can mostly be done in the TestResult class; those were bad examples. There's stuff like synthesizing arguments for test methods, or deciding to repeat a potentially flaky test method before reporting a failure, which are not possible to do from the result. I'm not sure that startTest and stopTest hooks help with those features, the ones which really need suites; it would seem it mostly gives you a hook to do stuff that could already be done in TestResult anyway.

On Tue, 2010-02-16 at 02:09 -0500, Glyph Lefkowitz wrote:
In a recent message he was talking about either breaking compatibility with TestSuite implementations that override run(), or test-reordering - both of which I consider important, core features of the unittest module.
Well, by "breaking compatibility with custom TestSuite implementations that override run" I mean that is one possible place to put the functionality. Code that does override it will *not* stop working, it just won't support the new features.
Ah, I see. This doesn't sound *too* bad, but I'd personally prefer it if the distinction were a bit more clearly drawn. I'd like frameworks to be able to implement extension functionality without having to first stub out functionality. In other words, if I want a test suite without setUpClass, I'd prefer to avoid having an abstraction inversion.
+1
If we chose this implementation strategy there would be no compatibility issues for existing tests / frameworks that don't use the new features.
That's very good to hear.
It does however get tougher to be 'stdlib compatible' for frameworks that extend the stdlib - at least with how extensions work today.
If tests do want to use the new features then the framework authors will need to ensure they are compatible with them. This seems like a reasonable trade-off to me. We can ensure that it is easy to write custom TestSuite objects that work with earlier versions of unittest but are also compatible with setUpClass in 2.7 (and document the recipe - although I expect it will just mean that TestSuite.run should call a single method if it exists).
This is something that I hope Jonathan Lange or Robert Collins will chime in to comment on: expanding the protocol between suite and test is an area which is fraught with peril, but it seems like it's something that test framework authors always want to do. (Personally, *I* really want to do it because I want to be able to run things asynchronously, so the semantics of 'run()' need to change pretty dramatically to support that...) It might be good to eventually develop a general mechanism for this, rather than building up an ad-hoc list of test-feature compatibility recipes which involve a list of if hasattr(...): foo(); checks in every suite implementation.
Please have a look at the testtools.TestCase.run - its incomplete, but its working towards making it possible for trial to not need to replace run, but instead provide a couple of hooks (registered during setUp) to handle what you need. What it currently offers is catching additional exceptions for you, which is a common form of extension. bzrlib is using this quite successfully, and we deleted a lot of code that overlapped the stdlib unittest run().
Perhaps a better idea might be to also add startTest and stopTest methods to TestSuite so that frameworks can build in features like timing tests (etc) without having to override run itself. This is already possible in the TestResult of course, which is a more common extensibility point in *my* experience.
I think timing and monitoring tests can mostly be done in the TestResult class; those were bad examples. There's stuff like synthesizing arguments for test methods, or deciding to repeat a potentially flaky test method before reporting a failure, which are not possible to do from the result. I'm not sure that startTest and stopTest hooks help with those features, the ones which really need suites; it would seem it mostly gives you a hook to do stuff that could already be done in TestResult anyway.
Also its not really possible to 'run one thing' around a test at the moment - theres no good place (without changing tests or doing somewhat convoluted stuff) to have custom code sit in the stack above the test code - this makes it harder to handle: - profiling - drop-into-a-debugger - $other use case This is also in my hit-list of things to solve-and-propose-for-stdlib-unittest that I blogged about a while back. -Rob

On 13/02/2010 04:01, Glyph Lefkowitz wrote: [snipping some good points...]
Regarding the objection that setUp/tearDown for classes would run into issues with subclassing, I propose to let the standard semantics of subclasses do their job. Thus a subclass that overrides setUpClass or tearDownClass is responsible for calling the base class's setUpClass and tearDownClass (and the TestCase base class should provide empty versions of both). The testrunner should only call setUpClass and tearDownClass for classes that have at least one test that is selected.
Yes, this would mean that if a base class has a test method and a setUpClass (and tearDownClass) method and a subclass also has a test method and overrides setUpClass (and/or tearDown), the base class's setUpClass and tearDown may be called twice. What's the big deal? If setUpClass and tearDownClass are written properly they should support this.
Just to be clear: by "written properly" you mean, written as classmethods, storing their data only on 'cls', right?
Heh, yes (answered several times in this thread already I think...).
If this behavior is undesired in a particular case, maybe what was really meant were module-level setUp and tearDown, or the class structure should be rearranged.
There's also a bit of an open question here for me: if subclassing is allowed, and module-level setup and teardown are allowed, then what if I define a test class with test methods in module 'a', as well as module setup and teardown, then subclass it in 'b' which *doesn't* have setup and teardown... is the subclass in 'b' always assumed to depend on the module-level setup in 'a'? Is there a way that it could be made not to if it weren't necessary? What if it stubs out all of its test methods? In the case of classes you've got the 'cls' variable to describe the dependency and the shared state, but in the case of modules, inheritance doesn't create an additional module object to hold on to.
This is also an interesting point. The 'naive' implementation, which I think I prefer, only runs the setUpModule of modules actually containing tests. Similarly setUpClass is only called on classes with actual tests, although they may call up to base class implementations. This has a couple of consequences, particularly for setUpModule. It makes the basic rule: * only use setUpModule for modules actually containing tests * don't mix concrete TestCases (with tests) in the same module (using setUpModule) as base classes for other tests The use case (that I can think of) that isn't supported (and of course all this needs to be documented): * Having setUpModule / tearDownModule in modules that define base classes and expecting them to be called around all tests that inherit from those base classes Having this in place makes the implementation simpler. If we explicitly document that this rule may change and so users shouldn't rely on setUpModule not being called for modules containing base classes, then we are free to rethink it later. Not having this restriction at all is possible, it just requires more introspection at TestSuite creation / ordering time. Note that setUpClass on a base class maybe called several times if several base classes inherit and all call up to the base class implementation. As it will be a class method the cls argument will be different for each call. Another question. If we are implementing TestCase.setUpClass as an additional test then should it be reported *even* if it is only the default (empty) implementation that is used? The reason to have setUpClass implemented as a test is so that you can report the failure *before* you run all the tests. Lots of unit test users want a consistent number of tests every run - so we shouldn't insert an extra test only on fail. The other alternative is to report a setUpClass failure as part of the first test that depends on it - this makes the implementation more, complex (having a pseudo-test represent setUpClass / tearDownClass is convenient for ordering and isolating the 'magic' in one place - the rest of the test running infrastructure doesn't need to know about setUpClass or Module). If we do add a default setUpClass test for all TestCases it means extra noise for test runs that don't use the new feature. Saying no it shouldn't be shown means that we have to introspect test classes to see if they inherit setUpClass from TestCase or from some intermediate base class. Not hard just an extra complexity. One solution would be for TestCase *not* to have default implementations, but it is probably nicer for them to appear in the API. I guess my preferred approach is to have a default implementation, but not to create pseudo-tests for them if they aren't used. All the best, Michael
testresources very neatly sidesteps this problem by just providing an API to say "this test case depends on that test resource", without relying on the grouping of tests within classes, modules, or packages. Of course you can just define a class-level or module-level resource and then have all your tests depend on it, which gives you the behavior of setUpClass and setUpModule in a more general way.
-glyph
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On 15/02/2010 17:05, Michael Foord wrote:
[snip] This is also an interesting point. The 'naive' implementation, which I think I prefer, only runs the setUpModule of modules actually containing tests. Similarly setUpClass is only called on classes with actual tests, although they may call up to base class implementations.
This has a couple of consequences, particularly for setUpModule. It makes the basic rule:
* only use setUpModule for modules actually containing tests * don't mix concrete TestCases (with tests) in the same module (using setUpModule) as base classes for other tests
The use case (that I can think of) that isn't supported (and of course all this needs to be documented):
* Having setUpModule / tearDownModule in modules that define base classes and expecting them to be called around all tests that inherit from those base classes
Having this in place makes the implementation simpler. If we explicitly document that this rule may change and so users shouldn't rely on setUpModule not being called for modules containing base classes, then we are free to rethink it later. Not having this restriction at all is possible, it just requires more introspection at TestSuite creation / ordering time.
Note that setUpClass on a base class maybe called several times if several base classes inherit and all call up to the base class implementation. As it will be a class method the cls argument will be different for each call.
Another question. If we are implementing TestCase.setUpClass as an additional test then should it be reported *even* if it is only the default (empty) implementation that is used?
The reason to have setUpClass implemented as a test is so that you can report the failure *before* you run all the tests. Lots of unit test users want a consistent number of tests every run - so we shouldn't insert an extra test only on fail. The other alternative is to report a setUpClass failure as part of the first test that depends on it - this makes the implementation more, complex (having a pseudo-test represent setUpClass / tearDownClass is convenient for ordering and isolating the 'magic' in one place - the rest of the test running infrastructure doesn't need to know about setUpClass or Module).
If we do add a default setUpClass test for all TestCases it means extra noise for test runs that don't use the new feature. Saying no it shouldn't be shown means that we have to introspect test classes to see if they inherit setUpClass from TestCase or from some intermediate base class. Not hard just an extra complexity. One solution would be for TestCase *not* to have default implementations, but it is probably nicer for them to appear in the API.
I guess my preferred approach is to have a default implementation, but not to create pseudo-tests for them if they aren't used.
One place to implement this is in the TestLoader (specifically in loadTestsFromModule and loadTestsFromTestCase) - especially as this is the place where test ordering is currently provided by unittest. The TestSuite may not need to change much. This isn't compatible with test frameworks that build custom suites without going through loadTestsFromModule - e.g. modules implementing load_tests that replace the standard test suite with a new suite with specific TestCases and still expect setUpModule / tearDownModule to be used [1]. Perhaps an API hook to make this easy? TestCase would need a change in run(...) to support automatic SetupFailed test failing when setUpClass / module fails. This *isn't* compatible with custom TestSuites that reorder tests. The alternative is for TestSuite.run(...) to change to support setUpClass / setUpModule by adding them at test run time by introspecting contained tests. That would make setUpClass (etc) incompatible with custom TestSuite implementations that override run. It would be compatible with TestSuites that do reordering but that don't overload run(...) - although the resulting test order may not be 'optimal' (the setUp and tearDown may end up separated farther in time than desired but at least they would work). Perhaps this approach is less likely to break? (Manually reordering tests or adding new tests to a TestSuite wouldn't break setUpClass as the TestSuite only adds them at run(...) time rather than placing them in the suite where the 'user' can move them around). All the best, Michael [1] or even load_tests that just addTests to the standard suite but still need tearDownModule to be run *after* all the tests. Even with the implementation in loadTestsFrom* it seems like the TestSuite may need to know something about the order. Perhaps we could have TestLoader.orderTests - the loader is available to load_tests functions.
All the best,
Michael
testresources very neatly sidesteps this problem by just providing an API to say "this test case depends on that test resource", without relying on the grouping of tests within classes, modules, or packages. Of course you can just define a class-level or module-level resource and then have all your tests depend on it, which gives you the behavior of setUpClass and setUpModule in a more general way.
-glyph
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
participants (15)
-
Antoine Pitrou
-
Ben Finney
-
Brian Curtin
-
Chris Withers
-
exarkun@twistedmatrix.com
-
Glyph Lefkowitz
-
Guido van Rossum
-
Holger Krekel
-
Michael Foord
-
Nick Coghlan
-
Olemis Lang
-
R. David Murray
-
Robert Collins
-
Robert Kern
-
Tres Seaver