[Twisted-Python] Thoughts about testing
From my experience, testing with Trial is too hard. My
Hi all, I just want to share some ideas that I have had. problems can be divided into three categories: 1. obscure error reporting, 2. unclean reactor 3. whoop whoop whoop. I am also pretty sure that I'm doing things that Trial does not want me to do, such as actually opening sockets and communication over the network. I'm saying this because it does not seem to happen in the Twisted test-suite. But I don't think there is a good reason for this limitation. Actually, in sum, I think there are too many limitations on writing tests with Trial, and they all boil down to the unclean reactor problem (or at least most of them). I want to suggest an alternative approach. How about not requiring the reactor to be clean at the end of a test? If anyone wants to make sure that anything they do leaves the reactor in a clean state, they can test for it themselves. This seems to me more like a constraint imposed by the implementation of Trial rather than a useful feature. Also, parts of Twisted itself are practically unusable inside a test because they leave the reactor dirty (such as threaded address resolution). An alternative feature could be enabling the user to specify that a certain delayed call or thread is allowed to remain after the test, and then Trial won't complain. The only question remaining is how to do it. Simple: use a different process. Run the tests in a different process, and create a new one each time the reactor is dirtied. py.execnet is a nice example of this concept. The second thought is this: there seem to be popping up different testing toolkits each with their own very nice extensions and features (http://testoob.sourceforge.net/, http://codespeak.net/py/current/doc/test.html). Trial cannot benefit from this, having branched away at the pyUnit level. I think Trial's special features can be relatively easily formulated as plugins to a plugin-oriented testing framework (especially if the clean reactor requirement is relieved), and so can the other testing packages. What this means, is that the best thing anyone who wants the world of unit testing to develop, and to benefit from it, is to push for a plugin-oriented pyUnit, and for an implementation of Trial (and the other tools) as a plugin for that framework. I think. Any comments? Antony Kummel __________________________________ Yahoo! FareChase: Search multiple travel sites in one click. http://farechase.yahoo.com
On Tue, 25 Oct 2005 05:19:43 -0700 (PDT), Antony Kummel <antonykummel@yahoo.com> wrote:
The network is a source of unpredictability. Unit tests that rely on it fail intermittently, mysteriously, and with no clear course of action for reproducing the failure, making debugging the problem extremely difficult, and reduces the overall utility of the test suite by introduces failures that aren't really failures, but which nevertheless must be investigated to determine whether they represent a real problem. So, there's a pretty good reason, I think. However, trial doesn't prevent you from doing this, so I'm not sure what the objection is.
If tests are allowed to leave things like connections and timers (in general, event sources) lying around, subsequent tests can fail through no fault of their own when one of these event sources misbehaves. If, for example, one causes an exception to be logged, trial will notice this and attribute it to some other hapless test, causing it to fail. These problems are even more difficult to track down than the ones I mentioned above, since it is not even clear in these cases _which_ test is /really/ failing.
I think it's a good idea. I don't know that, in its current form, it is complete. There are probably some improvements that could be made to ease the process of tracking down the sources of various problems it reports. I don't think that means the entire feature should be scrapped.
This should be addressed, certainly. However, I don't often find myself resolving names using the system resolver in unit tests. What if the system resolver is buggy? What if the system is misconfigured? What if there is a transient DNS failure? What if the DNS server for the host you are interested in is temporarily offline? These are not conditions I am happy to allow to cause my unit tests to fail.
Running tests in a child process is an interesting idea. It provides a much greater degree of isolation between tests than essentially any other approach, which is great. Isolation is great for unit tests. Unfortunately, it comes with a lot of overhead. While there are techniques for optimizing the implementation of such a feature, running each test method in a different process would probably add at least 4 minutes to Twisted's test suite. This is basically unacceptable (Twisted's suite takes way too long to run already). Beyond performance problems, there's also the issue of debugging. As in, how do you? I'm aware of remote debuggers for Python, but they're all third-party. This is not necessarily a killer drawback, but it is definitely a downside.
Rather than hearing about the plethora of new testing libraries appearing, I'd like to hear about features they provide that are valuable for writing tests. I would certainly like to borrow py.test's magical assert. What other features are test authors finding useful in some of these projects? Jp
Hi Jp, --- Jean-Paul Calderone <exarkun@divmod.com> wrote:
Well, it will certainly be non-optimal, but let me add some more points in its favour: 1. It may still be faster than debugging unit tests that fail because of a dirty reactor or other problems that it resolves. 2. You don't have to run each test method in a different process, only the ones who leave the environment dirty. Of course people are likely to be lazy if they get the chance, but it makes more sense to leave it up to them. And of course it could be optimized (is a pool of process what you had in mind?) 3. This feature brings us very close to completely distributed unit testing. This will make it possible to easily run tests simultaneously on different computers which would actually make it faster than the current method with very little effort (given enough computers). 4. A mechanism allowing for running unit tests at random locations (processes or machines) will probably be good for making the tests themselves span more than one computer. Despite your justified objections I think this could be quite a powerful tool. 5. The same mechanism can be useful for another problem I often run into: dirtying of the interactive prompt environment. When using a relatively rich tool such as PyCrust, for example, I have to close it and start it again every time I make a grave enough mistake (for example when I need to reload nested modules). If the code I write at the prompt was to run in a different process, I could simply click a button to restart that process without it having an annoying impact.
First of all, TestOOB already allows you to run test in a different process (although I didn't go down to the details -- this may be a little different than what I talked about). If the plugin-oriented framework I proposed existed, the feature we discussed above would have already been available to us. Secondly, it is not hard to think about ways to make unit testing easier. Take a graphical UI for example. In the current situation, any such improvement will be useful for only a small group of users, and that group is likely to grow ever smaller with the emergeance of new frameworks. And when doing unit tests, we all desparately want the same thing -- to make it easier. With a common base, properly designed, we can all benefit from everyone else's efforts, and I think this is good enough a reason. I join you in wanting to hear what test authors are finding useful in these projects. Antony Kummel __________________________________ Start your day with Yahoo! - Make it your home page! http://www.yahoo.com/r/hs
On Tue, Oct 25, 2005 at 12:59:24PM -0400, Jean-Paul Calderone wrote:
Robert Collins' subunit and testresources libraries for pyunit go a long way to solving this, I believe: http://www.robertcollins.net/unittest/
Trial is moving very rapidly towards pyunit compatibility once again, thanks to Jonathan Lange's efforts. It should be possible to run trial test suites from an ordinary unittest.py runner in the near future, if it's not already possible with SVN Trial. I'm not sure what you mean by "push for a plugin-oriented pyUnit" -- it already is extensible. I was under the impression that py.test wasn't at all pyunit compatible, though? The docs I've seen tell me that they explicitly aren't interested in being able to interoperate with unittest or unittest extensions.
FWIW, I don't like py.test's magical assert ;) -Andrew.
On 2005.10.25 12:59:24 -0400, Jean-Paul Calderone wrote:
Things I particularly like about py.test, vs. unittest: 1. assert, not self.failUnlessAllThisUglyCamelCaseActuallyKillsMe() 2. You can use module-level test functions. No need to make a test class unless it's actually helpful. (Yay Java.) 3. Tests execute in predictable order. 4. setup_module / setup_class / setup_method. Handy for reusing database connections. 5. A separate test runner binary, so you don't need __main__ boilerplate in every test module, and you can put tests in the same module as a script (if you don't care about them being found automatically). 6. You can fill your tests with print statements, and only see their output if the test fails (or you pass the -s option). -- David Ripton dripton@ripton.net
On Wed, 2005-10-26 at 04:56 -0700, David Ripton wrote:
Things I particularly like about py.test, vs. unittest:
Can we compare to trial as it stands please? I don't think anyone is seriously suggesting stdlib unittest for Twisted, and I am pretty sure trial has already subsumed every interesting feature from unittest itself :)
1. assert, not self.failUnlessAllThisUglyCamelCaseActuallyKillsMe()
This is cute - more in the way it deals with printing appropriate context than the fact that you actually use 'assert' - but it's a shame that they resorted to a language hack rather than a simple function like 'test()' or something. Using 'assert' makes it impossible to test with -O or -OO. I personally don't use those options, but some people do, and they should be testable.
2. You can use module-level test functions. No need to make a test class unless it's actually helpful. (Yay Java.)
Handy in some cases, but from what I've seen in my brief survey of py.test-based tests the only thing this does is make it more of a convention to smear unit test state across a bunch of shared globals ...
3. Tests execute in predictable order.
... and then since there is now global state smeared across your whole module, the tests have to run in an exact order because each one sets up things for the next, rather than proper setUp/tearDown encapsulation.
4. setup_module / setup_class / setup_method. Handy for reusing database connections.
Technically trial has similar features, but they were broken so horribly for so long that it became the common wisdom not to use them. I have to assume this has recently changed?
Trial has this already too.
6. You can fill your tests with print statements, and only see their output if the test fails (or you pass the -s option).
This is like a replication of trial's AND unittest's worst feature - the fact that tracebacks are deferred until execution is finished, so tests which hang intermittently are un-debuggable. I believe it is only for the buildbot that -e is not the default. Speaking of which, let me go update TwistedEmacs...
Glyph Lefkowitz wrote:
I do despise the unittest (and hence trial) camelcaseness. I don't know enough low level python to be able to answer whether a test() function would have the same drawback as assert, if -O testing is really an issue.
There is a use case for this, though. py.test has a -x option that will stop executing after one test returns an error. This is very much a time saver when you're running a bunch of lengthy tests on a function where one initial test failing may inevitably mean that a bunch of more sophisticated tests involving the same item will fail. If I can predictably say that test_x will always run before test_advanced_x, then I can try to fix the simple version before doing a lengthy test on the advanced version. One thing that I did like out of py.test was the ability to name a test or test object that you wanted to run and it would do magic pattern matching on the test's __name__ based on what you typed in. It still printed out non-matching tests as skipped, but it only ran what you wanted. It allows you to, again, focus on small functionality within a larger test file.
trial.unittest.setUp was changed to work after 2.0 was released on windows, as I needed to upgrade to twisted svn to get it working. Moof -- Giles Antonio Radford, alias Moof "Too old to be a chicken and too young to be a dirty old man" Serving up my ego over at <http://metamoof.net/>
On Wed, 26 Oct 2005 19:48:32 +0200, Moof <moof@metamoof.net> wrote:
Glyph Lefkowitz wrote:
Nope. A test() function could do all the same stack introspection and everything as assert, but would not create problems with -O. So it's a stealable idea.
3. Tests execute in predictable order.
The correct way to deal with this is to print errors immediately when they happen, like trial -e, or to run just the test that you want, like twisted-dev.el does. If your unit test spits out the error right away, you can fix it while the rest of the tests are running, then kill and restart them immediately, for maximum user/computer parallelism.
Hmm, wildcards might be handy, but in practice I've never wanted more than trial's hierarchical naming. One thing I wish trial had was a reporter mode that looked like 'make' output, eg: % trial --gcc -e foo trial --gcc -e foo.test trial --gcc -e foo.test.test_whatever trial --gcc -e foo.test.test_whatever.TestWhatever.test1 OK trial --gcc -e foo.test.test_whatever.TestWhatever.test2 OK trial --gcc -e foo.test.test_whatever.TestWhatever.test3 FAILED: FooError foo/bar.py:1: ... so I could easily jump to a failed test, then copy the exact trial line needed to run it, so that I could paste that into my buffer's test line, or run trial manually on the command line to 'zoom in' to the test or suite I want. I bet mumak has cleaned up the reporter API so this is now sane to do, I should have a look.
Serving up my ego over at <http://metamoof.net/>
BTW, this site seems to be down.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 glyph@divmod.com wrote:
+1 Vorpal, Bugslaying -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (MingW32) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDYALs3A5SrXAiHQcRAo4LAKCgUtvHUgYc9J1bMxc90jBGPHveDgCfXdWv T22mtWl6P7j7ijI0oSfYOE4= =c4nd -----END PGP SIGNATURE-----
On 2005.10.26 13:14:56 -0400, Glyph Lefkowitz wrote:
Sorry. Haven't used trial lately, and didn't want to make outdated comparisons. unittest is the lingua franca; if I accidentally say something useful, the trial guys can surely translate.
$ python -O `which py.test` test_assert.py inserting into sys.path: /home/dripton/py-dist /home/dripton/py-dist/py/test/cmdline.py:30: UserWarning: Assertions are turned off! (are you using python -O?) py.std.warnings.warn("Assertions are turned off!" ... So I think this is a purely theoretical problem. I can respect disliking the magic assert on general principles, though, even though it's very handy. (Kinda like the print statement.)
Well, you can abuse globals to carry state across test methods in unittest and trial as well. (Init them at the module level rather than in setUp.)
If someone insists on writing bad tests, I don't think there's much a Python framework is going to do to stop them. It's just not a very authoritarian language. "Stop, or I'll say 'stop' again!" But I think most slackers tend to skip writing bad tests and go straight to writing no tests. I admit to reusing expensive-to-create objects across multiple test functions/methods for performance, but I have been conditioned by years of xUnit to treat such objects as immutable. (Okay, in the case of database connections, I put a rollback call in the teardown, which is not quite the same thing.) There are other uses for predictable order, even if the tests are fully independent. Like running the test you're currently trying to fix first, or moving the slowest ones to the end. Knowing what your program is going to do is sometimes handy. (unittest is random enough that one of my co-workers insisted it must be multi-threaded.)
Yeah, that makes sense. But *if* you're going to have deferred tracebacks, then having print statements silenced when the test passes can be nice, so that you can lie in wait for intermittent bugs without generating too much noise. -- David Ripton dripton@ripton.net
On 26/10/05, David Ripton <dripton@ripton.net> wrote:
class FooTest(unittest.TestCase): failureException = AssertionError def test_foo(self): assert False #yay fails
2. You can use module-level test functions. No need to make a test class unless it's actually helpful. (Yay Java.)
This feature is present in unittest, and presumably works for free in Trial. Don't know about cmd line support.
3. Tests execute in predictable order.
Trial *ought* to execute tests in a predictable order. If it doesn't, please file a bug on http://www.twistedmatrix.com/bugs/, assign it to 'jml' with topic 'trial'. (Also, please include repro)
4. setup_module / setup_class / setup_method. Handy for reusing database connections.
We are working on something better than that.
*blink* Thanks for the feedback! jml
On Oct 26, 2005, at 8:38 PM, Jonathan Lange wrote:
It does, it just doesn't execute them in *source* order, which is sometimes annoying. I've groused about this before...I generally write tests in order from simple->complex as well, and python happily throws that order out before trial even gets a chance to see it. Easily fixable by giving your test cases a numeric prefix. Possibly "fixable" in trial with a horrible hack of looking up source line numbers of classes/methods and sorting according to that, but that seems rather...horrible. James
On Wed, 26 Oct 2005 21:11:06 -0400, James Y Knight <foom@fuhm.net> wrote:
On Oct 26, 2005, at 8:38 PM, Jonathan Lange wrote:
Why? It's just one attribute - no magical side effects or anything. It's documented, and this only affects sort order. Seems like a pretty reasonable change to me.
On Oct 27, 2005, at 2:10 AM, glyph@divmod.com wrote:
Well, here's the 5 minute solution. It works...no points for style though. James Index: runner.py =================================================================== --- runner.py (revision 14859) +++ runner.py (working copy) @@ -296,6 +296,34 @@ return thing.__name__ return thing.id() +def sourceOrder(thing): + if isinstance(thing, pyunit.TestCase): + # ?!?! + thing = thing._parents[0] + if hasattr(thing, 'im_func'): + thing = thing.im_func + if hasattr(thing, 'func_code'): + thing = thing.func_code + if hasattr(thing, 'co_firstlineno'): + return thing.co_firstlineno + + if isinstance(thing, (types.ClassType, type)): + so = None + for x in vars(thing).itervalues(): + try: + newso = sourceOrder(x) + except TypeError: + # Not a sourceorderable + pass + else: + if so is not None: + so = min(so, newso) + else: + so = newso + if so is None: + return 0 + return so + raise TypeError("Unknown test object type: %s %s %s" % (thing, type(thing), vars(thing))) def isTestCase(obj): try: @@ -316,6 +344,7 @@ self.suiteFactory = TestSuite self.classSuiteFactory = ClassSuite self.sorter = name + self.testSorter = sourceOrder self._importErrors = [] def _findTestClasses(self, module): @@ -324,7 +353,7 @@ for name, val in inspect.getmembers(module): if isTestCase(val): classes.append(val) - return dsu(classes, self.sorter) + return dsu(classes, self.testSorter) def _findTestModules(self, package): modGlob = os.path.join(os.path.dirname(package.__file__), self.moduleGlob) @@ -371,7 +400,7 @@ factory = self.classSuiteFactory names = reflect.prefixedMethodNames(klass, self.methodPrefix) tests = dsu([ klass(self.methodPrefix+name) for name in names ], - self.sorter) + self.testSorter) suite = factory(klass) suite.addTests(tests) return NamedSuite(klass.__name__, suite)
On Thu, 27 Oct 2005 11:38:01 +1100, Jonathan Lange <jml@mumak.net> wrote:
On 26/10/05, David Ripton <dripton@ripton.net> wrote:
1. assert, not self.failUnlessAllThisUglyCamelCaseActuallyKillsMe()
My understanding is that if you do t = True nil = False assert t == nil in py.test, the error reporting will show you "t != nil (True != False)" or somesuch, along with a bunch of other useful diagnostic information. My proposal was to steal this by doing 'self.failUnless(t == nil)' and stealing as much of their stack introspection junk (available as a hacked version of AssertionError) as we can.
On 10/25/05, Antony Kummel <antonykummel@yahoo.com> wrote:
For #1 and #3, it sounds like you're using trial from Twisted 1.3 (or perhaps even 2.0). I implore you to get 2.1.0, which has been massively improved in these areas (I'm pretty sure "whoop whoop" doesn't exist at all any more). As for #2, I'll let Jp and the others handle that. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | w----v----w-+
On 25/10/05, Antony Kummel <antonykummel@yahoo.com> wrote:
Thanks for your feedback Antony. As Chris pointed out, Trial has greatly improved between 2.0 and 2.1. Trial 2.1 has half the number of lines of code as Trial 2.0, for example. The "whoop whoop" error *is* gone, as is the class that contained it. But one of the biggest changes is that Trial is now (almost) entirely built as an extension to unittest. This is the way things should be. unittest is a great framework with a great design [1], and doesn't (I think) need to be pushed towards any large scale plugin changes. I'm definitely keen to think and discuss more about how Trial behaves wrt the reactor. And I will leap *like a ninja* upon any reproducible bugs you can file [2] with respect to obscure error reporting. cheers, jml -- trial maintainer [1] http://dirtsimple.org/2005/08/ruby-gems-python-eggs-and-beauty-of.html [2] http://www.twistedmatrix.com/bugs/ -- assign to 'jml' with topic 'trial'
On Tue, 25 Oct 2005 05:19:43 -0700 (PDT), Antony Kummel <antonykummel@yahoo.com> wrote:
The network is a source of unpredictability. Unit tests that rely on it fail intermittently, mysteriously, and with no clear course of action for reproducing the failure, making debugging the problem extremely difficult, and reduces the overall utility of the test suite by introduces failures that aren't really failures, but which nevertheless must be investigated to determine whether they represent a real problem. So, there's a pretty good reason, I think. However, trial doesn't prevent you from doing this, so I'm not sure what the objection is.
If tests are allowed to leave things like connections and timers (in general, event sources) lying around, subsequent tests can fail through no fault of their own when one of these event sources misbehaves. If, for example, one causes an exception to be logged, trial will notice this and attribute it to some other hapless test, causing it to fail. These problems are even more difficult to track down than the ones I mentioned above, since it is not even clear in these cases _which_ test is /really/ failing.
I think it's a good idea. I don't know that, in its current form, it is complete. There are probably some improvements that could be made to ease the process of tracking down the sources of various problems it reports. I don't think that means the entire feature should be scrapped.
This should be addressed, certainly. However, I don't often find myself resolving names using the system resolver in unit tests. What if the system resolver is buggy? What if the system is misconfigured? What if there is a transient DNS failure? What if the DNS server for the host you are interested in is temporarily offline? These are not conditions I am happy to allow to cause my unit tests to fail.
Running tests in a child process is an interesting idea. It provides a much greater degree of isolation between tests than essentially any other approach, which is great. Isolation is great for unit tests. Unfortunately, it comes with a lot of overhead. While there are techniques for optimizing the implementation of such a feature, running each test method in a different process would probably add at least 4 minutes to Twisted's test suite. This is basically unacceptable (Twisted's suite takes way too long to run already). Beyond performance problems, there's also the issue of debugging. As in, how do you? I'm aware of remote debuggers for Python, but they're all third-party. This is not necessarily a killer drawback, but it is definitely a downside.
Rather than hearing about the plethora of new testing libraries appearing, I'd like to hear about features they provide that are valuable for writing tests. I would certainly like to borrow py.test's magical assert. What other features are test authors finding useful in some of these projects? Jp
Hi Jp, --- Jean-Paul Calderone <exarkun@divmod.com> wrote:
Well, it will certainly be non-optimal, but let me add some more points in its favour: 1. It may still be faster than debugging unit tests that fail because of a dirty reactor or other problems that it resolves. 2. You don't have to run each test method in a different process, only the ones who leave the environment dirty. Of course people are likely to be lazy if they get the chance, but it makes more sense to leave it up to them. And of course it could be optimized (is a pool of process what you had in mind?) 3. This feature brings us very close to completely distributed unit testing. This will make it possible to easily run tests simultaneously on different computers which would actually make it faster than the current method with very little effort (given enough computers). 4. A mechanism allowing for running unit tests at random locations (processes or machines) will probably be good for making the tests themselves span more than one computer. Despite your justified objections I think this could be quite a powerful tool. 5. The same mechanism can be useful for another problem I often run into: dirtying of the interactive prompt environment. When using a relatively rich tool such as PyCrust, for example, I have to close it and start it again every time I make a grave enough mistake (for example when I need to reload nested modules). If the code I write at the prompt was to run in a different process, I could simply click a button to restart that process without it having an annoying impact.
First of all, TestOOB already allows you to run test in a different process (although I didn't go down to the details -- this may be a little different than what I talked about). If the plugin-oriented framework I proposed existed, the feature we discussed above would have already been available to us. Secondly, it is not hard to think about ways to make unit testing easier. Take a graphical UI for example. In the current situation, any such improvement will be useful for only a small group of users, and that group is likely to grow ever smaller with the emergeance of new frameworks. And when doing unit tests, we all desparately want the same thing -- to make it easier. With a common base, properly designed, we can all benefit from everyone else's efforts, and I think this is good enough a reason. I join you in wanting to hear what test authors are finding useful in these projects. Antony Kummel __________________________________ Start your day with Yahoo! - Make it your home page! http://www.yahoo.com/r/hs
On Tue, Oct 25, 2005 at 12:59:24PM -0400, Jean-Paul Calderone wrote:
Robert Collins' subunit and testresources libraries for pyunit go a long way to solving this, I believe: http://www.robertcollins.net/unittest/
Trial is moving very rapidly towards pyunit compatibility once again, thanks to Jonathan Lange's efforts. It should be possible to run trial test suites from an ordinary unittest.py runner in the near future, if it's not already possible with SVN Trial. I'm not sure what you mean by "push for a plugin-oriented pyUnit" -- it already is extensible. I was under the impression that py.test wasn't at all pyunit compatible, though? The docs I've seen tell me that they explicitly aren't interested in being able to interoperate with unittest or unittest extensions.
FWIW, I don't like py.test's magical assert ;) -Andrew.
On 2005.10.25 12:59:24 -0400, Jean-Paul Calderone wrote:
Things I particularly like about py.test, vs. unittest: 1. assert, not self.failUnlessAllThisUglyCamelCaseActuallyKillsMe() 2. You can use module-level test functions. No need to make a test class unless it's actually helpful. (Yay Java.) 3. Tests execute in predictable order. 4. setup_module / setup_class / setup_method. Handy for reusing database connections. 5. A separate test runner binary, so you don't need __main__ boilerplate in every test module, and you can put tests in the same module as a script (if you don't care about them being found automatically). 6. You can fill your tests with print statements, and only see their output if the test fails (or you pass the -s option). -- David Ripton dripton@ripton.net
On Wed, 2005-10-26 at 04:56 -0700, David Ripton wrote:
Things I particularly like about py.test, vs. unittest:
Can we compare to trial as it stands please? I don't think anyone is seriously suggesting stdlib unittest for Twisted, and I am pretty sure trial has already subsumed every interesting feature from unittest itself :)
1. assert, not self.failUnlessAllThisUglyCamelCaseActuallyKillsMe()
This is cute - more in the way it deals with printing appropriate context than the fact that you actually use 'assert' - but it's a shame that they resorted to a language hack rather than a simple function like 'test()' or something. Using 'assert' makes it impossible to test with -O or -OO. I personally don't use those options, but some people do, and they should be testable.
2. You can use module-level test functions. No need to make a test class unless it's actually helpful. (Yay Java.)
Handy in some cases, but from what I've seen in my brief survey of py.test-based tests the only thing this does is make it more of a convention to smear unit test state across a bunch of shared globals ...
3. Tests execute in predictable order.
... and then since there is now global state smeared across your whole module, the tests have to run in an exact order because each one sets up things for the next, rather than proper setUp/tearDown encapsulation.
4. setup_module / setup_class / setup_method. Handy for reusing database connections.
Technically trial has similar features, but they were broken so horribly for so long that it became the common wisdom not to use them. I have to assume this has recently changed?
Trial has this already too.
6. You can fill your tests with print statements, and only see their output if the test fails (or you pass the -s option).
This is like a replication of trial's AND unittest's worst feature - the fact that tracebacks are deferred until execution is finished, so tests which hang intermittently are un-debuggable. I believe it is only for the buildbot that -e is not the default. Speaking of which, let me go update TwistedEmacs...
Glyph Lefkowitz wrote:
I do despise the unittest (and hence trial) camelcaseness. I don't know enough low level python to be able to answer whether a test() function would have the same drawback as assert, if -O testing is really an issue.
There is a use case for this, though. py.test has a -x option that will stop executing after one test returns an error. This is very much a time saver when you're running a bunch of lengthy tests on a function where one initial test failing may inevitably mean that a bunch of more sophisticated tests involving the same item will fail. If I can predictably say that test_x will always run before test_advanced_x, then I can try to fix the simple version before doing a lengthy test on the advanced version. One thing that I did like out of py.test was the ability to name a test or test object that you wanted to run and it would do magic pattern matching on the test's __name__ based on what you typed in. It still printed out non-matching tests as skipped, but it only ran what you wanted. It allows you to, again, focus on small functionality within a larger test file.
trial.unittest.setUp was changed to work after 2.0 was released on windows, as I needed to upgrade to twisted svn to get it working. Moof -- Giles Antonio Radford, alias Moof "Too old to be a chicken and too young to be a dirty old man" Serving up my ego over at <http://metamoof.net/>
On Wed, 26 Oct 2005 19:48:32 +0200, Moof <moof@metamoof.net> wrote:
Glyph Lefkowitz wrote:
Nope. A test() function could do all the same stack introspection and everything as assert, but would not create problems with -O. So it's a stealable idea.
3. Tests execute in predictable order.
The correct way to deal with this is to print errors immediately when they happen, like trial -e, or to run just the test that you want, like twisted-dev.el does. If your unit test spits out the error right away, you can fix it while the rest of the tests are running, then kill and restart them immediately, for maximum user/computer parallelism.
Hmm, wildcards might be handy, but in practice I've never wanted more than trial's hierarchical naming. One thing I wish trial had was a reporter mode that looked like 'make' output, eg: % trial --gcc -e foo trial --gcc -e foo.test trial --gcc -e foo.test.test_whatever trial --gcc -e foo.test.test_whatever.TestWhatever.test1 OK trial --gcc -e foo.test.test_whatever.TestWhatever.test2 OK trial --gcc -e foo.test.test_whatever.TestWhatever.test3 FAILED: FooError foo/bar.py:1: ... so I could easily jump to a failed test, then copy the exact trial line needed to run it, so that I could paste that into my buffer's test line, or run trial manually on the command line to 'zoom in' to the test or suite I want. I bet mumak has cleaned up the reporter API so this is now sane to do, I should have a look.
Serving up my ego over at <http://metamoof.net/>
BTW, this site seems to be down.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 glyph@divmod.com wrote:
+1 Vorpal, Bugslaying -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (MingW32) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDYALs3A5SrXAiHQcRAo4LAKCgUtvHUgYc9J1bMxc90jBGPHveDgCfXdWv T22mtWl6P7j7ijI0oSfYOE4= =c4nd -----END PGP SIGNATURE-----
On 2005.10.26 13:14:56 -0400, Glyph Lefkowitz wrote:
Sorry. Haven't used trial lately, and didn't want to make outdated comparisons. unittest is the lingua franca; if I accidentally say something useful, the trial guys can surely translate.
$ python -O `which py.test` test_assert.py inserting into sys.path: /home/dripton/py-dist /home/dripton/py-dist/py/test/cmdline.py:30: UserWarning: Assertions are turned off! (are you using python -O?) py.std.warnings.warn("Assertions are turned off!" ... So I think this is a purely theoretical problem. I can respect disliking the magic assert on general principles, though, even though it's very handy. (Kinda like the print statement.)
Well, you can abuse globals to carry state across test methods in unittest and trial as well. (Init them at the module level rather than in setUp.)
If someone insists on writing bad tests, I don't think there's much a Python framework is going to do to stop them. It's just not a very authoritarian language. "Stop, or I'll say 'stop' again!" But I think most slackers tend to skip writing bad tests and go straight to writing no tests. I admit to reusing expensive-to-create objects across multiple test functions/methods for performance, but I have been conditioned by years of xUnit to treat such objects as immutable. (Okay, in the case of database connections, I put a rollback call in the teardown, which is not quite the same thing.) There are other uses for predictable order, even if the tests are fully independent. Like running the test you're currently trying to fix first, or moving the slowest ones to the end. Knowing what your program is going to do is sometimes handy. (unittest is random enough that one of my co-workers insisted it must be multi-threaded.)
Yeah, that makes sense. But *if* you're going to have deferred tracebacks, then having print statements silenced when the test passes can be nice, so that you can lie in wait for intermittent bugs without generating too much noise. -- David Ripton dripton@ripton.net
On 26/10/05, David Ripton <dripton@ripton.net> wrote:
class FooTest(unittest.TestCase): failureException = AssertionError def test_foo(self): assert False #yay fails
2. You can use module-level test functions. No need to make a test class unless it's actually helpful. (Yay Java.)
This feature is present in unittest, and presumably works for free in Trial. Don't know about cmd line support.
3. Tests execute in predictable order.
Trial *ought* to execute tests in a predictable order. If it doesn't, please file a bug on http://www.twistedmatrix.com/bugs/, assign it to 'jml' with topic 'trial'. (Also, please include repro)
4. setup_module / setup_class / setup_method. Handy for reusing database connections.
We are working on something better than that.
*blink* Thanks for the feedback! jml
On Oct 26, 2005, at 8:38 PM, Jonathan Lange wrote:
It does, it just doesn't execute them in *source* order, which is sometimes annoying. I've groused about this before...I generally write tests in order from simple->complex as well, and python happily throws that order out before trial even gets a chance to see it. Easily fixable by giving your test cases a numeric prefix. Possibly "fixable" in trial with a horrible hack of looking up source line numbers of classes/methods and sorting according to that, but that seems rather...horrible. James
On Wed, 26 Oct 2005 21:11:06 -0400, James Y Knight <foom@fuhm.net> wrote:
On Oct 26, 2005, at 8:38 PM, Jonathan Lange wrote:
Why? It's just one attribute - no magical side effects or anything. It's documented, and this only affects sort order. Seems like a pretty reasonable change to me.
On Oct 27, 2005, at 2:10 AM, glyph@divmod.com wrote:
Well, here's the 5 minute solution. It works...no points for style though. James Index: runner.py =================================================================== --- runner.py (revision 14859) +++ runner.py (working copy) @@ -296,6 +296,34 @@ return thing.__name__ return thing.id() +def sourceOrder(thing): + if isinstance(thing, pyunit.TestCase): + # ?!?! + thing = thing._parents[0] + if hasattr(thing, 'im_func'): + thing = thing.im_func + if hasattr(thing, 'func_code'): + thing = thing.func_code + if hasattr(thing, 'co_firstlineno'): + return thing.co_firstlineno + + if isinstance(thing, (types.ClassType, type)): + so = None + for x in vars(thing).itervalues(): + try: + newso = sourceOrder(x) + except TypeError: + # Not a sourceorderable + pass + else: + if so is not None: + so = min(so, newso) + else: + so = newso + if so is None: + return 0 + return so + raise TypeError("Unknown test object type: %s %s %s" % (thing, type(thing), vars(thing))) def isTestCase(obj): try: @@ -316,6 +344,7 @@ self.suiteFactory = TestSuite self.classSuiteFactory = ClassSuite self.sorter = name + self.testSorter = sourceOrder self._importErrors = [] def _findTestClasses(self, module): @@ -324,7 +353,7 @@ for name, val in inspect.getmembers(module): if isTestCase(val): classes.append(val) - return dsu(classes, self.sorter) + return dsu(classes, self.testSorter) def _findTestModules(self, package): modGlob = os.path.join(os.path.dirname(package.__file__), self.moduleGlob) @@ -371,7 +400,7 @@ factory = self.classSuiteFactory names = reflect.prefixedMethodNames(klass, self.methodPrefix) tests = dsu([ klass(self.methodPrefix+name) for name in names ], - self.sorter) + self.testSorter) suite = factory(klass) suite.addTests(tests) return NamedSuite(klass.__name__, suite)
On Thu, 27 Oct 2005 11:38:01 +1100, Jonathan Lange <jml@mumak.net> wrote:
On 26/10/05, David Ripton <dripton@ripton.net> wrote:
1. assert, not self.failUnlessAllThisUglyCamelCaseActuallyKillsMe()
My understanding is that if you do t = True nil = False assert t == nil in py.test, the error reporting will show you "t != nil (True != False)" or somesuch, along with a bunch of other useful diagnostic information. My proposal was to steal this by doing 'self.failUnless(t == nil)' and stealing as much of their stack introspection junk (available as a hacked version of AssertionError) as we can.
On 10/25/05, Antony Kummel <antonykummel@yahoo.com> wrote:
For #1 and #3, it sounds like you're using trial from Twisted 1.3 (or perhaps even 2.0). I implore you to get 2.1.0, which has been massively improved in these areas (I'm pretty sure "whoop whoop" doesn't exist at all any more). As for #2, I'll let Jp and the others handle that. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | w----v----w-+
On 25/10/05, Antony Kummel <antonykummel@yahoo.com> wrote:
Thanks for your feedback Antony. As Chris pointed out, Trial has greatly improved between 2.0 and 2.1. Trial 2.1 has half the number of lines of code as Trial 2.0, for example. The "whoop whoop" error *is* gone, as is the class that contained it. But one of the biggest changes is that Trial is now (almost) entirely built as an extension to unittest. This is the way things should be. unittest is a great framework with a great design [1], and doesn't (I think) need to be pushed towards any large scale plugin changes. I'm definitely keen to think and discuss more about how Trial behaves wrt the reactor. And I will leap *like a ninja* upon any reproducible bugs you can file [2] with respect to obscure error reporting. cheers, jml -- trial maintainer [1] http://dirtsimple.org/2005/08/ruby-gems-python-eggs-and-beauty-of.html [2] http://www.twistedmatrix.com/bugs/ -- assign to 'jml' with topic 'trial'
participants (12)
-
Andrew Bennetts -
Antony Kummel -
Christopher Armstrong -
Cory Dodt -
David Ripton -
Glyph Lefkowitz -
glyph＠divmod.com
-
James Y Knight -
Jean-Paul Calderone -
Jonathan Lange -
Moof -
Ralf Schmitt