[Twisted-Python] Running Twisted tests with testtools
Hello, I'm the nominal maintainer of Trial. I'm also an active maintainer of testtools[1]. testtools is, roughly speaking, a base TestCase that works across a bunch of Pythons and has a few nice features[2]. I started to prefer it over Twisted's TestCase some time ago, and now use it pretty consistently, even for my Twisted code. testtools has a way of letting you run tests within the Twisted reactor, without needing to use a particular base class. For example:: class MyTests(TestCase): run_tests_with = AsynchronousDeferredRunTest def test_foo(self): ... return some_deferred Or:: class MyTests(TestCase): def test_ordinary_thing(self): ... @run_test_with(AsynchronousDeferredRunTest) def test_twisted_thing(self): ... return some_deferred It doesn't behave exactly the way Trial does. Freed from backwards compatibility constraints, I tightened up the implementation quite a bit. There are options to make it emulate Trial's behaviour[3]. The documentation says that it's "highly experimental". This is not strictly true any more, as it has been used in numerous projects and has been found to be quite robust. The key source code is at:: http://bazaar.launchpad.net/~testtools-committers/testtools/trunk/view/head:... http://bazaar.launchpad.net/~testtools-committers/testtools/trunk/view/head:... And you can get the code from PyPI or Launchpad[4]. testtools has had this feature for some time now. It's past time that I officially informed the Twisted community about it. That's the main reason for this email. In my ideal world, Twisted's TestCase would inherit from testtools's TestCase and would continue to provide the same API it does today. This might not be everyone else's ideal though, and the path from here to there is unclear to me. I welcome your thoughts. cheers, jml [1] http://testtools.rtfd.org ; http://pypi.python.org/pypi/testtools [2] http://testtools.readthedocs.org/en/latest/overview.html [3] http://testtools.readthedocs.org/en/latest/for-test-authors.html#twisted-sup... [4] bzr branch lp:testtools
On Feb 6, 2012, at 10:48 AM, Jonathan Lange wrote:
Users still routinely struggle with the one dependency we allowed Twisted core to have - zope.interface. I do still think that's worth it, since it freed us from a significant and complex maintenance burden. And I do sometimes wish that we could make it an optional or bundled dependency, to give users who have to download Twisted themselves a gentler on-ramp. By your own admission, however, the person who should be doing maintenance on trial and the person doing maintenance on testtools are mostly the same guy :-). While testtools' features are nice, and I will likely even elect to use myself it for testing some Twisted applications in the future, I would set the bar very high for making testtools a required dependency for Twisted's own test suite. Just for starters, the Python packaging ecosystem disaster would need to be fixed; also, the name of the package should be changed to be more unique so that users wouldn't find things like <http://www.testtools.com/> and <https://github.com/ferruhy/testtools> when searching around the web for the contents of the inevitable packaging error message. -glyph
On Mon, Feb 6, 2012 at 6:37 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
...
I highly recommend trying it out.
Just for starters, the Python packaging ecosystem disaster would need to be fixed;
As much as I would like to see that happen, I am not ever going to do anything about it.
We've already renamed once (from pyunit3k). Would consider renaming again if someone suggested something sufficiently good. Thanks for the reply. jml
Hi Jonathan, |--==> On Mon, 6 Feb 2012 19:01:30 +0000, Jonathan Lange <jml@mumak.net> said: [...] JL> As much as I would like to see that happen, I am not ever going to do JL> anything about it. So what position can the Twisted project realistically take? Keep using trial for testing Twisted itself, but recommending projects based on twisted to use testtools instead (and deprecating trial somehow)? Also, from what I had heard it'd be technically not possible to provide a compatibility layer, is that accurate? Personally, if testtools ever becomes the blessed tool for testing twisted-based projects, I would not mind at all to have to replace from twisted.trial.unittest import TestCase with something like from testtools.trial import TestCase in my tests (where testtools.trial.TestCase clearly provides a compatibility layer with twisted.trial.unittest.TestCase implemented on top of testtools). At least for getting me going, and maybe gradually transition to pure testtools. However it'd be good if Twisted took a clear position on this, because the risk is to see trial languishing and testtools never really taking off (assuming that testtools is the way to go, which is your thinking afaiu). Cheers, Free
On Feb 7, 2012, at 3:49 PM, Free Ekanayaka wrote:
The position that Twisted can take is that Trial is awesome and you should use Trial. testtools is a separate project, and you may elect to use it to enhance your trial experience.
Also, from what I had heard it'd be technically not possible to provide a compatibility layer, is that accurate?
It's already compatible, as Jonathan said. You can just use the trial runner with testtools tests.
Personally, if testtools ever becomes the blessed tool for testing twisted-based projects, I would not mind at all to have to replace
For a nominal donation to the project, someone from the Twisted project can come to your office, put on some ridiculous priest costume and sanctify whatever toolchain you want to use. ;-)
However it'd be good if Twisted took a clear position on this,
Nope. We write software, not position papers :).
Trial is a supported tool that is part of Twisted. It works great for me every single day. It can work great for you. The fact that Jonathan is technically the "maintainer" and he hasn't been doing much work on it is irrelevant; other people have been working on it, bugs do get fixed, features do get added. There's not a lot of personality-driven development on Twisted; every patch is at least a bit of a community effort. (If someone else wanted to step forward and take the "maintainer" mantle I'm sure he'd be happy to give it up.) There are lots of parts of trial which work great with testtools, too. It has a plugin architecture, for example, which can do some interesting things (exarkun wrote a cool one called "merit" but I don't know if the code for that still exists anywhere...). It would be good to provide better documentation for those kinds of things, and maybe some community tools, so the sorts of people who write Nose plugins could also write Twisted plugins. Apropos of that, Trial's web page received some improved content a little while ago, in case anyone is interested in improving its presence on the web, and helping people understand what it can do: <http://twistedmatrix.com/trac/wiki/TwistedTrial>. I think that the main risks to "languishing" or "not taking off" for both Trial and testtools are issues with their own documentation and promotional materials, not in any conflict with each other. If you're worried about it, contribute code or documentation or blog-posts or what have you to one or both projects :). -glyph
Hi Jonathan, |--==> On Wed, 8 Feb 2012 08:32:27 +0000, Jonathan Lange <jml@mumak.net> said: [...] JL> I know it's bad form, but I endorse everything that Glyph said. Except JL> that I'd add that testtools is awesome and you should use that. Would you elaborate on that and explain why you come to prefer it over trial for your new projects? Cheers, Free
On Wed, Feb 8, 2012 at 5:24 PM, Free Ekanayaka <free@64studio.com> wrote:
That's mostly documented here: http://testtools.readthedocs.org/en/latest/overview.html and in the other docs. In a nutshell: details, fixtures, matchers, and a bunch of unittest extensions that make it much more flexible, making it easy to write things like parallel test runners and AsynchronousDeferredRunTest. Trial provides an excellent test runner, but for everything else is pretty much standard. I don't really have a summary of why ADRT is better than Trial's deferred test support. It's largely because you get better debugging information for failed tests (that's what "details" are all about) and because reactor cleanup issues are more localized to individual tests due to not having to support addCleanup and making no promises about what happens outside of test execution. jml
Hi Jonathan, |--==> On Wed, 8 Feb 2012 22:12:34 +0000, Jonathan Lange <jml@mumak.net> said: [...]
Would you elaborate on that and explain why you come to prefer it over trial for your new projects?
JL> That's mostly documented here: JL> http://testtools.readthedocs.org/en/latest/overview.html JL> and in the other docs. In a nutshell: details, fixtures, matchers, and JL> a bunch of unittest extensions that make it much more flexible, making JL> it easy to write things like parallel test runners Yeah, I like these features a lot too, but leaving out AsynchronousDeferredRunTest maybe one could use all these features on top of trial's TestCase, by using testtool's TestCase as a kind of mixin. Not sure it'd work in practice though. JL> I don't really have a summary of why ADRT is better than Trial's JL> deferred test support. It's largely because you get better debugging JL> information for failed tests (that's what "details" are all about) and JL> because reactor cleanup issues are more localized to individual tests JL> due to not having to support addCleanup and making no promises about JL> what happens outside of test execution. Fair enough and good to know, thanks. Cheers, Free
|--==> On Fri, 10 Feb 2012 09:50:10 +0000, Jonathan Lange <jml@mumak.net> said: JL> On Fri, Feb 10, 2012 at 8:43 AM, Free Ekanayaka <free@64studio.com> wrote:
JL> Why would you want to? Maybe be not really that, but I was mainly wondering, maybe Twisted's trial machinery for handling tests returning deferreds could be factored in a way that it can be re-used by AsynchronousDeferredRunTest (which would just be glue code for integrating with testtool's TestCase). This way testtool's twisted support would be based on twisted itself. Not sure it's worth doing if this generalization/refactoring would lead to more complex code, but I thought I'd mention it.
On Fri, Feb 10, 2012 at 11:21 AM, Free Ekanayaka <free@64studio.com> wrote:
testtools/_spinner.py could *probably* be moved into Twisted itself without too much technical hassle. It would have to be a public API for testtools to rely on it, but it would also have to be an API that Twisted strongly warned against using. Then, probably, Trial & testtools could use that. ADRT relies on the RunTest layer of indirection that testtools has, and has code that is genuinely useful to anyone running Twisted tests. Moving that into Twisted would be trickier, since it would mean either duplicating or depending on testtools. jml
Hi Glyph, |--==> On Wed, 8 Feb 2012 02:05:22 -0500, Glyph <glyph@twistedmatrix.com> said: [...] G> The position that Twisted can take is that Trial is awesome and you G> should use Trial. testtools is a separate project, and you may G> elect to use it to enhance your trial experience. That sounds good to me.
Also, from what I had heard it'd be technically not possible to provide a compatibility layer, is that accurate?
G> It's already compatible, as Jonathan said. You can just use the G> trial runner with testtools tests. Even those returning deferreds and using AsynchronousDeferredRunTest? I presume so, just asking to be sure. Also, testtools has an AsynchronousDeferredRunTestForBrokenTwisted class too which I believe mimics more closely what trial currently does, so I'm not sure what the difference between AsynchronousDeferredRunTest and AsynchronousDeferredRunTestForBrokenTwisted would be when using trial to run such tests. I didn't really look into code tough.
Personally, if testtools ever becomes the blessed tool for testing twisted-based projects, I would not mind at all to have to replace
G> For a nominal donation to the project, someone from the Twisted G> project can come to your office, put on some ridiculous priest G> costume and sanctify whatever toolchain you want to use. ;-)
However it'd be good if Twisted took a clear position on this,
G> Nope. We write software, not position papers :). Okay maybe I used the wrong wording. I was asking for a "position" because I got the impression that Jonathan is considering testtools a sort of trial2 that provides improvements over trial, but that was not likely going to replace only it because of compatibility reasons. Please Jonathan correct me if it's not the case. So *if* testtools is considered superior to trial for some reason, but it's problematic for Twisted to internally drop trial in favor of testtools, I was wondering if it was the case to (also) point people to testtools from the Twisted website/documentaiton, or something along those lines. If it's not the case and it's just a matter of preference, than fine. G> Trial is a supported tool that is part of Twisted. It works great G> for me every single day. It can work great for you. Sure, same here. [...] G> I think that the main risks to "languishing" or "not taking off" G> for both Trial and testtools are issues with their own G> documentation and promotional materials, not in any conflict with G> each other. If you're worried about it, contribute code or G> documentation or blog-posts or what have you to one or both G> projects :). Probably a good addition to the documentation of the two projects would be to describe the differences between the two and if there is any reason one should use one or the other, beside bare taste. I can surely help with that when I have better picture myself. Cheers, Free
On Feb 8, 2012, at 12:22 PM, Free Ekanayaka wrote:
I still disagree. This is the job for some other website, to compare python testing tools. After all, why not add in nose? py.test? They both also have Twisted support that works differently. Trial's website and documentation should focus on how Trial works and how to use it, not on providing a comprehensive comparison matrix. I'm not saying that such a comparison shouldn't exist, just that it shouldn't be the Twisted website's job and it's certainly not the Trial documentation's job. So feel free to write one up! (Plus, returning Deferreds should ideally be an edge case, even in the context of testing Twisted stuff - as much as possible, tests should run synchronously and simulate the events that cause asynchrony in the real world, so that they can test different orderings and edge cases and not just depend on random load variance on the build machines to get coverage of those cases.) -glyph
Hi Glyph, |--==> On Wed, 8 Feb 2012 12:30:14 -0500, Glyph <glyph@twistedmatrix.com> said: G> On Feb 8, 2012, at 12:22 PM, Free Ekanayaka wrote:
G> I still disagree. This is the job for some other website, to compare G> python testing tools. After all, why not add in nose? py.test? They G> both also have Twisted support that works differently. Trial's G> website and documentation should focus on how Trial works and how to G> use it, not on providing a comprehensive comparison matrix. G> I'm not saying that such a comparison shouldn't exist, just that it G> shouldn't be the Twisted website's job and it's certainly not the G> Trial documentation's job. So feel free to write one up! G> (Plus, returning Deferreds should ideally be an edge case, even in the G> context of testing Twisted stuff - as much as possible, tests should G> run synchronously and simulate the events that cause asynchrony in the G> real world, so that they can test different orderings and edge cases G> and not just depend on random load variance on the build machines to G> get coverage of those cases.) Both good points, agreed. Cheers, Free
On Wed, Feb 8, 2012 at 5:22 PM, Free Ekanayaka <free@64studio.com> wrote:
Hi Glyph,
|--==> On Wed, 8 Feb 2012 02:05:22 -0500, Glyph <glyph@twistedmatrix.com> said:
...
Yes. (Also, you don't get sure from asking, you get sure from trying).
From the documentation:
* Trial spins the reactor a couple of times before cleaning it up, ``AsynchronousDeferredRunTest`` does not. If you rely on this behavior, use ``AsynchronousDeferredRunTestForBrokenTwisted``. And the API documentation: """Test runner that works around Twisted brokenness re reactor junk. There are many APIs within Twisted itself where a Deferred fires but leaves cleanup work scheduled for the reactor to do. Arguably, many of these are bugs. This runner iterates the reactor event loop a number of times after every test, in order to shake out these buggy-but-commonplace events. """
The passive voice here is masking an ambiguity. I consider testtools superior to trial in many ways. Some others disagree, or don't have enough information to form an opinion. Perhaps it's worth a mention in the Twisted documentation, but I personally won't push too hard for that. ...
For testtools, we'd also have to do that for nose, nose2, unittest2, zope.testing and py.test for it to make any sense. Heck, just listing all of those makes me regret ever writing any test framework code in the first place. I'm not sure I'd want that documentation, to be honest. I would only ever do work towards it if it meant that one test framework could be killed, even if that ended up being testtools. cheers, jml
On Thu, Feb 9, 2012 at 1:54 PM, <exarkun@twistedmatrix.com> wrote:
Right. At the time it was discouraging because I felt like there were a hundred tests to fix one at a time, then add the deprecation warning and then a year later get to my actual goal of removing the extra iterates. I don't feel that way any more. jml
On Feb 9, 2012, at 8:57 AM, Jonathan Lange wrote:
This is also a great exercise for a Twisted contributing beginner, at least, if they have some help and guidance from a mentor familiar with the issues involved. If anyone out there is on the fence about getting involved with the project and looking for an on-ramp that involves manageable chunks of work, test maintenance fixes like these are a great place to get started. -glyph
Hi Glyph, |--==> On Thu, 9 Feb 2012 16:20:37 -0500, Glyph <glyph@twistedmatrix.com> said: [...]
G> This is also a great exercise for a Twisted contributing beginner, G> at least, if they have some help and guidance from a mentor G> familiar with the issues involved. If anyone out there is on the G> fence about getting involved with the project and looking for an G> on-ramp that involves manageable chunks of work, test maintenance G> fixes like these are a great place to get started. It indeed sounds like a nice learning experience of Twisted internals, and I presume fixing a test that relies on these reactor spins could be tricky/fun sometimes. If there are still hundreds of such tests this can be a long work and I may not be able to complete it all, however if it's fine/makes sense that I help fixing at least some of them, then I'll be glad to help. Cheers, Free
Hi Jonathan, |--==> On Wed, 8 Feb 2012 22:15:18 +0000, Jonathan Lange <jml@mumak.net> said: [...]
Even those returning deferreds and using AsynchronousDeferredRunTest? I presume so, just asking to be sure.
JL> Yes. JL> (Also, you don't get sure from asking, you get sure from trying). True, I've taken some time to read the code and try a few experiments, and it was an highly beneficial exercise. I now understand that the logic that deals with tests returning deferreds is all contained in twisted.trial.unittest.TestCase and trial-the-test-runner doesn't basically have any knowledge in this regards. So running a testtools.testscase.TestCase decorated with AsynchronousDeferredRunTest is pretty much transparent. This is a nice separation of concerns, I guess it was kind of obvious but I had never really looked into this.
JL> From the documentation: JL> * Trial spins the reactor a couple of times before cleaning it up, JL> ``AsynchronousDeferredRunTest`` does not. If you rely on this behavior, use JL> ``AsynchronousDeferredRunTestForBrokenTwisted``. I see, so if understand it correctly, the part that implements this behavior in trial is the twisted.trial.util._Janitor._cleanPending method that: # flush short-range timers reactor.iterate(0) reactor.iterate(0) which I guess is always called no matter what, while in testtools this is an optional behavior that you get by setting _OBLIGATORY_REACTOR_ITERATIONS like AsynchronousDeferredRunTestForBrokenTwisted does. Beside code organization/design is this the main substantial difference between twisted.trial.unittest.TestCase and testtools.testcase.TestCase plus AsynchronousDeferredRunTest? Or put it in a stronger form, if there was a compatibility class inheriting from testtools.testcase.TestCase/AsynchronousDeferredRunTestForBrokenTwisted and providing the same API as twisted.trial.unittest.TestCase (like assertFailure, getSkip, etc.) would it work as a seamless drop-in replacement, at least in principle? Cheers, Free
On Feb 6, 2012, at 10:48 AM, Jonathan Lange wrote:
Users still routinely struggle with the one dependency we allowed Twisted core to have - zope.interface. I do still think that's worth it, since it freed us from a significant and complex maintenance burden. And I do sometimes wish that we could make it an optional or bundled dependency, to give users who have to download Twisted themselves a gentler on-ramp. By your own admission, however, the person who should be doing maintenance on trial and the person doing maintenance on testtools are mostly the same guy :-). While testtools' features are nice, and I will likely even elect to use myself it for testing some Twisted applications in the future, I would set the bar very high for making testtools a required dependency for Twisted's own test suite. Just for starters, the Python packaging ecosystem disaster would need to be fixed; also, the name of the package should be changed to be more unique so that users wouldn't find things like <http://www.testtools.com/> and <https://github.com/ferruhy/testtools> when searching around the web for the contents of the inevitable packaging error message. -glyph
On Mon, Feb 6, 2012 at 6:37 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
...
I highly recommend trying it out.
Just for starters, the Python packaging ecosystem disaster would need to be fixed;
As much as I would like to see that happen, I am not ever going to do anything about it.
We've already renamed once (from pyunit3k). Would consider renaming again if someone suggested something sufficiently good. Thanks for the reply. jml
Hi Jonathan, |--==> On Mon, 6 Feb 2012 19:01:30 +0000, Jonathan Lange <jml@mumak.net> said: [...] JL> As much as I would like to see that happen, I am not ever going to do JL> anything about it. So what position can the Twisted project realistically take? Keep using trial for testing Twisted itself, but recommending projects based on twisted to use testtools instead (and deprecating trial somehow)? Also, from what I had heard it'd be technically not possible to provide a compatibility layer, is that accurate? Personally, if testtools ever becomes the blessed tool for testing twisted-based projects, I would not mind at all to have to replace from twisted.trial.unittest import TestCase with something like from testtools.trial import TestCase in my tests (where testtools.trial.TestCase clearly provides a compatibility layer with twisted.trial.unittest.TestCase implemented on top of testtools). At least for getting me going, and maybe gradually transition to pure testtools. However it'd be good if Twisted took a clear position on this, because the risk is to see trial languishing and testtools never really taking off (assuming that testtools is the way to go, which is your thinking afaiu). Cheers, Free
On Feb 7, 2012, at 3:49 PM, Free Ekanayaka wrote:
The position that Twisted can take is that Trial is awesome and you should use Trial. testtools is a separate project, and you may elect to use it to enhance your trial experience.
Also, from what I had heard it'd be technically not possible to provide a compatibility layer, is that accurate?
It's already compatible, as Jonathan said. You can just use the trial runner with testtools tests.
Personally, if testtools ever becomes the blessed tool for testing twisted-based projects, I would not mind at all to have to replace
For a nominal donation to the project, someone from the Twisted project can come to your office, put on some ridiculous priest costume and sanctify whatever toolchain you want to use. ;-)
However it'd be good if Twisted took a clear position on this,
Nope. We write software, not position papers :).
Trial is a supported tool that is part of Twisted. It works great for me every single day. It can work great for you. The fact that Jonathan is technically the "maintainer" and he hasn't been doing much work on it is irrelevant; other people have been working on it, bugs do get fixed, features do get added. There's not a lot of personality-driven development on Twisted; every patch is at least a bit of a community effort. (If someone else wanted to step forward and take the "maintainer" mantle I'm sure he'd be happy to give it up.) There are lots of parts of trial which work great with testtools, too. It has a plugin architecture, for example, which can do some interesting things (exarkun wrote a cool one called "merit" but I don't know if the code for that still exists anywhere...). It would be good to provide better documentation for those kinds of things, and maybe some community tools, so the sorts of people who write Nose plugins could also write Twisted plugins. Apropos of that, Trial's web page received some improved content a little while ago, in case anyone is interested in improving its presence on the web, and helping people understand what it can do: <http://twistedmatrix.com/trac/wiki/TwistedTrial>. I think that the main risks to "languishing" or "not taking off" for both Trial and testtools are issues with their own documentation and promotional materials, not in any conflict with each other. If you're worried about it, contribute code or documentation or blog-posts or what have you to one or both projects :). -glyph
Hi Jonathan, |--==> On Wed, 8 Feb 2012 08:32:27 +0000, Jonathan Lange <jml@mumak.net> said: [...] JL> I know it's bad form, but I endorse everything that Glyph said. Except JL> that I'd add that testtools is awesome and you should use that. Would you elaborate on that and explain why you come to prefer it over trial for your new projects? Cheers, Free
On Wed, Feb 8, 2012 at 5:24 PM, Free Ekanayaka <free@64studio.com> wrote:
That's mostly documented here: http://testtools.readthedocs.org/en/latest/overview.html and in the other docs. In a nutshell: details, fixtures, matchers, and a bunch of unittest extensions that make it much more flexible, making it easy to write things like parallel test runners and AsynchronousDeferredRunTest. Trial provides an excellent test runner, but for everything else is pretty much standard. I don't really have a summary of why ADRT is better than Trial's deferred test support. It's largely because you get better debugging information for failed tests (that's what "details" are all about) and because reactor cleanup issues are more localized to individual tests due to not having to support addCleanup and making no promises about what happens outside of test execution. jml
Hi Jonathan, |--==> On Wed, 8 Feb 2012 22:12:34 +0000, Jonathan Lange <jml@mumak.net> said: [...]
Would you elaborate on that and explain why you come to prefer it over trial for your new projects?
JL> That's mostly documented here: JL> http://testtools.readthedocs.org/en/latest/overview.html JL> and in the other docs. In a nutshell: details, fixtures, matchers, and JL> a bunch of unittest extensions that make it much more flexible, making JL> it easy to write things like parallel test runners Yeah, I like these features a lot too, but leaving out AsynchronousDeferredRunTest maybe one could use all these features on top of trial's TestCase, by using testtool's TestCase as a kind of mixin. Not sure it'd work in practice though. JL> I don't really have a summary of why ADRT is better than Trial's JL> deferred test support. It's largely because you get better debugging JL> information for failed tests (that's what "details" are all about) and JL> because reactor cleanup issues are more localized to individual tests JL> due to not having to support addCleanup and making no promises about JL> what happens outside of test execution. Fair enough and good to know, thanks. Cheers, Free
|--==> On Fri, 10 Feb 2012 09:50:10 +0000, Jonathan Lange <jml@mumak.net> said: JL> On Fri, Feb 10, 2012 at 8:43 AM, Free Ekanayaka <free@64studio.com> wrote:
JL> Why would you want to? Maybe be not really that, but I was mainly wondering, maybe Twisted's trial machinery for handling tests returning deferreds could be factored in a way that it can be re-used by AsynchronousDeferredRunTest (which would just be glue code for integrating with testtool's TestCase). This way testtool's twisted support would be based on twisted itself. Not sure it's worth doing if this generalization/refactoring would lead to more complex code, but I thought I'd mention it.
On Fri, Feb 10, 2012 at 11:21 AM, Free Ekanayaka <free@64studio.com> wrote:
testtools/_spinner.py could *probably* be moved into Twisted itself without too much technical hassle. It would have to be a public API for testtools to rely on it, but it would also have to be an API that Twisted strongly warned against using. Then, probably, Trial & testtools could use that. ADRT relies on the RunTest layer of indirection that testtools has, and has code that is genuinely useful to anyone running Twisted tests. Moving that into Twisted would be trickier, since it would mean either duplicating or depending on testtools. jml
Hi Glyph, |--==> On Wed, 8 Feb 2012 02:05:22 -0500, Glyph <glyph@twistedmatrix.com> said: [...] G> The position that Twisted can take is that Trial is awesome and you G> should use Trial. testtools is a separate project, and you may G> elect to use it to enhance your trial experience. That sounds good to me.
Also, from what I had heard it'd be technically not possible to provide a compatibility layer, is that accurate?
G> It's already compatible, as Jonathan said. You can just use the G> trial runner with testtools tests. Even those returning deferreds and using AsynchronousDeferredRunTest? I presume so, just asking to be sure. Also, testtools has an AsynchronousDeferredRunTestForBrokenTwisted class too which I believe mimics more closely what trial currently does, so I'm not sure what the difference between AsynchronousDeferredRunTest and AsynchronousDeferredRunTestForBrokenTwisted would be when using trial to run such tests. I didn't really look into code tough.
Personally, if testtools ever becomes the blessed tool for testing twisted-based projects, I would not mind at all to have to replace
G> For a nominal donation to the project, someone from the Twisted G> project can come to your office, put on some ridiculous priest G> costume and sanctify whatever toolchain you want to use. ;-)
However it'd be good if Twisted took a clear position on this,
G> Nope. We write software, not position papers :). Okay maybe I used the wrong wording. I was asking for a "position" because I got the impression that Jonathan is considering testtools a sort of trial2 that provides improvements over trial, but that was not likely going to replace only it because of compatibility reasons. Please Jonathan correct me if it's not the case. So *if* testtools is considered superior to trial for some reason, but it's problematic for Twisted to internally drop trial in favor of testtools, I was wondering if it was the case to (also) point people to testtools from the Twisted website/documentaiton, or something along those lines. If it's not the case and it's just a matter of preference, than fine. G> Trial is a supported tool that is part of Twisted. It works great G> for me every single day. It can work great for you. Sure, same here. [...] G> I think that the main risks to "languishing" or "not taking off" G> for both Trial and testtools are issues with their own G> documentation and promotional materials, not in any conflict with G> each other. If you're worried about it, contribute code or G> documentation or blog-posts or what have you to one or both G> projects :). Probably a good addition to the documentation of the two projects would be to describe the differences between the two and if there is any reason one should use one or the other, beside bare taste. I can surely help with that when I have better picture myself. Cheers, Free
On Feb 8, 2012, at 12:22 PM, Free Ekanayaka wrote:
I still disagree. This is the job for some other website, to compare python testing tools. After all, why not add in nose? py.test? They both also have Twisted support that works differently. Trial's website and documentation should focus on how Trial works and how to use it, not on providing a comprehensive comparison matrix. I'm not saying that such a comparison shouldn't exist, just that it shouldn't be the Twisted website's job and it's certainly not the Trial documentation's job. So feel free to write one up! (Plus, returning Deferreds should ideally be an edge case, even in the context of testing Twisted stuff - as much as possible, tests should run synchronously and simulate the events that cause asynchrony in the real world, so that they can test different orderings and edge cases and not just depend on random load variance on the build machines to get coverage of those cases.) -glyph
Hi Glyph, |--==> On Wed, 8 Feb 2012 12:30:14 -0500, Glyph <glyph@twistedmatrix.com> said: G> On Feb 8, 2012, at 12:22 PM, Free Ekanayaka wrote:
G> I still disagree. This is the job for some other website, to compare G> python testing tools. After all, why not add in nose? py.test? They G> both also have Twisted support that works differently. Trial's G> website and documentation should focus on how Trial works and how to G> use it, not on providing a comprehensive comparison matrix. G> I'm not saying that such a comparison shouldn't exist, just that it G> shouldn't be the Twisted website's job and it's certainly not the G> Trial documentation's job. So feel free to write one up! G> (Plus, returning Deferreds should ideally be an edge case, even in the G> context of testing Twisted stuff - as much as possible, tests should G> run synchronously and simulate the events that cause asynchrony in the G> real world, so that they can test different orderings and edge cases G> and not just depend on random load variance on the build machines to G> get coverage of those cases.) Both good points, agreed. Cheers, Free
On Wed, Feb 8, 2012 at 5:22 PM, Free Ekanayaka <free@64studio.com> wrote:
Hi Glyph,
|--==> On Wed, 8 Feb 2012 02:05:22 -0500, Glyph <glyph@twistedmatrix.com> said:
...
Yes. (Also, you don't get sure from asking, you get sure from trying).
From the documentation:
* Trial spins the reactor a couple of times before cleaning it up, ``AsynchronousDeferredRunTest`` does not. If you rely on this behavior, use ``AsynchronousDeferredRunTestForBrokenTwisted``. And the API documentation: """Test runner that works around Twisted brokenness re reactor junk. There are many APIs within Twisted itself where a Deferred fires but leaves cleanup work scheduled for the reactor to do. Arguably, many of these are bugs. This runner iterates the reactor event loop a number of times after every test, in order to shake out these buggy-but-commonplace events. """
The passive voice here is masking an ambiguity. I consider testtools superior to trial in many ways. Some others disagree, or don't have enough information to form an opinion. Perhaps it's worth a mention in the Twisted documentation, but I personally won't push too hard for that. ...
For testtools, we'd also have to do that for nose, nose2, unittest2, zope.testing and py.test for it to make any sense. Heck, just listing all of those makes me regret ever writing any test framework code in the first place. I'm not sure I'd want that documentation, to be honest. I would only ever do work towards it if it meant that one test framework could be killed, even if that ended up being testtools. cheers, jml
On Thu, Feb 9, 2012 at 1:54 PM, <exarkun@twistedmatrix.com> wrote:
Right. At the time it was discouraging because I felt like there were a hundred tests to fix one at a time, then add the deprecation warning and then a year later get to my actual goal of removing the extra iterates. I don't feel that way any more. jml
On Feb 9, 2012, at 8:57 AM, Jonathan Lange wrote:
This is also a great exercise for a Twisted contributing beginner, at least, if they have some help and guidance from a mentor familiar with the issues involved. If anyone out there is on the fence about getting involved with the project and looking for an on-ramp that involves manageable chunks of work, test maintenance fixes like these are a great place to get started. -glyph
Hi Glyph, |--==> On Thu, 9 Feb 2012 16:20:37 -0500, Glyph <glyph@twistedmatrix.com> said: [...]
G> This is also a great exercise for a Twisted contributing beginner, G> at least, if they have some help and guidance from a mentor G> familiar with the issues involved. If anyone out there is on the G> fence about getting involved with the project and looking for an G> on-ramp that involves manageable chunks of work, test maintenance G> fixes like these are a great place to get started. It indeed sounds like a nice learning experience of Twisted internals, and I presume fixing a test that relies on these reactor spins could be tricky/fun sometimes. If there are still hundreds of such tests this can be a long work and I may not be able to complete it all, however if it's fine/makes sense that I help fixing at least some of them, then I'll be glad to help. Cheers, Free
Hi Jonathan, |--==> On Wed, 8 Feb 2012 22:15:18 +0000, Jonathan Lange <jml@mumak.net> said: [...]
Even those returning deferreds and using AsynchronousDeferredRunTest? I presume so, just asking to be sure.
JL> Yes. JL> (Also, you don't get sure from asking, you get sure from trying). True, I've taken some time to read the code and try a few experiments, and it was an highly beneficial exercise. I now understand that the logic that deals with tests returning deferreds is all contained in twisted.trial.unittest.TestCase and trial-the-test-runner doesn't basically have any knowledge in this regards. So running a testtools.testscase.TestCase decorated with AsynchronousDeferredRunTest is pretty much transparent. This is a nice separation of concerns, I guess it was kind of obvious but I had never really looked into this.
JL> From the documentation: JL> * Trial spins the reactor a couple of times before cleaning it up, JL> ``AsynchronousDeferredRunTest`` does not. If you rely on this behavior, use JL> ``AsynchronousDeferredRunTestForBrokenTwisted``. I see, so if understand it correctly, the part that implements this behavior in trial is the twisted.trial.util._Janitor._cleanPending method that: # flush short-range timers reactor.iterate(0) reactor.iterate(0) which I guess is always called no matter what, while in testtools this is an optional behavior that you get by setting _OBLIGATORY_REACTOR_ITERATIONS like AsynchronousDeferredRunTestForBrokenTwisted does. Beside code organization/design is this the main substantial difference between twisted.trial.unittest.TestCase and testtools.testcase.TestCase plus AsynchronousDeferredRunTest? Or put it in a stronger form, if there was a compatibility class inheriting from testtools.testcase.TestCase/AsynchronousDeferredRunTestForBrokenTwisted and providing the same API as twisted.trial.unittest.TestCase (like assertFailure, getSkip, etc.) would it work as a seamless drop-in replacement, at least in principle? Cheers, Free
participants (5)
-
exarkun＠twistedmatrix.com -
Free Ekanayaka -
Glyph -
Glyph Lefkowitz
-
Jonathan Lange