[Twisted-Python] probably stupid question about automated testing and reactor.run
Hi All, Sorry, the silly questions about testing keep coming... Is it legit for a unit testable piece of an application's code to call reactor.run()? How do you feed a reactor.stop() into such a test? Would that be a sane thing to try and do? How do people deal with this kind of testing? cheers, Chris
Hi, First of all please take what I'm about to say with a pinch of salt, as I really don't know what I'm talking about half the time. As reactors can only be started once, I would imagine calling it from a test function would be largely pointless. Depending on what test suite you're using there should be a way of calling reactor.stop() at the end though... or something. HTH, On Wed, 7 Nov 2018 at 16:14, Chris Withers <chris@withers.org> wrote:
-- Take care, Chris Norman
`reactor` is global mutable state. So all of the usual tragedies of such a thing apply when you consider how unit tests will interact with it. Twisted has some APIs that will jump through some hoops for you to try to mitigate the damage done by this early design mistake. For example, there's a context manager that will swap out the current reactor with a new object specified by you. *If* the code under test doesn't import the reactor until it is invoked, this can help (this is just one reason a lot of modern Twisted code avoids importing the reactor until it absolutely needs it - of course, another thing modern Twisted code often does is accept the reactor as an argument instead of importing it and if you do this then you can escape some of the "global" part of the problem). But you still may not want to have `reactor.run` in your application code. There are other problems - such as the fact that it's common for the *failure* mode of tests for such code to be to hang indefinitely. Or for test failures to be obscure in other ways due to the fact that you're invoking pretty much your *whole* stack at once - not so much a "unit" test - and so finding the cause of the problem is a challenge. Or for tests to work on one run but fail on another due to non-determinism in event delivery/dispatch. Or for tests to work on one platform but fail on another for similar reasons. So, think about the goals you have for the tests you want to write. You may indeed want *some* tests for reactor.run-calling code but my guess is that you want the majority to be for "pure" (in the functional sense) code that stays pretty far from the reactor. Of course, if you're using a Twisted-based library that hasn't thought out the testing story for application code, you may have some roadblocks in your way. Personally, the most recent Twisted-using application I wrote, I factored all of my "start up Twisted and configure the process and such" code into one re-usable component and manually verified it works, then wrote automated tests (mostly unit) for everything *else*. In some ways, this is what `IService` gives you, though if you find yourself wanting to go beyond what it offers you may have to write something like what I did. Finally, Twisted has a bunch of blackbox tests *for reactor implementations* which essentially must call reactor.run. It's not clear how applicable the strategies used by these tests might be to tests for application code (rather than reactor implementation code) but it might be worth looking at. These tests are mostly in twisted/internet/test/ - eg test_tcp.py in that directory. Jean-Paul On Wed, Nov 7, 2018 at 11:14 AM Chris Withers <chris@withers.org> wrote:
Hi, First of all please take what I'm about to say with a pinch of salt, as I really don't know what I'm talking about half the time. As reactors can only be started once, I would imagine calling it from a test function would be largely pointless. Depending on what test suite you're using there should be a way of calling reactor.stop() at the end though... or something. HTH, On Wed, 7 Nov 2018 at 16:14, Chris Withers <chris@withers.org> wrote:
-- Take care, Chris Norman
`reactor` is global mutable state. So all of the usual tragedies of such a thing apply when you consider how unit tests will interact with it. Twisted has some APIs that will jump through some hoops for you to try to mitigate the damage done by this early design mistake. For example, there's a context manager that will swap out the current reactor with a new object specified by you. *If* the code under test doesn't import the reactor until it is invoked, this can help (this is just one reason a lot of modern Twisted code avoids importing the reactor until it absolutely needs it - of course, another thing modern Twisted code often does is accept the reactor as an argument instead of importing it and if you do this then you can escape some of the "global" part of the problem). But you still may not want to have `reactor.run` in your application code. There are other problems - such as the fact that it's common for the *failure* mode of tests for such code to be to hang indefinitely. Or for test failures to be obscure in other ways due to the fact that you're invoking pretty much your *whole* stack at once - not so much a "unit" test - and so finding the cause of the problem is a challenge. Or for tests to work on one run but fail on another due to non-determinism in event delivery/dispatch. Or for tests to work on one platform but fail on another for similar reasons. So, think about the goals you have for the tests you want to write. You may indeed want *some* tests for reactor.run-calling code but my guess is that you want the majority to be for "pure" (in the functional sense) code that stays pretty far from the reactor. Of course, if you're using a Twisted-based library that hasn't thought out the testing story for application code, you may have some roadblocks in your way. Personally, the most recent Twisted-using application I wrote, I factored all of my "start up Twisted and configure the process and such" code into one re-usable component and manually verified it works, then wrote automated tests (mostly unit) for everything *else*. In some ways, this is what `IService` gives you, though if you find yourself wanting to go beyond what it offers you may have to write something like what I did. Finally, Twisted has a bunch of blackbox tests *for reactor implementations* which essentially must call reactor.run. It's not clear how applicable the strategies used by these tests might be to tests for application code (rather than reactor implementation code) but it might be worth looking at. These tests are mostly in twisted/internet/test/ - eg test_tcp.py in that directory. Jean-Paul On Wed, Nov 7, 2018 at 11:14 AM Chris Withers <chris@withers.org> wrote:
participants (3)
-
Chris Norman -
Chris Withers -
Jean-Paul Calderone