[Twisted-Python] Using a custom reactor in twisted trial for test cases?
![](https://secure.gravatar.com/avatar/c1857ef8af415f6e8510cef704b67c48.jpg?s=120&d=mm&r=g)
Hi there, I am using twisted trial to run test cases for an application. The application however uses stackless python and has a custom stackless reactor. I implemented this reactor like this... -------------------- stacklessreactor.py ----------------------- # Use epoll() as our base reactor from twisted.internet.epollreactor import EPollReactor as StacklessBaseReactor import stackless # seconds between running the greenthreads. 0.0 for flat out 100% CPU STACKLESS_MAX_PUMP_RATE = 0.1 class StacklessReactor(StacklessBaseReactor): """This reactor does the stackless greenthread pumping in the main thread, interwoven with the reactor pump""" def doIteration(self, timeout): """Calls the base reactors doIteration, and then fires off all the stackless threads""" if timeout > STACKLESS_MAX_PUMP_RATE: timeout = STACKLESS_MAX_PUMP_RATE stackless.schedule() return StacklessBaseReactor.doIteration(self,timeout) def install(): """ Install the stackless() reactor. """ p = StacklessReactor() from twisted.internet.main import installReactor installReactor(p) ------------------------------------------------------------------- And I install this as my reactor in my application with... import stacklessreactor stacklessreactor.install() ...placed right at the top of my .tac python file. And this all works. Running the app with twistd, the custom reactor is installed and is used as the reactor for the app. Now however, I come to write tests and run them with trial. I *need* the tests to be run under the stackless reactor or things simply wont work (a lot of the code I need to test are stackless tasklets). When I go "/usr/local/stackless/bin/trial --help-reactors" I get the following list: kqueue kqueue(2)-based reactor. win32 Win32 WaitForMultipleObjects-based reactor. epoll epoll(4)-based reactor. iocp Win32 IO Completion Ports-based reactor. gtk Gtk1 integration reactor. cf CoreFoundation integration reactor. gtk2 Gtk2 integration reactor. default The best reactor for the current platform. debug-gui Semi-functional debugging/introspection reactor. poll poll(2)-based reactor. glib2 GLib2 event-loop integration reactor. select select(2)-based reactor. wx wxPython integration reactor. qt QT integration reactor One of these I can use by passing in --reactor=name. So the question is, is there a way of getting the trial framework to use my custom reactor? Is there a way to get my reactor into that list somehow? Is this not a supported feature of trial? And... if this isn't a supported feature, what is the best way to get a TestCase that will run under that reactor? Look forward to any help people can offer me. With kind regards Crispin Wellington
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On 09:38 am, cwellington@ccg.murdoch.edu.au wrote:
It looks like your custom reactor is mainly in charge of making sure stackless.schedule() gets called at least once every 0.1 seconds. Is that right? If so, a much better approach would be to use twisted.internet.task.LoopingCall rather than implementing a custom reactor. Is there something undesirable about that (much simpler, less fragile) approach? As for your actual question, if you want a new reactor to be as usable as one of the existing ones, you need to write a plugin declaring its available. Take a look at twisted/plugins/twisted_reactors.py for some examples. Jean-Paul
![](https://secure.gravatar.com/avatar/c1857ef8af415f6e8510cef704b67c48.jpg?s=120&d=mm&r=g)
On Fri, 2009-10-30 at 14:06 +0000, exarkun@twistedmatrix.com wrote:
I tried using LoopingCall, but it does not work. It only calls the scheduler once. I think this has to do with the fact that the stackless scheduler needs to be interwoven with the twisted reactor pump. There is more info about why it has to be done like this here: http://code.google.com/p/stacklessexamples/wiki/StacklessTwisted
I shall have a look at that approach. Thanks. Crispin
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Nov 1, 2009, at 11:17 PM, Crispin Wellington wrote:
LoopingCall does "interweave" the function that you pass to it with the "twisted reactor pump". If you were using it correctly, it would work, as the page that you link to indicates :). What do you mean "does not work"?
This example, linked from that page: <http://stacklessexamples.googlecode.com/svn/trunk/examples/twisted/TwistedTi...
is roughly the same as what exarkun recommended. These examples aren't the greatest, because they tend to assume that you _always_ have Stackless code that's ready to run, and you want to run at some number of "frames per second". For simple examples this makes sense, but for a system architecture this is a limiting approach. If your stackless application wants to sit idle and wait for input, you're still going to wake up once every 1/30 second and checking to see if there's anything to do, burning CPU cycles and battery life. Also, if you want to run *faster* than the arbitrary timeout you've selected (1/30 second can be a very long time, especially if you're doing something pseudo-realtime, like audio playback) you're out of luck. Better would be to have tasklets schedule *themselves* for when they want to run. If you just want to allow the rest of the reactor a time- slice, twisted.internet.task.cooperate() will allow you to schedule a potentially arbitrary number of tasks which want to run "as often as possible" without completely swamping the reactor; you can suspend that task by returning a Deferred which only fires when more stackless stuff is actually ready to run.
![](https://secure.gravatar.com/avatar/c1857ef8af415f6e8510cef704b67c48.jpg?s=120&d=mm&r=g)
On Mon, 2009-11-02 at 01:12 -0500, Glyph Lefkowitz wrote:
On Nov 1, 2009, at 11:17 PM, Crispin Wellington wrote:
What do you mean "does not work"?
OK. Having a closer look, its not that looping call doesn't work, its that there is some unknown number of "reactor pumps" between starting the test, and finishing it. What I need is a way for the reactor to be pumping away while a particular test function of a testcase continues working. As an example, here is a non-working test case. Notice the comment "#what to do here to pump the reactor?". ------------------ from twisted.trial import unittest, reporter, runner import os, time import stackless from twisted.internet import reactor, task def example_func(t=10.0): """wait for t seconds then return""" exit_time = time.time()+t while time.time()<exit_time: stackless.schedule() t = task.LoopingCall(stackless.schedule) t.start(0.1) class StacklessTest(unittest.TestCase): def setUpClass(self): pass def test_stackless(self): """Test that we can successfuly create a user proxy cert""" # get the time now... start = time.time() task = stackless.tasklet(example_func) task.setup(10.0) task.run() while task.alive: pass #what to do here to pump the reactor? # end time end = time.time() self.assert_( endtime - starttime >= 10.0 ) --------------------- Running this under trial, it just hangs, inside the while task.alive: loop. So I guess my problem is my approach. How do I test long running "tasklets" that use twisted calls (unlike this contrived example that only sleeps) within the twisted trial framework? How would the Twisted experts write test code for a case like this? Any help is much appreciated! Kind Regards Crispin Wellington
![](https://secure.gravatar.com/avatar/0da8d6ba99ebd77ed8a31c889a28f542.jpg?s=120&d=mm&r=g)
On 2 Nov 2009, at 07:22, Crispin Wellington wrote:
I'm no expert, but I have some tests against objects that use spawnProcess() which has the similar problem that I need to pump the reactor. The key is to return a deferred from the trial test method - then trial does the pump for you. I'm actually using @inlineCallbacks and the sleep() function mentioned in a previous post - you'll probably want task.deferLater() or similar.
![](https://secure.gravatar.com/avatar/c1857ef8af415f6e8510cef704b67c48.jpg?s=120&d=mm&r=g)
On Fri, 2009-10-30 at 14:06 +0000, exarkun@twistedmatrix.com wrote:
So if I make a 'plugin' that declares a twisted.application.reactors.Reactor in the way done so inside twisted/plugins/twisted_reactors.py, how do I make twisted trial pick it up? Or do I actually have to add it into twisted/plugins/twisted_reactors.py? Or monkey punch it in? If so, where do I bootstap it in? Trial seems to need the reactor before TestCases are even loaded. Thanks again for any help Crispin
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On 09:38 am, cwellington@ccg.murdoch.edu.au wrote:
It looks like your custom reactor is mainly in charge of making sure stackless.schedule() gets called at least once every 0.1 seconds. Is that right? If so, a much better approach would be to use twisted.internet.task.LoopingCall rather than implementing a custom reactor. Is there something undesirable about that (much simpler, less fragile) approach? As for your actual question, if you want a new reactor to be as usable as one of the existing ones, you need to write a plugin declaring its available. Take a look at twisted/plugins/twisted_reactors.py for some examples. Jean-Paul
![](https://secure.gravatar.com/avatar/c1857ef8af415f6e8510cef704b67c48.jpg?s=120&d=mm&r=g)
On Fri, 2009-10-30 at 14:06 +0000, exarkun@twistedmatrix.com wrote:
I tried using LoopingCall, but it does not work. It only calls the scheduler once. I think this has to do with the fact that the stackless scheduler needs to be interwoven with the twisted reactor pump. There is more info about why it has to be done like this here: http://code.google.com/p/stacklessexamples/wiki/StacklessTwisted
I shall have a look at that approach. Thanks. Crispin
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Nov 1, 2009, at 11:17 PM, Crispin Wellington wrote:
LoopingCall does "interweave" the function that you pass to it with the "twisted reactor pump". If you were using it correctly, it would work, as the page that you link to indicates :). What do you mean "does not work"?
This example, linked from that page: <http://stacklessexamples.googlecode.com/svn/trunk/examples/twisted/TwistedTi...
is roughly the same as what exarkun recommended. These examples aren't the greatest, because they tend to assume that you _always_ have Stackless code that's ready to run, and you want to run at some number of "frames per second". For simple examples this makes sense, but for a system architecture this is a limiting approach. If your stackless application wants to sit idle and wait for input, you're still going to wake up once every 1/30 second and checking to see if there's anything to do, burning CPU cycles and battery life. Also, if you want to run *faster* than the arbitrary timeout you've selected (1/30 second can be a very long time, especially if you're doing something pseudo-realtime, like audio playback) you're out of luck. Better would be to have tasklets schedule *themselves* for when they want to run. If you just want to allow the rest of the reactor a time- slice, twisted.internet.task.cooperate() will allow you to schedule a potentially arbitrary number of tasks which want to run "as often as possible" without completely swamping the reactor; you can suspend that task by returning a Deferred which only fires when more stackless stuff is actually ready to run.
![](https://secure.gravatar.com/avatar/c1857ef8af415f6e8510cef704b67c48.jpg?s=120&d=mm&r=g)
On Mon, 2009-11-02 at 01:12 -0500, Glyph Lefkowitz wrote:
On Nov 1, 2009, at 11:17 PM, Crispin Wellington wrote:
What do you mean "does not work"?
OK. Having a closer look, its not that looping call doesn't work, its that there is some unknown number of "reactor pumps" between starting the test, and finishing it. What I need is a way for the reactor to be pumping away while a particular test function of a testcase continues working. As an example, here is a non-working test case. Notice the comment "#what to do here to pump the reactor?". ------------------ from twisted.trial import unittest, reporter, runner import os, time import stackless from twisted.internet import reactor, task def example_func(t=10.0): """wait for t seconds then return""" exit_time = time.time()+t while time.time()<exit_time: stackless.schedule() t = task.LoopingCall(stackless.schedule) t.start(0.1) class StacklessTest(unittest.TestCase): def setUpClass(self): pass def test_stackless(self): """Test that we can successfuly create a user proxy cert""" # get the time now... start = time.time() task = stackless.tasklet(example_func) task.setup(10.0) task.run() while task.alive: pass #what to do here to pump the reactor? # end time end = time.time() self.assert_( endtime - starttime >= 10.0 ) --------------------- Running this under trial, it just hangs, inside the while task.alive: loop. So I guess my problem is my approach. How do I test long running "tasklets" that use twisted calls (unlike this contrived example that only sleeps) within the twisted trial framework? How would the Twisted experts write test code for a case like this? Any help is much appreciated! Kind Regards Crispin Wellington
![](https://secure.gravatar.com/avatar/0da8d6ba99ebd77ed8a31c889a28f542.jpg?s=120&d=mm&r=g)
On 2 Nov 2009, at 07:22, Crispin Wellington wrote:
I'm no expert, but I have some tests against objects that use spawnProcess() which has the similar problem that I need to pump the reactor. The key is to return a deferred from the trial test method - then trial does the pump for you. I'm actually using @inlineCallbacks and the sleep() function mentioned in a previous post - you'll probably want task.deferLater() or similar.
![](https://secure.gravatar.com/avatar/c1857ef8af415f6e8510cef704b67c48.jpg?s=120&d=mm&r=g)
On Fri, 2009-10-30 at 14:06 +0000, exarkun@twistedmatrix.com wrote:
So if I make a 'plugin' that declares a twisted.application.reactors.Reactor in the way done so inside twisted/plugins/twisted_reactors.py, how do I make twisted trial pick it up? Or do I actually have to add it into twisted/plugins/twisted_reactors.py? Or monkey punch it in? If so, where do I bootstap it in? Trial seems to need the reactor before TestCases are even loaded. Thanks again for any help Crispin
participants (4)
-
Crispin Wellington
-
exarkun@twistedmatrix.com
-
Glyph Lefkowitz
-
Paul Thomas