[Twisted-Python] Calling deferToThread from a thread
Hi All, I have a data store api that mandates the return of deferreds for each call. I have a number of implementations for the api - an in-memory one for testing, and a number of other network + disk based apis. My issue is this - I'm using py.test for unit testing (I'm aware of trial, but for compatibility reasons with the rest of the project I haven't looked into using it). To test deferred-based code I have set up a separate thread which the reactor runs in (the setup and teardown code is shown below for anyone who is interested). def net(): reactor.run(installSignalHandlers=0) def setup_module(mod): global net_thread net_thread = threading.Thread(target=net) net_thread.start() time.sleep(1) def teardown_module(module): reactor.callFromThread(reactor.stop) This seems to work generally fine. The problem (or at least I *think* it's the problem) is that in one of my implementations uses deferToThread internally to create a deferred which it then returns to the caller. Is it thread safe to call a deferToThread in a separate thread? Normally I'd do a reactor.callFromThread, but I'm not sure how to get a deferred by this construct. I see two options: 1. In my unit test I write a function f which makes the api call and gets the deferred, and sets it to a local variable within the test instance. Then I schedule f to run in the reactor thread via reactor.callFromThread 2. I look into trial which has support for this kind of stuff?? Thanks for any help. Cheers, Reza
On Thu, 18 Sep 2008 10:42:35 +0100, Reza Lotun <rlotun@gmail.com> wrote:
Hi All,
I have a data store api that mandates the return of deferreds for each call. I have a number of implementations for the api - an in-memory one for testing, and a number of other network + disk based apis.
deferToThread isn't special. Like pretty much any Twisted API, it is only safe to call in the reactor thread. It doesn't matter if the reactor thread is the main thread or another one - whichever, that's the only thread Twisted APIs are safe in.
[snip]
I see two options: 1. In my unit test I write a function f which makes the api call and gets the deferred, and sets it to a local variable within the test instance. Then I schedule f to run in the reactor thread via reactor.callFromThread 2. I look into trial which has support for this kind of stuff??
Your best bet is to look at py.test's support for xunit style tests. Whenever you need to deal with Deferreds, you should be using trial's TestCase. Anything else you do will be fragile and prone to break (as trial is, but at least we will fix it when it breaks so that you don't have to). Jean-Paul
On 2008.09.18 10:42:35 +0100, Reza Lotun wrote:
My issue is this - I'm using py.test for unit testing (I'm aware of trial, but for compatibility reasons with the rest of the project I haven't looked into using it). To test deferred-based code I have set up a separate thread which the reactor runs in (the setup and teardown code is shown below for anyone who is interested).
I see two options: 1. In my unit test I write a function f which makes the api call and gets the deferred, and sets it to a local variable within the test instance. Then I schedule f to run in the reactor thread via reactor.callFromThread 2. I look into trial which has support for this kind of stuff??
trial has very good support for deferreds. If your test method returns a deferred then trial magically does the right thing. You don't need to manage the reactor yourself. See Twisted's docs and unit tests for examples. trial's API is based on unittest's, which is based on JUnit's. Works fine, but it has the Java nature. I recommend trying nose. It has support for deferreds, using a decorator. And its API is based on the py.test API. Unfortunately they chose to spell setup and teardown differently, so you can't use your py.test test modules unchanged, but the changes are usually trivial. -- David Ripton dripton@ripton.net
participants (3)
-
David Ripton -
Jean-Paul Calderone -
Reza Lotun