[Twisted-Python] Returning a Deferred as a result from another Deferred
Returning a Deferred as a result from another Deferred seems to be disallowed by an assertion at the start of the callback chain. Why is this? Right now I have worked around this limitation by wrapping my Deferred in a list and then unwrapping it in the callback. Maybe someone can suggest a better approach. My application takes a command from one connection and defers processing to a worker thread to permit potentially-blocking database access. That worker thread often will need to send data over another connection to fulfill the command, and await a reply before the reply can be sent back over the original command connection. The worker thread sends data over the second connection using callFromThread, and this send operation itself returns a Deferred which fires when the reply is received. It is this Deferred that I want to return as a result from the original Deferred created by deferring to the worker thread in the command handler, so I can wait until the reply comes from the second connection, and then reply on the original command connection. Can someone suggest a better approach? Thanks. Ryan
On 10/6/07, Ryan Fugger <rfugger@gmail.com> wrote:
Returning a Deferred as a result from another Deferred seems to be disallowed by an assertion at the start of the callback chain. Why is this? Right now I have worked around this limitation by wrapping my Deferred in a list and then unwrapping it in the callback.
No it's not. I can guarantee this. What's the content of the AssertionError that you are getting? jml
On 10/5/07, Jonathan Lange <jml@mumak.net> wrote:
On 10/6/07, Ryan Fugger <rfugger@gmail.com> wrote:
Returning a Deferred as a result from another Deferred seems to be disallowed by an assertion at the start of the callback chain. Why is this? Right now I have worked around this limitation by wrapping my Deferred in a list and then unwrapping it in the callback.
No it's not. I can guarantee this.
What's the content of the AssertionError that you are getting?
No content in the error. Line 238 of twisted/internet/defer.py (first line in Deferred.callback) is: assert not isinstance(result, Deferred) That's pretty explicit in disallowing Deferreds... I'm working with the version 2.5 release. Ryan
Hi Ryan, On Oct 5, 2007, at 2:41 PM, Ryan Fugger wrote:
Returning a Deferred as a result from another Deferred seems to be disallowed by an assertion at the start of the callback chain. Why is this? Right now I have worked around this limitation by wrapping my Deferred in a list and then unwrapping it in the callback.
The following will cause an AssertionError d = Deferred() d2 = Deferred() d2.callback(d) There isn't really a good reason to do this anyway. Since there is likely no reason to not just return d with your callbacks attached. If you do need two deferreds here for some reason, you _can_ return a deferred from a callback. In which case further callbacks will not be called until the second deferred has fired. -David
On 10/5/07, David Reid <dreid@dreid.org> wrote:
Hi Ryan, On Oct 5, 2007, at 2:41 PM, Ryan Fugger wrote:
Returning a Deferred as a result from another Deferred seems to be disallowed by an assertion at the start of the callback chain. Why is this? Right now I have worked around this limitation by wrapping my Deferred in a list and then unwrapping it in the callback.
The following will cause an AssertionError
d = Deferred() d2 = Deferred()
d2.callback(d)
There isn't really a good reason to do this anyway. Since there is likely no reason to not just return d with your callbacks attached. If you do need two deferreds here for some reason, you _can_ return a deferred from a callback. In which case further callbacks will not be called until the second deferred has fired.
What if I need to return a Deferred from a deferToThread? Ryan
On Oct 5, 2007, at 7:46 PM, Ryan Fugger wrote:
What if I need to return a Deferred from a deferToThread?
That isn't really a "What if I" question, that's a "Why do you think you" question. deferToThread is best used for taking uncontrollably blocking APIs (preferably C APIs that release GIL) and getting a Deferred back. I personally can not think of a single reason why you would ever take a Deferred returning API and shuffle it off to a thread. -David
On 10/5/07, David Reid <dreid@dreid.org> wrote:
On Oct 5, 2007, at 7:46 PM, Ryan Fugger wrote:
What if I need to return a Deferred from a deferToThread?
That isn't really a "What if I" question, that's a "Why do you think you" question.
deferToThread is best used for taking uncontrollably blocking APIs (preferably C APIs that release GIL) and getting a Deferred back. I personally can not think of a single reason why you would ever take a Deferred returning API and shuffle it off to a thread.
See my original email for the specifics. I receive a command on a connection, deferToThread for handling, which involves database access, and then callFromThread to send a message on another connection, which returns a Deferred that fires when the reply is received, triggering a reply to the command on the original connection. Ryan
On 06:34 am, rfugger@gmail.com wrote:
On 10/5/07, David Reid <dreid@dreid.org> wrote:
deferToThread is best used for taking uncontrollably blocking APIs (preferably C APIs that release GIL) and getting a Deferred back. I personally can not think of a single reason why you would ever take a Deferred returning API and shuffle it off to a thread.
See my original email for the specifics. I receive a command on a connection, deferToThread for handling, which involves database access, and then callFromThread to send a message on another connection, which returns a Deferred that fires when the reply is received, triggering a reply to the command on the original connection.
I assume the briefest Python summary of your code looks like this: def thingInThread(): databaseResult = databaseWork() deferred = reactor.callFromThread(otherConnection.sendRequest, databaseResult) return deferred def thingInReactor(): return threads.deferToThread(thingInThread) This isn't really idiomatic Twisted code, because while it is technically safe (thingInThread doesn't *do* anything with its Deferred) it is slightly misleading. Deferreds are not thread safe, and there's nothing you *could* do with that Deferred in thingInThread; if you wanted to add a callback to it, for example, you can't. What you actually want is something more like this: def thingInReactor(): deferred = threads.deferToThread(databaseWork) def tellOtherConnection(databaseResult): return otherConnection.sendRequest(databaseResult) deferred.addCallback(tellOtherConnection) return deferred This deferred can safely have callbacks added to it and generally be treated normally, since it never sees a thread. Also, now you don't have the problem where .callback() is being called with Deferred as an argument; instead, you are returning one Deferred to another, which results in the outer Deferred ("deferred") receiving the inner Deferred's result (whatever "otherConnection.sendRequest" would have fired with).
participants (4)
-
David Reid -
glyph＠divmod.com -
Jonathan Lange -
Ryan Fugger