[Twisted-Python] Twisted and PEP 3148 support
There has been some discussion over email between myself and several Twisted developers regarding the possibility of adding support for PEP 3148 futures and executors to Twisted. I'm starting this thread to move the conversation to this mailing list to gain a broader audience. For those who do not know what PEP 3148 is about, suffice to say that it's an API for deferred execution of code. The reference implementation can be found in the Python 3.2 standard library under the package name concurrent.futures. A backport that works on Python 2.5, 2.6, 2.7 and 3.1 has been made available by Brian Quinlan and myself and is available for download at the Python Package Index. A brief summary of the email conversations: * The purpose of integrating PEP 3148 support is the intended shift towards a common API for creating asynchronous applications * Deferreds and Futures are alike in purpose but very different under the hood * Support for the Futures API cannot be integrated directly to Deferreds because of the "result" attribute which is a method in Future but a value in Deferred * Some people suggested wrapping Futures in Deferreds, though I have yet to hear a rationale for this * PEP 3148 executor support could be directly integrated to Twisted's thread pools (and by extension, reactors) as there are no naming conflicts that I'm aware of The main focus of the discussion is to figure out how to best integrate support for this new API to Twisted. If possible, existing protocols should remain compatible through the use of adapters or some other mechanism. If not, a way to port them over with a minimal amount of work would be the next target.
2011/1/10 Alex Grönholm <alex.gronholm@nextday.fi>
I just want to make clear up-front the subtle point that Twisted's Deferreds are not tools for "deferred execution of code", but rather for "deferred receipt of results". It's easy to forget this and believe that the similar jargon implies similarity in purpose. -- Christopher Armstrong http://radix.twistedmatrix.com/ http://planet-if.com/
On Mon, Jan 10, 2011 at 11:57:14PM +0200, Alex Grönholm wrote:
I think I've read PEP 3148 before, but that was a while ago. Looking over it again now, here are my initial thoughts as just a user of Twisted. - Executor.submit() seems to be an analogue to Twisted's reactor.callInThread(), that should be easy enough to add to Twisted's implementation. - Executor.map() doesn't havve a Twisted analogue, as far as I know, but seems a useful addition. - Executor.shutdown() seems like a Bad Idea, at least for implementation in Twisted's reactor - I don't want to call some third-party library and have it shut down my entire process when I'm done! Perhaps, rather than making Twisted's reactor implement the Executor interface, there should be .makeExecutor() method that returns an Executor instance that can support the .shutdown() semantics without actually shutting down the Twisted event loop. - Futures are obviously fairly similar to Deferreds, but there are some differences of approach: - Futures support cancellation; Twisted finally managed to get rid of cancellation support in Deferreds. - Futures pass themselves to a series of callbacks, and have .result() and .exception() methods so that the callbacks can discover what happened. Deferreds pass results or Failure objects through a double list of callbacks and errbacks. - Futures store the results of the function call that was run in a thread, and pass the same information to each calback. Deferreds allow each callback to transform the result and even pause the callback-chain to wait for more asynchronous results. - Futures allow code to wait indefinitely for a result or exception to appear, which makes sense if the result is being calculated in a thread, but which would cause a deadlock in an event-based system like Twisted. It should be pretty simple to create a Deferred that wraps a Future: from twisted.internet import defer def deferredFromFuture(future): d = defer.Deferred() def callback(future): e = future.exception() if e: d.fail(e) return d.succeed(future.result()) future.add_done_callback(callback) return d I guess a creating a future that wraps a Deferred wouldn't be hard either: from concurrent.futures import Future def futureFromDeferred(deferred): f = Future() f.set_running_or_notify_cancel() def callback(result): f.set_result(result) return result d.addcallback(callback) def errback(failure): f.set_exception(failure.value) return failure d.adderrback(errback) return f Of course, if somebody added another callback after the ones added by futureFromDeferred(), the new values wouldn't be passed to the Future, and if somebody called future.result() or future.exception() without a timeout, that could cause a deadlock as mentioned above. I'm not sure it's possible to work around either of those problems, so perhaps futureFromDeferred() is a thing that should not be included in a library, at least not without big warnings all over it like Twisted's integration with the stdlib "logging" module has. I also can't immediately see a way to make an object that functions as both a Deferred and a Future, or build Deferred's functionality on top of Futures. Perhaps I just haven't thought it through sufficiently - I'd be interested to hear if anyone else has any ideas.
On 01:54 am, screwtape@froup.com wrote: Just a couple quick (minor) factual corrections. I'll snip everything that does not appear to need correction (of which there was a lot) to make it easier to read.
- Futures support cancellation; Twisted finally managed to get rid of cancellation support in Deferreds.
We only got rid of Deferred.setTimeout. In exchange, we added generalized cancellation support.
Futures may call their callbacks in any thread. So the line:
d.fail(e)
must instead be something like: reactor.callFromThread(d.errback, e) (notice also `d.callback`, not `d.fail`). A similar change is necessary for the success case below.
Jean-Paul
On Tue, Jan 11, 2011 at 02:15:59AM -0000, exarkun@twistedmatrix.com wrote:
Ah, yes, I misremembered. Thank you for the correction. I haven't used Deferred's cancellation support; I don't know how compatible it is with Future's cancellation support.
PEP 3148 says: # Added callables are called in the order that they were added and are # always called in a thread belonging to the process that added them. I assumed that implied some kind of internal .callFromThread() magic; I could be wrong.
2011/1/10 Alex Grönholm <alex.gronholm@nextday.fi>
I just want to make clear up-front the subtle point that Twisted's Deferreds are not tools for "deferred execution of code", but rather for "deferred receipt of results". It's easy to forget this and believe that the similar jargon implies similarity in purpose. -- Christopher Armstrong http://radix.twistedmatrix.com/ http://planet-if.com/
On Mon, Jan 10, 2011 at 11:57:14PM +0200, Alex Grönholm wrote:
I think I've read PEP 3148 before, but that was a while ago. Looking over it again now, here are my initial thoughts as just a user of Twisted. - Executor.submit() seems to be an analogue to Twisted's reactor.callInThread(), that should be easy enough to add to Twisted's implementation. - Executor.map() doesn't havve a Twisted analogue, as far as I know, but seems a useful addition. - Executor.shutdown() seems like a Bad Idea, at least for implementation in Twisted's reactor - I don't want to call some third-party library and have it shut down my entire process when I'm done! Perhaps, rather than making Twisted's reactor implement the Executor interface, there should be .makeExecutor() method that returns an Executor instance that can support the .shutdown() semantics without actually shutting down the Twisted event loop. - Futures are obviously fairly similar to Deferreds, but there are some differences of approach: - Futures support cancellation; Twisted finally managed to get rid of cancellation support in Deferreds. - Futures pass themselves to a series of callbacks, and have .result() and .exception() methods so that the callbacks can discover what happened. Deferreds pass results or Failure objects through a double list of callbacks and errbacks. - Futures store the results of the function call that was run in a thread, and pass the same information to each calback. Deferreds allow each callback to transform the result and even pause the callback-chain to wait for more asynchronous results. - Futures allow code to wait indefinitely for a result or exception to appear, which makes sense if the result is being calculated in a thread, but which would cause a deadlock in an event-based system like Twisted. It should be pretty simple to create a Deferred that wraps a Future: from twisted.internet import defer def deferredFromFuture(future): d = defer.Deferred() def callback(future): e = future.exception() if e: d.fail(e) return d.succeed(future.result()) future.add_done_callback(callback) return d I guess a creating a future that wraps a Deferred wouldn't be hard either: from concurrent.futures import Future def futureFromDeferred(deferred): f = Future() f.set_running_or_notify_cancel() def callback(result): f.set_result(result) return result d.addcallback(callback) def errback(failure): f.set_exception(failure.value) return failure d.adderrback(errback) return f Of course, if somebody added another callback after the ones added by futureFromDeferred(), the new values wouldn't be passed to the Future, and if somebody called future.result() or future.exception() without a timeout, that could cause a deadlock as mentioned above. I'm not sure it's possible to work around either of those problems, so perhaps futureFromDeferred() is a thing that should not be included in a library, at least not without big warnings all over it like Twisted's integration with the stdlib "logging" module has. I also can't immediately see a way to make an object that functions as both a Deferred and a Future, or build Deferred's functionality on top of Futures. Perhaps I just haven't thought it through sufficiently - I'd be interested to hear if anyone else has any ideas.
On 01:54 am, screwtape@froup.com wrote: Just a couple quick (minor) factual corrections. I'll snip everything that does not appear to need correction (of which there was a lot) to make it easier to read.
- Futures support cancellation; Twisted finally managed to get rid of cancellation support in Deferreds.
We only got rid of Deferred.setTimeout. In exchange, we added generalized cancellation support.
Futures may call their callbacks in any thread. So the line:
d.fail(e)
must instead be something like: reactor.callFromThread(d.errback, e) (notice also `d.callback`, not `d.fail`). A similar change is necessary for the success case below.
Jean-Paul
On Tue, Jan 11, 2011 at 02:15:59AM -0000, exarkun@twistedmatrix.com wrote:
Ah, yes, I misremembered. Thank you for the correction. I haven't used Deferred's cancellation support; I don't know how compatible it is with Future's cancellation support.
PEP 3148 says: # Added callables are called in the order that they were added and are # always called in a thread belonging to the process that added them. I assumed that implied some kind of internal .callFromThread() magic; I could be wrong.
participants (5)
-
Alex Grönholm -
Christopher Armstrong -
exarkun＠twistedmatrix.com -
Jason J. W. Williams -
Tim Allen