Re: [Python-ideas] [Python-Dev] Python needs a standard asynchronous return object
Moving to python-ideas. Have you seen http://www.python.org/dev/peps/pep-3148/ ? That seems exactly what you want. --Guido On Fri, Sep 10, 2010 at 4:00 PM, James Yonan <james@openvpn.net> wrote:
-- --Guido van Rossum (python.org/~guido)
On Sat, Sep 11, 2010 at 9:25 AM, Guido van Rossum <guido@python.org> wrote:
James did mention that in the post, although he didn't say what deferreds really added beyond what futures provide, and why the "add_done_callback" method isn't adequate to provide interoperability between futures and deferreds (which would be odd, since Brian made changes to that part of PEP 3148 to help with that interoperability after discussions with Glyph). Between PEP 380 and PEP 3148 I'm not really seeing a lot more scope for standardisation in this space though. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
(Summary: I want to make an apology, and reopen the debate. Possibly relevant: PEP 342, PEP 380, PEP 3148, PEP 3152.) On Sat, Sep 11, 2010 at 9:03 AM, Jesse Noller <jnoller@gmail.com> wrote:
Whoops. I was a bit quick at the trigger there.
Re-reading the OP's post[0] and the blog[1] he references, I notice that he did not mention PEP 380 (which for the blog's example doesn't actually add much except adding a nicer way to return a value from a generator) but he did mention the awesomeness of not needing threads when using deferreds. He sounds as if the python-dev community had never heard of that style of handling concurrency, which seems backwards: the generator-based style of doing it was introduced in PEP 342 which enabled Twisted's inline callbacks. (Though he does mention Python Enhanced Generators which could be an implicit reference to PEP 342 -- "Coroutines via Enhanced Generators".) But thinking about this more I don't know that it will be easy to mix PEP 3148, which is solidly thread-based, with a PEP 342 style scheduler (whether or not the PEP 380 enhancements are applied, or even PEP 3152). And if we take the OP's message at face value, his point isn't so much that Twisted is great, but that in order to benefit maximally from PEP 342 there needs to be a standard way of using callbacks. I think that's probably true. And comparing the blog's examples to PEP 3148, I find Twisted's terminology rather confusing compared to the PEP's clean Futures API (where IMO you can ignore almost everything except result()). Maybe it's possible to write a little framework that lets you create Futures using either threads, processes (both supported by PEP 3148) or generators. But I haven't tried it. And maybe the need to use 'yield' for everything that may block when using generators, but not when using threads or processes, will make this awkward. So maybe we'll be stuck with at least two Future-like APIs: PEP 3148 and something else, generator-based. Or maybe PEP 3152. So, yes, there may be something here, and let's reopen the discussion. And I apologize for shooting first and asking questions second. [0] http://mail.python.org/pipermail/python-dev/2010-September/103576.html [1] http://blog.mekk.waw.pl/archives/14-Twisted-inlineCallbacks-and-deferredGene... -- --Guido van Rossum (python.org/~guido)
On Sat, 11 Sep 2010 19:26:50 -0700 Guido van Rossum <guido@python.org> wrote:
I'm not sure why. The implementation is certainly thread-based, but functions such as `wait(fs, timeout=None, return_when=ALL_COMPLETED)` could be implemented in termes of a single-threaded event loop / job scheduler. Actually, Twisted has a similar primitive in DeferredList, although more powerful since the DeferredList itself is a Deferred, and can therefore be further combined, etc.: http://twistedmatrix.com/documents/10.0.0/api/twisted.internet.defer.Deferre...
Well, apart from the API which may be considered a taste issue (I have used Deferreds long before I heard about Futures, so perhaps I'm a bit biased), the following API doc in PEP 3148 shows that the Future model of callbacks is less rich than Twisted's: “add_done_callback(fn) Attaches a callable fn to the future that will be called when the future is cancelled or finishes running. fn will be called with the future as its only argument. Added callables are called in the order that they were added and are always called in a thread belonging to the process that added them. If the callable raises an Exception then it will be logged and ignored. If the callable raises another BaseException then behavior is not defined.” With Twisted Deferreds, when a callback or errback raises an error, its exception isn't “logged and ignored”, it is passed to the remaining errback chain attached to the Deferred. This is part of what makes Deferreds more complicated to understand, but it also makes them more powerful. Another key point is that a callback can itself return another Deferred object, in which case the next callback (or errback, in case of error) will be called only once the other Deferred produces a result. This is all handled transparently and you can freely mix callbacks that immediately return a value, and callbacks that return a Deferred whose final value will be available later. And the other Deferred can have its own callback/errback chain, etc. (just for the record, the “final value” of a Deferred is the value returned by the last callback in the chain) I think the main reason, though, that people find Deferreds inconvenient is that they force you to think in terms of asynchronicity (well, almost: you can of course hack yourself some code which blocks until a Deferred has a value, but it's extremely discouraged). They would like to have officially supported methods like `result(timeout=None)` which make simple things (like quick scripts to fetch a bunch of URLs) simpler. Twisted is generally used for server applications where such code is out of question (in an async model, that is). Regards Antoine.
On Sun, Sep 12, 2010 at 4:03 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Sure, but the tricky thing is to make it pluggable so that PEP 3148 and Twisted and other frameworks can use it all together, and a single call will accept a mixture of Futures. I also worry that "impure" code will have a hard time -- e.g. when mixing generator-based coroutines and thread-based futures, it would be quite bad if a coroutine called .result() on a Future or the .wait() function instead of yielding to the scheduler.
This sounds similar to the way you can create derived futures in Java.
I heard of Deferred long before PEP 3148 was even conceived, but I find Twisted's terminology terribly confusing while I find the PEP's names easy to understand.
Yeah, please do explain why Twisted has so much machinery to handle exceptions? ISTM that the main difference is that add_done_callback() isn't meant for callbacks that return a value. So then the exceptions that might be raised are kind of "out of band". For any API that returns a value I agree that raising an exception should be handled -- but in the PEP 342 world we can do that by passing exceptions back into coroutine using throw(), so no separate "success" and "failure" callbacks are needed.
Yeah, that is part of what makes it so utterly confusing. PEP 380 supports a similar thing but much cleaner, without ever using callbacks.
Actually I think the main reason is historic: Twisted introduced callback-based asynchronous (thread-less) programming when there was no alternative in Python, and they invented both the mechanisms and the terminology as they were figuring it all out. That is no mean feat. But with PEP 342 (generator-based coroutines) and especially PEP 380 (yield from) there *is* an alternative, and while Twisted has added APIs to support generators, it hasn't started to deprecate its other APIs, and its terminology becomes hard to follow for people (like me, frankly) who first learned this stuff through PEP 342. -- --Guido van Rossum (python.org/~guido)
On Sun, 12 Sep 2010 08:49:56 -0700 Guido van Rossum <guido@python.org> wrote:
Having a common abstraction (Future or Deferred) allows for scheduling-agnostic libraries which consume and/or produce these abstractions (*). I'm not sure it is desireable to mix scheduling models in a single process (let alone a single thread), though. (*) Of course, the abstraction is somehow leaky since being called from different threads, depending on the scheduling model, could have adverse consequences
It implies that it's mostly useful for simple callbacks (which would e.g. print out a success report, or set an Event to wake up another thread). The Twisted model allows the major part of processing to occur in the callbacks themselves, in which case proper error handling and propagation is mandatory. Regards Antoine.
On Sun, Sep 12, 2010 at 9:17 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
IIRC even Twisted supports putting stuff in a thread if you really need it. And have you looked at Go's Goroutines? They are a hybrid -- they don't map 1:1 to OS threads, but they aren't pure coroutines either, so that if a goroutine blocks on I/O the others will still make progress.
Yeah, this is always a problem with pure async frameworks -- if one callback or coroutine blocks by mistake, the whole world is blocked. (So Goroutines attempt to fix this; I have no idea how successful they are.)
A generator-based coroutines approach can do this too (just put the work between the yields in the generator) and has all the proper exception-propagation stuff built in since PEP 342 (PEP 380 will just make it easier). And a Futures-based approach can do it too -- it's not described in PEP 3148, but you can easily design an API for wrappable Futures. -- --Guido van Rossum (python.org/~guido)
Thanks for the ping about this (I don't think I subscribe to python-ideas, so someone may have to moderate my post in). Sorry for the delay in responding, but I've been kinda busy and cooking up these examples took a bit of thinking. And thanks, James, for restarting this discussion. I obviously find it interesting :). I'm going to mix in some other stuff I found on the web archives, since it's easiest just to reply in one message. I'm sorry that this response is a bit sprawling and doesn't have a single clear narrative, the thread thus far didn't seem to lend it to one. For those of you who don't want to read my usual novel-length post, you can probably stop shortly after the end of the first block of code examples. On Sep 11, 2010, at 10:26 PM, Guido van Rossum wrote:
That blog post was written to demonstrate why programs using generators are "... far easier to read and write ..." than ones using Deferreds, so it stands to reason it would choose an example where that helps :). When you want to write systems that manage varying levels of parallelism within a single computation, generators can start to get pretty hairy and the "normal" Deferred way of doing things looks more straightforward. Thinking in terms of asynchronicity is tricky, and generators can be a useful tool for promoting that understanding, but they only make it superficially easier. For example:
If you're writing an application whose parallelism calls for an asynchronous approach, after all, you presumably don't want to be standing around waiting for each network round trip to complete. How do you re-write this so that there are always at least N outstanding do_something_async calls running in parallel? You can sorta do it like this:
but that will always block on one particular do_something_async, when you really want to say "let me know when any outstanding call is complete". So I could handwave about 'yield any_completed(outstanding)'...
but that just begs the question of how you implement any_completed(), and I can't think of a way to do that with generators, without getting into the specifics of some Deferred-or-Future-like asynchronous result object. You could implement such a function with such primitives, and here's what it looks like with Deferreds:
Here's how you do the top-level task in Twisted, without generators, in the truly-parallel fashion (keep in mind this combines the functionality of 'any_completed' and 'parallel', so it's a bit shorter):
Some informal benchmarking has shown this method to be considerably faster (on the order of 1/2 to 1/3 as much CPU time) than at least our own inlineCallbacks generator-scheduling method. Take this with the usual fist-sized grain of salt that you do any 'informal' benchmarks, but the difference is significant enough that I do try to refactor into this style in my own code, and I have seen performance benefits from doing this on more specific benchmarks. This is all untested, and that's far too many lines of code to expect to work without testing, but hopefully it gives a pretty good impression of the differences in flavor between the different styles.
Yeah, please do explain why Twisted has so much machinery to handle exceptions?
There are a lot of different implied questions here, so I'll answer a few of those. Why does twisted.python.failure exist? The answer to that is that we wanted an object that represented an exception as raised at a particular point, associated with a particular stack, that could live on without necessarily capturing all the state in that stack. If you're going to report failures asynchronously, you don't necessarily want to hold a reference to every single thing in a potentially giant stack while you're waiting to send it to some network endpoint. Also, in 1.5.2 we had no way of chaining exceptions, and this code is that old. Finally, even if you can chain exceptions, it's a serious performance hit to have to re-raise and re-catch the same exception 4 or 5 times in order to translate it or handle it at many different layers of the stack, so a Failure is intended to encapsulate that state such that it can just be returned, in performance-sensitive areas. (This is sort of a weak point though, since the performance of Failure itself is so terrible, for unrelated reasons.) Why is twisted.python.failure such a god damned mess? The answer to that is ... uh, sorry. Yes, it is. We should clean it up. It was written a long time ago and the equivalent module now could be _much_ shorter, simpler, and less of a performance problem. It just never seems to be the highest priority. Maybe after we're done porting to py3 :). My one defense here is that still a slight improvement over the stdlib 'traceback' module ;-). Why do Deferreds have an errback chain rather than just handing you an exception object in the callback chain? Basically, this is for the same reason that Python has exceptions instead of just making you check return codes. We wanted it to be easy to say:
and know that the argument to 'ok' would always be what getPage promised (you don't need to typecheck it for exception-ness) and the default error behavior would be to simply bail out with a traceback, not to barrel through your success-path code wreaking havoc.
ISTM that the main difference is that add_done_callback() isn't meant for callbacks that return a value.
add_done_callback works fine with callbacks that return a value. If it didn't, I'd be concerned, because then it would have the barrel-through-the-success-path flaw. But, I assume the idiomatic asynchronous-code-using-Futures would look like this:
This is one extra line of code as compared to the Twisted version, and chaining involves a bit more gymnastics (somehow creating more futures to return further up the stack, I guess, I haven't thought about it too hard), but it does allow you to handle exceptions with a simple 'except:', rather than calling some exception-handling methods, so I can see why some people would prefer it.
Maybe it's possible to write a little framework that lets you create Futures using either threads, processes (both supported by PEP 3148) or generators. But I haven't tried it. And maybe the need to use 'yield' for everything that may block when using generators, but not when using threads or processes, will make this awkward.
You've already addressed the main point that I really wanted to mention here, but I'd like to emphasize it. Blocking and not-blocking are fundamentally different programming styles, and if you sometimes allow blocking on asynchronous results, that means you are effectively always programming in the blocking-and-threaded style and not getting much benefit from the code which does choose to be politely non-blocking. I was somewhat pleased with the changes made to the Futures PEP because you could use them as an asynchronous result, and have things that implemented the Future API but raised an exception if you tried to wait on them. That would at least allow some layer of stdlib compatibility. If you are disciplined and careful, this would let you write async code which used a common interoperability mechanism, and if you weren't careful, it would blow up when you tried to use it the wrong way. But - and I am guessing that this is the main thrust of this discussion - I do think that having Deferred in the standard library would be much, much better if we can do that.
So maybe we'll be stuck with at least two Future-like APIs: PEP 3148 and something else, generator-based.
Having something "generator-based" is, in my opinion, an abstraction inversion. The things which you are yielding from these generators are asynchronous results. There should be a specific type for asynchronous results which can be easily interacted with. Generators are syntactic sugar for doing that interaction in a way which doesn't involve defining tons of little functions. This is useful, and it makes the concept more accessible, so I don't say "just" syntactic sugar: but nevertheless, the generators need to be 'yield'ing something, and the type of thing that they're yielding is a Deferred-or-something-like-it. I don't think that this is really two 'Future-like APIs'. At least, they're not redundant, any more than having both socket.makefile() and socket.recv() is redundant. If Future had a deferred() method rather than an add_done_callback() method, then it would always be very clear whether you had a synchronous-but-possibly-not-ready or a purely-asynchronous result. Although it would be equally easy to just have a function that turned a Future into a Deferred by calling add_done_callback(). You can go from any arbitrary Future to a full-featured Deferred, but not the other way around.
Or maybe PEP 3152.
I don't like PEP 3152 aesthetically on many levels, but I can't deny that it would do the job. 'cocall', though, really? It would be nice if it read like an actual word, i.e. "yield to" or "invoke" or even just "call" or something. In another message, where Guido is replying to Antoine:
I think the main reason, though, that people find Deferreds inconvenient is that they force you to think in terms of asynchronicity (...)
Actually I think the main reason is historic: Twisted introduced callback-based asynchronous (thread-less) programming when there was no alternative in Python, and they invented both the mechanisms and the terminology as they were figuring it all out. That is no mean feat. But with PEP 342 (generator-based coroutines) and especially PEP 380 (yield from) there *is* an alternative, and while Twisted has added APIs to support generators, it hasn't started to deprecate its other APIs, and its terminology becomes hard to follow for people (like me, frankly) who first learned this stuff through PEP 342.
I really have to go with Antoine on this one: people were confused about Deferreds long before PEP 342 came along :). Given that Javascript environments have mostly adopted the Twisted terminology (oddly, Node.js doesn't, but Dojo and MochiKit both have pretty literal-minded Deferred translations), there are plenty of people who are familiar with the terminology but still get confused. See the beginning of the message for why we're not deprecating our own APIs. Once again, sorry for not compressing this down further! If you got this far, you win a prize :).
On Sat, Sep 11, 2010 at 9:25 AM, Guido van Rossum <guido@python.org> wrote:
James did mention that in the post, although he didn't say what deferreds really added beyond what futures provide, and why the "add_done_callback" method isn't adequate to provide interoperability between futures and deferreds (which would be odd, since Brian made changes to that part of PEP 3148 to help with that interoperability after discussions with Glyph). Between PEP 380 and PEP 3148 I'm not really seeing a lot more scope for standardisation in this space though. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
(Summary: I want to make an apology, and reopen the debate. Possibly relevant: PEP 342, PEP 380, PEP 3148, PEP 3152.) On Sat, Sep 11, 2010 at 9:03 AM, Jesse Noller <jnoller@gmail.com> wrote:
Whoops. I was a bit quick at the trigger there.
Re-reading the OP's post[0] and the blog[1] he references, I notice that he did not mention PEP 380 (which for the blog's example doesn't actually add much except adding a nicer way to return a value from a generator) but he did mention the awesomeness of not needing threads when using deferreds. He sounds as if the python-dev community had never heard of that style of handling concurrency, which seems backwards: the generator-based style of doing it was introduced in PEP 342 which enabled Twisted's inline callbacks. (Though he does mention Python Enhanced Generators which could be an implicit reference to PEP 342 -- "Coroutines via Enhanced Generators".) But thinking about this more I don't know that it will be easy to mix PEP 3148, which is solidly thread-based, with a PEP 342 style scheduler (whether or not the PEP 380 enhancements are applied, or even PEP 3152). And if we take the OP's message at face value, his point isn't so much that Twisted is great, but that in order to benefit maximally from PEP 342 there needs to be a standard way of using callbacks. I think that's probably true. And comparing the blog's examples to PEP 3148, I find Twisted's terminology rather confusing compared to the PEP's clean Futures API (where IMO you can ignore almost everything except result()). Maybe it's possible to write a little framework that lets you create Futures using either threads, processes (both supported by PEP 3148) or generators. But I haven't tried it. And maybe the need to use 'yield' for everything that may block when using generators, but not when using threads or processes, will make this awkward. So maybe we'll be stuck with at least two Future-like APIs: PEP 3148 and something else, generator-based. Or maybe PEP 3152. So, yes, there may be something here, and let's reopen the discussion. And I apologize for shooting first and asking questions second. [0] http://mail.python.org/pipermail/python-dev/2010-September/103576.html [1] http://blog.mekk.waw.pl/archives/14-Twisted-inlineCallbacks-and-deferredGene... -- --Guido van Rossum (python.org/~guido)
On Sat, 11 Sep 2010 19:26:50 -0700 Guido van Rossum <guido@python.org> wrote:
I'm not sure why. The implementation is certainly thread-based, but functions such as `wait(fs, timeout=None, return_when=ALL_COMPLETED)` could be implemented in termes of a single-threaded event loop / job scheduler. Actually, Twisted has a similar primitive in DeferredList, although more powerful since the DeferredList itself is a Deferred, and can therefore be further combined, etc.: http://twistedmatrix.com/documents/10.0.0/api/twisted.internet.defer.Deferre...
Well, apart from the API which may be considered a taste issue (I have used Deferreds long before I heard about Futures, so perhaps I'm a bit biased), the following API doc in PEP 3148 shows that the Future model of callbacks is less rich than Twisted's: “add_done_callback(fn) Attaches a callable fn to the future that will be called when the future is cancelled or finishes running. fn will be called with the future as its only argument. Added callables are called in the order that they were added and are always called in a thread belonging to the process that added them. If the callable raises an Exception then it will be logged and ignored. If the callable raises another BaseException then behavior is not defined.” With Twisted Deferreds, when a callback or errback raises an error, its exception isn't “logged and ignored”, it is passed to the remaining errback chain attached to the Deferred. This is part of what makes Deferreds more complicated to understand, but it also makes them more powerful. Another key point is that a callback can itself return another Deferred object, in which case the next callback (or errback, in case of error) will be called only once the other Deferred produces a result. This is all handled transparently and you can freely mix callbacks that immediately return a value, and callbacks that return a Deferred whose final value will be available later. And the other Deferred can have its own callback/errback chain, etc. (just for the record, the “final value” of a Deferred is the value returned by the last callback in the chain) I think the main reason, though, that people find Deferreds inconvenient is that they force you to think in terms of asynchronicity (well, almost: you can of course hack yourself some code which blocks until a Deferred has a value, but it's extremely discouraged). They would like to have officially supported methods like `result(timeout=None)` which make simple things (like quick scripts to fetch a bunch of URLs) simpler. Twisted is generally used for server applications where such code is out of question (in an async model, that is). Regards Antoine.
On Sun, Sep 12, 2010 at 4:03 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Sure, but the tricky thing is to make it pluggable so that PEP 3148 and Twisted and other frameworks can use it all together, and a single call will accept a mixture of Futures. I also worry that "impure" code will have a hard time -- e.g. when mixing generator-based coroutines and thread-based futures, it would be quite bad if a coroutine called .result() on a Future or the .wait() function instead of yielding to the scheduler.
This sounds similar to the way you can create derived futures in Java.
I heard of Deferred long before PEP 3148 was even conceived, but I find Twisted's terminology terribly confusing while I find the PEP's names easy to understand.
Yeah, please do explain why Twisted has so much machinery to handle exceptions? ISTM that the main difference is that add_done_callback() isn't meant for callbacks that return a value. So then the exceptions that might be raised are kind of "out of band". For any API that returns a value I agree that raising an exception should be handled -- but in the PEP 342 world we can do that by passing exceptions back into coroutine using throw(), so no separate "success" and "failure" callbacks are needed.
Yeah, that is part of what makes it so utterly confusing. PEP 380 supports a similar thing but much cleaner, without ever using callbacks.
Actually I think the main reason is historic: Twisted introduced callback-based asynchronous (thread-less) programming when there was no alternative in Python, and they invented both the mechanisms and the terminology as they were figuring it all out. That is no mean feat. But with PEP 342 (generator-based coroutines) and especially PEP 380 (yield from) there *is* an alternative, and while Twisted has added APIs to support generators, it hasn't started to deprecate its other APIs, and its terminology becomes hard to follow for people (like me, frankly) who first learned this stuff through PEP 342. -- --Guido van Rossum (python.org/~guido)
On Sun, 12 Sep 2010 08:49:56 -0700 Guido van Rossum <guido@python.org> wrote:
Having a common abstraction (Future or Deferred) allows for scheduling-agnostic libraries which consume and/or produce these abstractions (*). I'm not sure it is desireable to mix scheduling models in a single process (let alone a single thread), though. (*) Of course, the abstraction is somehow leaky since being called from different threads, depending on the scheduling model, could have adverse consequences
It implies that it's mostly useful for simple callbacks (which would e.g. print out a success report, or set an Event to wake up another thread). The Twisted model allows the major part of processing to occur in the callbacks themselves, in which case proper error handling and propagation is mandatory. Regards Antoine.
On Sun, Sep 12, 2010 at 9:17 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
IIRC even Twisted supports putting stuff in a thread if you really need it. And have you looked at Go's Goroutines? They are a hybrid -- they don't map 1:1 to OS threads, but they aren't pure coroutines either, so that if a goroutine blocks on I/O the others will still make progress.
Yeah, this is always a problem with pure async frameworks -- if one callback or coroutine blocks by mistake, the whole world is blocked. (So Goroutines attempt to fix this; I have no idea how successful they are.)
A generator-based coroutines approach can do this too (just put the work between the yields in the generator) and has all the proper exception-propagation stuff built in since PEP 342 (PEP 380 will just make it easier). And a Futures-based approach can do it too -- it's not described in PEP 3148, but you can easily design an API for wrappable Futures. -- --Guido van Rossum (python.org/~guido)
Thanks for the ping about this (I don't think I subscribe to python-ideas, so someone may have to moderate my post in). Sorry for the delay in responding, but I've been kinda busy and cooking up these examples took a bit of thinking. And thanks, James, for restarting this discussion. I obviously find it interesting :). I'm going to mix in some other stuff I found on the web archives, since it's easiest just to reply in one message. I'm sorry that this response is a bit sprawling and doesn't have a single clear narrative, the thread thus far didn't seem to lend it to one. For those of you who don't want to read my usual novel-length post, you can probably stop shortly after the end of the first block of code examples. On Sep 11, 2010, at 10:26 PM, Guido van Rossum wrote:
That blog post was written to demonstrate why programs using generators are "... far easier to read and write ..." than ones using Deferreds, so it stands to reason it would choose an example where that helps :). When you want to write systems that manage varying levels of parallelism within a single computation, generators can start to get pretty hairy and the "normal" Deferred way of doing things looks more straightforward. Thinking in terms of asynchronicity is tricky, and generators can be a useful tool for promoting that understanding, but they only make it superficially easier. For example:
If you're writing an application whose parallelism calls for an asynchronous approach, after all, you presumably don't want to be standing around waiting for each network round trip to complete. How do you re-write this so that there are always at least N outstanding do_something_async calls running in parallel? You can sorta do it like this:
but that will always block on one particular do_something_async, when you really want to say "let me know when any outstanding call is complete". So I could handwave about 'yield any_completed(outstanding)'...
but that just begs the question of how you implement any_completed(), and I can't think of a way to do that with generators, without getting into the specifics of some Deferred-or-Future-like asynchronous result object. You could implement such a function with such primitives, and here's what it looks like with Deferreds:
Here's how you do the top-level task in Twisted, without generators, in the truly-parallel fashion (keep in mind this combines the functionality of 'any_completed' and 'parallel', so it's a bit shorter):
Some informal benchmarking has shown this method to be considerably faster (on the order of 1/2 to 1/3 as much CPU time) than at least our own inlineCallbacks generator-scheduling method. Take this with the usual fist-sized grain of salt that you do any 'informal' benchmarks, but the difference is significant enough that I do try to refactor into this style in my own code, and I have seen performance benefits from doing this on more specific benchmarks. This is all untested, and that's far too many lines of code to expect to work without testing, but hopefully it gives a pretty good impression of the differences in flavor between the different styles.
Yeah, please do explain why Twisted has so much machinery to handle exceptions?
There are a lot of different implied questions here, so I'll answer a few of those. Why does twisted.python.failure exist? The answer to that is that we wanted an object that represented an exception as raised at a particular point, associated with a particular stack, that could live on without necessarily capturing all the state in that stack. If you're going to report failures asynchronously, you don't necessarily want to hold a reference to every single thing in a potentially giant stack while you're waiting to send it to some network endpoint. Also, in 1.5.2 we had no way of chaining exceptions, and this code is that old. Finally, even if you can chain exceptions, it's a serious performance hit to have to re-raise and re-catch the same exception 4 or 5 times in order to translate it or handle it at many different layers of the stack, so a Failure is intended to encapsulate that state such that it can just be returned, in performance-sensitive areas. (This is sort of a weak point though, since the performance of Failure itself is so terrible, for unrelated reasons.) Why is twisted.python.failure such a god damned mess? The answer to that is ... uh, sorry. Yes, it is. We should clean it up. It was written a long time ago and the equivalent module now could be _much_ shorter, simpler, and less of a performance problem. It just never seems to be the highest priority. Maybe after we're done porting to py3 :). My one defense here is that still a slight improvement over the stdlib 'traceback' module ;-). Why do Deferreds have an errback chain rather than just handing you an exception object in the callback chain? Basically, this is for the same reason that Python has exceptions instead of just making you check return codes. We wanted it to be easy to say:
and know that the argument to 'ok' would always be what getPage promised (you don't need to typecheck it for exception-ness) and the default error behavior would be to simply bail out with a traceback, not to barrel through your success-path code wreaking havoc.
ISTM that the main difference is that add_done_callback() isn't meant for callbacks that return a value.
add_done_callback works fine with callbacks that return a value. If it didn't, I'd be concerned, because then it would have the barrel-through-the-success-path flaw. But, I assume the idiomatic asynchronous-code-using-Futures would look like this:
This is one extra line of code as compared to the Twisted version, and chaining involves a bit more gymnastics (somehow creating more futures to return further up the stack, I guess, I haven't thought about it too hard), but it does allow you to handle exceptions with a simple 'except:', rather than calling some exception-handling methods, so I can see why some people would prefer it.
Maybe it's possible to write a little framework that lets you create Futures using either threads, processes (both supported by PEP 3148) or generators. But I haven't tried it. And maybe the need to use 'yield' for everything that may block when using generators, but not when using threads or processes, will make this awkward.
You've already addressed the main point that I really wanted to mention here, but I'd like to emphasize it. Blocking and not-blocking are fundamentally different programming styles, and if you sometimes allow blocking on asynchronous results, that means you are effectively always programming in the blocking-and-threaded style and not getting much benefit from the code which does choose to be politely non-blocking. I was somewhat pleased with the changes made to the Futures PEP because you could use them as an asynchronous result, and have things that implemented the Future API but raised an exception if you tried to wait on them. That would at least allow some layer of stdlib compatibility. If you are disciplined and careful, this would let you write async code which used a common interoperability mechanism, and if you weren't careful, it would blow up when you tried to use it the wrong way. But - and I am guessing that this is the main thrust of this discussion - I do think that having Deferred in the standard library would be much, much better if we can do that.
So maybe we'll be stuck with at least two Future-like APIs: PEP 3148 and something else, generator-based.
Having something "generator-based" is, in my opinion, an abstraction inversion. The things which you are yielding from these generators are asynchronous results. There should be a specific type for asynchronous results which can be easily interacted with. Generators are syntactic sugar for doing that interaction in a way which doesn't involve defining tons of little functions. This is useful, and it makes the concept more accessible, so I don't say "just" syntactic sugar: but nevertheless, the generators need to be 'yield'ing something, and the type of thing that they're yielding is a Deferred-or-something-like-it. I don't think that this is really two 'Future-like APIs'. At least, they're not redundant, any more than having both socket.makefile() and socket.recv() is redundant. If Future had a deferred() method rather than an add_done_callback() method, then it would always be very clear whether you had a synchronous-but-possibly-not-ready or a purely-asynchronous result. Although it would be equally easy to just have a function that turned a Future into a Deferred by calling add_done_callback(). You can go from any arbitrary Future to a full-featured Deferred, but not the other way around.
Or maybe PEP 3152.
I don't like PEP 3152 aesthetically on many levels, but I can't deny that it would do the job. 'cocall', though, really? It would be nice if it read like an actual word, i.e. "yield to" or "invoke" or even just "call" or something. In another message, where Guido is replying to Antoine:
I think the main reason, though, that people find Deferreds inconvenient is that they force you to think in terms of asynchronicity (...)
Actually I think the main reason is historic: Twisted introduced callback-based asynchronous (thread-less) programming when there was no alternative in Python, and they invented both the mechanisms and the terminology as they were figuring it all out. That is no mean feat. But with PEP 342 (generator-based coroutines) and especially PEP 380 (yield from) there *is* an alternative, and while Twisted has added APIs to support generators, it hasn't started to deprecate its other APIs, and its terminology becomes hard to follow for people (like me, frankly) who first learned this stuff through PEP 342.
I really have to go with Antoine on this one: people were confused about Deferreds long before PEP 342 came along :). Given that Javascript environments have mostly adopted the Twisted terminology (oddly, Node.js doesn't, but Dojo and MochiKit both have pretty literal-minded Deferred translations), there are plenty of people who are familiar with the terminology but still get confused. See the beginning of the message for why we're not deprecating our own APIs. Once again, sorry for not compressing this down further! If you got this far, you win a prize :).
participants (5)
-
Antoine Pitrou -
Glyph Lefkowitz -
Guido van Rossum -
Jesse Noller -
Nick Coghlan