On Thu, Oct 11, 2012 at 3:42 PM, Devin Jeanpierre
On Thu, Oct 11, 2012 at 5:18 PM, Guido van Rossum
wrote: [...] Twisted Deferred is pretty arcane, and I would much rather not use it as the basis of a forward-looking design. I'd much rather see what we can mooch off PEP 3148 (Futures).
Could you be more specific? I've never heard Deferreds in particular called "arcane". They're very popular in e.g. the JS world,
Really? Twisted is used in the JS world? Or do you just mean the pervasiveness of callback style async programming? That's one of the things I am desperately trying to keep out of Python, I find that style unreadable and unmanageable (whenever I click on a button in a website and nothing happens I know someone has a bug in their callbacks). I understand you feel different; but I feel the general sentiment is that callback-based async programming is even harder than multi-threaded programming (and nobody is claiming that threads are easy :-).
and possibly elsewhere. Moreover, they're extremely similar to futures, so if one is arcane so is the other.
I love Futures, they represent a nice simple programming model. But I especially love that you can write async code using Futures and yield-based coroutines (what you call inlineCallbacks) and never have to write an explicit callback function. Ever.
Maybe if you could elaborate on features of their designs that are better/worse?
As far as I know, they mostly differ in that:
- Callbacks are added in a pipeline, rather than "in parallel" - Deferreds pass in values along the pipeline, rather than self (and have a separate pipeline for error values).
These two combined are indeed what mostly feels arcane to me.
Neither is clearly better or more obvious than the other. If anything I generally find deferred composition more useful than deferred tee-ing, so I feel like composition is the correct base operator, but you could pick another.
If you're writing long complicated chains of callbacks that benefit from these features, IMO you are already doing it wrong. I understand that this is a matter of style where I won't be able to convince you. But style is important to me, so let's agree to disagree.
Either way, each is implementable in terms of the other (ish?). The pipeline approach is particularly nice for the errback pipeline, because it allows chained exception (Failure) handling on the deferred to be very simple. The larger issue is that futures don't make chaining easy at all, even if it is theoretically possible.
But as soon as you switch from callbacks to yield-based coroutines the chaining becomes natural, error handling is just a matter of try/except statements (or not if you want the error to bubble up) and (IMO) the code becomes much more readable.
For example, look at the following Twisted code: http://bpaste.net/show/RfEwoaflO0qY76N8NjHx/ , and imagine how that might generalize to more realistic error handling scenarios.
Looks fine to me. I have a lot of code like that in NDB and it works great. (Note that NDB's Futures are not the same as PEP 3148 Futures, although they have some things in common; in particular NDB Futures are not tied to threads.)
The equivalent Futures code would involve creating one Future per callback in the pipeline and manually hooking them up with a special callback that passes values to the next future. And if we add that to the futures API, the API will almost certainly be somewhat similar to what Twisted has with deferreds and chaining and such. So then, equally arcane.
The *implementation* of this stuff in NDB is certainly hairy; I already posted the link to the code: http://code.google.com/p/appengine-ndb-experiment/source/browse/ndb/tasklets... However, this is internal code and doesn't affect the Future API at all.
To my mind, it is Futures that need to mooch off of Deferreds, not the other way around. Twisted's Deferreds have a lot of history with making asynchronous computation pleasant, and Futures are missing a lot of good tools.
I am totally open to learning from Twisted's experience. I hope that you are willing to share even the end result might not look like Twisted at all -- after all in Python 3.3 we have "yield from" and return from a generator and many years of experience with different styles of async APIs. In addition to Twisted, there's Tornado and Monocle, and then there's the whole greenlets/gevent and Stackless/microthreads community that we can't completely ignore. I believe somewhere is an ideal async architecture, and I hope you can help us discover it. (For example, I am very interested in Twisted's experiences writing real-world performant, robust reactors.) -- --Guido van Rossum (python.org/~guido)