First of all, sorry for not snipping the reply I made previously. Noticed that only after I sent it :( On Thu, Oct 11, 2012 at 7:37 PM, Guido van Rossum <guido@python.org> wrote:
On Thu, Oct 11, 2012 at 3:42 PM, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
Could you be more specific? I've never heard Deferreds in particular called "arcane". They're very popular in e.g. the JS world,
Really? Twisted is used in the JS world? Or do you just mean the pervasiveness of callback style async programming?
Ah, I mean Deferreds. I attended a talk earlier this year all about deferreds in JS, and not a single reference to Python or Twisted was made! These are the examples I remember mentioned in the talk: - http://api.jquery.com/category/deferred-object/ (not very twistedish at all, ill-liked by the speaker) - http://mochi.github.com/mochikit/doc/html/MochiKit/Async.html (maybe not a good example, mochikit tries to be "python in JS") - http://dojotoolkit.org/reference-guide/1.8/dojo/Deferred.html - https://github.com/kriskowal/q (also includes an explanation of why the author likes deferreds) There were a few more that the speaker mentioned, but didn't cover. One of his points was that the various systems of deferreds are subtly different, some very badly so, and that it was a mess, but that deferreds were still awesome. JS is a language where async programming is mainstream, so lots of people try to make it easier, and they all do it slightly differently.
That's one of the things I am desperately trying to keep out of Python, I find that style unreadable and unmanageable (whenever I click on a button in a website and nothing happens I know someone has a bug in their callbacks). I understand you feel different; but I feel the general sentiment is that callback-based async programming is even harder than multi-threaded programming (and nobody is claiming that threads are easy :-).
:S There are (at least?) four different styles of asynchronous computation used in Twisted, and you seem to be confused as to which ones I'm talking about. 1. Explicit callbacks: For example, reactor.callLater(t, lambda: print("woo hoo")) 2. Method dispatch callbacks: Similar to the above, the reactor or somebody has a handle on your object, and calls methods that you've defined when events happen e.g. IProtocol's dataReceived method 3. Deferred callbacks: When you ask for something to be done, it's set up, and you get an object back, which you can add a pipeline of callbacks to that will be called whenever whatever happens e.g. twisted.internet.threads.deferToThread(print, "x").addCallback(print, "x was printed in some other thread!") 4. Generator coroutines These are a syntactic wrapper around deferreds. If you yield a deferred, you will be sent the result if the deferred succeeds, or an exception if the deferred fails. e.g. examples from previous message I don't see a reason for the first to exist at all, the second one is kind of nice in some circumstances (see below), but perhaps overused. I feel like you're railing on the first and second when I'm talking about the third and fourth. I could be wrong.
and possibly elsewhere. Moreover, they're extremely similar to futures, so if one is arcane so is the other.
I love Futures, they represent a nice simple programming model. But I especially love that you can write async code using Futures and yield-based coroutines (what you call inlineCallbacks) and never have to write an explicit callback function. Ever.
The reason explicit non-deferred callbacks are involved in Twisted is because of situations in which deferreds are not present, because of past history in Twisted. It is not at all a limitation of deferreds or something futures are better at, best as I'm aware. (In case that's what you're getting at.) Anyway, one big issue is that generator coroutines can't really effectively replace callbacks everywhere. Consider the GUI button example you gave. How do you write that as a coroutine? I can see it being written like this: def mycoroutine(gui): while True: clickevent = yield gui.mybutton1.on_click() # handle clickevent But that's probably worse than using callbacks.
Neither is clearly better or more obvious than the other. If anything I generally find deferred composition more useful than deferred tee-ing, so I feel like composition is the correct base operator, but you could pick another.
If you're writing long complicated chains of callbacks that benefit from these features, IMO you are already doing it wrong. I understand that this is a matter of style where I won't be able to convince you. But style is important to me, so let's agree to disagree.
This is more than a matter of style, so at least for now I'd like to hold off on calling it even. In my day to day silly, synchronous, python code, I do lots of synchronous requests. For example, it's not unreasonable for me to want to load two different files from disk, or make several database interactions, etc. If I want to make this asynchronous, I have to find a way to execute multiple things that could hypothetically block, at the same time. If I can't do that easily, then the asynchronous solution has failed, because its entire purpose is to do everything that I do synchronously, except without blocking the main thread. Here's an example with lots of synchronous requests in Django: def view_paste(request, filekey): try: fileinfo= Pastes.objects.get(key=filekey) except DoesNotExist: t = loader.get_template('pastebin/error.html') return HttpResponse(t.render(Context(dict(error='File does not exist')))) f = open(fileinfo.filename) fcontents = f.read() t = loader.get_template('pastebin/paste.html') return HttpResponse(t.render(Context(dict(file=fcontents)))) How many blocking requests are there? Lots. This is, in a word, a long, complicated chain of synchronous requests. This is also very similar to what actual django code might look like in some circumstances. Even if we might think this is unreasonable, some subset of alteration of this is reasonable. Certainly we should be able to, say, load multiple (!) objects from the database, and open the template (possibly from disk), all potentially-blocking operations. This is inherently a long, complicated chain of requests, whether we implement it asynchronously or synchronously, or use Deferreds or Futures, or write it in Java or Python. Some parts can be done at any time before the end (loader.get_template(...)), some need to be done in a certain order, and there's branching depending on what happens in different cases. In order to even write this code _at all_, we need a way to chain these IO actions together. If we can't chain them together, we can't produce that final synthesis of results at the end. We _need_ a pipeline or something computationally equivalent or more powerful. Results from past "deferred computations" need to be passed forward into future "deferred computations", in order to implement this at all. This is not a style issue, this is an issue of needing to be able to solve problems that involve more than one computation where the results of every computation matters somewhere. It's just that in this case, some of the computations are computed asynchronously.
I am totally open to learning from Twisted's experience. I hope that you are willing to share even the end result might not look like Twisted at all -- after all in Python 3.3 we have "yield from" and return from a generator and many years of experience with different styles of async APIs. In addition to Twisted, there's Tornado and Monocle, and then there's the whole greenlets/gevent and Stackless/microthreads community that we can't completely ignore. I believe somewhere is an ideal async architecture, and I hope you can help us discover it.
(For example, I am very interested in Twisted's experiences writing real-world performant, robust reactors.)
For that stuff, you'd have to speak to the main authors of Twisted. I'm just a twisted user. :( In the end it really doesn't matter what API you go with. The Twisted people will wrap it up so that they are compatible, as far as that is possible. I hope I haven't detracted too much from the main thrust of the surrounding discussion. Futures/deferreds are a pretty big tangent, so sorry. I justified it to myself by figuring that it'd probably come up anyway, somehow, since these are useful abstractions for asynchronous programming. -- Devin