[Python-ideas] The async API of the future: Twisted and Deferreds

[Ben Darnell]

On Fri, Oct 12, 2012 at 11:14 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Fri, 12 Oct 2012 15:11:54 -0700
> Guido van Rossum <guido at python.org> wrote:
>>
>> > 2. Method dispatch callbacks:
>> >
>> >     Similar to the above, the reactor or somebody has a handle on your
>> > object, and calls methods that you've defined when events happen
>> >     e.g. IProtocol's dataReceived method
>>
>> While I'm sure it's expedient and captures certain common patterns
>> well, I like this the least of all -- calling fixed methods on an
>> object sounds like a step back; it smells of the old Java way (before
>> it had some equivalent of anonymous functions), and of asyncore, which
>> (nearly) everybody agrees is kind of bad due to its insistence that
>> you subclass its classes. (Notice how subclassing as the prevalent
>> approach to structuring your code has gotten into a lot of discredit
>> since 1996.)
>
> But how would you write a dataReceived equivalent then? Would you have
> a "task" looping on a read() call, e.g.
>
> @task
> def my_protocol_main_loop(conn):
>     while <some_condition>:
>         try:
>             data = yield conn.read(1024)
>         except ConnectionError:
>             conn.close()
>             break
>
> I'm not sure I understand the problem with subclassing. It works fine
> in Twisted. Even in Python 3 we don't shy away from subclassing, for
> example the IO stack is based on subclassing RawIOBase, BufferedIOBase,
> etc.

Subclassing per se isn't a problem, but requiring a single
dataReceived method per class can be awkward.  Many protocols are
effectively state machines, and modeling each state as a function can
be cleaner than a big if/switch block in dataReceived.  For example,
here's a simplistic HTTP client using tornado's IOStream:

       from tornado import ioloop
        from tornado import iostream
        import socket

        def send_request():
            stream.write("GET / HTTP/1.0\r\nHost: friendfeed.com\r\n\r\n")
            stream.read_until("\r\n\r\n", on_headers)

        def on_headers(data):
            headers = {}
            for line in data.split("\r\n"):
               parts = line.split(":")
               if len(parts) == 2:
                   headers[parts[0].strip()] = parts[1].strip()
            stream.read_bytes(int(headers["Content-Length"]), on_body)

        def on_body(data):
            print data
            stream.close()
            ioloop.IOLoop.instance().stop()

        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
        stream = iostream.IOStream(s)
        stream.connect(("friendfeed.com", 80), send_request)
        ioloop.IOLoop.instance().start()


Classes allow and encourage broader interfaces, which are sometimes a
good thing, but interact poorly with coroutines.  Both twisted and
tornado use separate callbacks for incoming data and for the
connection being closed, but for coroutines it's probably better to
just treat a closed connection as an error on the read.  Futures (and
yield from) give us a nice way to do that.

-Ben