[Python-ideas] The async API of the future: Twisted and Deferreds

[Ben Darnell]

On Sat, Oct 13, 2012 at 10:49 AM, Laurens Van Houtven <_ at lvh.cc> wrote:
> Interesting. That's certainly a nice API, but that then again (read_until)
> sounds like something I'd implement using dataReceived... You know,
> read_until clears the buffer, logs the requested callback. data_received
> adds something to the buffer, and checks if it triggered the (one of the?)
> registered callbacks.

Right, that's how IOStream is implemented internally.  The
transport/protocol split works a little differently in Tornado:
IOStream is implemented something like a Protocol subclass, but we
consider it a part of the transport layer.  The "protocols" are
arbitrary classes that don't share any particular interface, but
instead just call methods on the IOStream.

-Ben

>
> Of course, I may just be rusted in my ways and trying to implement
> everything in terms of things I know (then again, that might be just what's
> needed when you're trying to make a useful general API).
>
> I guess it's time for me to go deep-diving into Tornado :)
>
>
> On Sat, Oct 13, 2012 at 7:27 PM, Ben Darnell <ben at bendarnell.com> wrote:
>>
>> On Sat, Oct 13, 2012 at 10:18 AM, Laurens Van Houtven <_ at lvh.cc> wrote:
>> > What calls on_headers in this example? Coming from twisted, that seems
>> > like
>> > dataReceived's responsibility, but given your introductory paragraph
>> > that's
>> > not actually what goes on here?
>>
>> The IOStream does, after send_request calls
>> stream.read_until("\r\n\r\n", on_headers).  Inside IOStream, there is
>> a _handle_read method that is registered with the IOLoop and fills up
>> a buffer.  When the read condition is satisfied the IOStream calls
>> back into application code.
>>
>> -Ben
>>
>> >
>> >
>> > On Sat, Oct 13, 2012 at 7:07 PM, Ben Darnell <ben at bendarnell.com> wrote:
>> >>
>> >> On Fri, Oct 12, 2012 at 11:14 PM, Antoine Pitrou <solipsis at pitrou.net>
>> >> wrote:
>> >> > On Fri, 12 Oct 2012 15:11:54 -0700
>> >> > Guido van Rossum <guido at python.org> wrote:
>> >> >>
>> >> >> > 2. Method dispatch callbacks:
>> >> >> >
>> >> >> >     Similar to the above, the reactor or somebody has a handle on
>> >> >> > your
>> >> >> > object, and calls methods that you've defined when events happen
>> >> >> >     e.g. IProtocol's dataReceived method
>> >> >>
>> >> >> While I'm sure it's expedient and captures certain common patterns
>> >> >> well, I like this the least of all -- calling fixed methods on an
>> >> >> object sounds like a step back; it smells of the old Java way
>> >> >> (before
>> >> >> it had some equivalent of anonymous functions), and of asyncore,
>> >> >> which
>> >> >> (nearly) everybody agrees is kind of bad due to its insistence that
>> >> >> you subclass its classes. (Notice how subclassing as the prevalent
>> >> >> approach to structuring your code has gotten into a lot of discredit
>> >> >> since 1996.)
>> >> >
>> >> > But how would you write a dataReceived equivalent then? Would you
>> >> > have
>> >> > a "task" looping on a read() call, e.g.
>> >> >
>> >> > @task
>> >> > def my_protocol_main_loop(conn):
>> >> >     while <some_condition>:
>> >> >         try:
>> >> >             data = yield conn.read(1024)
>> >> >         except ConnectionError:
>> >> >             conn.close()
>> >> >             break
>> >> >
>> >> > I'm not sure I understand the problem with subclassing. It works fine
>> >> > in Twisted. Even in Python 3 we don't shy away from subclassing, for
>> >> > example the IO stack is based on subclassing RawIOBase,
>> >> > BufferedIOBase,
>> >> > etc.
>> >>
>> >> Subclassing per se isn't a problem, but requiring a single
>> >> dataReceived method per class can be awkward.  Many protocols are
>> >> effectively state machines, and modeling each state as a function can
>> >> be cleaner than a big if/switch block in dataReceived.  For example,
>> >> here's a simplistic HTTP client using tornado's IOStream:
>> >>
>> >>        from tornado import ioloop
>> >>         from tornado import iostream
>> >>         import socket
>> >>
>> >>         def send_request():
>> >>             stream.write("GET / HTTP/1.0\r\nHost:
>> >> friendfeed.com\r\n\r\n")
>> >>             stream.read_until("\r\n\r\n", on_headers)
>> >>
>> >>         def on_headers(data):
>> >>             headers = {}
>> >>             for line in data.split("\r\n"):
>> >>                parts = line.split(":")
>> >>                if len(parts) == 2:
>> >>                    headers[parts[0].strip()] = parts[1].strip()
>> >>             stream.read_bytes(int(headers["Content-Length"]), on_body)
>> >>
>> >>         def on_body(data):
>> >>             print data
>> >>             stream.close()
>> >>             ioloop.IOLoop.instance().stop()
>> >>
>> >>         s = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
>> >>         stream = iostream.IOStream(s)
>> >>         stream.connect(("friendfeed.com", 80), send_request)
>> >>         ioloop.IOLoop.instance().start()
>> >>
>> >>
>> >> Classes allow and encourage broader interfaces, which are sometimes a
>> >> good thing, but interact poorly with coroutines.  Both twisted and
>> >> tornado use separate callbacks for incoming data and for the
>> >> connection being closed, but for coroutines it's probably better to
>> >> just treat a closed connection as an error on the read.  Futures (and
>> >> yield from) give us a nice way to do that.
>> >>
>> >> -Ben
>> >> _______________________________________________
>> >> Python-ideas mailing list
>> >> Python-ideas at python.org
>> >> http://mail.python.org/mailman/listinfo/python-ideas
>> >
>> >
>> >
>> >
>> > --
>> > cheers
>> > lvh
>> >
>
>
>
>
> --
> cheers
> lvh
>