[Python-ideas] The async API of the future: Twisted and Deferreds

[Ben Darnell]

On Sat, Oct 13, 2012 at 12:13 PM, Laurens Van Houtven <_ at lvh.cc> wrote:
> I quite like IOStream's interface, actually. If that's part of the transport
> layer, how do you prevent from having duplicating its behavior (read_until
> etc)? If there's just another separate object that would be the ITransport
> in twisted, I think the difference is purely one of labeling.

So far we haven't actually needed much flexibility in the transport
layer - most of the functionality is in the BaseIOStream class, and
then there are subclasses IOStream (regular sockets), SSLIOStream, and
PipeIOStream that actually call recv(), read(), connect(), etc.  We
might need a little refactoring if we introduce dramatically different
types of transports, but the plan is that we'd represent transports as
classes in the IOStream hierarchy.

-Ben

>
>
> On Sat, Oct 13, 2012 at 8:54 PM, Ben Darnell <ben at bendarnell.com> wrote:
>>
>> On Sat, Oct 13, 2012 at 10:49 AM, Laurens Van Houtven <_ at lvh.cc> wrote:
>> > Interesting. That's certainly a nice API, but that then again
>> > (read_until)
>> > sounds like something I'd implement using dataReceived... You know,
>> > read_until clears the buffer, logs the requested callback. data_received
>> > adds something to the buffer, and checks if it triggered the (one of
>> > the?)
>> > registered callbacks.
>>
>> Right, that's how IOStream is implemented internally.  The
>> transport/protocol split works a little differently in Tornado:
>> IOStream is implemented something like a Protocol subclass, but we
>> consider it a part of the transport layer.  The "protocols" are
>> arbitrary classes that don't share any particular interface, but
>> instead just call methods on the IOStream.
>>
>> -Ben
>>
>> >
>> > Of course, I may just be rusted in my ways and trying to implement
>> > everything in terms of things I know (then again, that might be just
>> > what's
>> > needed when you're trying to make a useful general API).
>> >
>> > I guess it's time for me to go deep-diving into Tornado :)
>> >
>> >
>> > On Sat, Oct 13, 2012 at 7:27 PM, Ben Darnell <ben at bendarnell.com> wrote:
>> >>
>> >> On Sat, Oct 13, 2012 at 10:18 AM, Laurens Van Houtven <_ at lvh.cc> wrote:
>> >> > What calls on_headers in this example? Coming from twisted, that
>> >> > seems
>> >> > like
>> >> > dataReceived's responsibility, but given your introductory paragraph
>> >> > that's
>> >> > not actually what goes on here?
>> >>
>> >> The IOStream does, after send_request calls
>> >> stream.read_until("\r\n\r\n", on_headers).  Inside IOStream, there is
>> >> a _handle_read method that is registered with the IOLoop and fills up
>> >> a buffer.  When the read condition is satisfied the IOStream calls
>> >> back into application code.
>> >>
>> >> -Ben
>> >>
>> >> >
>> >> >
>> >> > On Sat, Oct 13, 2012 at 7:07 PM, Ben Darnell <ben at bendarnell.com>
>> >> > wrote:
>> >> >>
>> >> >> On Fri, Oct 12, 2012 at 11:14 PM, Antoine Pitrou
>> >> >> <solipsis at pitrou.net>
>> >> >> wrote:
>> >> >> > On Fri, 12 Oct 2012 15:11:54 -0700
>> >> >> > Guido van Rossum <guido at python.org> wrote:
>> >> >> >>
>> >> >> >> > 2. Method dispatch callbacks:
>> >> >> >> >
>> >> >> >> >     Similar to the above, the reactor or somebody has a handle
>> >> >> >> > on
>> >> >> >> > your
>> >> >> >> > object, and calls methods that you've defined when events
>> >> >> >> > happen
>> >> >> >> >     e.g. IProtocol's dataReceived method
>> >> >> >>
>> >> >> >> While I'm sure it's expedient and captures certain common
>> >> >> >> patterns
>> >> >> >> well, I like this the least of all -- calling fixed methods on an
>> >> >> >> object sounds like a step back; it smells of the old Java way
>> >> >> >> (before
>> >> >> >> it had some equivalent of anonymous functions), and of asyncore,
>> >> >> >> which
>> >> >> >> (nearly) everybody agrees is kind of bad due to its insistence
>> >> >> >> that
>> >> >> >> you subclass its classes. (Notice how subclassing as the
>> >> >> >> prevalent
>> >> >> >> approach to structuring your code has gotten into a lot of
>> >> >> >> discredit
>> >> >> >> since 1996.)
>> >> >> >
>> >> >> > But how would you write a dataReceived equivalent then? Would you
>> >> >> > have
>> >> >> > a "task" looping on a read() call, e.g.
>> >> >> >
>> >> >> > @task
>> >> >> > def my_protocol_main_loop(conn):
>> >> >> >     while <some_condition>:
>> >> >> >         try:
>> >> >> >             data = yield conn.read(1024)
>> >> >> >         except ConnectionError:
>> >> >> >             conn.close()
>> >> >> >             break
>> >> >> >
>> >> >> > I'm not sure I understand the problem with subclassing. It works
>> >> >> > fine
>> >> >> > in Twisted. Even in Python 3 we don't shy away from subclassing,
>> >> >> > for
>> >> >> > example the IO stack is based on subclassing RawIOBase,
>> >> >> > BufferedIOBase,
>> >> >> > etc.
>> >> >>
>> >> >> Subclassing per se isn't a problem, but requiring a single
>> >> >> dataReceived method per class can be awkward.  Many protocols are
>> >> >> effectively state machines, and modeling each state as a function
>> >> >> can
>> >> >> be cleaner than a big if/switch block in dataReceived.  For example,
>> >> >> here's a simplistic HTTP client using tornado's IOStream:
>> >> >>
>> >> >>        from tornado import ioloop
>> >> >>         from tornado import iostream
>> >> >>         import socket
>> >> >>
>> >> >>         def send_request():
>> >> >>             stream.write("GET / HTTP/1.0\r\nHost:
>> >> >> friendfeed.com\r\n\r\n")
>> >> >>             stream.read_until("\r\n\r\n", on_headers)
>> >> >>
>> >> >>         def on_headers(data):
>> >> >>             headers = {}
>> >> >>             for line in data.split("\r\n"):
>> >> >>                parts = line.split(":")
>> >> >>                if len(parts) == 2:
>> >> >>                    headers[parts[0].strip()] = parts[1].strip()
>> >> >>             stream.read_bytes(int(headers["Content-Length"]),
>> >> >> on_body)
>> >> >>
>> >> >>         def on_body(data):
>> >> >>             print data
>> >> >>             stream.close()
>> >> >>             ioloop.IOLoop.instance().stop()
>> >> >>
>> >> >>         s = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
>> >> >>         stream = iostream.IOStream(s)
>> >> >>         stream.connect(("friendfeed.com", 80), send_request)
>> >> >>         ioloop.IOLoop.instance().start()
>> >> >>
>> >> >>
>> >> >> Classes allow and encourage broader interfaces, which are sometimes
>> >> >> a
>> >> >> good thing, but interact poorly with coroutines.  Both twisted and
>> >> >> tornado use separate callbacks for incoming data and for the
>> >> >> connection being closed, but for coroutines it's probably better to
>> >> >> just treat a closed connection as an error on the read.  Futures
>> >> >> (and
>> >> >> yield from) give us a nice way to do that.
>> >> >>
>> >> >> -Ben
>> >> >> _______________________________________________
>> >> >> Python-ideas mailing list
>> >> >> Python-ideas at python.org
>> >> >> http://mail.python.org/mailman/listinfo/python-ideas
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > cheers
>> >> > lvh
>> >> >
>> >
>> >
>> >
>> >
>> > --
>> > cheers
>> > lvh
>> >
>
>
>
>
> --
> cheers
> lvh
>