[Python-ideas] The async API of the future: Twisted and Deferreds

[Guido van Rossum]

[Quick, I know I'm way behind, especially on this thread; more tomorrow.]

On Fri, Oct 12, 2012 at 11:14 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>
> On Fri, 12 Oct 2012 15:11:54 -0700
> Guido van Rossum <guido at python.org> wrote:
> >
> > > 2. Method dispatch callbacks:
> > >
> > >     Similar to the above, the reactor or somebody has a handle on your
> > > object, and calls methods that you've defined when events happen
> > >     e.g. IProtocol's dataReceived method
> >
> > While I'm sure it's expedient and captures certain common patterns
> > well, I like this the least of all -- calling fixed methods on an
> > object sounds like a step back; it smells of the old Java way (before
> > it had some equivalent of anonymous functions), and of asyncore, which
> > (nearly) everybody agrees is kind of bad due to its insistence that
> > you subclass its classes. (Notice how subclassing as the prevalent
> > approach to structuring your code has gotten into a lot of discredit
> > since 1996.)
>
> But how would you write a dataReceived equivalent then? Would you have
> a "task" looping on a read() call, e.g.
>
> @task
> def my_protocol_main_loop(conn):
>     while <some_condition>:
>         try:
>             data = yield conn.read(1024)
>         except ConnectionError:
>             conn.close()
>             break

No, I would use plain callbacks. There would be some kind of IOObject
class defined by the stdlib that wraps a socket (it would make it
non-blocking, and possibly to other things), and the user would make a
registration call to the event loop giving it the IOOjbect and the
user's callback function plus *args and **kwds; the event loop would
call callback(*args, **kwds) each time the IOObject became readable.
(Oh, and there would be separate registration (and unregistration)
functions for reading and writing.)

Apparently my rants about callbacks have made people assume that I
don't want to see them anywhere. In fact I am comfortable with
callbacks for a number of situations -- I just think we have several
other tools in our toolbox that are way underused, whereas callbacks
are way overused, in part because the alternative tools are relatively
new.

This way the user could switch to a different callback when a
different phase of the protocol is reached. I realize there are other
shapes this API could take. But I really don't want the user to have
to subclass IOObject.

> I'm not sure I understand the problem with subclassing. It works fine
> in Twisted. Even in Python 3 we don't shy away from subclassing, for
> example the IO stack is based on subclassing RawIOBase, BufferedIOBase,
> etc.


I'm fine with using subclassing for the internal structure of a
library. (The IOObject I am postulating would almost certainly have a
bunch of subclasses used for different types of sockets, IOCP, SSL,
etc.) The thing that I've soured upon (and many others too) is to tell
users "and to use this fine feature, just subclass this handy base
class and override or extend the following three methods". Because in
practice (certainly in Python, where the compiler doesn't enforce
privacy) users always start overriding other methods, or using
internal state, or add state that clashes with the base class's state,
or forget to call mandatory super calls, or make incorrect assumptions
about thread-safety, or whatever else they can do to screw things up.
And duck typing isn't ideal either for this situation.

--
--Guido van Rossum (python.org/~guido)