[Python-ideas] PEP 525: Asynchronous Generators

[Cory Benfield]

> On 11 Aug 2016, at 13:00, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Twisted callbacks are still red functions - you call them via the event loop rather than directly, and only event loop aware functions know how to make that request of the event loop.
> 
> async/await just makes the function colour locally visible with a clear syntax, rather than needing to be inferred from behavioural details of the function implementation.
> 

That's not really true, though its trueness depends on what you mean when you say “Twisted callbacks”. Like asyncio (which adopted this model from Twisted), Twisted has two separate things that might be called “callbacks”. The first can be called “callbacks-by-convention”: these are things like IProtocol, which define a model that Twisted will use to deliver data to your application. The second can be called “callbacks-by-implementation”, which are Deferreds.

For “callbacks-by-convention”, I think the best way to describe it is that Twisted functions have a *convention* of being red, but they aren’t *actually* red. They can be called whenever you like. For example, consider the following Twisted protocol:

    from twisted.internet.protocol import Protocol

    class PrinterProtocol(Protocol):
        def connectionMade(self, transport):
            self.transport = transport
            self.transport.write(
                b'GET / HTTP/1.1\r\n'
                b'Host: http2bin.org\r\n'
                b'Connection: close\r\n'
                b'\r\n'
            )

        def dataReceived(self, data):
            print data,

        def connectionLost(self, reason):
            print "Connection lost”

There is nothing here that requires execution inside a reactor. In fact, if you change the spelling of “write” to “sendall”, you don’t need anything that the Python 2.7 standard library doesn’t give you:

    import socket

    def main():
        p = PrinterProtocol()
        s = socket.create_connection(('http2bin.org', 80))
        p.connectionMade(s)

        while True:
            data = s.recv(8192)
            if data:
                p.dataReceived(data)
            else:
                p.connectionLost(None)
                break

        s.close()

What matters here is that these functions are *just functions*. Unlike Python 3’s async functions, they do not require a coroutine runner that understands their special yield values. You call them, they do a thing, they return synchronously. This is how callback code *has* to be constructed, and it’s why callbacks are the lowest-common-denominator kind of async code. Conversely, async functions as defined in Python 3 really do need a coroutine runner to execute them or nothing happens.

But I presume you were really talking about the second kind of Twisted callback-function, which is the kind that uses Deferreds. And once again, there is nothing inherent in these functions that gives them a colour. Consider this bit of code:

    def showConnection(conn):
        print conn
        conn.close()

    def doStuff():
        deferredConnection = makeConnection(‘http2bin.org’, 80)
        deferredConnection.addCallback(showConnection)
        return deferredConnection

Your argument is that those functions are red. My argument is that they are uncoloured. For example, you can write makeConnection() like this:

    def makeConnection(host, port):
        s = socket.create_connection((host, port))
        d = Deferred()
        d.callback(s)
        return d

The simple fact of returning a Deferred doesn’t make your function async. This is one of Deferred’s fairly profound differences from asyncio.Future, which genuinely *does* require an event loop: Deferred can be evaluated entirely synchronously, with no recourse to an event loop. This has the effect of *uncolouring* functions that work with Deferreds. Their control flow is obscured, sure, but they are nonetheless not like async functions. In fact, it would be entirely possible to replace Twisted’s core event loop functions with ones that don’t delegate to a reactor, and then have a synchronous version of Twisted. It would be pretty gloriously useless, but is entirely do-able.

Cory