[Python-ideas] Are there asynchronous generators?

[Nick Coghlan]

On 1 July 2015 at 14:25, Ron Adam <ron3200 at gmail.com> wrote:
>
> That's really very nice.  Are there advantages to asyncio over the
> multiprocessing module?

I find it most useful to conceive of asyncio as an implementation of
an "event driven programming" development paradigm. This means that
after starting out with imperative procedural programming in Python,
you can branch out into other more advanced complexity management
models like object-oriented programming (class statements, dunder
protocols, descriptors, metaclasses, type hinting), functional
programming (comprehensions, generators, decorators, closures,
functools, itertools), array oriented programming (memoryview,
__matmul__, NumPy, SciPy), and event driven programming (asyncio,
Twisted, Tornado).

The stark difference between event driven programming and the first
three alternate development models I noted is that you can readily
implement the notions of "imperative shell, OO core", "imperative
shell, functional core", and "imperative shell, array oriented core",
where you expose a regular procedural API to other code, and implement
it internally using whichever approach makes the most sense for your
particular component. Even generators follow this same basic notion of
having a clear "start of iteration" and "end of iteration".

The concurrent execution model that most readily aligns with this
"imperative shell" approach is concurrent.futures
(https://docs.python.org/3/library/concurrent.futures.html) - it's
designed to let you easily take particular input->output operations
and dispatch them for execution in parallel in separate threads or
processes.

By contrast, event driven programming fundamentally changes your
execution model from "I will accept inputs at the beginning of the
program, and produce outputs at the end of the program" to "I will
start waiting for events, responding to them as they arrive, until one
of them indicates I should cease operation". "Waiting for an event"
becomes a core development concept, as now indicated by the "await"
keyword in PEP 492. The "async" keyword in that same PEP indicates
that the marked construct may need to wait for events as part of its
operation (async def, async for, async with), but exactly *where*
those wait points are depends on the specific construct (await
expressions in the function body for async def, protocol method
invocations for async for and async with).

For the design of asyncio (and similar frameworks) to make any sense
at all, it's necessary to approach them with that "event driven
programming" mindset - they seem entirely nonsensical when approached
with an inputs -> outputs algorithmic mindset, but more logical when
considered from a "receive request -> send other requests -> receive
responses -> send response" perspective.

For folks that primarily deal with algorithmic problems where inputs
are converted to outputs, the event driven model addresses a kind of
problem that *they don't have*, so it can seem entirely pointless.
However, there really are significant categories of problems (such as
network service development) where the event driven model is a
genuinely superior design tool. Like array oriented programming (and
even object-oriented and functional programming), the benefits can
unfortunately be hard to explain to folks that haven't personally
experienced the problems these tools address, so folks end up having
to take it on faith that we're applying the "Complex is better than
complicated" line from the Zen of Python when introducing new
modelling techniques into the core language.

Regards,
Nick.

P.S. It *is* possible to implement the "imperative shell, event driven
core" model, but it requires a synchronous-to-asynchronous adapter
like gevent, or an event-loop-per-thread model and extensive use of
"run_until_complete()". It's much more complex than "just use
concurrent.futures".

P.P.S. Credit to Gary Bernhardt for the "imperative shell, <X> core"
phrasing for low coupling component design where the external API
design is distinct from the internal architectural design

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia