[Python-ideas] asyncio: factoring-out of the general mechanism

[Christoph Groth]

Guido van Rossum wrote:

> Well, what else did you have in mind? Remember that until we fix the GIL
> (which won't happen until Python 4) you won't get benefit from async
> programming unless you are overlapping with I/O, and (due to the
> limitations of select/poll/etc.) that is pretty much limited to network and
> IPC (disk I/O in particular cannot be overlapped in this way).

I recently realized that asynchronous programming is also useful for
scientific computation: It allows to quite naturally express a complex
algorithm that at various stages launches background computation.

A good application of this technique is adaptive numerical integration.
See [1] for an example script that uses asyncio.  There, the
asynchronous master Python process performs the integration algorithm
(that is computationally cheap but has quite complex control flow),
while the integrand function is evaluated by worker processes.

Now, this particular example uses concurrent.futures, but one could also
use MPI, scoop, or 0mq.  This is what made me wonder about the
monolithic design of asyncio.  We do not want to add support for all
these libraries to the asyncio package?  Already now importing asyncio
triggers a considerable avalanche of imports that do not all seem
necessary for most problems.

> Note that the async/await keywords themselves have no direct link to an
> event loop -- they are just a different way to spell generators. It's just
> that they are only useful with asyncio or some other event loop library.

That's clear.  I was just wondering whether it would make sense to split
up asyncio into a basic package that only provides the event loop itself
(and does not depend on any particular protocol), and support for
various "applications".

The parts that relate to various "applications" could then be added to
other modules.  For example, the subprocess package could be extended by
some coroutines for launching processes.

But perhaps coroutines (or at least their Python flavor) only make sense
within the context of a specific event loop and the language does not
want to mandate the use of a specific one?  Then indeed it seems to be a
good idea to structure async functionality around specific event loops.

Christoph
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 810 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151002/82faae46/attachment.sig>