asyncio: factoring-out of the general mechanism
Hello, I have a question/idea about the support for asynchronous programming in Python. With Python 3.5, support for asynchronous programming has been added to the core Python language. The new language features, however, are only useful with an event loop, as provided (only?) by the asyncio stdlib module. I notice that already the class asyncio.AbstractEventLoop contains many things that to me seem very specific to particular applications. For examples that class contains placeholders for methods that expect IP host names and IP ports as arguments. In a world where networking is done in other ways than through IP, Python would be still useful. One could keep using most of the standard library that is not specific to IP networking. Asynchronous programming is a concept that is way more universal than many of the things in asyncio (IP ports, unix pipes, etc.). Has there been a discussion about separating the abstract bits needed for asynchronous programming in Python from the application-specific ones? Or would this be a bad idea for some reason? Christoph
Well, what else did you have in mind? Remember that until we fix the GIL
(which won't happen until Python 4) you won't get benefit from async
programming unless you are overlapping with I/O, and (due to the
limitations of select/poll/etc.) that is pretty much limited to network and
IPC (disk I/O in particular cannot be overlapped in this way).
The main other area of interest would be UI events (as in a browser or GUI)
and for that I think an adaptation of the asyncio event loop will be
suitable. (If you're not going to use them it's fine not to implement the
network-specific methods on an event loop.)
Note that the async/await keywords themselves have no direct link to an
event loop -- they are just a different way to spell generators. It's just
that they are only useful with asyncio or some other event loop library.
On Thu, Oct 1, 2015 at 9:27 AM, Christoph Groth
Hello,
I have a question/idea about the support for asynchronous programming in Python.
With Python 3.5, support for asynchronous programming has been added to the core Python language. The new language features, however, are only useful with an event loop, as provided (only?) by the asyncio stdlib module.
I notice that already the class asyncio.AbstractEventLoop contains many things that to me seem very specific to particular applications. For examples that class contains placeholders for methods that expect IP host names and IP ports as arguments.
In a world where networking is done in other ways than through IP, Python would be still useful. One could keep using most of the standard library that is not specific to IP networking.
Asynchronous programming is a concept that is way more universal than many of the things in asyncio (IP ports, unix pipes, etc.). Has there been a discussion about separating the abstract bits needed for asynchronous programming in Python from the application-specific ones? Or would this be a bad idea for some reason?
Christoph
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido)
Guido van Rossum wrote:
Well, what else did you have in mind? Remember that until we fix the GIL (which won't happen until Python 4) you won't get benefit from async programming unless you are overlapping with I/O, and (due to the limitations of select/poll/etc.) that is pretty much limited to network and IPC (disk I/O in particular cannot be overlapped in this way).
I recently realized that asynchronous programming is also useful for scientific computation: It allows to quite naturally express a complex algorithm that at various stages launches background computation. A good application of this technique is adaptive numerical integration. See [1] for an example script that uses asyncio. There, the asynchronous master Python process performs the integration algorithm (that is computationally cheap but has quite complex control flow), while the integrand function is evaluated by worker processes. Now, this particular example uses concurrent.futures, but one could also use MPI, scoop, or 0mq. This is what made me wonder about the monolithic design of asyncio. We do not want to add support for all these libraries to the asyncio package? Already now importing asyncio triggers a considerable avalanche of imports that do not all seem necessary for most problems.
Note that the async/await keywords themselves have no direct link to an event loop -- they are just a different way to spell generators. It's just that they are only useful with asyncio or some other event loop library.
That's clear. I was just wondering whether it would make sense to split up asyncio into a basic package that only provides the event loop itself (and does not depend on any particular protocol), and support for various "applications". The parts that relate to various "applications" could then be added to other modules. For example, the subprocess package could be extended by some coroutines for launching processes. But perhaps coroutines (or at least their Python flavor) only make sense within the context of a specific event loop and the language does not want to mandate the use of a specific one? Then indeed it seems to be a good idea to structure async functionality around specific event loops. Christoph
Here is the missing link [1]: https://mail.scipy.org/pipermail/scipy-dev/2015-September/020932.html
On Oct 2, 2015, at 14:41, Christoph Groth
Guido van Rossum wrote:
Well, what else did you have in mind? Remember that until we fix the GIL (which won't happen until Python 4) you won't get benefit from async programming unless you are overlapping with I/O, and (due to the limitations of select/poll/etc.) that is pretty much limited to network and IPC (disk I/O in particular cannot be overlapped in this way).
I recently realized that asynchronous programming is also useful for scientific computation: It allows to quite naturally express a complex algorithm that at various stages launches background computation.
A good application of this technique is adaptive numerical integration. See [1] for an example script that uses asyncio. There, the asynchronous master Python process performs the integration algorithm (that is computationally cheap but has quite complex control flow), while the integrand function is evaluated by worker processes.
Now, this particular example uses concurrent.futures, but one could also use MPI, scoop, or 0mq. This is what made me wonder about the monolithic design of asyncio.
Why would you want asyncio here? You explicitly need parallelism, and independent threads/processes rather than coroutines (with or without an event loop). And it seems like composable futures are exactly the abstraction you want. It seems like what you really need here is to extend (or make it easier for end users and third party libs to extend) concurrent.futures to work with 0mq, etc., so you can leave the controller code alone while swapping out the dispatch mechanism.
participants (3)
-
Andrew Barnert -
Christoph Groth -
Guido van Rossum