News from asyncio

Hi, I'm working for eNovance on the asyncio module, the goal is to use it in the huge OpenStack project (2.5 millions line of code) which currently uses eventlet. I'm trying to fix remaining issues in the asyncio module before Python 3.4 final. The asyncio project is very active but discussions are splitted between its own dedicated mailing list (python-tulip Google group), Tulip bug tracker and Python bug tracker. Please join Tulip mailing list if you are interested to contribute. http://code.google.com/p/tulip/ I would like to share with you the status of the module. Many bugs have been fixed recently. I suppose that new bugs are found because new developers started to play with asyncio since Python 3.4 beta 1. asyncio issues fixed in Python 3.4 beta 3, in a random order: - I wrote most of the asyncio documentation, please help me to improve it! I tried to add many short examples, each time explaining one feature or concept (ex: callbacks, signals, futures, etc.): http://docs.python.org/dev/library/asyncio.html - Characters devices (TTY/PTY) are now supported, useful to control real terminal (not pipes) for subprocesses. On Mac OS X older than Maverick (10.9), the SelectSelector should be used instead of KqueueSelector (kqueue didnd't support character devices) - Tulip #111: StreamReader.readexactly() now raises an IncompleteReadError if the end of stream is reached before we received enough bytes, instead of returning less bytes than requested. - Python #20311: asyncio had a performance issue related to the resolution of selectors and clocks. For example, selectors.EpollSelector has a resolution of 1 millisecond (10^-3), whereas asyncio uses arbitrary timestamps. The issue was fixed by adding a resolution attribute to selectors and a private granularity attribute to asyncio.BaseEventLoop, and use the granularity in asyncio event loop to round times. - New Task.current_task() class method - Guido wrote a web crawler, see examples/crawl.py in Tulip - More symbols are exported in the main asyncio module (ex: Queue, iscouroutine(), etc.) - Charles-François improved the signal handlers: SA_RESTART flag is now set to limit EINTR errors in syscalls - Some optimizations (ex: don't call logger.log() when it's not needed) - Many bugfixes - (sorry if I forgot other changes, see also Tulip history and history of the asyncio module in Python) I also would like to change asyncio to support a "stream-like" API for subprocesses, see Tulip issue #115 (and Python issue #20400): http://code.google.com/p/tulip/issues/detail?id=115 I ported ayncio on Python 2.6 and 2.7, because today OpenStack only uses these Python versions. I created a new project called "Trollius" (instead of "Tulip") because the syntax is a little bit different. "yield from" becomes "yield", and "return x" becomes "raise Return(x)": https://bitbucket.org/enovance/trollius https://pypi.python.org/pypi/trollius If you are interested by the OpenStack part, see my blueprint (something similar to PEPs but for smaller changes) for Oslo Messaing: https://wiki.openstack.org/wiki/Oslo/blueprints/asyncio There is an ongoing effort to port OpenStack to Python 3, eNovance is also working on the portage: https://wiki.openstack.org/wiki/Python3 Victor

2014-01-27 Antoine Pitrou <solipsis@pitrou.net>:
IncompleteReadError has two additionnal attributes: - partial: "incomplete" received bytes - expected: total number of expected bytes (n parameter of readexactly) I prefer to use a different exception to ensure that these attributes are present. I don't like having to check "hasattr(exc, ...)". Before this change, readexactly(n) returned the partial received bytes if the end of the stream was reached. Victor

On 27 January 2014 10:55, Victor Stinner <victor.stinner@gmail.com> wrote:
I had the same doubt. Note also that IncompleteReadError is a subclass of EOFError, so you can catch EOFError if you like.
-- Gustavo J. A. M. Carneiro Gambit Research LLC "The universe is always one step beyond logic." -- Frank Herbert

2014-01-27 Serhiy Storchaka <storchaka@gmail.com>:
Please read the original issue for more information: http://code.google.com/p/tulip/issues/detail?id=111 I mentionned http.client.IncompleRead exception there. The HTTP exception is similar but also different: - asyncio.IncompleReadError inherits from EOFError, not from HTTPException (which inherits from Exception) - asyncio.IncompleReadError.expected is the total expected size, not the remaining size Victor

On Mon, Jan 27, 2014 at 5:21 AM, Victor Stinner <victor.stinner@gmail.com> wrote:
- asyncio.IncompleReadError.expected is the total expected size, not the remaining size
Why not be consistent with the meaning of http.client.IncompleteRead.expected? The current meaning can be recovered via len(e.partial) + e.expected. -- Devin

On 27Jan2014 09:15, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
Which you could also expose as a property name ".requested", since this value must have been thought useful or Victor wouldn't have defined .expected to be it:-) Seems reasonable, for all that the above computation is trivial. Just a thought, -- Cameron Simpson <cs@zip.com.au> Artificial intelligence won't make a micrometer out of a monkeywrench. - Rick Gordon <rickg@crl.com>

Latest asyncio update: it has a new asyncio.subprocess submodule which provides a high-level API to control subprocesses, similar to subprocess.Popen but using asyncio event loop (and so is asynchronous). It solves the following old and tricky issue: http://bugs.python.org/issue12187 "subprocess.wait() with a timeout uses polling on POSIX" "yield from asyncio.wait_for(proc.wait(), timeout)" is no more a busy loop ;-) Victor

2014-01-27 Antoine Pitrou <solipsis@pitrou.net>:
IncompleteReadError has two additionnal attributes: - partial: "incomplete" received bytes - expected: total number of expected bytes (n parameter of readexactly) I prefer to use a different exception to ensure that these attributes are present. I don't like having to check "hasattr(exc, ...)". Before this change, readexactly(n) returned the partial received bytes if the end of the stream was reached. Victor

On 27 January 2014 10:55, Victor Stinner <victor.stinner@gmail.com> wrote:
I had the same doubt. Note also that IncompleteReadError is a subclass of EOFError, so you can catch EOFError if you like.
-- Gustavo J. A. M. Carneiro Gambit Research LLC "The universe is always one step beyond logic." -- Frank Herbert

2014-01-27 Serhiy Storchaka <storchaka@gmail.com>:
Please read the original issue for more information: http://code.google.com/p/tulip/issues/detail?id=111 I mentionned http.client.IncompleRead exception there. The HTTP exception is similar but also different: - asyncio.IncompleReadError inherits from EOFError, not from HTTPException (which inherits from Exception) - asyncio.IncompleReadError.expected is the total expected size, not the remaining size Victor

On Mon, Jan 27, 2014 at 5:21 AM, Victor Stinner <victor.stinner@gmail.com> wrote:
- asyncio.IncompleReadError.expected is the total expected size, not the remaining size
Why not be consistent with the meaning of http.client.IncompleteRead.expected? The current meaning can be recovered via len(e.partial) + e.expected. -- Devin

On 27Jan2014 09:15, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
Which you could also expose as a property name ".requested", since this value must have been thought useful or Victor wouldn't have defined .expected to be it:-) Seems reasonable, for all that the above computation is trivial. Just a thought, -- Cameron Simpson <cs@zip.com.au> Artificial intelligence won't make a micrometer out of a monkeywrench. - Rick Gordon <rickg@crl.com>

Latest asyncio update: it has a new asyncio.subprocess submodule which provides a high-level API to control subprocesses, similar to subprocess.Popen but using asyncio event loop (and so is asynchronous). It solves the following old and tricky issue: http://bugs.python.org/issue12187 "subprocess.wait() with a timeout uses polling on POSIX" "yield from asyncio.wait_for(proc.wait(), timeout)" is no more a busy loop ;-) Victor
participants (6)
-
Antoine Pitrou
-
Cameron Simpson
-
Devin Jeanpierre
-
Gustavo Carneiro
-
Serhiy Storchaka
-
Victor Stinner