Mailman 3 June 2016 - Async-sig

async executors
by Christoph Groth June 13, 2016

June 13, 2016

Hi, A few days ago I posted a message [1] on python-ideas to which no one replied except Brett who suggested re-posting it here, which I do now. The following is a complete rewrite/update of the original message. I promise: I will stop the spamming if no one replies this time! Async programming with executors seems to be a useful approach for number-crunching problems where the main algorithm (the one launching the futures) is quite complicated and possibly concurrent, but light enough to run on a single core (the heavy lifting is done “inside” the futures). This use case is addressed by asyncio's run_in_executor that uses the executors provided by the concurrent.futures package. There is, however, a lot of duplication of functionality between concurrent.futures and asyncio. (The main difference is that concurrent.futures relies on threading and locks.) If one is going to use concurrent.futures only from coroutines that run under the control of the asyncio event loop, it should be possible to replace the concurrent.futures executors by async versions of them. Why could such an async executor be interesting? • It can be a lot simpler (and thus easier to understand and extend) because it builds on top of asyncio. There have been already discussions on unifying the Future type from concurrent.futures with that of asyncio [2]. • Performance is better, especially when many worker processes are involved, because of the simpler code that uses less locking (the only locking that remains is inside the multiprocessing module). As far as I can see the original concurrent.futures.ProcessPoolExecutor has no advantage when used in asyncio-based programs except when some coroutine blocks for long enough for the call queue to become empty. (But hey, async programming is about non-blocking coroutines!) Are there other possible advantages or disadvantages? Based on concurrent.futures.ProcessPoolExecutor, I’ve made a proof-of-concept implementation [3] that demonstrates that the idea works. (There have been some improvements compared to the version that I posted on python-ideas.) Christoph PS. I would be grateful if any asyncio experts could have a look at the part of the main loop of the process management coroutine where the coroutine awaits new data [4]. Currently, I am using an asyncio.Event that is set by a callback via asyncio’s add_reader(). Is this the most natural way to do it currently? I’m particularly puzzled by the fact that reader.poll(0) is not always true after the event has been set. If there is no better way to combine asyncio and multiprocessing.Process, perhaps this is a thing that could be improved in a future version of the Python stdlib? [1] http://thread.gmane.org/gmane.comp.python.ideas/40709 [2] http://thread.gmane.org/gmane.comp.python.ideas/24551/focus=24557 [3] https://gitlab.kwant-project.org/cwg/aexecutor [4] https://gitlab.kwant-project.org/cwg/aexecutor/blob/9290504e779c543d9ee93ad…

3 5

Re: [Async-sig] asyncio.timeout() is not portable
by Nathaniel Smith June 12, 2016

June 12, 2016

On Jun 6, 2016 10:19 AM, "Yury Selivanov" <yselivanov(a)gmail.com> wrote: > > > > On Jun 5, 2016, at 11:56 AM, Ben Darnell <ben(a)bendarnell.com> wrote: > > > > It has come to my attention ( https://github.com/KeepSafe/aiohttp/issues/877) that Python 3.5.2 (!) introduced a new context manager asyncio.timeout, which attaches a timeout to the current asyncio.Task. Any library or application that uses this feature will not be portable to other coroutine runners. > > > > with asyncio.timeout(1): > > result = await aiohttp.get('http://www.example.com') > > > > It's difficult to make this interface portable. We'd need to introduce a global thread-specific object that all of the coroutine runners could register the current task on, and define an interface that asyncio.timeout would call on that object to set the timeout. We could get rid of the global if the context manager used `async with` instead of `with`, since that would give us a way to communicate with the coroutine runner directly, but the net result is still that each runner needs to implement hooks for this feature (the hooks may or may not be generalizable to other features in the future). > > We’ve discussed this and decided that we should revert asyncio.Timeout from 3.5.2. I’ll do that later today. This is tangential and somewhat irrelevant given the above, but since I'm not sure where else to put it: having thought about this issue a bit more last night, I also think that it's important that the timeout manager (if/when it's reintroduced) should inject a meaningful exception like TimeoutError, and it should be legal for the coroutine to catch this and continue. This is in contrast to the one that's currently being reverted, which uses the task cancellation feature and is indistinguishable from any other use of the task cancellation feature. -n

4 5

asyncio.timeout() is not portable
by Ben Darnell June 12, 2016

June 12, 2016

It has come to my attention (https://github.com/KeepSafe/aiohttp/issues/877) that Python 3.5.2 (!) introduced a new context manager asyncio.timeout, which attaches a timeout to the current asyncio.Task. Any library or application that uses this feature will not be portable to other coroutine runners. with asyncio.timeout(1): result = await aiohttp.get('http://www.example.com') It's difficult to make this interface portable. We'd need to introduce a global thread-specific object that all of the coroutine runners could register the current task on, and define an interface that asyncio.timeout would call on that object to set the timeout. We could get rid of the global if the context manager used `async with` instead of `with`, since that would give us a way to communicate with the coroutine runner directly, but the net result is still that each runner needs to implement hooks for this feature (the hooks may or may not be generalizable to other features in the future). If our goal is portability across coroutine runners, then asyncio.timeout() needs to go back to the drawing board. If the goal is to extend asyncio.Task to be the one true coroutine runner then we could leave it as-is. For comparison, Tornado's equivalent to asyncio.timeout is tornado.gen.with_timeout(), and it has no special interactions with the coroutine runner: result = await with_timeout(timedelta(seconds=1), convert_yielded(aiohttp.get("http://www.example.com"))) (The convert_yielded call won't be needed in the next release of Tornado. The use of timedelta is because Tornado prefers absolute deadlines instead of relative timeouts, so that's how plain numbers are interpreted). -Ben

5 7

asyncio <-> twisted interoperability
by Yury Selivanov June 8, 2016

June 8, 2016

During the asyncio open space a few days ago we had a number of important questions raised, including Twisted/asyncio interop. I’d like to start the discussion of a few related things: 1. key difference between a Deferred and a Future; 2. make Deferred objects compatible with asyncio code; 3. overall strategy to make Twisted code compatible with asyncio code. In detail: 1. Unlike Futures, Deferred objects know nothing about the event loop. Deferred calls its callbacks directly, while Future uses ‘loop.call_soon’ to ensure fair scheduling. 2. I see two issues with Deferred/Future compatibility: 2.1. There should be a way for Deferred objects to get the currently running event loop reliably. There is a pending PR to add the new ‘asyncio.get_running_loop’ function. If we land it in 3.5.2, Deferred object will be able to get the currently running event loop and schedule callbacks in a fair fashion, similarly to Futures. 2.2. asyncio uses ‘isinstance(o, Future)’ calls in many key places, such as ‘asyncio.ensure_future’ and in the Task class. The latter makes it impossible for anything but Future and coroutines to be awaitable in asyncio code. We have two options to fix this: make Future an instance of ABCMeta, or introduce a new protocol for Future-like objects. The former option will not be considered because it makes isinstance calls much slower. As for the second option — a protocol for Future-like objects — I propose to add a new dunder attribute ‘__asyncio_future__ = True’ to asyncio.Future class, and replace all ‘isinstance(o, Future)’ calls with ‘hasattr(o, “__asyncio_future__”)’ checks. (2.1) and (2.2) should make it possible for Twisted to make Deferred objects fully compatible with asyncio code. Glyph and Amber, please confirm this. 3. I’ve been on Amber’s talk about Twisted and asyncio, and wanted to share my ideas on how Twisted should approach compatibility with asyncio and async/await syntax: - I don’t think we need ‘twisted.deferredCoroutine’ decorator — that will make async/await a lot slower. Instead, Twisted should follow asyncio and create an alternative, tailored for Twisted, implementation of ‘asyncio.Task’, along with ‘ensure_deferred’ and/or ‘create_task’ APIs. This way users can create Deferreds for coroutines only when they need it. - ‘ensure_deferred’ could also accept instances of ‘asyncio.Future’, and wrap them in a Deferred object. This should make it possible to re-use asyncio-specific code in Twisted code. Thoughts? Thanks, Yury

6 18

Re: [Async-sig] async executors
by Nathaniel Smith June 6, 2016

June 6, 2016

On Jun 6, 2016 5:37 AM, "Christoph Groth" <christoph(a)grothesque.org> wrote: > [...] > Async programming with executors seems to be a useful approach for number-crunching problems where the main algorithm (the one launching the futures) is quite complicated and possibly concurrent, but light enough to run on a single core (the heavy lifting is done “inside” the futures). > > This use case is addressed by asyncio's run_in_executor that uses the executors provided by the concurrent.futures package. There is, however, a lot of duplication of functionality between concurrent.futures and asyncio. (The main difference is that concurrent.futures relies on threading and locks.) If one is going to use concurrent.futures only from coroutines that run under the control of the asyncio event loop, it should be possible to replace the concurrent.futures executors by async versions of them. Sounds like an interesting project :-) FYI, the latest versions of curio also have their own thread/process pool implementations that replace concurrent.futures, I think for similar reasons to the ones you allude to: http://curio.readthedocs.io/en/latest/reference.html#performing-external-wo… There are certainly reasons you might prefer to use asyncio instead of curio, but if you're implementing such a thing on asyncio then curio's implementation and semantics might be interesting for reference. -n

1 0

A possible tiny step towards event loop interoperability
by Nathaniel Smith June 4, 2016

June 4, 2016

Convincing everyone to agree on a single event loop API sounds hard. But maybe it would be useful to start with a baby step of defining a standard way for a coroutine to ask the event loop who it is, e.g. await what_event_loop_is_this() -> "asyncio" -> "curio" -> ... ? -n -- Nathaniel J. Smith -- https://vorpus.org

7 21

Inviting people to this list
by Brett Cannon June 4, 2016

June 4, 2016

We have folks from asyncio, h2/h11, tornado, and Twisted. David Beazley knows about the list but he told me he isn't a mailing list guy so we will see if he joins. There is no one from gevent that I know of, so if someone thinks they should be here then please invite them to join. If there are other specific projects that you think should have someone here then please reach out to them. Otherwise I'm personally not going out of my way to broadcast this list as I'm happy with who's here and the level of discourse (although no need to hide the list's existence either). 😃

1 0