On Fri, Jun 3, 2016 at 3:54 PM, Cory Benfield <cory@lukasa.co.uk> wrote:

...

On 3 Jun 2016, at 15:26, Nathaniel Smith <njs@pobox.com> wrote:

Convincing everyone to agree on a single event loop API sounds hard.

But maybe it would be useful to start with a baby step of defining a standard way for a coroutine to ask the event loop who it is, e.g.

await what_event_loop_is_this() -> "asyncio" -> "curio" -> ...

How do we ensure that this function is always available in the global namespace, or alternatively that it’s always available in a *consistent* namespace?

It could go into asyncio, but also we could just document that the function is literally @types.coroutine def what_event_loop_is_this(): return (yield "WHAT-EVENT-LOOP-IS-THIS") (actual value subject to bikeshedding -- the point is just that it's some well-known value that the event loop can watch for and which wouldn't be used for anything else.) -n -- Nathaniel J. Smith -- https://vorpus.org

asyncio functions works with asyncio-compatible code only, isn't it? For example asyncio.get_event_loop() should return a loop instance which conforms asyncio.AbstractEventLoop ABC. Third-party libraries like aiohttp just cannot work with arbitrary event loop type. They are always require at least *compatible* loop instance. On Fri, Jun 3, 2016 at 4:25 PM Nathaniel Smith <njs@pobox.com> wrote:

On Fri, Jun 3, 2016 at 3:54 PM, Cory Benfield <cory@lukasa.co.uk> wrote:

...

...

On 3 Jun 2016, at 15:26, Nathaniel Smith <njs@pobox.com> wrote:

Convincing everyone to agree on a single event loop API sounds hard.

But maybe it would be useful to start with a baby step of defining a standard way for a coroutine to ask the event loop who it is, e.g.

await what_event_loop_is_this() -> "asyncio" -> "curio" -> ...

How do we ensure that this function is always available in the global

namespace, or alternatively that it’s always available in a *consistent* namespace?

It could go into asyncio, but also we could just document that the function is literally

@types.coroutine def what_event_loop_is_this(): return (yield "WHAT-EVENT-LOOP-IS-THIS")

(actual value subject to bikeshedding -- the point is just that it's some well-known value that the event loop can watch for and which wouldn't be used for anything else.)

-n

-- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/

-- Thanks, Andrew Svetlov

I think this could be useful, but's answering the question of "what coroutine runner is this", not "what event loop is this". It's already possible to stack one event loop on top of another so that the question of event loop is ambiguous, but any `async def` function or other coroutine has exactly one coroutine runner. The identity of the coroutine runner matters more in practice than the event loop (PEP 484 has standardized the event loop interface but left coroutine runners unspecified). For more on this see https://github.com/tornadoweb/tornado/issues/1493 (in which differences between coroutine runners became an issue in the context of the event loop integration between Tornado and asyncio), and https://github.com/tornadoweb/tornado/pull/1716 (in which asyncio.sleep relies on certain behavior of the asyncio coroutine runner that prevents its use in the tornado coroutine runner). -Ben On Fri, Jun 3, 2016 at 6:26 PM, Nathaniel Smith <njs@pobox.com> wrote:

Convincing everyone to agree on a single event loop API sounds hard.

But maybe it would be useful to start with a baby step of defining a standard way for a coroutine to ask the event loop who it is, e.g.

await what_event_loop_is_this() -> "asyncio" -> "curio" -> ...

?

-n

-- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/

On Fri, Jun 3, 2016 at 5:14 PM, Ben Darnell <ben@bendarnell.com> wrote:

I think this could be useful, but's answering the question of "what coroutine runner is this", not "what event loop is this".

Thanks, that's definitely a better way to put it. -n -- Nathaniel J. Smith -- https://vorpus.org

On Fri, Jun 3, 2016 at 8:43 PM, Guido van Rossum <guido@python.org> wrote:

Could someone point me to the specific code that's considered the coroutine runner in asyncio and Tornado? I've been immersed in asyncio for so long that I don't know which part you're talking about. :-(

asyncio.Task and tornado.gen.Runner. Basically the thing that calls next() and send() on generator objects.

On Fri, Jun 3, 2016 at 5:26 PM, Nathaniel Smith <njs@pobox.com> wrote:

...

On Fri, Jun 3, 2016 at 5:14 PM, Ben Darnell <ben@bendarnell.com> wrote:

...

I think this could be useful, but's answering the question of "what coroutine runner is this", not "what event loop is this".

Thanks, that's definitely a better way to put it.

-n

-- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido)

On Fri, Jun 3, 2016 at 8:48 PM, Guido van Rossum <guido@python.org> wrote:

OK, that helps. Could you talk me through the crucial incompatibilities here? (If that's too much to ask I'm also happy to leave this up to the experts -- I trust that there's more than enough brainpower here to come up with a solution that works for all concerned.)

Coroutines that are chained together with either `yield from` or `await` are essentially one big generator with one shared coroutine runner. At the bottom, though, are the actual objects that are yielded by this generator (in asyncio, this happens in `Future.__iter__`). If the coroutine runner is asyncio.Task, it will raise if any part of this chain of coroutines yields anything but None or an asyncio.Future. The Tornado coroutine runner is more liberal: it accepts its own futures as well as asyncio and concurrent futures or twisted deferreds. In other words, this works with tornado, but not with asyncio: import asyncio import tornado.gen import tornado.ioloop import tornado.platform.asyncio use_tornado = False async def f(): await asyncio.sleep(0.1) await tornado.gen.sleep(0.1) if use_tornado: tornado.platform.asyncio.AsyncIOMainLoop().install() tornado.ioloop.IOLoop.current().run_sync(f) else: asyncio.get_event_loop().run_until_complete(f()) There were more incompatibilities with the yield-based syntax than with the await-based syntax. Tornado's coroutine runner allowed you to yield objects like lists and dicts (yielding a container was a kind of fork/join) that were not allowed in asyncio (thanks mainly to asyncio's use of `yield from`). The introduction of `await` has imposed similar restrictions on both systems, so there is no more yielding of lists or dicts in either case; explicit wrapper objects must be used. (but there's still a difference in that asyncio.Task allows None but tornado coroutines do not)

On Fri, Jun 3, 2016 at 5:45 PM, Ben Darnell <ben@bendarnell.com> wrote:

...

On Fri, Jun 3, 2016 at 8:43 PM, Guido van Rossum <guido@python.org> wrote:

...

Could someone point me to the specific code that's considered the coroutine runner in asyncio and Tornado? I've been immersed in asyncio for so long that I don't know which part you're talking about. :-(

asyncio.Task and tornado.gen.Runner. Basically the thing that calls next() and send() on generator objects.

...

On Fri, Jun 3, 2016 at 5:26 PM, Nathaniel Smith <njs@pobox.com> wrote:

...

On Fri, Jun 3, 2016 at 5:14 PM, Ben Darnell <ben@bendarnell.com>

wrote:

...

...

...

I think this could be useful, but's answering the question of "what coroutine runner is this", not "what event loop is this".

Thanks, that's definitely a better way to put it.

-n

-- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido)

-- --Guido van Rossum (python.org/~guido)

On Fri, Jun 3, 2016 at 9:14 PM, Nathaniel Smith <njs@pobox.com> wrote:

On Fri, Jun 3, 2016 at 6:10 PM, Ben Darnell <ben@bendarnell.com> wrote:

...

The introduction of `await` has imposed similar restrictions on both systems, so there is no more yielding of lists or dicts in either case; explicit wrapper objects must be used. (but there's still a difference in that asyncio.Task allows None but tornado coroutines do not)

I'm not saying that yielding lists or dicts is useful, but it sounds like you're saying that somehow it's actually incompatible with async/await, and I don't understand why that would be -- e.g., this works fine?:

Yes, this works fine as long as you have the extra hop into a decorated coroutine. What you can't do is `await [1, 2, 3]` in an `async def` native coroutine.

In [2]: @types.coroutine ...: def yield_list(): ...: yield [1, 2, 3] ...:

In [3]: async def f(): ...: await yield_list() ...:

In [4]: next(f().__await__()) Out[4]: [1, 2, 3]

-n

-- Nathaniel J. Smith -- https://vorpus.org

On Jun 3, 2016, at 8:14 PM, Ben Darnell <ben@bendarnell.com> wrote:

I think this could be useful, but's answering the question of "what coroutine runner is this", not "what event loop is this". It's already possible to stack one event loop on top of another so that the question of event loop is ambiguous, but any `async def` function or other coroutine has exactly one coroutine runner. The identity of the coroutine runner matters more in practice than the event loop (PEP 484 has standardized the event loop interface but left coroutine runners unspecified).

Right, let’s discuss the hypothetical ‘await what_coroutine_runner_is_this()’. Which, in case of asyncio, would return an instance of the current Task, right?

For more on this see https://github.com/tornadoweb/tornado/issues/1493 (in which differences between coroutine runners became an issue in the context of the event loop integration between Tornado and asyncio), and https://github.com/tornadoweb/tornado/pull/1716 (in which asyncio.sleep relies on certain behavior of the asyncio coroutine runner that prevents its use in the tornado coroutine runner).

I’ve read through both of those issues, but I still don’t understand what problem (or how the linked problems) can be solved by getting the current coroutine runner. Could you please provide an example? Thanks, Yury

On Fri, Jun 3, 2016 at 7:40 PM, Yury Selivanov <yselivanov@gmail.com> wrote:

...

On Jun 3, 2016, at 8:14 PM, Ben Darnell <ben@bendarnell.com> wrote:

I think this could be useful, but's answering the question of "what coroutine runner is this", not "what event loop is this". It's already possible to stack one event loop on top of another so that the question of event loop is ambiguous, but any `async def` function or other coroutine has exactly one coroutine runner. The identity of the coroutine runner matters more in practice than the event loop (PEP 484 has standardized the event loop interface but left coroutine runners unspecified).

Right, let’s discuss the hypothetical ‘await what_coroutine_runner_is_this()’. Which, in case of asyncio, would return an instance of the current Task, right?

I guess what I was really thinking of was await what_api_does_this_coroutine_runner_implement() so if you really needed to you could write code that does asyncio-things on asyncio, curio-things on curio, etc., and the caller doesn't have to care. I'll emphasize again that this is a trivial tiny suggestion :-) -n -- Nathaniel J. Smith -- https://vorpus.org

On Jun 3, 2016, at 11:29 PM, Nathaniel Smith <njs@pobox.com> wrote:

On Fri, Jun 3, 2016 at 7:40 PM, Yury Selivanov <yselivanov@gmail.com> wrote:

...

...

On Jun 3, 2016, at 8:14 PM, Ben Darnell <ben@bendarnell.com> wrote:

I think this could be useful, but's answering the question of "what coroutine runner is this", not "what event loop is this". It's already possible to stack one event loop on top of another so that the question of event loop is ambiguous, but any `async def` function or other coroutine has exactly one coroutine runner. The identity of the coroutine runner matters more in practice than the event loop (PEP 484 has standardized the event loop interface but left coroutine runners unspecified).

Right, let’s discuss the hypothetical ‘await what_coroutine_runner_is_this()’. Which, in case of asyncio, would return an instance of the current Task, right?

I guess what I was really thinking of was

await what_api_does_this_coroutine_runner_implement()

so if you really needed to you could write code that does asyncio-things on asyncio, curio-things on curio, etc., and the caller doesn't have to care.

I'll emphasize again that this is a trivial tiny suggestion :-)

Glyph and Ben want information about the coroutine runner in __await__, where awaiting on something is not possible. If we are talking about coroutines that users of curio/twisted/tornado/asyncio will write, then I don’t think they will need that thing either — it’s too low level. What makes interoperability between asyncio, tornado and twisted possible is that they all use similar event loops designs. They all have something similar to call_soon and other callbacks related APIs, Future/Deferred etc. What’s good about Futures and Deferreds is that they are well-defined objects with APIs etc. In curio, David intentionally didn’t want to have anything similar to Futures and Deferreds. Instead, he introduced a concept of “traps” (see [1]), which are generators wrapped with “@types.coroutine” yielding hardcoded tuples, that curio “kernel” can understand. Which is a very neat and clever design, except that it doesn’t compose with anything else. With that design, it’s essentially impossible for asyncio coroutines to await on curio coroutines. The only way would be to implement all of curio APIs in asyncio in a wrapper to do this: async def asyncio_coroutine(): await curio_adapter(curio_coroutine()) I’m not sure if it’s possible for curio coroutines to transparently await on asyncio coroutines without a significant refactoring of curio. But even if David decides to do that, for compatibility with Futures he’ll only need to add ‘isinstance(trap, Future)’ in his kernel code, no new APIs are needed (IMO). [1] https://github.com/dabeaz/curio/blob/master/curio/traps.py#L25 Yury