[Async-sig] "Coroutines" sometimes run without being scheduled on an event loop

[twisteroid ambassador]

Hi,

tl;dr: coroutine functions and regular functions returning Futures
behave differently: the latter may start running immediately without
being scheduled on a loop, or even with no loop running. This might be
bad since the two are sometimes advertised to be interchangeable.


I find that sometimes I want to construct a coroutine object, store it
for some time, and run it later. Most times it works like one would
expect: I call a coroutine function which gives me a coroutine object,
I hold on to the coroutine object, I later await it or use
loop.create_task(), asyncio.gather(), etc. on it, and only then it
starts to run.

However, I have found some cases where the "coroutine" starts running
immediately. The first example is loop.run_in_executor(). I guess this
is somewhat unsurprising since the passed function don't actually run
in the event loop. Demonstrated below with strace and the interactive
console:

$ strace -e connect -f python3
Python 3.6.5 (default, Apr  4 2018, 15:01:18)
[GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> import socket
>>> s = socket.socket()
>>> loop = asyncio.get_event_loop()
>>> coro = loop.sock_connect(s, ('127.0.0.1', 80))
>>> loop.run_until_complete(asyncio.sleep(1))
>>> task = loop.create_task(coro)
>>> loop.run_until_complete(asyncio.sleep(1))
connect(3, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("127.0.0.1")}, 16) = -1 ECONNREFUSED (Connection
refused)
>>> s.close()
>>> s = socket.socket()
>>> coro2 = loop.run_in_executor(None, s.connect, ('127.0.0.1', 80))
strace: Process 13739 attached
>>> [pid 13739] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 ECONNREFUSED (Connection refused)

>>> coro2
<Future pending cb=[_chain_future.<locals>._call_check_cancel() at
/usr/lib64/python3.6/asyncio/futures.py:403]>
>>> loop.run_until_complete(asyncio.sleep(1))
>>> coro2
<Future finished exception=ConnectionRefusedError(111, 'Connection refused')>
>>>

Note that with loop.sock_connect(), the connect syscall is only run
after loop.create_task() is called on the coroutine AND the loop is
running. On the other hand, as soon as loop.run_in_executor() is
called on socket.connect, the connect syscall gets called, without the
event loop running at all.

Another such case is with Python 3.4.2, where even loop.sock_connect()
will run immediately:

$ strace -e connect -f python3
Python 3.4.2 (default, Oct  8 2014, 10:45:20)
[GCC 4.9.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> import asyncio
>>> loop = asyncio.get_event_loop()
>>> s = socket.socket()
>>> c = loop.sock_connect(s, ('127.0.0.1', 82))
connect(7, {sa_family=AF_INET, sin_port=htons(82),
sin_addr=inet_addr("127.0.0.1")}, 16) = -1ECONNREFUSED (Connection
refused)
>>> c
<Future finished exception=ConnectionRefusedError(111, 'Connection refused')>
>>>

In both these cases, the misbehaving "coroutine" aren't actually
defined as coroutine functions, but regular functions returning a
Future, which is probably why they don't act like coroutines. However,
coroutine functions and regular functions returning Futures are often
used interchangeably: Python docs Section 18.5.3.1 even says:

> Note: In this documentation, some methods are documented as coroutines, even if they are plain Python functions returning a Future. This is intentional to have a freedom of tweaking the implementation of these functions in the future.

In particular, both run_in_executor() and sock_connect() are
documented as coroutines.

If an asyncio API may change from a function returning Future to a
coroutine function and vice versa any time, then one cannot rely on
the behavior of creating the "coroutine object" not running the
coroutine immediately. This seems like an important Gotcha waiting to
bite someone.

Back to the scenario in the beginning. If I want to write a function
that takes coroutine objects and schedule them to run later, and some
coroutine objects turn out to be misbehaving like above, then they
will run too early. To avoid this, I could either 1. pass the
coroutine functions and their arguments separately "callback style",
2. use functools.partial or lambdas, or 3. always pass in real
coroutine objects returned from coroutine functions defined with
"async def". Does this sound right?

Thanks,

twistero