[Async-sig] "Coroutines" sometimes run without being scheduled on an event loop

Thu May 3 16:24:04 EDT 2018

Depending on the coroutine*not* running sounds like asking for trouble.

On Thu, May 3, 2018, 09:38 Andrew Svetlov <andrew.svetlov at gmail.com> wrote:

> What real problem do you want to solve?
> Correct code should always use `await loop.sock_connect(sock, addr)`, it
> this case the behavior difference never hurts you.
>
> On Thu, May 3, 2018 at 7:04 PM twisteroid ambassador <
> twisteroid.ambassador at gmail.com> wrote:
>
>> Hi,
>>
>> tl;dr: coroutine functions and regular functions returning Futures
>> behave differently: the latter may start running immediately without
>> being scheduled on a loop, or even with no loop running. This might be
>> bad since the two are sometimes advertised to be interchangeable.
>>
>>
>> I find that sometimes I want to construct a coroutine object, store it
>> for some time, and run it later. Most times it works like one would
>> expect: I call a coroutine function which gives me a coroutine object,
>> I hold on to the coroutine object, I later await it or use
>> loop.create_task(), asyncio.gather(), etc. on it, and only then it
>> starts to run.
>>
>> However, I have found some cases where the "coroutine" starts running
>> immediately. The first example is loop.run_in_executor(). I guess this
>> is somewhat unsurprising since the passed function don't actually run
>> in the event loop. Demonstrated below with strace and the interactive
>> console:
>>
>> $ strace -e connect -f python3
>> Python 3.6.5 (default, Apr  4 2018, 15:01:18)
>> [GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> import asyncio
>> >>> import socket
>> >>> s = socket.socket()
>> >>> loop = asyncio.get_event_loop()
>> >>> coro = loop.sock_connect(s, ('127.0.0.1', 80))
>> >>> loop.run_until_complete(asyncio.sleep(1))
>> >>> task = loop.create_task(coro)
>> >>> loop.run_until_complete(asyncio.sleep(1))
>> connect(3, {sa_family=AF_INET, sin_port=htons(80),
>> sin_addr=inet_addr("127.0.0.1")}, 16) = -1 ECONNREFUSED (Connection
>> refused)
>> >>> s.close()
>> >>> s = socket.socket()
>> >>> coro2 = loop.run_in_executor(None, s.connect, ('127.0.0.1', 80))
>> strace: Process 13739 attached
>> >>> [pid 13739] connect(3, {sa_family=AF_INET, sin_port=htons(80),
>> sin_addr=inet_addr("127.0.0.1")}, 16) = -1 ECONNREFUSED (Connection refused)
>>
>> >>> coro2
>> <Future pending cb=[_chain_future.<locals>._call_check_cancel() at
>> /usr/lib64/python3.6/asyncio/futures.py:403]>
>> >>> loop.run_until_complete(asyncio.sleep(1))
>> >>> coro2
>> <Future finished exception=ConnectionRefusedError(111, 'Connection
>> refused')>
>> >>>
>>
>> Note that with loop.sock_connect(), the connect syscall is only run
>> after loop.create_task() is called on the coroutine AND the loop is
>> running. On the other hand, as soon as loop.run_in_executor() is
>> called on socket.connect, the connect syscall gets called, without the
>> event loop running at all.
>>
>> Another such case is with Python 3.4.2, where even loop.sock_connect()
>> will run immediately:
>>
>> $ strace -e connect -f python3
>> Python 3.4.2 (default, Oct  8 2014, 10:45:20)
>> [GCC 4.9.1] on linux
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> import socket
>> >>> import asyncio
>> >>> loop = asyncio.get_event_loop()
>> >>> s = socket.socket()
>> >>> c = loop.sock_connect(s, ('127.0.0.1', 82))
>> connect(7, {sa_family=AF_INET, sin_port=htons(82),
>> sin_addr=inet_addr("127.0.0.1")}, 16) = -1ECONNREFUSED (Connection
>> refused)
>> >>> c
>> <Future finished exception=ConnectionRefusedError(111, 'Connection
>> refused')>
>> >>>
>>
>> In both these cases, the misbehaving "coroutine" aren't actually
>> defined as coroutine functions, but regular functions returning a
>> Future, which is probably why they don't act like coroutines. However,
>> coroutine functions and regular functions returning Futures are often
>> used interchangeably: Python docs Section 18.5.3.1 even says:
>>
>> > Note: In this documentation, some methods are documented as coroutines,
>> even if they are plain Python functions returning a Future. This is
>> intentional to have a freedom of tweaking the implementation of these
>> functions in the future.
>>
>> In particular, both run_in_executor() and sock_connect() are
>> documented as coroutines.
>>
>> If an asyncio API may change from a function returning Future to a
>> coroutine function and vice versa any time, then one cannot rely on
>> the behavior of creating the "coroutine object" not running the
>> coroutine immediately. This seems like an important Gotcha waiting to
>> bite someone.
>>
>> Back to the scenario in the beginning. If I want to write a function
>> that takes coroutine objects and schedule them to run later, and some
>> coroutine objects turn out to be misbehaving like above, then they
>> will run too early. To avoid this, I could either 1. pass the
>> coroutine functions and their arguments separately "callback style",
>> 2. use functools.partial or lambdas, or 3. always pass in real
>> coroutine objects returned from coroutine functions defined with
>> "async def". Does this sound right?
>>
>> Thanks,
>>
>> twistero
>> _______________________________________________
>> Async-sig mailing list
>> Async-sig at python.org
>> https://mail.python.org/mailman/listinfo/async-sig
>> Code of Conduct: https://www.python.org/psf/codeofconduct/
>>
> --
> Thanks,
> Andrew Svetlov
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20180503/46a6098c/attachment-0001.html>