Question about asyncio and blocking operations
Frank Millman
frank at chagford.com
Mon Jan 25 04:04:58 EST 2016
"Ian Kelly" wrote in message
news:CALwzidnGOgpx+CpMVBA8vpEFuq4-BwMVS0gZ3ShB0oWZi0Bw+Q at mail.gmail.com...
>
> On Sat, Jan 23, 2016 at 7:38 AM, Frank Millman <frank at chagford.com> wrote:
> > Here is the difficulty. The recommended way to handle a blocking
> > operation
> > is to run it as task in a different thread, using run_in_executor().
> > This
> > method is a coroutine. An implication of this is that any method that
> > calls
> > it must also be a coroutine, so I end up with a chain of coroutines
> > stretching all the way back to the initial event that triggered it.
>
> This seems to be a common misapprehension about asyncio programming.
> While coroutines are the focus of the library, they're based on
> futures, and so by working at a slightly lower level you can also
> handle them as such. So while this would be the typical way to use
> run_in_executor:
>
> async def my_coroutine(stuff):
> value = await get_event_loop().run_in_executor(None,
> blocking_function, stuff)
> result = await do_something_else_with(value)
> return result
>
> This is also a perfectly valid way to use it:
>
> def normal_function(stuff):
> loop = get_event_loop()
> coro = loop.run_in_executor(None, blocking_function, stuff)
> task = loop.create_task(coro)
> task.add_done_callback(do_something_else)
> return task
I am struggling to get my head around this.
1. In the second function, AFAICT coro is already a future. Why is it
necessary to turn it into a task? In fact when I tried that in my testing, I
got an assertion error -
File: "C:\Python35\lib\asyncio\base_events.py", line 211, in create_task
task = tasks.Task(coro, loop=self)
File: "C:\Python35\lib\asyncio\tasks.py", line 70, in __init__
assert coroutines.iscoroutine(coro), repr(coro)
AssertionError: <Future pending ... >
2. In the first function, calling 'run_in_executor' unblocks the main loop
so that it can continue with other tasks, but the function itself is
suspended until the blocking function returns. In the second function, I
cannot see how the function gets suspended. It looks as if the blocking
function will run in the background, and the main function will continue.
I would like to experiment with this further, but I would need to see the
broader context - IOW see the 'caller' of normal_function(), and see what it
does with the return value.
I feel I am getting closer to an 'aha' moment, but I am not there yet, so
all info is appreciated.
Frank
More information about the Python-list
mailing list