Learning from the shell in supporting asyncio background calls

Hi folks, Based on the recent discussions Sven kicked off regarding the complexity of interacting with asyncio from otherwise synchronous code, I came up with an API design that I like inspired by the way background and foreground tasks in the POSIX shell work. My blog post about this design is at http://www.curiousefficiency.org/posts/2015/07/asyncio-background-calls.html, but the essential components are the following two APIs: def run_in_background(target, *, loop=None): """Schedules target as a background task Returns the scheduled task. If target is a future or coroutine, equivalent to asyncio.ensure_future If target is a callable, it is scheduled in the default executor """ ... def run_in_foreground(task, *, loop=None): """Runs event loop in current thread until the given task completes Returns the result of the task. For more complex conditions, combine with asyncio.wait() To include a timeout, combine with asyncio.wait_for() """ ... run_in_background is akin to invoking a shell command with a trailing "&" - it puts the operation into the background, leaving the current thread to move on to the next operation (or wait for input at the REPL). When coroutines are scheduled, they won't start running until you start a foreground task, while callables delegated to the default executor will start running immediately. To actually get the *results* of that task, you have to run it in the foreground of the current thread using run_in_foreground - this is akin to bringing a background process to the foreground of a shell session using "fg". To relate this idea back to some of the examples Sven was discussing, here's how translating some old serialised synchronous code to use those APIs might look in practice: # Serial synchronous data loading def load_and_process_data(): data1 = load_remote_data_set1() data2 = load_remote_data_set2() return process_data(data1, data2) # Parallel asynchronous data loading def load_and_process_data(): future1 = asyncio.run_in_background(load_remote_data_set1_async()) future2 = asyncio.run_in_background(load_remote_data_set2_async()) data1 = asyncio.run_in_foreground(future1) data2 = asyncio.run_in_foreground(future2) return process_data(data1, data2) The application remains fundamentally synchronous, but the asyncio event loop is exploited to obtain some local concurrency in waiting for client IO operations. Regards, Nick. P.S. time.sleep() and asyncio.sleep() are rather handy as standins for blocking and non-blocking IO operations. I wish I'd remembered that earlier :) -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 10 July 2015 at 11:49, Nick Coghlan <ncoghlan@gmail.com> wrote:
Why is that better than something like: data1, data2 = asyncio.run([future1, future2]) IIUC your proposal is that run_in_background adds the tasks to an implicit global variable. Then the first call to run_in_foreground runs both tasks returning when future1 is ready. At that point it suspends future2 if incomplete? Then the second call to run_in_foreground returns immediately if future2 is ready or otherwise runs that task until complete? -- Oscar

On 10 July 2015 at 21:48, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
It just schedules them normally using asyncio.get_event_loop().create_task() (or run_in_executor if you pass in a callable). For folks that don't want to click through to the blog post (which has a full worked example using asynchronous timers), the full implementations of the two functions (with a slight tweak to run_in_background to make the executor configurable as well) are: def run_in_background(target, *, loop=None, executor=None): if loop is None: loop = asyncio.get_event_loop() try: return asyncio.ensure_future(target, loop=loop) except TypeError: pass if callable(target): return loop.run_in_executor(executor, target) raise TypeError("background task must be future, coroutine or " "callable, not {!r}".format(type(target))) def run_in_foreground(task, *, loop=None): if loop is None: loop = asyncio.get_event_loop() return loop.run_until_complete(asyncio.ensure_future(task))
No, it's all driven by the main asyncio event loop - the suggested functions are relatively thin shims designed to let people make effective use of asyncio with just the POSIX shell foreground & background task mental model, rather than having to learn how asyncio *really* works in order to benefit from it. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

As I wrote on the issue, I'm -1 on this proposal. Not only does this API encourage beginners to ignore the essential difference between synchronous functions meant to run in a thread (using synchronous I/O and pre-emptive CPU scheduling) and asyncio coroutines/tasks (which use overlapped I/O and require explicit scheduling), it also encourages avoiding the "await" primitive (formerly "yield from") in favor of a function call which cannot be used from within a coroutine/task. This particular spelling moreover introduces a "similarity" between foreground and background tasks that doesn't actually exist. The example suggests that this should really be a pair of convenience functions in collections.futures, as it does not make any use of asyncio. On Fri, Jul 10, 2015 at 12:49 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)

On 10 July 2015 at 21:51, Guido van Rossum <guido@python.org> wrote:
My apologies for the confusion - the revised proposal focuses on coroutines, not threads. With the benefit of hindight, leaving the implementation details out of the python-ideas post was clearly a mistake, as my previous posts had been more focused on threads. I've added the full implementation details in my reply to Oscar, which will hopefully make the revised proposal clearer. The blog post goes into detail on this - it specifically takes a synchronous function, replaces it with an asynchronous coroutine using the await syntax, and then uses run_in_background() to manipulate the asynchronous version from the REPL. The main operation I use with "run_in_foreground" in the post is actually asyncio.sleep, as "run_in_foreground(asyncio.sleep(0))" was the simplest way I found to single step the event loop, and it also allows you to trivially say "run the event loop for 5 seconds", etc. Concatenating some of the example code from the post together gives this demonstration of the basic UX: >>> async def ticker(): ... for i in itertools.count(): ... print(i) ... await asyncio.sleep(1) ... >>> ticker1 = run_in_background(ticker()) >>> ticker1 <Task pending coro=<ticker() running at <stdin>:1>> >>> run_in_foreground(asyncio.sleep(5)) 0 1 2 3 4 If there isn't a coroutine currently running in the foreground, then background coroutines don't run either. All of the currently running tasks can be interrogated through the existing asyncio.Task.all_tasks() class method.
This particular spelling moreover introduces a "similarity" between foreground and background tasks that doesn't actually exist.
The concept behind the revised proposal is layering the simpler foreground/background task representational model on top of the full complexity of the asyncio implementation model. The "run_in_foreground" naming is technically a lie - what actually gets run in the foreground is the current thread's event loop. However, I think it's an acceptable and useful lie, as what it does is run the event loop in the current thread until the supplied future produces a result, which means the current thread isn't going to be doing anything other than running the event loop until the specified operation is completed. This approach *doesn't* expose the full power of asyncio and native coroutines, but it exposes a lot of it, and it should be relatively easy to grasp for anyone that's already familiar with background processes in POSIX shell environments.
The example suggests that this should really be a pair of convenience functions in collections.futures, as it does not make any use of asyncio.
While that was true of the previous proposal (which always used the executor), this new proposal only falls back to using run_in_executor if asyncio.ensure_future fails with TypeError and the supplied background task target is a callable. Regards, Nick. P.S. If anyone reading this isn't already familiar with the concept of representational models vs implementation models, then I highly recommend http://www.uxpassion.com/blog/implementation-mental-representation-models-ux... as a good introduction to the idea -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Jul 11, 2015 12:04 AM, "Nick Coghlan" <ncoghlan@gmail.com> wrote:
[...]
For what it's worth, I find it extraordinarily confusing that background tasks don't run in the background and foreground tasks don't run in the foreground. The functionality doesn't strike me as obviously unreasonable, it just doesn't at all match my expectations from encountering these words in the shell context. Maybe start_suspended(...) and unsuspend_all_until(task)? -n

On 11 July 2015 at 15:16, Nathaniel Smith <njs@pobox.com> wrote:
I think the core problem lies in trying to reduce "the bare minimum you need to know to work effectively with native coroutines" to only two concepts, when there's actually three: * scheduling a coroutine in the event loop without waiting for it * executing a call in a background thread or process * running the event loop in the foreground while waiting for one of the above two operations I'm trying to do this *without* requiring folks to actually know what a future is: I want them to be able to use asyncio *without* learning about all the moving parts first. Once they appreciate what it can do for them, *then* they may have the motivation to tackle the task of figuring out how all the pieces fit together. However, In the design I put together for the blog posts, "run_in_background" currently handles both of the first two tasks, and it likely makes more sense to split them, which would give: def run_in_foreground(task, *, loop=None): """Runs the given event loop in the current thread until the task completes If not given, *loop* defaults to the current thread's event loop Returns the result of the task. For more complex conditions, combine with asyncio.wait() To include a timeout, combine with asyncio.wait_for() """ # Convenience wrapper around get_event_loop + # ensure_future + run_until_complete ... def schedule_coroutine(target, *, loop=None): """Schedules target coroutine in the given event loop If not given, *loop* defaults to the current thread's event loop Returns the scheduled task. Use run_in_foreground to wait for the result. """ # This just means extracting the coroutine part of # asyncio.ensure_future out to its own function. ... def call_in_background(target, *, loop=None, executor=None): """Schedules and starts target callable as a background task If not given, *loop* defaults to the current thread's event loop If not given, *executor* defaults to the loop's default executor Returns the scheduled task. Use run_in_foreground to wait for the result. """ # Convenience wrapper around get_event_loop + run_in_executor ... I'll sleep on that, and if I still like that structure in the morning, I'll look at revising my coroutine posts. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 11 July 2015 at 20:17, Nick Coghlan <ncoghlan@gmail.com> wrote:
I'll sleep on that, and if I still like that structure in the morning, I'll look at revising my coroutine posts.
I've revised both of my asyncio posts to use this three part helper API to work with coroutines and the event loop from the interactive prompt: * run_in_foreground * schedule_coroutine * call_in_background I think the revised TCP echo client and server post is the better of the two descriptions, since it uses actual network IO operations as its underlying example, rather than toy background timers: http://www.curiousefficiency.org/posts/2015/07/asyncio-tcp-echo-server.html As with most of the main asyncio API, "run" in this revised setup now refers specifically to running the event loop. ("run_in_executor" is still an anomaly, which I now believe might have been better named "call_in_executor" to align with the call_soon, call_soon_threadsafe and call_later callback management APIs, rather than the run_* event loop invocation APIs) The foreground/background split is now intended to refer primarily to "main thread in the main process" (e.g. the interactive prompt, the GUI thread in a desktop application, the main server process in a network application) vs "worker threads and processes" (whether managed by the default executor, or another executor passed in specifically to "call_in_background"). This is much closer in spirit to the shell meaning. The connection that "call_in_background" has to asyncio over using concurrent.futures directly is that, just like schedule_coroutine, it's designed to be used in tandem with run_in_foreground (either standalone, or in combination with asyncio.wait, or asyncio.wait_for) to determine if the results are available yet. Both schedule_coroutine and call_in_background are deliberately restricted in the kinds of objects they accept - unlike ensure_future, schedule_coroutine will complain if given an existing future, while call_in_background will complain immediately if given something that isn't some kind of callable. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

@Nick It seems like, we are not alone in our thinking that asyncio still needs many more convenience wrappers. https://mail.python.org/pipermail/python-list/2015-August/694859.html Same conclusion as yours and mine: "I think python's non blocking I/O is far from being something useful for developers till non-async code can invoke async code transparently. Duplicating all code/libs when you realize that something not fits asyncio is not a solution and even less a pythonic solution." On 12.07.2015 04:48, Nick Coghlan wrote:

Honestly, I don't understand the issue. (Or maybe I'm too tired right now.) But I think asyncio is actually very well designed. We just have to keep in mind that code is either synchronous or asynchronous. Code that consumes too much CPU or does blocking calls does definitely not belong in an event driven system. By the way, the shell equivalent of "&" is definitely a thread, not a coroutine. And "fg" (foreground) equals "thread.join". You were also talking about a Repl. Not sure if this helps, but "ptpython", the REPL that I develop can be embedded into any application as a coroutine. There is no blocking I/O in the Repl. https://github.com/jonathanslenders/ptpython/blob/master/examples/asyncio-py... @nick: About the discussion you are referring to. For solving the producer/consumer problem, the answer is probably to use asyncio Queues (Have a look at the put and get method.) Jonathan 2015-08-11 23:26 GMT+02:00 Sven R. Kunze <srkunze@mail.de>:

On 12.08.2015 00:37, Jonathan Slenders wrote:
Nobody says asyncio is not well designed if that is what you were thinking about others who have issues with asyncio were thinking (did that make sense?).
That is exactly what people complain about. They don't like this thinking. They don't like this "all in or nothing" attitude. You may ask why? Because people want to try stuff out. But when in order to do so, they need to convert 10 mio lines of code in order to *see some results*, it just looks insane to them (looking at you, too, Python 3). The point is not to have some toy projects show-casting the abilities of asyncio, but trying it out among the lines of formerly 100% synchronous code. I am sorry but that is the world we live in, so, we need to compromise; otherwise people will disagree and not follow. This is not about doing 100% right and perfect design but taking people with you on the journey. Best, Sven

About this. I think I absolutely understand the difficulties, but in reality the "problem" is broader. Code is always written using certain paradigms. And Python allows a lot of these, so depending on the background, use case and interests of the author, he decides what paradigm to use. These days, we have functional code, reactive code, imperative, declarative, object oriented, event-driven, etc... All paradigms have different answers to questions like "Where do we keep our state?" (the variables), "Where do we do our I/O?", "how do we reuse our code?", "how do we write our logic?". Not everything is compatible. Mixing two styles can sometimes be very ugly and confusing. I guess the question is more about finding the right glue to put things together, without forcing code to fit into another paradigm.

On 12 August 2015 at 07:26, Sven R. Kunze <srkunze@mail.de> wrote:
Catching up on email after travelling last week, I want to explicitly note that don't agree with this any more - there's one method name on the event loop I think needs tweaking (for background blocking calls in another thread or process), but PEP 492 otherwise delivers all the pieces needed to make it straightforward to run the event loop as needed from synchronous code. I did a lightning talk about that at PyCon Australia, which I'll turn into another asyncio-in-your-synchronous-test-suite blog post at some point: https://www.youtube.com/watch?v=_pfJZfdwkgI Explicitly asynchronous code is as much a tool for thinking as it is an execution model, so I've come to realise that folks wanting to hide the conceptual modelling is akin to the complaints we hear from folks learning imaginary numbers for the first time, and insisting that real numbers ought to be enough for anyone. Yes, asyncio (like Twisted before it) does stretch our brains in new and interesting ways - that's the main reason it's worth having in the standard library :) Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 19.08.2015 11:24, Nick Coghlan wrote:
Nice video!
That is where I disagree. It is not about insisting that it would be enough. It is about insisting that giving it a try is not equal to rewrite 100% of your code.
That certainly is very true. During all these discussions, I really learned a lot. However, the main intention has not been changed: lowering the entry barriers.

Sven R. Kunze writes:
However, the main intention has not been changed: lowering the entry barriers.
AFAICS, the entry barriers to concurrency are quite low for the class of problems most like your examples as they were posted: just use process pools. Sure, the API is function calls or method applications rather than dedicated syntax, but that's very Pythonic: don't do in syntax what can be done in a function. (Cf. print.) And it's true that a bit of boilerplate is involved, but it seems to me that if that's an entry barrier, a "SimpleProcessPool" class that handles the boilerplate is the way to go: from multiprocessing import SimpleProcessPool as TaskPool p = TaskPool() list_of_futures = [p.start_task(task) for task in tasks] If it's suitable for threads, just change "multiprocessing" to "threading" and "SimpleProcessPool" to "SimpleThreadPool". Asyncio is less mature, but I suppose it could probably be shoehorned into this "TaskPool" framework, too. How much lower can the entry barrier get? I know you said you're willing to give up some generality to handle common cases, but (speaking for myself, maybe Andrew and Nick have a clue I don't) I don't see at all what those cases are. The ones I can think of, that are simple enough to be automatically handled by the compiler and runtime I imagine, fit the multiprocessing/message-passing model well. The complicated use cases are just hard to do concisely, safely, and efficiently in any model. Although different models make different aspects easy, none of them make everything easy, and the "optimal" choice of model is a matter of the "art of programming". I think you need to come up with one or more compelling examples of real use cases that are dramatically simplified by "fork" syntax or a similar device.

On Aug 20, 2015, at 19:51, Stephen J. Turnbull <stephen@xemacs.org> wrote:
You can already do this without adding anything to the stdlib, except the method is called submit instead of start_task. Just change the first line to: from concurrent.futures import ProcessPoolExecutor as TaskPool And if you want threads, just ThreadPoolExecutor. It also does exactly what you'd hope if you put the pool in a with statement. And if you need more power or flexibility later, it's there. So, I'm pretty sure this was completely solved back in Python 3.2.

On 11 July 2015 at 15:04, Nick Coghlan <ncoghlan@gmail.com> wrote:
I wrote a second post about this foreground/background task idea, which presents a hopefully more compelling example: setting up two TCP echo servers from the interactive prompt, and then interacting with them using asynchronous clients, including an example using run_in_background, run_in_foreground and asyncio.wait to run parallel client commands. This is all done using the main thread in the REPL, and takes advantage of the fact that it's all running in the same thread to dynamically allocate the server ports and pass that information to the demonstration clients. I believe this particular example effectively demonstrates the power of asyncio to dramatically simplify the testing of both network clients and network servers, as you don't need to mess about with synchronising across threads or processes. The full post is at http://www.curiousefficiency.org/posts/2015/07/asyncio-tcp-echo-server.html, but I'll include the examples of usage inline. The code for setting up the servers and retrieving their chosen ports looks like: >>> make_server = asyncio.start_server(handle_tcp_echo, '127.0.0.1') >>> server = run_in_foreground(make_server) >>> port = server.sockets[0].getsockname()[1] >>> make_server2 = asyncio.start_server(handle_tcp_echo, '127.0.0.1') >>> server2 = run_in_foreground(make_server2) >>> port2 = server2.sockets[0].getsockname()[1] This is an effectively synchronous operation, so it could be readily encapsulated in a normal function call for invocation from a test suite to set up local test servers, and report the port number to connect to. The code for running parallel clients looks like: >>> echo1 = run_in_background(tcp_echo_client('Hello World!', port)) >>> echo2 = run_in_background(tcp_echo_client('Hello World!', port2)) >>> run_in_foreground(asyncio.wait([echo1, echo2])) >>> echo1.result() 'Hello World!' >>> echo2.result() 'Hello World!' While I don't go into it in the post, blocking clients could also be tested in much the same way, by using run_in_background's callable support to run them as call-and-response operations through the default executor, while running the asynchronous server components in the main thread. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 10 July 2015 at 11:49, Nick Coghlan <ncoghlan@gmail.com> wrote:
Why is that better than something like: data1, data2 = asyncio.run([future1, future2]) IIUC your proposal is that run_in_background adds the tasks to an implicit global variable. Then the first call to run_in_foreground runs both tasks returning when future1 is ready. At that point it suspends future2 if incomplete? Then the second call to run_in_foreground returns immediately if future2 is ready or otherwise runs that task until complete? -- Oscar

On 10 July 2015 at 21:48, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
It just schedules them normally using asyncio.get_event_loop().create_task() (or run_in_executor if you pass in a callable). For folks that don't want to click through to the blog post (which has a full worked example using asynchronous timers), the full implementations of the two functions (with a slight tweak to run_in_background to make the executor configurable as well) are: def run_in_background(target, *, loop=None, executor=None): if loop is None: loop = asyncio.get_event_loop() try: return asyncio.ensure_future(target, loop=loop) except TypeError: pass if callable(target): return loop.run_in_executor(executor, target) raise TypeError("background task must be future, coroutine or " "callable, not {!r}".format(type(target))) def run_in_foreground(task, *, loop=None): if loop is None: loop = asyncio.get_event_loop() return loop.run_until_complete(asyncio.ensure_future(task))
No, it's all driven by the main asyncio event loop - the suggested functions are relatively thin shims designed to let people make effective use of asyncio with just the POSIX shell foreground & background task mental model, rather than having to learn how asyncio *really* works in order to benefit from it. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

As I wrote on the issue, I'm -1 on this proposal. Not only does this API encourage beginners to ignore the essential difference between synchronous functions meant to run in a thread (using synchronous I/O and pre-emptive CPU scheduling) and asyncio coroutines/tasks (which use overlapped I/O and require explicit scheduling), it also encourages avoiding the "await" primitive (formerly "yield from") in favor of a function call which cannot be used from within a coroutine/task. This particular spelling moreover introduces a "similarity" between foreground and background tasks that doesn't actually exist. The example suggests that this should really be a pair of convenience functions in collections.futures, as it does not make any use of asyncio. On Fri, Jul 10, 2015 at 12:49 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)

On 10 July 2015 at 21:51, Guido van Rossum <guido@python.org> wrote:
My apologies for the confusion - the revised proposal focuses on coroutines, not threads. With the benefit of hindight, leaving the implementation details out of the python-ideas post was clearly a mistake, as my previous posts had been more focused on threads. I've added the full implementation details in my reply to Oscar, which will hopefully make the revised proposal clearer. The blog post goes into detail on this - it specifically takes a synchronous function, replaces it with an asynchronous coroutine using the await syntax, and then uses run_in_background() to manipulate the asynchronous version from the REPL. The main operation I use with "run_in_foreground" in the post is actually asyncio.sleep, as "run_in_foreground(asyncio.sleep(0))" was the simplest way I found to single step the event loop, and it also allows you to trivially say "run the event loop for 5 seconds", etc. Concatenating some of the example code from the post together gives this demonstration of the basic UX: >>> async def ticker(): ... for i in itertools.count(): ... print(i) ... await asyncio.sleep(1) ... >>> ticker1 = run_in_background(ticker()) >>> ticker1 <Task pending coro=<ticker() running at <stdin>:1>> >>> run_in_foreground(asyncio.sleep(5)) 0 1 2 3 4 If there isn't a coroutine currently running in the foreground, then background coroutines don't run either. All of the currently running tasks can be interrogated through the existing asyncio.Task.all_tasks() class method.
This particular spelling moreover introduces a "similarity" between foreground and background tasks that doesn't actually exist.
The concept behind the revised proposal is layering the simpler foreground/background task representational model on top of the full complexity of the asyncio implementation model. The "run_in_foreground" naming is technically a lie - what actually gets run in the foreground is the current thread's event loop. However, I think it's an acceptable and useful lie, as what it does is run the event loop in the current thread until the supplied future produces a result, which means the current thread isn't going to be doing anything other than running the event loop until the specified operation is completed. This approach *doesn't* expose the full power of asyncio and native coroutines, but it exposes a lot of it, and it should be relatively easy to grasp for anyone that's already familiar with background processes in POSIX shell environments.
The example suggests that this should really be a pair of convenience functions in collections.futures, as it does not make any use of asyncio.
While that was true of the previous proposal (which always used the executor), this new proposal only falls back to using run_in_executor if asyncio.ensure_future fails with TypeError and the supplied background task target is a callable. Regards, Nick. P.S. If anyone reading this isn't already familiar with the concept of representational models vs implementation models, then I highly recommend http://www.uxpassion.com/blog/implementation-mental-representation-models-ux... as a good introduction to the idea -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Jul 11, 2015 12:04 AM, "Nick Coghlan" <ncoghlan@gmail.com> wrote:
[...]
For what it's worth, I find it extraordinarily confusing that background tasks don't run in the background and foreground tasks don't run in the foreground. The functionality doesn't strike me as obviously unreasonable, it just doesn't at all match my expectations from encountering these words in the shell context. Maybe start_suspended(...) and unsuspend_all_until(task)? -n

On 11 July 2015 at 15:16, Nathaniel Smith <njs@pobox.com> wrote:
I think the core problem lies in trying to reduce "the bare minimum you need to know to work effectively with native coroutines" to only two concepts, when there's actually three: * scheduling a coroutine in the event loop without waiting for it * executing a call in a background thread or process * running the event loop in the foreground while waiting for one of the above two operations I'm trying to do this *without* requiring folks to actually know what a future is: I want them to be able to use asyncio *without* learning about all the moving parts first. Once they appreciate what it can do for them, *then* they may have the motivation to tackle the task of figuring out how all the pieces fit together. However, In the design I put together for the blog posts, "run_in_background" currently handles both of the first two tasks, and it likely makes more sense to split them, which would give: def run_in_foreground(task, *, loop=None): """Runs the given event loop in the current thread until the task completes If not given, *loop* defaults to the current thread's event loop Returns the result of the task. For more complex conditions, combine with asyncio.wait() To include a timeout, combine with asyncio.wait_for() """ # Convenience wrapper around get_event_loop + # ensure_future + run_until_complete ... def schedule_coroutine(target, *, loop=None): """Schedules target coroutine in the given event loop If not given, *loop* defaults to the current thread's event loop Returns the scheduled task. Use run_in_foreground to wait for the result. """ # This just means extracting the coroutine part of # asyncio.ensure_future out to its own function. ... def call_in_background(target, *, loop=None, executor=None): """Schedules and starts target callable as a background task If not given, *loop* defaults to the current thread's event loop If not given, *executor* defaults to the loop's default executor Returns the scheduled task. Use run_in_foreground to wait for the result. """ # Convenience wrapper around get_event_loop + run_in_executor ... I'll sleep on that, and if I still like that structure in the morning, I'll look at revising my coroutine posts. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 11 July 2015 at 20:17, Nick Coghlan <ncoghlan@gmail.com> wrote:
I'll sleep on that, and if I still like that structure in the morning, I'll look at revising my coroutine posts.
I've revised both of my asyncio posts to use this three part helper API to work with coroutines and the event loop from the interactive prompt: * run_in_foreground * schedule_coroutine * call_in_background I think the revised TCP echo client and server post is the better of the two descriptions, since it uses actual network IO operations as its underlying example, rather than toy background timers: http://www.curiousefficiency.org/posts/2015/07/asyncio-tcp-echo-server.html As with most of the main asyncio API, "run" in this revised setup now refers specifically to running the event loop. ("run_in_executor" is still an anomaly, which I now believe might have been better named "call_in_executor" to align with the call_soon, call_soon_threadsafe and call_later callback management APIs, rather than the run_* event loop invocation APIs) The foreground/background split is now intended to refer primarily to "main thread in the main process" (e.g. the interactive prompt, the GUI thread in a desktop application, the main server process in a network application) vs "worker threads and processes" (whether managed by the default executor, or another executor passed in specifically to "call_in_background"). This is much closer in spirit to the shell meaning. The connection that "call_in_background" has to asyncio over using concurrent.futures directly is that, just like schedule_coroutine, it's designed to be used in tandem with run_in_foreground (either standalone, or in combination with asyncio.wait, or asyncio.wait_for) to determine if the results are available yet. Both schedule_coroutine and call_in_background are deliberately restricted in the kinds of objects they accept - unlike ensure_future, schedule_coroutine will complain if given an existing future, while call_in_background will complain immediately if given something that isn't some kind of callable. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

@Nick It seems like, we are not alone in our thinking that asyncio still needs many more convenience wrappers. https://mail.python.org/pipermail/python-list/2015-August/694859.html Same conclusion as yours and mine: "I think python's non blocking I/O is far from being something useful for developers till non-async code can invoke async code transparently. Duplicating all code/libs when you realize that something not fits asyncio is not a solution and even less a pythonic solution." On 12.07.2015 04:48, Nick Coghlan wrote:

Honestly, I don't understand the issue. (Or maybe I'm too tired right now.) But I think asyncio is actually very well designed. We just have to keep in mind that code is either synchronous or asynchronous. Code that consumes too much CPU or does blocking calls does definitely not belong in an event driven system. By the way, the shell equivalent of "&" is definitely a thread, not a coroutine. And "fg" (foreground) equals "thread.join". You were also talking about a Repl. Not sure if this helps, but "ptpython", the REPL that I develop can be embedded into any application as a coroutine. There is no blocking I/O in the Repl. https://github.com/jonathanslenders/ptpython/blob/master/examples/asyncio-py... @nick: About the discussion you are referring to. For solving the producer/consumer problem, the answer is probably to use asyncio Queues (Have a look at the put and get method.) Jonathan 2015-08-11 23:26 GMT+02:00 Sven R. Kunze <srkunze@mail.de>:

On 12.08.2015 00:37, Jonathan Slenders wrote:
Nobody says asyncio is not well designed if that is what you were thinking about others who have issues with asyncio were thinking (did that make sense?).
That is exactly what people complain about. They don't like this thinking. They don't like this "all in or nothing" attitude. You may ask why? Because people want to try stuff out. But when in order to do so, they need to convert 10 mio lines of code in order to *see some results*, it just looks insane to them (looking at you, too, Python 3). The point is not to have some toy projects show-casting the abilities of asyncio, but trying it out among the lines of formerly 100% synchronous code. I am sorry but that is the world we live in, so, we need to compromise; otherwise people will disagree and not follow. This is not about doing 100% right and perfect design but taking people with you on the journey. Best, Sven

About this. I think I absolutely understand the difficulties, but in reality the "problem" is broader. Code is always written using certain paradigms. And Python allows a lot of these, so depending on the background, use case and interests of the author, he decides what paradigm to use. These days, we have functional code, reactive code, imperative, declarative, object oriented, event-driven, etc... All paradigms have different answers to questions like "Where do we keep our state?" (the variables), "Where do we do our I/O?", "how do we reuse our code?", "how do we write our logic?". Not everything is compatible. Mixing two styles can sometimes be very ugly and confusing. I guess the question is more about finding the right glue to put things together, without forcing code to fit into another paradigm.

On 12 August 2015 at 07:26, Sven R. Kunze <srkunze@mail.de> wrote:
Catching up on email after travelling last week, I want to explicitly note that don't agree with this any more - there's one method name on the event loop I think needs tweaking (for background blocking calls in another thread or process), but PEP 492 otherwise delivers all the pieces needed to make it straightforward to run the event loop as needed from synchronous code. I did a lightning talk about that at PyCon Australia, which I'll turn into another asyncio-in-your-synchronous-test-suite blog post at some point: https://www.youtube.com/watch?v=_pfJZfdwkgI Explicitly asynchronous code is as much a tool for thinking as it is an execution model, so I've come to realise that folks wanting to hide the conceptual modelling is akin to the complaints we hear from folks learning imaginary numbers for the first time, and insisting that real numbers ought to be enough for anyone. Yes, asyncio (like Twisted before it) does stretch our brains in new and interesting ways - that's the main reason it's worth having in the standard library :) Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 19.08.2015 11:24, Nick Coghlan wrote:
Nice video!
That is where I disagree. It is not about insisting that it would be enough. It is about insisting that giving it a try is not equal to rewrite 100% of your code.
That certainly is very true. During all these discussions, I really learned a lot. However, the main intention has not been changed: lowering the entry barriers.

Sven R. Kunze writes:
However, the main intention has not been changed: lowering the entry barriers.
AFAICS, the entry barriers to concurrency are quite low for the class of problems most like your examples as they were posted: just use process pools. Sure, the API is function calls or method applications rather than dedicated syntax, but that's very Pythonic: don't do in syntax what can be done in a function. (Cf. print.) And it's true that a bit of boilerplate is involved, but it seems to me that if that's an entry barrier, a "SimpleProcessPool" class that handles the boilerplate is the way to go: from multiprocessing import SimpleProcessPool as TaskPool p = TaskPool() list_of_futures = [p.start_task(task) for task in tasks] If it's suitable for threads, just change "multiprocessing" to "threading" and "SimpleProcessPool" to "SimpleThreadPool". Asyncio is less mature, but I suppose it could probably be shoehorned into this "TaskPool" framework, too. How much lower can the entry barrier get? I know you said you're willing to give up some generality to handle common cases, but (speaking for myself, maybe Andrew and Nick have a clue I don't) I don't see at all what those cases are. The ones I can think of, that are simple enough to be automatically handled by the compiler and runtime I imagine, fit the multiprocessing/message-passing model well. The complicated use cases are just hard to do concisely, safely, and efficiently in any model. Although different models make different aspects easy, none of them make everything easy, and the "optimal" choice of model is a matter of the "art of programming". I think you need to come up with one or more compelling examples of real use cases that are dramatically simplified by "fork" syntax or a similar device.

On Aug 20, 2015, at 19:51, Stephen J. Turnbull <stephen@xemacs.org> wrote:
You can already do this without adding anything to the stdlib, except the method is called submit instead of start_task. Just change the first line to: from concurrent.futures import ProcessPoolExecutor as TaskPool And if you want threads, just ThreadPoolExecutor. It also does exactly what you'd hope if you put the pool in a with statement. And if you need more power or flexibility later, it's there. So, I'm pretty sure this was completely solved back in Python 3.2.

On 11 July 2015 at 15:04, Nick Coghlan <ncoghlan@gmail.com> wrote:
I wrote a second post about this foreground/background task idea, which presents a hopefully more compelling example: setting up two TCP echo servers from the interactive prompt, and then interacting with them using asynchronous clients, including an example using run_in_background, run_in_foreground and asyncio.wait to run parallel client commands. This is all done using the main thread in the REPL, and takes advantage of the fact that it's all running in the same thread to dynamically allocate the server ports and pass that information to the demonstration clients. I believe this particular example effectively demonstrates the power of asyncio to dramatically simplify the testing of both network clients and network servers, as you don't need to mess about with synchronising across threads or processes. The full post is at http://www.curiousefficiency.org/posts/2015/07/asyncio-tcp-echo-server.html, but I'll include the examples of usage inline. The code for setting up the servers and retrieving their chosen ports looks like: >>> make_server = asyncio.start_server(handle_tcp_echo, '127.0.0.1') >>> server = run_in_foreground(make_server) >>> port = server.sockets[0].getsockname()[1] >>> make_server2 = asyncio.start_server(handle_tcp_echo, '127.0.0.1') >>> server2 = run_in_foreground(make_server2) >>> port2 = server2.sockets[0].getsockname()[1] This is an effectively synchronous operation, so it could be readily encapsulated in a normal function call for invocation from a test suite to set up local test servers, and report the port number to connect to. The code for running parallel clients looks like: >>> echo1 = run_in_background(tcp_echo_client('Hello World!', port)) >>> echo2 = run_in_background(tcp_echo_client('Hello World!', port2)) >>> run_in_foreground(asyncio.wait([echo1, echo2])) >>> echo1.result() 'Hello World!' >>> echo2.result() 'Hello World!' While I don't go into it in the post, blocking clients could also be tested in much the same way, by using run_in_background's callable support to run them as call-and-response operations through the default executor, while running the asynchronous server components in the main thread. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (8)
-
Andrew Barnert
-
Guido van Rossum
-
Jonathan Slenders
-
Nathaniel Smith
-
Nick Coghlan
-
Oscar Benjamin
-
Stephen J. Turnbull
-
Sven R. Kunze