Mailman 3 November 2016 - Async-sig

A modest terminological proposal
by Nathaniel Smith 28 Dec '16

28 Dec '16

I've found that when talking about async/await stuff recently, I've mostly dropped the word "coroutine" from my vocabulary and replaced it with "async function". I'm writing to suggest that we might want to make this switch as a community, and do it now, before the next 10x increase in the async/await userbase. Rationale: *Consistency*: If the only two words to keep track of were function vs coroutine, that would be one thing. But here's a list of syntax constructs we have now, and how we talk about them in English: with -> applies a *context manager* to a *with block* async with -> applies an *async context manager* to an *async with block* for -> does a *loop* over an *iterator* async for -> does an *async loop* over an *async iterator* def + yield -> makes a *generator* async def + yield -> makes an *async generator* [... for ...] -> is a *comprehension* [... async for ...] -> is an *async comprehension* def -> creates a *function* async def -> creates a *coroutine* One of these is not like the others. Why? It's a pointless toe-stub for newcomers. *Expressivity:* Once I started talking about async functions, I've found my vocabulary suddenly expanding with useful new bonus terms, like "async method" (= an async def inside a class body), or "async callable" (= any callable that returns an awaitable, emphasizing the duck type rather than the concrete type). *Approachability*: What is a "coroutine" anyway? Is it like a monad? Am I going to have to learn about endofunctors and continuations next? Of course we've all gotten used to the term now so we don't notice, but for a new user this is an obviously jargony "if you don't have a PhD then this isn't *for* you" kind of word. It's not hospitable. Plus, it emphasizes the implementor under-the-hood point of view, not the user point of view. (Which is totally understandable! Because we're just now transitioning from the internal implementors hacking things out phase to the broad general user base phase, but that's why we should fix this now.) For an end-user, especially one starting out, they don't need to know or care about all the coroutine stuff that async/await is using to make the magic happen, they can learn that later. To get started, all you need to know is that you do async stuff by using async functions and sticking an 'await' onto your I/O calls. A lot of work went into abstracting away that magic and giving it a friendly Python syntax; the English syntax should follow suit. *Accuracy*: Speaking of jargon, the term "coroutine" *is* an existing piece of jargon in computer science, and our term and their term don't quite match up. This isn't a huge deal, but it's unnecessary confusion. According to Wikipedia I guess technically we don't even have "true" coroutines, just "semicoroutines"? And every generator has just as much claim to being a coroutine-in-the-CS-sense as an async function does, but when we say coroutine we don't mean generators. (Except when we do.) This confusion might partly reflect the somewhat confusing transition from 'yield from' to async/await, as demonstrated by the official doc's somewhat confusing definition of "coroutine": https://docs.python.org/3/library/asyncio-task.html#coroutine But going forward, yield-from-based coroutines and @asyncio.coroutine and all that are quickly becoming a historical curiosity, and most of the time what we really want to be talking about are just async functions. *So:* I'm not suggesting we modify Python itself to rename types.coroutine or the .cr_* fields or anything, that's all expert internal stuff (possible exception: the __repr__ for async functions). But I do propose that we all agree to do a search/replace for s/coroutine/async function/g on our end-user docs and when generally talking about these things, and announce that we're doing so. It's a small thing and a bit of work, but if there's a general agreement to do it in a coordinated way then I think it could be pretty straightforward transition that will pay dividends for a long time. -n -- Nathaniel J. Smith -- https://vorpus.org

5 10

Re: [Async-sig] A modest terminological proposal
by Thomas Nyberg 30 Nov '16

30 Nov '16

This is in reply to this message in the archives: https://mail.python.org/pipermail/async-sig/2016-October/000141.html Nathaniel Smith: > I've found that when talking about async/await stuff recently, I've > mostly dropped the word "coroutine" from my vocabulary and replaced it > with "async function". I'm writing to suggest that we might want > to > make this switch as a community, and do it now, before the next 10x > increase in the async/await userbase. For what it's worth, I 100% agree. > *Accuracy*: Speaking of jargon, the term "coroutine" *is* an existing > piece of jargon in computer science, and our term and their term don't > quite match up. This isn't a huge deal, but it's unnecessary > confusion. According to Wikipedia I guess technically we don't even > have "true" coroutines, just "semicoroutines"? And every generator has > just as much claim to being a coroutine-in-the-CS-sense as an async > function does, but when we say coroutine we don't mean generators. > (Except when we do.) This confusion might partly reflect the somewhat > confusing transition from 'yield from' to async/await, as demonstrated > by the official doc's somewhat confusing definition of "coroutine": I've personally been very confused lately by python's terminology. I've always understood coroutines to have nothing a priori to do with event loops, scheduling. For example, I've understood coroutines as being objects that allow the following kind of flow (taken from https://en.wikipedia.org/wiki/Coroutine#Comparison_with_subroutines): var q := new queue coroutine produce loop while q is not full create some new items add the items to q yield to consume coroutine consume loop while q is not empty remove some items from q use the items yield to produce Python has basically supported this flow since 2.x (apart from the terminology issue of semi-coroutines mentioned above which I feel is relatively minor). This big point though is that being scheduled or asyncronous is kind of a separate property. Posix threads are implemented by feeding it a routine to start things off just like much of python's asyncronous stuff is built around coroutines, but I don't feel like the term "coroutines" should be merged into both just as routines aren't automatically tied to threads. Personally I think if anything the best possible comparison else where in computer science are the haskell thunks which delay execution until needed. That's the first thing I think about when I see how curio is setup for example. I recognize that it is very unlikely that this will change (hey language meaning changes over time), but I felt like I should reply after finding this thread because I personally have been very confused by the terminology used by python. (Personally I've been using coroutines by my definition for years using yield/send.) Thanks a lot for the write-up though. You saved my sanity a little bit... Cheers, Thomas

2 1

[ANN] async_generator v1.2 released
by Nathaniel Smith 26 Nov '16

26 Nov '16

Hi all, I just released v1.2 of my async_generator package: https://pypi.org/pypi/async_generator/ This package makes it easy to write PEP 525-style "async generators", even on Python 3.5. It looks like: from async_generator import async_generator, yield_, yield_from_ @async_generator async def parse_json_separated_newlines_from_network(streamreader): async for line in streamreader: await yield_(json.loads(line)) Starting in v1.1 (released last week but not announced widely), the generators are now highly compatible with 3.6's native async generators, including full support for athrow / asend / aclose. They also fully support "yield from", which is otherwise unavailable even on Python 3.6. v1.2 fixed some edge cases in "yield from" -- but the headline feature is that we're now *so* compatible with 3.6's native async generators that you can now use our yield from support directly inside a native generator: from async_generator import yield_from_ async def f(): yield 2 yield 3 # yields [1, 2, 3, 4] async def g(): yield 1 await yield_from_(f()) yield 4 Have fun, -n -- Nathaniel J. Smith -- https://vorpus.org

2 16

A tasks list and run every element again when finish
by David Pineda 25 Nov '16

25 Nov '16

Hello. I'm using the asyncio module. I have a list with different task, every tasks has different time to be done. In a figure: fn 1 -> -> -> -> fn 2 --> --> --> --> --> . . . fn N -----> -----> -----> -----> So i need to repeat every task when its done. I tried to use 'while' but while have to complete all list first to repeat the list, this is not useful for me. Y ask you, fellows, about if there are a way to do that. In particular these tasks are with socket and communication methods. I ve tried with multiprocess but the broke because are different loop events. Yes, i tryyo mix asyncio with multiprocess, but in this case didn't work I will be very thankfull of you. Best Regards! -- David A. Pineda Osorio F:+56 9 82142267 Ingeniero Civil Electricista Universidad de Chile

1 1

aiostream release
by Vincent Michel 21 Nov '16

21 Nov '16

Hi all, I just released aiostream, a collection of generator-based operators for asynchronous iteration. - GitHub: https://github.com/vxgmichel/aiostream - PyPI: https://pypi.org/project/aiostream - Docs: http://pythonhosted.org/aiostream It can be seen as an asynchronous version of itertools, although some aspects are slightly different. The stream operators rely heavily on asynchronous generators (PEP 525), so python 3.6 is required. The project is fully tested, and the documentation provides a couple of examples: http://pythonhosted.org/aiostream/examples.html Cheers, /Vincent

1 0

Re: [Async-sig] [python-tulip] Python 3.6b2 will have C implemented Future
by Ben Darnell 19 Nov '16

19 Nov '16

[+async-sig(a)python.org <async-sig(a)python.org>, which is the new home for these kinds of discussions] Tornado's tests are now failing on nightly with "TypeError: can't send non-None value to a FutureIter": https://travis-ci.org/tornadoweb/tornado/jobs/167252979 -Ben On Tue, Oct 11, 2016 at 3:55 PM INADA Naoki <songofacandy(a)gmail.com> wrote: > > If you have asyncio based project, and it uses Travis-CI, > > > please add "nightly" to your .travis.cnf [2]. > > > > > > [2] > > > > https://docs.travis-ci.com/user/languages/python/#Choosing-Python-versions-… > > > > Travis changed the "nightly" version to 3.7 > > Now "3.6-dev" is for Python 3.6beta (Still 3.6b1, it may be upgraded soon). > > > > -- > > INADA Naoki <songofacandy(a)gmail.com> > >

4 8

Adding asyncio.run() function in Python 3.6
by Yury Selivanov 18 Nov '16

18 Nov '16

One of the remaining problems with the event loop in asyncio is bootstrapping/finalizing asyncio programs. Currently, there are two different scenarios: [1] Running a coroutine: async def main(): # your program loop = asyncio.get_event_loop() try: loop.run_until_complete(main()) finally: loop.close() [2] Running a server: loop = asyncio.get_event_loop() srv = loop.run_until_complete( loop.create_server(…)) try: loop.run_forever() finally: try: srv.close() loop.run_until_complete(srv.wait_closed()) finally: loop.close() Both cases are *incomplete*: they don’t do correct finalization of asynchronous generators. To do that we’ll need to add another 1-3 lines of code (extra try-finally). This manual approach is really painful: * It makes bootstrapping asyncio code unnecessarily hard. * It makes the documentation hard to follow. And we can’t restructure the docs to cover the loop only in the advanced section. * Most of the people will never know about `loop.shutdown_asyncgen` coroutine. * We don’t have a place to add debug code to let people know that their asyncio program didn’t clean all resources properly (a lot of unordered warnings will be spit out instead). In https://github.com/python/asyncio/pull/465 I propose to add a new function to asyncio in Python 3.6: asyncio.run(). The function can either accept a coroutine, which solves [1]: async def main(): # your program asyncio.run(main()) Or it can accept an asynchronous generator, which solves [2]: async def main(): srv = await loop.create_server(…)) try: yield # let the loop run forever finally: srv.close() await srv.wait_closed() asyncio.run(main()) asyncio.run() solves the following: * An easy way to start an asyncio program that properly takes care of loop instantiation and finalization. * It looks much better in the docs. With asyncio.run people don’t need to care about the loop at all, most probably will never use it. * Easier to experiment with asyncio in REPL. * The loop and asynchronous generators will be cleaned up properly. * We can add robust debug output to the function, listing the unclosed tasks, servers, connections, asynchronous generators etc, helping people with the cleanup logic. * Later, if we need to add more cleanup code to asyncio, we will have a function to add the logic to. I feel that we should add this to asyncio. One of the arguments against that, is that overloading asyncio.run to accept both coroutines and asynchronous generators makes the API more complex. If that’s really the case, we can add two functions: asyncio.run(coro) and asyncio.run_forever(async_generator). Also take a look at https://github.com/python/asyncio/pull/465. Thanks, Yury

4 4

Some thoughts on asynchronous API design in a post-async/await world
by Nathaniel Smith 10 Nov '16

10 Nov '16

I just posted a long blog/essay that's probably of interest to folks here: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-… The short version: I think curio something important to teach us; I tried to figure out what that is and how we can learn from it. -n -- Nathaniel J. Smith -- https://vorpus.org

9 22

sans io implementation
by MultiSosnooley . 07 Nov '16

07 Nov '16

Hi. I'm trying to make sans-io library. 1. Protocol works over http, so I've just passing Request object with method, url, data, etc. fields to user and receive Response object (json in this case). 2. I'm using generators for simplification of holding state of multirequest actions. Something like this: def login(): response = yield Request(...) # check server state # update state with response response = yield Request(...) # actually login yield None def act(generator): response = None while True: request = generator.send(response) if request is None: return response response = # do io act(login()) This solve problem multistep actions. The downside is that all your protocol functions, event if they have only one request and don't need response at all, must be generators. Is it ok to send just abstract Request and receive abstract Response (not just bytes)? Is there a better solution for multirequest actions?

2 2

Status of asyncio.readthedocs.io ?
by Ludovic Gasc 01 Nov '16

01 Nov '16

Hi everybody, For me, http://asyncio.readthedocs.io/ is enough stable now to promote the project. Ok for everybody, or you want to add content or change the structure ? BTW, thanks to all contributors, especially Victor Stinner and Mike Müller from Python-Academy: https://github.com/asyncio-doc/asyncio-doc/graphs/contributors Have a nice week. PS: No answer before Sunday = approved by people in the mailing-lists ;-) -- Ludovic Gasc (GMLudo) http://www.gmludo.eu/

1 1