Mailman 3 - Async-sig

async/sync library reusage
by manuel miranda 12 Jun '17

12 Jun '17

Hello everyone, After using asyncio for a while, I'm struggling to find information about how to support both synchronous and asynchronous use cases for the same library. I.e. imagine you have a package for http requests and you want to give the user the choice to use a synchronous or an asynchronous interface. Right now the approach the community is following is creating separate libraries one for each version. This is far from ideal for several reasons, some I can think of: - Code duplication, most of the functionality is the same in both libraries, only difference is the sync/async behaviors - Some new async libraries lack functionality compared to their sync siblings. Others will introduce bugs that the sync version already solved long ago, etc. - Different interfaces for the user for the same exact functionality. In summary, in some cases it looks like reinventing the wheel. So now comes the question, is there any documentation, guide on what would be best practice supporting this kind of duality? I've been playing a bit with that on my own but I really don't know if I'm doing something stupid or not. Simple example: """ import asyncio class MyConnector: @classmethod async def get(cls, key): return key class AsyncClient: async def get(self, key): return await MyConnector.get(key) class SyncClient: def __init__(self): self.loop = asyncio.get_event_loop() def get(self, key): return self.loop.run_until_complete(MyConnector.get(key)) def sync_call(): client = SyncClient() print(client.get("sync_key")) async def async_call(): client = AsyncClient() print(await client.get("async_key")) if __name__ == "__main__": loop = asyncio.get_event_loop() loop.run_until_complete(async_call()) sync_call() """ This is in case the underlying connector is asynchronous already. If its synchronous and you want to support both modes, you have to rewrite the IO interactions of MyConnector into a new AsyncMyConnector to support asyncio and then use one or the other accordingly in the upper classes. Am I doing it right or there is another better/alternative way? Thanks for your time, Manuel

10 28

PyCon US 2017 open space on Saturday sometime after lunch
by Brett Cannon 20 May '17

20 May '17

I have not picked a time yet and I can't book it until tomorrow, but the plan is sometime after lunch on Saturday I will book an async open space.

4 4

[ann] sphinxcontrib-trio: make sphinx better at documenting functions/methods, esp. for async/await code
by Nathaniel Smith 12 May '17

12 May '17

Hi all, I just released a new package, sphinxcontrib-trio: https://sphinxcontrib-trio.readthedocs.io/ It makes it easier to document many kinds of functions/methods in sphinx, including async functions, abstract methods, generators, etc. I originally wrote it for the trio [1] project, hence the name, but don't let that put you off -- there's nothing about it that's specific to trio, or even to async/await (except that projects that use async/await *really need* an extension like this). Really I think this extension ought to be a standard feature of sphinx. But in the mean time, it's pretty handy. -n [1] https://trio.readthedocs.io -- Nathaniel J. Smith -- https://vorpus.org

3 5

Control-C handling in Python and Trio
by Nathaniel Smith 23 Apr '17

23 Apr '17

Hi async-sig, Some of you have probably already seen this, but in case it's of interest, I just posted some discussion of signal handling and how I approach it in trio: https://vorpus.org/blog/control-c-handling-in-python-and-trio/ If nothing else at least it's the signal handling documentation that I wish I had when I started :-). -n -- Nathaniel J. Smith -- https://vorpus.org

3 2

Trio: async I/O for humans and snake people
by Nathaniel Smith 15 Mar '17

15 Mar '17

Hi all, I'd like to announce the initial release of Trio, a new permissively-licensed async I/O library for Python 3.5+. Blog post with more details: https://vorpus.org/blog/announcing-trio/ Or you can jump straight to the repo: https://github.com/python-trio/trio/ Cheers, -n -- Nathaniel J. Smith -- https://vorpus.org

2 2

Debugging destroyed, pending tasks
by Mark E. Haase 02 Feb '17

02 Feb '17

I'm still climbing up the asyncio learning curve. I'm building an application on top of asyncio, and one of my goals is that when the application is interrupted (e.g. SIGINT), it should carefully clean up all running tasks before closing the event loop. Some tasks should be cancelled right away, but other tasks should be allowed to complete. As an example of the latter, I may have several file download tasks that I want to finish before closing the event loop. (If I get a 2nd SIGINT, then I close the loop immediately and exit.) I have a hard time debugging the "Task was destroyed..." errors. For example: Task was destroyed but it is pending! task: <Task pending coro=<sleep() done, defined at /usr/lib/python3.5/asyncio/tasks.py:502> wait_for=<Future pending cb=[Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:414]> I'm pretty sure that I understand why I get this error: this is a task (e.g. created with asyncio.ensure_future()) that was never awaited. To prevent this error, I should await the task (or call its result() method, right?). But how do I figure out where the task was created? I have PYTHONASYNCIODEBUG=1 in my environment, and I see the "defined at" text, which is helpful if the coroutine is in my own code, but for asyncio.sleep(), it's not very helpful. Is it possible to see where this task was created, e.g. the line containing asyncio.ensure_future(asyncio.sleep(...)) ? If not, is this a capability that could be added to debug mode? E.g. ensure_future would add its own stack trace to the task object so that it could be retrieved later when a pending task is destroyed. In my current scenario, I have few enough tasks that there's only one place I think this particular error could come from, but as my application grows, this will get harder and harder to debug. Thoughts?

2 2

Re: [Async-sig] Some thoughts on asynchronous API design in a post-async/await world
by Dima Tisnek 02 Feb '17

02 Feb '17

Hi list, sorry for being a bit late, but I only discovered that post recently... I've a couple concerns: 1. update now 3.6 is out, the post should be rewritten or new one made, mainly because this post if what inquisitive minds find... 2. asyncio for whom? I've presented asyncio tutorial of sourts at a local Python meetup. I've noticed that asyncio documentation is pretty hard to understand. There are many concepts and a ton of caveats. I now finally understand why, and I think it comes down to asyncio being 2 distinct things: a. asyncio is a standard library meant to be used as a framework by other, higher-level, 3rd party libraries that are still written by very smart developers, and b. asyncio is a way for average joe developers to write async code. I used to think I was in cat.a., but clearly I'm more of a cat.b. dev :( I really with there was decent advice how to use asyncio/aiohttp/pytest-asyncio/curio? etc. Something along the lines of: * if you want fire-and-forget (e.g. async logging), do xyz1 * if you want a worker task, do xyz2 * if you want a synchronisation point, do xyz3 * if you want ... I'd love these to be reflected in the API that Python advertises. For example, I'd rather think of a Future() as a thing that provides a result, meaning it's commonly shared by multiple tasks, meaning it would by default be shielded. Should a Future() die if it's garbage-collected, like a generator or should it conitnue? These are areas where Python should provide best practice. Task cancellation (just like thread cancellation) is by definition(?) arcane, and I'd argue average developer should be advised against... I for one, still can't make up my mind what's better -- to cancel bottom or top frame when task being cancelled awaits something that awaits something else, etc... Should provision to block cancellation even exist? What would be the best quorum for these discussions? Cheers, d. P.S. I'm trying to avoid "coroutine" nomenclature on purpose

1 0

A modest terminological proposal
by Nathaniel Smith 28 Dec '16

28 Dec '16

I've found that when talking about async/await stuff recently, I've mostly dropped the word "coroutine" from my vocabulary and replaced it with "async function". I'm writing to suggest that we might want to make this switch as a community, and do it now, before the next 10x increase in the async/await userbase. Rationale: *Consistency*: If the only two words to keep track of were function vs coroutine, that would be one thing. But here's a list of syntax constructs we have now, and how we talk about them in English: with -> applies a *context manager* to a *with block* async with -> applies an *async context manager* to an *async with block* for -> does a *loop* over an *iterator* async for -> does an *async loop* over an *async iterator* def + yield -> makes a *generator* async def + yield -> makes an *async generator* [... for ...] -> is a *comprehension* [... async for ...] -> is an *async comprehension* def -> creates a *function* async def -> creates a *coroutine* One of these is not like the others. Why? It's a pointless toe-stub for newcomers. *Expressivity:* Once I started talking about async functions, I've found my vocabulary suddenly expanding with useful new bonus terms, like "async method" (= an async def inside a class body), or "async callable" (= any callable that returns an awaitable, emphasizing the duck type rather than the concrete type). *Approachability*: What is a "coroutine" anyway? Is it like a monad? Am I going to have to learn about endofunctors and continuations next? Of course we've all gotten used to the term now so we don't notice, but for a new user this is an obviously jargony "if you don't have a PhD then this isn't *for* you" kind of word. It's not hospitable. Plus, it emphasizes the implementor under-the-hood point of view, not the user point of view. (Which is totally understandable! Because we're just now transitioning from the internal implementors hacking things out phase to the broad general user base phase, but that's why we should fix this now.) For an end-user, especially one starting out, they don't need to know or care about all the coroutine stuff that async/await is using to make the magic happen, they can learn that later. To get started, all you need to know is that you do async stuff by using async functions and sticking an 'await' onto your I/O calls. A lot of work went into abstracting away that magic and giving it a friendly Python syntax; the English syntax should follow suit. *Accuracy*: Speaking of jargon, the term "coroutine" *is* an existing piece of jargon in computer science, and our term and their term don't quite match up. This isn't a huge deal, but it's unnecessary confusion. According to Wikipedia I guess technically we don't even have "true" coroutines, just "semicoroutines"? And every generator has just as much claim to being a coroutine-in-the-CS-sense as an async function does, but when we say coroutine we don't mean generators. (Except when we do.) This confusion might partly reflect the somewhat confusing transition from 'yield from' to async/await, as demonstrated by the official doc's somewhat confusing definition of "coroutine": https://docs.python.org/3/library/asyncio-task.html#coroutine But going forward, yield-from-based coroutines and @asyncio.coroutine and all that are quickly becoming a historical curiosity, and most of the time what we really want to be talking about are just async functions. *So:* I'm not suggesting we modify Python itself to rename types.coroutine or the .cr_* fields or anything, that's all expert internal stuff (possible exception: the __repr__ for async functions). But I do propose that we all agree to do a search/replace for s/coroutine/async function/g on our end-user docs and when generally talking about these things, and announce that we're doing so. It's a small thing and a bit of work, but if there's a general agreement to do it in a coordinated way then I think it could be pretty straightforward transition that will pay dividends for a long time. -n -- Nathaniel J. Smith -- https://vorpus.org

5 10

Re: [Async-sig] A modest terminological proposal
by Thomas Nyberg 30 Nov '16

30 Nov '16

This is in reply to this message in the archives: https://mail.python.org/pipermail/async-sig/2016-October/000141.html Nathaniel Smith: > I've found that when talking about async/await stuff recently, I've > mostly dropped the word "coroutine" from my vocabulary and replaced it > with "async function". I'm writing to suggest that we might want > to > make this switch as a community, and do it now, before the next 10x > increase in the async/await userbase. For what it's worth, I 100% agree. > *Accuracy*: Speaking of jargon, the term "coroutine" *is* an existing > piece of jargon in computer science, and our term and their term don't > quite match up. This isn't a huge deal, but it's unnecessary > confusion. According to Wikipedia I guess technically we don't even > have "true" coroutines, just "semicoroutines"? And every generator has > just as much claim to being a coroutine-in-the-CS-sense as an async > function does, but when we say coroutine we don't mean generators. > (Except when we do.) This confusion might partly reflect the somewhat > confusing transition from 'yield from' to async/await, as demonstrated > by the official doc's somewhat confusing definition of "coroutine": I've personally been very confused lately by python's terminology. I've always understood coroutines to have nothing a priori to do with event loops, scheduling. For example, I've understood coroutines as being objects that allow the following kind of flow (taken from https://en.wikipedia.org/wiki/Coroutine#Comparison_with_subroutines): var q := new queue coroutine produce loop while q is not full create some new items add the items to q yield to consume coroutine consume loop while q is not empty remove some items from q use the items yield to produce Python has basically supported this flow since 2.x (apart from the terminology issue of semi-coroutines mentioned above which I feel is relatively minor). This big point though is that being scheduled or asyncronous is kind of a separate property. Posix threads are implemented by feeding it a routine to start things off just like much of python's asyncronous stuff is built around coroutines, but I don't feel like the term "coroutines" should be merged into both just as routines aren't automatically tied to threads. Personally I think if anything the best possible comparison else where in computer science are the haskell thunks which delay execution until needed. That's the first thing I think about when I see how curio is setup for example. I recognize that it is very unlikely that this will change (hey language meaning changes over time), but I felt like I should reply after finding this thread because I personally have been very confused by the terminology used by python. (Personally I've been using coroutines by my definition for years using yield/send.) Thanks a lot for the write-up though. You saved my sanity a little bit... Cheers, Thomas

2 1

[ANN] async_generator v1.2 released
by Nathaniel Smith 26 Nov '16

26 Nov '16

Hi all, I just released v1.2 of my async_generator package: https://pypi.org/pypi/async_generator/ This package makes it easy to write PEP 525-style "async generators", even on Python 3.5. It looks like: from async_generator import async_generator, yield_, yield_from_ @async_generator async def parse_json_separated_newlines_from_network(streamreader): async for line in streamreader: await yield_(json.loads(line)) Starting in v1.1 (released last week but not announced widely), the generators are now highly compatible with 3.6's native async generators, including full support for athrow / asend / aclose. They also fully support "yield from", which is otherwise unavailable even on Python 3.6. v1.2 fixed some edge cases in "yield from" -- but the headline feature is that we're now *so* compatible with 3.6's native async generators that you can now use our yield from support directly inside a native generator: from async_generator import yield_from_ async def f(): yield 2 yield 3 # yields [1, 2, 3, 4] async def g(): yield 1 await yield_from_(f()) yield 4 Have fun, -n -- Nathaniel J. Smith -- https://vorpus.org

2 16