Big +1 here, and an implicit -1 on the other suggestions. asyncio != async/await Cheers, Steve Top-posted from my Windows Phone -----Original Message----- From: "Nathaniel Smith" Sent: ‎2/‎28/‎2017 23:19 To: "Nick Coghlan" Cc: "python-dev@python.org" Subject: Re: [Python-Dev] API design: where to add async variants ofexisting stdlib APIs? On Tue, Feb 28, 2017 at 9:42 PM, Nick Coghlan wrote:

Short version:

- there are some reasonable requests for async variants of contextlib APIs for 3.7 - prompted by Raymond, I'm thinking it actually makes more sense to add these in a new `asyncio.contextlib` module than it does to add them directly to the existing module - would anyone object strongly to my asking authors of the affected PRs to take their changes in that direction?

IMHO this is a good idea *iff* the new APIs really are bound to asyncio, rather than being generic across all uses of async/await. It sounds like that's not the case, though? There are definitely use cases for acontextmanager in programs that don't use asyncio at all (but rather use twisted, curio, ...). Guido's even suggested that he'd like to see a PEP for an "asyncio2" within the 3.7/3.8 timeframe: https://mail.python.org/pipermail/async-sig/2016-November/000175.html asyncio is an important use case for async/await, but it's definitely not the only one. In cases where it's possible to write generic machinery in terms of async/await semantics, without assuming any particular coroutine runner's semantics, then I strongly urge you to do so.

Longer version:

There are a couple of open issues requesting async variants of some contextlib APIs (asynccontextmanager and AsyncExitStack). I'm inclined to accept both of them, but Raymond raised a good question regarding our general design philosophy for these kinds of additions: would it make more sense to put these in an "asyncio.contextlib" module than it would to add them directly to contextlib itself?

The main advantage I see to the idea is that if someone proposed adding an "asyncio" dependency to contextlib, I'd say no. For the existing asynccontextmanager PR, I even said no to adding that dependency to the standard contextlib test suite, and instead asked that the new tests be moved out to a separate file, so the existing tests could continue to run even if asyncio was unavailable for some reason.

asyncio is a stable, non-provisional part of the standard library; it's not going anywhere. Personally I wouldn't be bothered about depending on it for tests. (My async_generator library is in a similar position: it isn't tied to any particular framework, and I don't even use asyncio myself, but the test suite depends on asyncio because hey, whatever, everyone already has it and it plays the role of generic coroutine runner as well as anything else does.) OTOH if you don't need to do any I/O then it's actually pretty easy to write a trivial coroutine runner. I think something like this should be sufficient to write any test you might want: @types.coroutine def send_me(value): return yield ("value", value) @types.coroutine def throw_me(exc): yield ("error", exc) async def yield_briefly(): await send_me(None) def run(async_fn, *args, **kwargs): coro = async_fn(*args, **kwargs) next_msg = ("value", None) try: while True: if next_msg[0] == "value": next_msg = coro.send(next_msg[1]) else: next_msg = coro.throw(next_msg[1]) except StopIteration as exc: return exc.value

While rejecting the idea of an asyncio dependency isn't a problem for asyncontextmanager specifically (it's low level enough for it not to matter), it's likely to be more of a concern for the AsyncExitStack API, where the "asyncio.iscoroutinefunction" introspection API is likely to be quite helpful, as are other APIs like `asyncio.ensure_future()`.

FYI FWIW, every time I've tried to use iscoroutinefunction so far I've ended up regretting it and ripping it out again :-). The problem is that people will do things like apply a decorator to a coroutine function, and get a wrapped function that returns a coroutine object and which is interchangeable with a real coroutine function in every way except that iscoroutinefunction returns False. And there's no collections.abc.CoroutineFunction (not sure how that would even work). Better to just call the thing and then do an isinstance(..., collections.abc.Coroutine) on the return value. I haven't had a reason to try porting ExitStack to handle async context managers yet, so I can't speak to it beyond that :-). -n -- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org

On Mar 1, 2017, at 8:47 AM, Yury Selivanov wrote:

...

IMHO this is a good idea*iff* the new APIs really are bound to asyncio, rather than being generic across all uses of async/await.

I agree. There is no need to make asynccontextmanager and AsyncExitStack dependent on asyncio or specific to asyncio.

They should both stay framework agnostic (use only protocols defined by PEP 492 and PEP 525) and both shouldn't be put into asyncio package.

Of course, it makes sense that anything not specific to asyncio should go outside of asyncio. What I'm more concerned about is what the other places actually are. Rather than putting async variants of everything sprinkled all over the standard library, I suggest collecting them all together, perhaps in a new asynctools module. Raymond

Last week I had to download a CSV from an FTP and push any update on it using websocket so asyncio was a natural fit and the network part went well. The surprise was that the CSV part would not work as expected. Usually I read csv doing: import csv file_like_object = csv_crawler.get_file() for row in csv.DictReader(file_like_object) But it didn't work because file_like_object.read() was a coroutine which the csv module doesn't handle. So I had to do: import csv import io raw_bytes = await stream.read(10000000) wrapped_bytes = io.BytesIO(raw_bytes) text = io.TextIOWrapper(wrapped_bytes, encoding=encoding, errors='replace') for i, row in enumerate(csv.DictReader(text)): Turns out I used asyncio a bit, and I now the stdlib, the io AIP, etc. But for somebody that doesn't, it's not very easy to figure out. Plus it's not as elegant as traditional Python. Not to mention it loads the entire CSV in memory. So I wondered if I could fix the csv module so it accept async. But the question arised. Where should I put it ? - Create AsyncDictReader and AsyncReader ? - Add inspect.iscoroutine calls widh it in the regular Readers and some __aiter__ and __aenter__ ? - add a csv.async namespace ? What API design are we recommanding for expose both sync and async behaviors ? Le 07/03/2017 à 03:08, Guido van Rossum a écrit :

On Mon, Mar 6, 2017 at 5:57 PM, Raymond Hettinger mailto:raymond.hettinger@gmail.com> wrote:

Of course, it makes sense that anything not specific to asyncio should go outside of asyncio.

What I'm more concerned about is what the other places actually are. Rather than putting async variants of everything sprinkled all over the standard library, I suggest collecting them all together, perhaps in a new asynctools module.

That's a tough design choice. I think neither extreme is particularly attractive -- having everything in an asynctools package might also bundle together thing that are entirely unrelated. In the extreme it would be like proposing that all metaclasses should go in a new "metaclasstools" package. I think we did a reasonable job with ABCs: core support goes in abc.py, support for collections ABCs goes into the collections package (in a submodule), and other packages and modules sometimes define ABCs for their own users.

Also, in some cases I expect we'll have to create a whole new module instead of updating some ancient piece of code with newfangled async variants to its outdated APIs.

-- --Guido van Rossum (python.org/~guido http://python.org/~guido)

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/desmoulinmichel%40gmail.c...

On 03/07/2017 09:41 AM, Brett Cannon wrote:

I don't think a common practice has bubbled up yet for when there's both synchronous and asynchronous versions of an API (closest I have seen is appending an "a" to the async version but that just looks like a spelling mistake to me most of the time). This is why the question of whether separate modules are a better idea is coming up.

I'm undoubtedly going to show my ignorance with this question, but is it feasible to have both sync and async support in the same object? -- ~Ethan~

On 07.03.2017 19:37, Jelle Zijlstra wrote:

2017-03-07 10:15 GMT-08:00 Ethan Furman mailto:ethan@stoneleaf.us>:

On 03/07/2017 09:41 AM, Brett Cannon wrote:

I don't think a common practice has bubbled up yet for when there's both synchronous and asynchronous versions of an API (closest I have seen is appending an "a" to the async version but that just looks like a spelling mistake to me most of the time). This is why the question of whether separate modules are a better idea is coming up.

I'm undoubtedly going to show my ignorance with this question, but is it feasible to have both sync and async support in the same object?

It's possible, but it quickly gets awkward and will require a lot of code duplication.

Correct me if I'm wrong, but we would get the code duplication anyway. async intrinsically does the same thing (just a little bit different) as its sync counterpart. Otherwise, you wouldn't use it.

For example, we could make @contextmanager work for async functions by making the _GeneratorContextManager class implement both enter/exit and aenter/aexit, but then you'd get an obscure error if you used with on an async contextmanager or async with on a non-async contextmanager.

On Tue, Mar 7, 2017 at 9:41 AM, Brett Cannon wrote:

I don't think a common practice has bubbled up yet for when there's both synchronous and asynchronous versions of an API (closest I have seen is appending an "a" to the async version but that just looks like a spelling mistake to me most of the time). This is why the question of whether separate modules are a better idea is coming up.

For the CSV case, it might be sensible to factor out the io. Like, provide an API that looks like: pushdictreader = csv.PushDictReader() while pushdictreader: chunk = read_some(...) pushdictreader.push(chunk) for row in pushdictreader: ... This API can now straightforwardly be used with sync and async code. Of course you'd want to wrap it up in a nicer interface, somewhere in the ballpark of: def sync_rows(read_some): pushdictreader = csv.PushDictReader() while pushdictreader: chunk = read_some(...) pushdictreader.push(chunk) for row in pushdictreader: yield row async def async_rows(read_some): pushdictreader = csv.PushDictReader() while pushdictreader: chunk = await read_some(...) pushdictreader.push(chunk) for row in pushdictreader: yield row So there'd still be a bit of code duplication, but much much less. Essentially the idea here is to convert the csv module to sans-io style (http://sans-io.readthedocs.io/). Another option is to make it all-async internally, and then offer a sync facade around it. So like start with the natural all-async interface: class AsyncFileLike(ABC): async def async_read(...): ... class AsyncDictReader: def __init__(self, async_file_like): self._async_file_like = async_file_like async def __anext__(self): ... And (crucially!) let's assume that the only way AsyncDictReader interacts with the coroutine runner is by calls to self._async_file_like.async_read. Now we can pass in a secretly-actually-synchronous AsyncFileLike and make a synchronous facade around the whole thing: class AsyncSyncAdapter(AsyncFileLike): def __init__(self, sync_file_like): self._sync_file_like = sync_file_like # Technically an async function, but guaranteed to never yield async def read(self, *args, **kwargs): return self._sync_file_like.read(*args, **kwargs) # Minimal coroutine supervisor: runs async_fn(*args, **kwargs), which must never yield def syncify(async_fn, *args, **kwargs): coro = async_fn(*args, **kwargs) it = coro.__await__() return next(it) class DictReader: def __init__(self, sync_file_like): # Technically an AsyncDictReader, but guaranteed to never yield self._async_dict_reader = AsyncDictReader(AsyncSyncAdapter(sync_file_like)) def __next__(self): return syncify(self._async_dict_reader.__anext__) So here we still have some goo around the edges of the module, but the actual CSV logic only has to be written once, and can still be written in a "pull" style where it does its own I/O, just like it is now. This is basically another approach to writing sans-io protocols, with the annoying trade-off that it means even your synchronous version requires Python 3.5+. But for a stdlib module that's no big deal... -n

On Tue, 7 Mar 2017 at 02:24 Michel Desmoulin wrote:

...

Last week I had to download a CSV from an FTP and push any update on it using websocket so asyncio was a natural fit and the network part went well.

The surprise was that the CSV part would not work as expected. Usually I read csv doing:

import csv

file_like_object = csv_crawler.get_file() for row in csv.DictReader(file_like_object)

But it didn't work because file_like_object.read() was a coroutine which the csv module doesn't handle.

So I had to do:

import csv import io

raw_bytes = await stream.read(10000000) wrapped_bytes = io.BytesIO(raw_bytes) text = io.TextIOWrapper(wrapped_bytes, encoding=encoding, errors='replace')

for i, row in enumerate(csv.DictReader(text)):

Turns out I used asyncio a bit, and I now the stdlib, the io AIP, etc. But for somebody that doesn't, it's not very easy to figure out. Plus it's not as elegant as traditional Python. Not to mention it loads the entire CSV in memory.

So I wondered if I could fix the csv module so it accept async. But the question arised. Where should I put it ?

- Create AsyncDictReader and AsyncReader ? - Add inspect.iscoroutine calls widh it in the regular Readers and some __aiter__ and __aenter__ ? - add a csv.async namespace ?

What API design are we recommanding for expose both sync and async behaviors ?

Le 07/03/2017 à 03:08, Guido van Rossum a écrit :

...

On Mon, Mar 6, 2017 at 5:57 PM, Raymond Hettinger mailto:raymond.hettinger@gmail.com> wrote:

Of course, it makes sense that anything not specific to asyncio should go outside of asyncio.

What I'm more concerned about is what the other places actually are. Rather than putting async variants of everything sprinkled all over the standard library, I suggest collecting them all together, perhaps in a new asynctools module.

That's a tough design choice. I think neither extreme is particularly attractive -- having everything in an asynctools package might also bundle together thing that are entirely unrelated. In the extreme it would be like proposing that all metaclasses should go in a new "metaclasstools" package. I think we did a reasonable job with ABCs: core support goes in abc.py, support for collections ABCs goes into the collections package (in a submodule), and other packages and modules sometimes define ABCs for their own users.

Also, in some cases I expect we'll have to create a whole new module instead of updating some ancient piece of code with newfangled async variants to its outdated APIs.

-- --Guido van Rossum (python.org/~guido http://python.org/~guido)

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/desmoulinmichel%40gmail.c...

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/njs%40pobox.com

-- Nathaniel J. Smith -- https://vorpus.org

On Mar 01, 2017, at 10:55 AM, Victor Stinner wrote:

I suggest to create 3rd party modules on PyPI. It became easy to pull dependencies using pip and virtualenv.

It seems like https://github.com/aio-libs is the home of many asyncio libraries.

This is what we did for aiosmtpd, an asyncio-based replacement for smtpd. It's worked out great on all fronts so far (good community contributions, rapid development, API flexibility as we move toward 1.0, good visibility inside the more general aio-libs umbrella). Cheers, -Barry