PEP 492: async/await in Python; version 4
Hi python-dev,
New version of the PEP is attached. Summary of updates:
1. Terminology:
- *native coroutine* term is used for "async def" functions.
- *generator-based coroutine* term is used for PEP 380
coroutines used in asyncio.
- *coroutine* is used when *native coroutine* or
*generator based coroutine* can be used in the same
context.
I think that it's not really productive to discuss the
terminology that we use in the PEP. Its only purpose is
to disambiguate concepts used in the PEP. We should discuss
how we will name new 'async def' coroutines in Python
Documentation if the PEP is accepted. Although if you
notice that somewhere in the PEP it is not crystal clear
what "coroutine" means please give me a heads up!
2. Syntax of await expressions is now thoroghly defined
in the PEP. See "Await Expression", "Updated operator
precedence table", and "Examples of "await" expressions"
sections.
I like the current approach. I'm still not convinced
that we should make 'await' the same grammatically as
unary minus.
I don't understand why we should allow parsing things
like 'await -fut'; this should be a SyntaxError.
I'm fine to modify the grammar to allow 'await await fut'
syntax, though. And I'm open to discussion :)
3. CO_NATIVE_COROUTINE flag. This enables us to disable
__iter__ and __next__ on native coroutines while maintaining
full backwards compatibility.
4. asyncio / Migration strategy. Existing code can
be used with PEP 492 as is, everything will work as
expected.
However, since *plain generator*s (not decorated with
@asyncio.coroutine) cannot 'yield from' native coroutines
(async def functions), it might break some code
*while adapting it to the new syntax*.
I'm open to just throw a RuntimeWarning in this case
in 3.5, and raise a TypeError in 3.6.
Please see the "Differences from generators" section of
the PEP.
5. inspect.isawaitable() function. And, all new functions
are now listed in the "New Standard Library Functions"
section.
6. Lot's of small updates and tweaks throughout the PEP.
Thanks,
Yury
PEP: 492
Title: Coroutines with async and await syntax
Version: $Revision$
Last-Modified: $Date$
Author: Yury Selivanov
One more thing to discuss: 7. StopAsyncIteration vs AsyncStopIteration. I don't have a strong opinion on this, I prefer the former because it reads better. There was no consensus on which one we should use. Thanks, Yury
Yury Selivanov wrote:
3. CO_NATIVE_COROUTINE flag. This enables us to disable __iter__ and __next__ on native coroutines while maintaining full backwards compatibility.
I don't think you can honestly claim "full backwards compatibility" as long as there are some combinations of old-style and new-style code that won't work together. You seem to be using your own personal definition of "full" here. -- Greg
On 2015-04-30 5:16 AM, Greg Ewing wrote:
Yury Selivanov wrote:
3. CO_NATIVE_COROUTINE flag. This enables us to disable __iter__ and __next__ on native coroutines while maintaining full backwards compatibility.
I don't think you can honestly claim "full backwards compatibility" as long as there are some combinations of old-style and new-style code that won't work together. You seem to be using your own personal definition of "full" here.
Well, using next() and iter() on coroutines in asyncio code is something esoteric. I can't even imagine why you would want to do that. Yury
Yury Selivanov wrote:
Well, using next() and iter() on coroutines in asyncio code is something esoteric. I can't even imagine why you would want to do that.
I'm talking about the fact that existing generator- based coroutines that aren't decorated with @coroutine won't be able to call new ones that use async def. This means that converting one body of code to the new style can force changes in other code that interacts with it. Maybe this is not considered a problem, but as long as it's true, I don't think it's accurate to claim "full backwards compatibility". -- Greg
On 2015-04-30 7:24 PM, Greg Ewing wrote:
Yury Selivanov wrote:
Well, using next() and iter() on coroutines in asyncio code is something esoteric. I can't even imagine why you would want to do that.
I'm talking about the fact that existing generator- based coroutines that aren't decorated with @coroutine won't be able to call new ones that use async def.
Ah, alright. You quoted this: 3. CO_NATIVE_COROUTINE flag. This enables us to disable __iter__ and __next__ on native coroutines while maintaining full backwards compatibility. I wrote "full backwards compatibility" for that particular point #3 -- existing @asyncio.coroutines will have __iter__ and __next__ working just fine. Sorry if this was misleading.
This means that converting one body of code to the new style can force changes in other code that interacts with it.
Maybe this is not considered a problem, but as long as it's true, I don't think it's accurate to claim "full backwards compatibility".
I covered this in point #4. I also touched this in https://www.python.org/dev/peps/pep-0492/#migration-strategy I'm still waiting for feedback on this from Guido. If he decides to go with RuntimeWarnings, then it's 100% backwards compatible. If we keep TypeErrors -- then *existing code will work on 3.5*, but something *might* break during adopting new syntax. I'll update the Backwards Compatibility section. Thanks, Yury
On Thu, Apr 30, 2015 at 4:24 PM, Greg Ewing
Yury Selivanov wrote:
Well, using next() and iter() on coroutines in asyncio code is something esoteric. I can't even imagine why you would want to do that.
I'm talking about the fact that existing generator- based coroutines that aren't decorated with @coroutine won't be able to call new ones that use async def.
This means that converting one body of code to the new style can force changes in other code that interacts with it.
Maybe this is not considered a problem, but as long as it's true, I don't think it's accurate to claim "full backwards compatibility".
Greg, you seem to have an odd notion of "full backwards compatibility". The term means that old code won't break. It doesn't imply that old and new code can always seamlessly interact (that would be an impossibly high bar for almost any change). That said, interoperability between old code and new code is an area of interest. But if the only thing that's standing between old code and new code is the @coroutine decorator, things are looking pretty good -- that decorator is already strongly required for coroutines intended for use with the asyncio package, and older versions of the asyncio package also define that decorator, so if there's old code out there that needs to be able to call the new coroutines (by whatever name, e.g. async functions :-), adding the @coroutine decorator to the old code doesn't look like too much of a burden. I assume there might be code out there that uses yield-from-based coroutines but does not use the asyncio package, but I doubt there is much code like that (I haven't seen much mention of yield-from outside its use in asyncio). So I think the interop problem is mostly limited to asyncio-using code that plays loose with the @coroutine decorator requirement and now wants to work with the new async functions. That's easy enough to address. -- --Guido van Rossum (python.org/~guido)
Yury Selivanov schrieb am 30.04.2015 um 03:30:
Asynchronous Iterators and "async for" --------------------------------------
An *asynchronous iterable* is able to call asynchronous code in its *iter* implementation, and *asynchronous iterator* can call asynchronous code in its *next* method. To support asynchronous iteration:
1. An object must implement an ``__aiter__`` method returning an *awaitable* resulting in an *asynchronous iterator object*.
2. An *asynchronous iterator object* must implement an ``__anext__`` method returning an *awaitable*.
3. To stop iteration ``__anext__`` must raise a ``StopAsyncIteration`` exception.
What this section does not explain, AFAICT, nor the section on design considerations, is why the iterator protocol needs to be duplicated entirely. Can't we just assume (or even validate) that any 'regular' iterator returned from "__aiter__()" (as opposed to "__iter__()") properly obeys to the new protocol? Why additionally duplicate "__next__()" and "StopIteration"? ISTM that all this really depends on is that "__next__()" returns an awaitable. Renaming the method doesn't help with that guarantee. Stefan
Yury Selivanov schrieb am 30.04.2015 um 03:30:
1. Terminology: - *native coroutine* term is used for "async def" functions.
When I read "native", I think of native (binary) code. So "native coroutine" sounds like it's implemented in some compiled low-level language. That might be the case (certainly in the CPython world), but it's not related to this PEP nor its set of examples.
We should discuss how we will name new 'async def' coroutines in Python Documentation if the PEP is accepted.
Well, it doesn't hurt to avoid obvious misleading terminology upfront. Stefan
On Fri, May 1, 2015 at 5:39 AM, Stefan Behnel
Yury Selivanov schrieb am 30.04.2015 um 03:30:
Asynchronous Iterators and "async for" --------------------------------------
An *asynchronous iterable* is able to call asynchronous code in its *iter* implementation, and *asynchronous iterator* can call asynchronous code in its *next* method. To support asynchronous iteration:
1. An object must implement an ``__aiter__`` method returning an *awaitable* resulting in an *asynchronous iterator object*.
2. An *asynchronous iterator object* must implement an ``__anext__`` method returning an *awaitable*.
3. To stop iteration ``__anext__`` must raise a ``StopAsyncIteration`` exception.
What this section does not explain, AFAICT, nor the section on design considerations, is why the iterator protocol needs to be duplicated entirely. Can't we just assume (or even validate) that any 'regular' iterator returned from "__aiter__()" (as opposed to "__iter__()") properly obeys to the new protocol? Why additionally duplicate "__next__()" and "StopIteration"?
ISTM that all this really depends on is that "__next__()" returns an awaitable. Renaming the method doesn't help with that guarantee.
This is an astute observation. I think its flaw (if any) is the situation where we want a single object to be both a regular iterator and an async iterator (say when migrating code to the new world). The __next__ method might want to return a result while __anext__ has to return an awaitable. The solution to that would be to have __aiter__ return an instance of a different class than __iter__, but that's not always convenient. Thus, aware of the choice, I would still prefer a separate __anext__ method. -- --Guido van Rossum (python.org/~guido)
On Fri, May 1, 2015 at 5:50 AM, Stefan Behnel
Yury Selivanov schrieb am 30.04.2015 um 03:30:
1. Terminology: - *native coroutine* term is used for "async def" functions.
When I read "native", I think of native (binary) code. So "native coroutine" sounds like it's implemented in some compiled low-level language. That might be the case (certainly in the CPython world), but it's not related to this PEP nor its set of examples.
We should discuss how we will name new 'async def' coroutines in Python Documentation if the PEP is accepted.
Well, it doesn't hurt to avoid obvious misleading terminology upfront.
I think "obvious[ly] misleading" is too strong, nobody is trying to mislead anybody, we just have different associations with the same word. Given the feedback I'd call "native coroutine" suboptimal (even though I proposed it myself) and I am now in favor of using "async function". -- --Guido van Rossum (python.org/~guido)
On 1 May 2015 at 16:31, Guido van Rossum
On Fri, May 1, 2015 at 5:50 AM, Stefan Behnel
wrote: Yury Selivanov schrieb am 30.04.2015 um 03:30:
1. Terminology: - *native coroutine* term is used for "async def" functions.
When I read "native", I think of native (binary) code. So "native coroutine" sounds like it's implemented in some compiled low-level language. That might be the case (certainly in the CPython world), but it's not related to this PEP nor its set of examples.
We should discuss how we will name new 'async def' coroutines in Python Documentation if the PEP is accepted.
Well, it doesn't hurt to avoid obvious misleading terminology upfront.
I think "obvious[ly] misleading" is too strong, nobody is trying to mislead anybody, we just have different associations with the same word. Given the feedback I'd call "native coroutine" suboptimal (even though I proposed it myself) and I am now in favor of using "async function".
But what if you have async methods? I know, a method is almost a function, but still, sounds slightly confusing. IMHO, these are really classical coroutines. If gevent calls them coroutines, I don't think asyncio has any less right to call them coroutines. You have to ask yourself this: a new programmer, when he sees mentions of coroutines, how likely is he to understand what he is dealing with? What about "async function"? The former is a well known concept, since decades ago, while the latter is something he probably (at least me) never heard of before. For me, an async function is just as likely to be an API that is asynchronous in the sense that it takes an extra "callback" parameter to be called when the asynchronous work is done. I think coroutine is the name of a concept, not a specific implementation. Cheers, -- Gustavo J. A. M. Carneiro Gambit Research "The universe is always one step beyond logic." -- Frank Herbert
On Fri, May 1, 2015 at 8:55 AM, Gustavo Carneiro
On 1 May 2015 at 16:31, Guido van Rossum
wrote: On Fri, May 1, 2015 at 5:50 AM, Stefan Behnel
wrote: Yury Selivanov schrieb am 30.04.2015 um 03:30:
1. Terminology: - *native coroutine* term is used for "async def" functions.
When I read "native", I think of native (binary) code. So "native coroutine" sounds like it's implemented in some compiled low-level language. That might be the case (certainly in the CPython world), but it's not related to this PEP nor its set of examples.
We should discuss how we will name new 'async def' coroutines in Python Documentation if the PEP is accepted.
Well, it doesn't hurt to avoid obvious misleading terminology upfront.
I think "obvious[ly] misleading" is too strong, nobody is trying to mislead anybody, we just have different associations with the same word. Given the feedback I'd call "native coroutine" suboptimal (even though I proposed it myself) and I am now in favor of using "async function".
But what if you have async methods? I know, a method is almost a function, but still, sounds slightly confusing.
IMHO, these are really classical coroutines. If gevent calls them coroutines, I don't think asyncio has any less right to call them coroutines.
You have to ask yourself this: a new programmer, when he sees mentions of coroutines, how likely is he to understand what he is dealing with? What about "async function"? The former is a well known concept, since decades ago, while the latter is something he probably (at least me) never heard of before.
For me, an async function is just as likely to be an API that is asynchronous in the sense that it takes an extra "callback" parameter to be called when the asynchronous work is done.
I think coroutine is the name of a concept, not a specific implementation.
Cheers,
Cheers indeed! I agree that the *concept* is best called coroutine -- and
we have used this term ever since PEP 342. But when we're talking specifics and trying to distinguish e.g. a function declared with 'async def' from a regular function or from a regular generator function, using 'async function' sounds right. And 'async method' if it's a method. -- --Guido van Rossum (python.org/~guido)
Guido van Rossum schrieb am 01.05.2015 um 17:28:
On Fri, May 1, 2015 at 5:39 AM, Stefan Behnel wrote:
Yury Selivanov schrieb am 30.04.2015 um 03:30:
Asynchronous Iterators and "async for" --------------------------------------
An *asynchronous iterable* is able to call asynchronous code in its *iter* implementation, and *asynchronous iterator* can call asynchronous code in its *next* method. To support asynchronous iteration:
1. An object must implement an ``__aiter__`` method returning an *awaitable* resulting in an *asynchronous iterator object*.
2. An *asynchronous iterator object* must implement an ``__anext__`` method returning an *awaitable*.
3. To stop iteration ``__anext__`` must raise a ``StopAsyncIteration`` exception.
What this section does not explain, AFAICT, nor the section on design considerations, is why the iterator protocol needs to be duplicated entirely. Can't we just assume (or even validate) that any 'regular' iterator returned from "__aiter__()" (as opposed to "__iter__()") properly obeys to the new protocol? Why additionally duplicate "__next__()" and "StopIteration"?
ISTM that all this really depends on is that "__next__()" returns an awaitable. Renaming the method doesn't help with that guarantee.
This is an astute observation. I think its flaw (if any) is the situation where we want a single object to be both a regular iterator and an async iterator (say when migrating code to the new world). The __next__ method might want to return a result while __anext__ has to return an awaitable. The solution to that would be to have __aiter__ return an instance of a different class than __iter__, but that's not always convenient.
My personal gut feeling is that this case would best be handled by a generic wrapper class. Both are well defined protocols, so I don't expect people to change all of their classes and instead return a wrapped object either from __iter__() or __aiter__(), depending on which they want to optimise for, or which will eventually turn out to be easier to wrap. But that's trying to predict the [Ff]uture, obviously. It just feels like unnecessary complexity for now. And we already have a type slot for __next__ ("tp_iternext"), but not for __anext__. Stefan
Stefan, I don't like the idea of combining __next__ and __anext__. In this case explicit is better than implicit. __next__ returning coroutines is a perfectly normal thing for a normal 'for' loop (it wouldn't to anything with them), whereas 'async for' will interpret that differently, and will try to await those coroutines. Yury On 2015-05-01 1:10 PM, Stefan Behnel wrote:
On Fri, May 1, 2015 at 5:39 AM, Stefan Behnel wrote:
Yury Selivanov schrieb am 30.04.2015 um 03:30:
Asynchronous Iterators and "async for" --------------------------------------
An *asynchronous iterable* is able to call asynchronous code in its *iter* implementation, and *asynchronous iterator* can call asynchronous code in its *next* method. To support asynchronous iteration:
1. An object must implement an ``__aiter__`` method returning an *awaitable* resulting in an *asynchronous iterator object*.
2. An *asynchronous iterator object* must implement an ``__anext__`` method returning an *awaitable*.
3. To stop iteration ``__anext__`` must raise a ``StopAsyncIteration`` exception. What this section does not explain, AFAICT, nor the section on design considerations, is why the iterator protocol needs to be duplicated entirely. Can't we just assume (or even validate) that any 'regular' iterator returned from "__aiter__()" (as opposed to "__iter__()") properly obeys to the new protocol? Why additionally duplicate "__next__()" and "StopIteration"?
ISTM that all this really depends on is that "__next__()" returns an awaitable. Renaming the method doesn't help with that guarantee.
This is an astute observation. I think its flaw (if any) is the situation where we want a single object to be both a regular iterator and an async iterator (say when migrating code to the new world). The __next__ method might want to return a result while __anext__ has to return an awaitable. The solution to that would be to have __aiter__ return an instance of a different class than __iter__, but that's not always convenient. My personal gut feeling is that this case would best be handled by a generic wrapper class. Both are well defined protocols, so I don't expect
Guido van Rossum schrieb am 01.05.2015 um 17:28: people to change all of their classes and instead return a wrapped object either from __iter__() or __aiter__(), depending on which they want to optimise for, or which will eventually turn out to be easier to wrap.
But that's trying to predict the [Ff]uture, obviously. It just feels like unnecessary complexity for now. And we already have a type slot for __next__ ("tp_iternext"), but not for __anext__.
Stefan
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com
Yury Selivanov schrieb am 01.05.2015 um 20:52:
I don't like the idea of combining __next__ and __anext__. In this case explicit is better than implicit. __next__ returning coroutines is a perfectly normal thing for a normal 'for' loop (it wouldn't to anything with them), whereas 'async for' will interpret that differently, and will try to await those coroutines.
Sure, but the difference is that one would have called __aiter__() first and the other __iter__(). Normally, either of the two would not exist, so using the wrong loop on an object will just fail. However, after we passed that barrier, we already know that the object that was returned is supposed to obey to the expected protocol, so it doesn't matter whether we call __next__() or name it __anext__(), except that the second requires us to duplicate an existing protocol. This has nothing to do with implicit vs. explicit. Stefan
On 05/01, Stefan Behnel wrote:
Yury Selivanov schrieb am 01.05.2015 um 20:52:
I don't like the idea of combining __next__ and __anext__. In this case explicit is better than implicit. __next__ returning coroutines is a perfectly normal thing for a normal 'for' loop (it wouldn't to anything with them), whereas 'async for' will interpret that differently, and will try to await those coroutines.
Sure, but the difference is that one would have called __aiter__() first and the other __iter__(). Normally, either of the two would not exist, so using the wrong loop on an object will just fail. However, after we passed that barrier, we already know that the object that was returned is supposed to obey to the expected protocol, so it doesn't matter whether we call __next__() or name it __anext__(), except that the second requires us to duplicate an existing protocol.
If we must have __aiter__, then we may as well also have __anext__; besides being more consistent, it also allows an object to be both a normol iterator and an asynch iterator. -- ~Ethan~
Let's say it this way: I want to know what I am looking at when I browse through the code -- an asynchronous iterator, or a normal iterator. I want an explicit difference between these protocols, because they are different. Moreover, the below code is a perfectly valid, infinite iterable: class SomeIterable: def __iter__(self): return self async def __next__(self): return 'spam' I'm strong -1 on this idea. Yury On 2015-05-01 3:03 PM, Stefan Behnel wrote:
I don't like the idea of combining __next__ and __anext__. In this case explicit is better than implicit. __next__ returning coroutines is a perfectly normal thing for a normal 'for' loop (it wouldn't to anything with them), whereas 'async for' will interpret that differently, and will try to await those coroutines. Sure, but the difference is that one would have called __aiter__() first and the other __iter__(). Normally, either of the two would not exist, so using the wrong loop on an object will just fail. However, after we passed
Yury Selivanov schrieb am 01.05.2015 um 20:52: that barrier, we already know that the object that was returned is supposed to obey to the expected protocol, so it doesn't matter whether we call __next__() or name it __anext__(), except that the second requires us to duplicate an existing protocol.
This has nothing to do with implicit vs. explicit.
Stefan
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com
On 2015-05-01 3:19 PM, Ethan Furman wrote:
Sure, but the difference is that one would have called __aiter__() first
and the other __iter__(). Normally, either of the two would not exist, so using the wrong loop on an object will just fail. However, after we passed that barrier, we already know that the object that was returned is supposed to obey to the expected protocol, so it doesn't matter whether we call __next__() or name it __anext__(), except that the second requires us to duplicate an existing protocol. If we must have __aiter__, then we may as well also have __anext__; besides being more consistent, it also allows an object to be both a normol iterator and an asynch iterator.
And this is a good point too. Thanks, Yury
On 2015-05-01 3:23 PM, Yury Selivanov wrote:
Let's say it this way: I want to know what I am looking at when I browse through the code -- an asynchronous iterator, or a normal iterator. I want an explicit difference between these protocols, because they are different.
Moreover, the below code is a perfectly valid, infinite iterable:
class SomeIterable: def __iter__(self): return self async def __next__(self): return 'spam'
I'm strong -1 on this idea.
To further clarify on the example: class SomeIterable: def __iter__(self): return self async def __aiter__(self): return self async def __next__(self): print('hello') raise StopAsyncIteration If you pass this to 'async for' you will get 'hello' printed and the loop will be over. If you pass this to 'for', you will get an infinite loop, because '__next__' will return a coroutine object (that has to be also awaited, but it wouldn't, because it's a plain 'for' statement). This is something that we shouldn't let happen. Yury
On 1 May 2015 at 20:24, Yury Selivanov
On 2015-05-01 3:19 PM, Ethan Furman wrote: [...]
If we must have __aiter__, then we may as well also have __anext__; besides being more consistent, it also allows an object to be both a normol iterator and an asynch iterator.
And this is a good point too.
I'm not convinced that allowing an object to be both a normal and an async iterator is a good thing. It could be a recipe for confusion. -- Arnaud
On 2015-05-01 4:24 PM, Arnaud Delobelle wrote:
On 1 May 2015 at 20:24, Yury Selivanov
wrote: On 2015-05-01 3:19 PM, Ethan Furman wrote: [...]
If we must have __aiter__, then we may as well also have __anext__; besides being more consistent, it also allows an object to be both a normol iterator and an asynch iterator.
And this is a good point too. I'm not convinced that allowing an object to be both a normal and an async iterator is a good thing. It could be a recipe for confusion.
I doubt that it will be a popular thing. But disallowing this by merging two different protocols in one isn't a good idea either. Yury
On 5/1/2015 9:59 AM, Guido van Rossum wrote:
I think coroutine is the name of a concept, not a specific implementation.
Cheers,
Cheers indeed! I agree that the *concept* is best called coroutine -- and we have used this term ever since PEP 342. But when we're talking specifics and trying to distinguish e.g. a function declared with 'async def' from a regular function or from a regular generator function, using 'async function' sounds right. And 'async method' if it's a method.
Exactly. The async function/method is an implementation technique for a specific kind/subset of coroutine functionality. So the term coroutine, qualified by a description of its best usage and limitationsof async function, can be used in defining async function, thus appealing to what people know or have heard of and vaguely understand and can read more about in the literature. A glossary entry for coroutine in the docs seems appropriate, which could point out the 16† ways to implement coroutine-style functionalities in Python, and perhaps make recommendations for different types of usage. †OK, not 16 ways, but it is 3 now, or 4?
Yury Selivanov schrieb am 01.05.2015 um 20:52:
I don't like the idea of combining __next__ and __anext__.
Ok, fair enough. So, how would you use this new protocol manually then? Say, I already know that I won't need to await the next item that the iterator will return. For normal iterators, I could just call next() on it and continue the for-loop. How would I do it for AIterators? Stefan
On 1 May 2015 at 21:27, Yury Selivanov
On 2015-05-01 4:24 PM, Arnaud Delobelle wrote:
On 1 May 2015 at 20:24, Yury Selivanov
wrote: On 2015-05-01 3:19 PM, Ethan Furman wrote:
[...]
If we must have __aiter__, then we may as well also have __anext__; besides being more consistent, it also allows an object to be both a normol iterator and an asynch iterator.
And this is a good point too.
I'm not convinced that allowing an object to be both a normal and an async iterator is a good thing. It could be a recipe for confusion.
I doubt that it will be a popular thing. But disallowing this by merging two different protocols in one isn't a good idea either.
I having been arguing for merging two different protocols. I'm saying that allowing an object to be both normal and async iterable is not an argument for having separate protocols because it's not a good thing. Cheers, -- Arnaud
Stefan Behnel schrieb am 02.05.2015 um 06:54:
Yury Selivanov schrieb am 01.05.2015 um 20:52:
I don't like the idea of combining __next__ and __anext__.
Ok, fair enough. So, how would you use this new protocol manually then? Say, I already know that I won't need to await the next item that the iterator will return. For normal iterators, I could just call next() on it and continue the for-loop. How would I do it for AIterators?
BTW, I guess that this "AIterator", or rather "AsyncIterator", needs to be a separate protocol (and ABC) then. Implementing "__aiter__()" and "__anext__()" seems perfectly reasonable without implementing (or using) a Coroutine. That means we also need an "AsyncIterable" as a base class for it. Would Coroutine then inherit from both Iterator and AsyncIterator? Or should we try to separate the protocol hierarchy completely? The section on "Coroutine objects" seems to suggest that inheritance from Iterator is not intended. OTOH, I'm not sure if inheriting from AsyncIterator is intended for Coroutine. The latter might just be a stand-alone ABC with send/throw/close, after all. I think that in order to get a better understanding of the protocol(s) that this PEP proposes, and the terminology that it should use, it would help to implement these ABCs. That might even help us to decide if we need new builtins (or helpers) aiter() and anext() in order to deal with these protocols. Stefan
On Fri, May 01, 2015 at 09:24:47PM +0100, Arnaud Delobelle wrote:
I'm not convinced that allowing an object to be both a normal and an async iterator is a good thing. It could be a recipe for confusion.
In what way? I'm thinking that the only confusion would be if you wrote "async for" instead of "for", or vice versa, and instead of getting an exception you got the (a)syncronous behaviour you didn't want. But I have no intuition for how likely it is that you could write an asyncronous for loop, leave out the async, and still have the code do something meaningful. Other than that, I think it would be fine to have an object be both a syncronous and asyncronous iterator. You specify the behaviour you want by how you use it. We can already do that, e.g. unittest's assertRaises is both a test assertion and a context manager. Objects can have multiple roles, and it's not usually abused, or confusing. I'm not sure that async iterables will be any different. -- Steve
On 3 May 2015 at 16:24, Steven D'Aprano
On Fri, May 01, 2015 at 09:24:47PM +0100, Arnaud Delobelle wrote:
I'm not convinced that allowing an object to be both a normal and an async iterator is a good thing. It could be a recipe for confusion.
In what way?
I'm thinking that the only confusion would be if you wrote "async for" instead of "for", or vice versa, and instead of getting an exception you got the (a)syncronous behaviour you didn't want.
Yes. This is the same kind of confusion that this PEP is trying hard to get rid of in other parts (i.e. the confusion between 'yield' and 'yield from' in current coroutines).
But I have no intuition for how likely it is that you could write an asyncronous for loop, leave out the async, and still have the code do something meaningful.
Well if you've made you object both iterable and 'async iterable' then it's very likely that you're going to get something meaningful out of either kind of iteration. Just not the way you want it if you mistakenly left out (or added) the 'async' keyword in your loop.
Other than that, I think it would be fine to have an object be both a syncronous and asyncronous iterator. You specify the behaviour you want by how you use it. We can already do that, e.g. unittest's assertRaises is both a test assertion and a context manager.
The latter is fine, because there is no danger of mistaking one for the other, unlike iterators and 'async iterators'. But my argument is simply that there is no good reason to aim for the ability to have object conform to both protocols. So it shouldn't be a criterion when looking at the merits of a proposal. I may very well be wrong but I haven't yet seen a compelling case for an object implementing both protocols. Cheers, -- Arnaud
On Sun May 3 08:32:02 CEST 2015, Stefan Behnel wrote:
Ok, fair enough. So, how would you use this new protocol manually then? Say, I already know that I won't need to await the next item that the iterator will return. For normal iterators, I could just call next() on it and continue the for-loop. How would I do it for AIterators?
Call next, then stick it somewhere it be waited on. Or is that syntactically illegal, because of the separation between sync and async? The "asych for" seems to assume that you want to do the waiting right now, at each step. (At least as far as this thread of the logic goes; something else might be happening in parallel via other threads of control.)
BTW, I guess that this "AIterator", or rather "AsyncIterator", needs to be a separate protocol (and ABC) then. Implementing "__aiter__()" and "__anext__()" seems perfectly reasonable without implementing (or using) a Coroutine.
That means we also need an "AsyncIterable" as a base class for it.
Agreed.
That might even help us to decide if we need new builtins (or helpers) aiter() and anext() in order to deal with these protocols.
I hope not; they seem more like specialized versions of functions, such as are found in math or cmath. Ideally, as much as possible of this PEP should live in asycio, rather than appearing globally. Which reminds me ... *should* the "await" keyword work with any future, or is it really intentionally restricted to use with a single library module and 3rd party replacements? -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ
Hi, still watching progress here. Read all posts and changes. Everything improved and I know it is a lot of work. Thx for doing this. But I still think this PEP goes to far. 1. To have "native coroutines" and await, __await__ is very good and useful. Also for beginners to see and mark coroutines are a different concept than generators. Also this is easy to learn. Easier as to explain why a generator is in this case a coroutine and so on. 2. I still don't like to sprinkle async everywhere. At all we don't need it for the first step. We can handle coroutines similar to generators, when there is a await in it then it is one. Same as for yield. Or to be more explicit, if it is marked as one with @coroutine it is enough. But then it makes also sense to do the same for generators with @generator. We should be consistent here. 3. async with is not needed, there are rare use cases for it and every can be written with a try: except: finally: Every async framework lived without it over years. No problem because for the seldom need try: ... was enough. 4. async for can be implemented with a while loop. For me this is even more explicit and clear. Every time I see the async for I try to find out what is done in async manner. For what I do an implicit await ? Also in most of my uses cases it was enough to produce Futures in a normal loop and yield (await) for them. Even for database interfaces. Most of the time they do prefetching under the hood and I don't have to care at all. 5. For async with/for a lot of new special methods are needed all only prefixed with "a". Easy to confuse with "a" for abstract. Often used to prefix abstract classes. Still think __async_iter__, __async_enter__ is better for this. Yes more to write but not so easy to confuse with the sync __iter__ or __enter__, ... And matches more to I must use async to call them. I am not new to the async land, have done a lot of stuff with twisted and even looked at tornado. Also has tried to teach some people the asnc world. This is not easy and you learn most only by doing and using it. For my final conclusion, we should not rush all this into the language. Do it step by step and also help other Python implementations to support it. For now only a very low percentage are at Python 3.4. And if you compare the statistics in PyPi, you see most are still at Python 2.7 and using async frameworks like twisted/tornado. We do such a fundamental language change only because a few people need it for a niche in the whole Python universe? We confuse the rest with new stuff they never need? Even the discussion on python-dev suggests there is some time needed to finalize all this. We forget to address the major problems here. How can someone in a "sync" script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way. And for all this we still can't tell them "oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5". It does not and it is only most useful for networking stuff nothing more. Don't get me wrong, I like async programming and I think it is very useful. But had to learn not everyone thinks so and most only want to solve there problems in an easy way and not get a new one called "async". Now I shut up. Go to my normal mode and be quiet and read. ;-) Regards, Wolfgang
Hi, still watching progress here. Read all posts and changes. Everything improved and I know it is a lot of work. Thx for doing this. But I still think this PEP goes to far. 1. To have "native coroutines" and await, __await__ is very good and useful. Also for beginners to see and mark coroutines are a different concept than generators. Also this is easy to learn. Easier as to explain why a generator is in this case a coroutine and so on. 2. I still don't like to sprinkle async everywhere. At all we don't need it for the first step. We can handle coroutines similar to generators, when there is a await in it then it is one. Same as for yield. Or to be more explicit, if it is marked as one with @coroutine it is enough. But then it makes also sense to do the same for generators with @generator. We should be consistent here. 3. async with is not needed, there are rare use cases for it and every can be written with a try: except: finally: Every async framework lived without it over years. No problem because for the seldom need try: ... was enough. 4. async for can be implemented with a while loop. For me this is even more explicit and clear. Every time I see the async for I try to find out what is done in async manner. For what I do an implicit await ? Also in most of my uses cases it was enough to produce Futures in a normal loop and yield (await) for them. Even for database interfaces. Most of the time they do prefetching under the hood and I don't have to care at all. 5. For async with/for a lot of new special methods are needed all only prefixed with "a". Easy to confuse with "a" for abstract. Often used to prefix abstract classes. Still think __async_iter__, __async_enter__ is better for this. Yes more to write but not so easy to confuse with the sync __iter__ or __enter__, ... And matches more to I must use async to call them. I am not new to the async land, have done a lot of stuff with twisted and even looked at tornado. Also has tried to teach some people the asnc world. This is not easy and you learn most only by doing and using it. For my final conclusion, we should not rush all this into the language. Do it step by step and also help other Python implementations to support it. For now only a very low percentage are at Python 3.4. And if you compare the statistics in PyPi, you see most are still at Python 2.7 and using async frameworks like twisted/tornado. We do such a fundamental language change only because a few people need it for a niche in the whole Python universe? We confuse the rest with new stuff they never need? Even the discussion on python-dev suggests there is some time needed to finalize all this. We forget to address the major problems here. How can someone in a "sync" script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way. And for all this we still can't tell them "oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5". It does not and it is only most useful for networking stuff nothing more. Don't get me wrong, I like async programming and I think it is very useful. But had to learn not everyone thinks so and most only want to solve there problems in an easy way and not get a new one called "async". Now I shut up. Go to my normal mode and be quiet and read. ;-) Regards, Wolfgang
tds333 <at> gmail.com
Hi,
still watching progress here. Read all posts and changes.
Everything improved and I know it is a lot of work. Thx for doing this.
But I still think this PEP goes to far.
[...]
We forget to address the major problems here. How can someone in a "sync" script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way.
[...]
Hi Wolfgang, You may want to see what I just posted on python-ideas. What I wrote about is related to several things you mention, and might provide a remedy. -- Koos
Hi Wolfgang, On 2015-05-05 7:27 AM, Wolfgang wrote:
Hi, [..] Even the discussion on python-dev suggests there is some time needed to finalize all this.
I'd say that: 80% of the recent discussion of the PEP is about terminology. 10% is about whether we should have __future__ import or not.
We forget to address the major problems here. How can someone in a "sync" script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way.
asyncio and twisted answered these questions ;) The answer is that you have to write async implementations. gevent has a different answer, but greenlents/stackless is something that will never be merged in CPython and other implementations.
And for all this we still can't tell them "oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5". It does not and it is only most useful for networking stuff nothing more.
"networking stuff", and in particular, web, is a huge part of current Python usage. Please don't underestimate that. Thanks, Yury
On Mon, May 4, 2015 at 10:35 AM, Jim J. Jewett
Which reminds me ... *should* the "await" keyword work with any future, or is it really intentionally restricted to use with a single library module and 3rd party replacements?
You can make any Future type work with await by adding an __await__ method. -- --Guido van Rossum (python.org/~guido)
On 5 May 2015 at 19:25, Yury Selivanov
On 2015-05-05 7:27 AM, Wolfgang wrote:
Even the discussion on python-dev suggests there is some time needed to finalize all this.
I'd say that:
80% of the recent discussion of the PEP is about terminology. 10% is about whether we should have __future__ import or not.
But the terminology discussion appears to revolve around people finding the various concepts involved in asyncio (particularly the new PEP, but also to an extent the existing implementation) confusing. I can confirm, having tried to work through the asyncio docs, that the underlying concepts and how they are explained, are confusing to an outsider. That's not to say that everything needs to be beginner-friendly, but it *does* mean that it's hard for the wider Python community to meaningfully comment, or evaluate or sanity-check the design. We're left with a sense of "trust us, it makes sense if you need it, everyone else can ignore it". Personally, I feel as if PEP 492 is looking a little premature - maybe the focus should be on making asyncio more accessible first, and *then* adding syntax. You can argue that the syntax is needed to help make async more accessible - but if that's the case then the terminology debates and confusion are clear evidence that it's not succeeding in that goal. Of course, that's based on my perception of one of the goals of the PEP as being "make coroutines and asyncio more accessible", If the actual goals are different, my conclusion is invalid.
We forget to address the major problems here. How can someone in a "sync" script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way.
asyncio and twisted answered these questions ;) The answer is that you have to write async implementations.
Well, twisted always had defer_to_thread. Asyncio has run_in_executor, but that seems to be callback-based rather than coroutine-based? Many people use requests for their web access. There are good reasons for this. Are you saying that until someone steps up and writes an async implementation of requests, I have to make a choice - requests or asyncio? Unfortunately, I can't see myself choosing asyncio in that situation. Which again means that asyncio becomes "something that the average user can't use". Which in turn further entrenches it as a specialist-only tool. As another example, in Twisted I could use defer_to_thread to integrate Oracle database access into a twisted application (that's what the twisted database stuff did under the hood). Can I do that with asyncio? Will the syntax in the PEP help, hinder or be irrelevant to that?
And for all this we still can't tell them "oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5". It does not and it is only most useful for networking stuff nothing more.
"networking stuff", and in particular, web, is a huge part of current Python usage. Please don't underestimate that.
Without async versions of requests and similar, how much of a chunk of the networking/web area will asyncio take? (Genuine question, I have no idea). And how much extra will this PEP add? Those may not be fair questions (and even if they are fair, the answers are probably unknowable), but as an outsider, I feel only the costs of the asyncio implementation (a new library that I don't understand, and now a relatively large amount of new syntax and special methods I have to ignore because they don't make sense to me). That's OK, but I think I am being reasonable to ask for some sense of the level of benefits others are getting to balance out the costs I incur. Paul
On Tue, May 5, 2015 at 3:14 PM Paul Moore
On 5 May 2015 at 19:25, Yury Selivanov
wrote: On 2015-05-05 7:27 AM, Wolfgang wrote:
Even the discussion on python-dev suggests there is some time needed to finalize all this.
I'd say that:
80% of the recent discussion of the PEP is about terminology. 10% is about whether we should have __future__ import or not.
But the terminology discussion appears to revolve around people finding the various concepts involved in asyncio (particularly the new PEP, but also to an extent the existing implementation) confusing. I can confirm, having tried to work through the asyncio docs, that the underlying concepts and how they are explained, are confusing to an outsider.
That's not to say that everything needs to be beginner-friendly, but it *does* mean that it's hard for the wider Python community to meaningfully comment, or evaluate or sanity-check the design. We're left with a sense of "trust us, it makes sense if you need it, everyone else can ignore it".
Watch David Beazley's talk from PyCon this year and you can watch him basically re-implement asyncio on stage in under 45 minutes. It's not as complicated as it seems when you realize there is an event loop driving everything (which people have been leaving out of the conversation since it doesn't tie into the syntax directly).
Personally, I feel as if PEP 492 is looking a little premature - maybe the focus should be on making asyncio more accessible first, and *then* adding syntax.
I think this ties the concept of adding syntax to Python to make coroutine-based programming easier too much to asyncio; the latter is just an implementation of the former. This PEP doesn't require asyncio beyond the fact that will be what provides the initial event loop in the stdlib.
You can argue that the syntax is needed to help make async more accessible - but if that's the case then the terminology debates and confusion are clear evidence that it's not succeeding in that goal.
Perhaps, but arguing about the nitty-gritty details of something doesn't automatically lead to a clearer understanding of the higher level concept. Discussing how turning a steering wheel in a car might help you grasp how cars turn, but it isn't a requirement to get "turn the wheel left to make the car go left".
Of course, that's based on my perception of one of the goals of the PEP as being "make coroutines and asyncio more accessible", If the actual goals are different, my conclusion is invalid.
I think the goal is "make coroutines easier to use" and does not directly relate to asyncio.
We forget to address the major problems here. How can someone in a "sync" script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way.
asyncio and twisted answered these questions ;) The answer is that you have to write async implementations.
Well, twisted always had defer_to_thread. Asyncio has run_in_executor, but that seems to be callback-based rather than coroutine-based?
Yep.
Many people use requests for their web access. There are good reasons for this. Are you saying that until someone steps up and writes an async implementation of requests, I have to make a choice - requests or asyncio?
I believe so; you need something to implement __await__. This is true in any language that implements co-routines. Unfortunately, I can't see myself choosing asyncio in that
situation. Which again means that asyncio becomes "something that the average user can't use". Which in turn further entrenches it as a specialist-only tool.
You forgot to append "... yet" to that statement. Just because something isn't available out of the box without some effort to support doesn't mean it will never happen, else there would be absolutely no Python 3 users out there.
As another example, in Twisted I could use defer_to_thread to integrate Oracle database access into a twisted application (that's what the twisted database stuff did under the hood). Can I do that with asyncio? Will the syntax in the PEP help, hinder or be irrelevant to that?
And for all this we still can't tell them "oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5". It does not and it is only most useful for networking stuff nothing more.
"networking stuff", and in particular, web, is a huge part of current Python usage. Please don't underestimate that.
Without async versions of requests and similar, how much of a chunk of the networking/web area will asyncio take? (Genuine question, I have no idea). And how much extra will this PEP add? Those may not be fair questions (and even if they are fair, the answers are probably unknowable), but as an outsider, I feel only the costs of the asyncio implementation (a new library that I don't understand, and now a relatively large amount of new syntax and special methods I have to ignore because they don't make sense to me). That's OK, but I think I am being reasonable to ask for some sense of the level of benefits others are getting to balance out the costs I incur.
Co-routine-based asynchronous programming is new to Python, so as a community we don't have it as something everyone learns over time. If you don't come from an area that supports it then it will be foreign to you and not make sense without someone giving you a good tutorial on it. But considering C#, Dart, and Ecmascript 6 (will) have co-routine support -- and those are just the languages I can name off the top of my head -- using the exact same keywords suggests to me that it isn't *that* difficult of a topic to teach people. This is just one of those PEPs where you have to trust the people with experience in the area are making good design decisions for those of us who aren't in a position to contribute directly without more experience in the domain. -Brett
Paul _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
Paul, On 2015-05-05 3:14 PM, Paul Moore wrote:
On 5 May 2015 at 19:25, Yury Selivanov
wrote: On 2015-05-05 7:27 AM, Wolfgang wrote:
Even the discussion on python-dev suggests there is some time needed to finalize all this. I'd say that:
80% of the recent discussion of the PEP is about terminology. 10% is about whether we should have __future__ import or not. But the terminology discussion appears to revolve around people finding the various concepts involved in asyncio (particularly the new PEP, but also to an extent the existing implementation) confusing. I can confirm, having tried to work through the asyncio docs, that the underlying concepts and how they are explained, are confusing to an outsider.
I agree. We have to improve asyncio docs in this area.
That's not to say that everything needs to be beginner-friendly, but it *does* mean that it's hard for the wider Python community to meaningfully comment, or evaluate or sanity-check the design. We're left with a sense of "trust us, it makes sense if you need it, everyone else can ignore it".
Personally, I feel as if PEP 492 is looking a little premature - maybe the focus should be on making asyncio more accessible first, and *then* adding syntax. You can argue that the syntax is needed to help make async more accessible - but if that's the case then the terminology debates and confusion are clear evidence that it's not succeeding in that goal. Of course, that's based on my perception of one of the goals of the PEP as being "make coroutines and asyncio more accessible", If the actual goals are different, my conclusion is invalid.
Again, PEP 492 is not only for asyncio. *Any* framework can use it, including Twisted. As for terminology, I view this discussion differently. It's not about the technical details (Python has asymmetric coroutines, that's it), but rather on how to disambiguate coroutines implemented with generators and yield-from, from new 'async def' coroutines. I can't see any fundamental problem with the PEP behind such discussions.
We forget to address the major problems here. How can someone in a "sync" script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way. asyncio and twisted answered these questions ;) The answer is that you have to write async implementations. Well, twisted always had defer_to_thread. Asyncio has run_in_executor, but that seems to be callback-based rather than coroutine-based?
Many people use requests for their web access. There are good reasons for this. Are you saying that until someone steps up and writes an async implementation of requests, I have to make a choice - requests or asyncio? Unfortunately, I can't see myself choosing asyncio in that situation. Which again means that asyncio becomes "something that the average user can't use". Which in turn further entrenches it as a specialist-only tool.
There is aiohttp library [1], which provides a client API similar to requests. And if you want to write high performance networking server in python3 you *will* choose asyncio (or gevent/twisted in python2). And PEP 492 is aimed to make this whole async stuff more accessible to an average user.
As another example, in Twisted I could use defer_to_thread to integrate Oracle database access into a twisted application (that's what the twisted database stuff did under the hood). Can I do that with asyncio? Will the syntax in the PEP help, hinder or be irrelevant to that?
You can use 'loop.run_in_executor' in asyncio. It returns a future that you can await on. You can also provide a nice facade for your Oracle-database code that provides a nice API but uses asyncio thread executor behind the scenes.
And for all this we still can't tell them "oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5". It does not and it is only most useful for networking stuff nothing more. "networking stuff", and in particular, web, is a huge part of current Python usage. Please don't underestimate that. Without async versions of requests and similar, how much of a chunk of the networking/web area will asyncio take? (Genuine question, I have no idea).
There are some things (like websockets) that are hard to implement correctly in existing frameworks like django and flask. And these kind of things are becoming more and more important. Languages like Go were designed specifically to allow writing efficient
And how much extra will this PEP add? Those may not be fair questions (and even if they are fair, the answers are probably unknowable), but as an outsider, I feel only the costs of the asyncio implementation (a new library that I don't understand, and now a relatively large amount of new syntax and special methods I have to ignore because they don't make sense to me). That's OK, but I think I am being reasonable to ask for some sense of the level of benefits others are getting to balance out the costs I incur.
Paul
It's chicken and egg problem. Right now, current coroutines via generators approach is cumbersome, it's harder to write async code than it should be. It stops the innovation in this area. Some languages like Go were specifically designed to make network programming easier, and they now steal users from Python. There is no absence of libraries for Go (and it's a new language!), btw. Give people the right tools and they will build what they need. Yury [1] https://github.com/KeepSafe/aiohttp
Jumping in to correct one fact.
On Tue, May 5, 2015 at 12:44 PM, Brett Cannon
On Tue, May 5, 2015 at 3:14 PM Paul Moore
wrote: Well, twisted always had defer_to_thread. Asyncio has run_in_executor, but that seems to be callback-based rather than coroutine-based?
Yep.
The run_in_executor call is not callback-based -- the confusion probably stems from the name of the function argument ('callback'). It actually returns a Future representing the result (or error) of an operation, where the operation is represented by the function argument. So if you have e.g. a function def factorial(n): return 1 if n <= 0 else n*factorial(n-1) you can run it in an executor from your async(io) code like this: loop = asyncio.get_event_loop() result = yield from loop.run_in_executor(factorial, 100) (In a PEP 492 coroutine substitute await for yield from.) -- --Guido van Rossum (python.org/~guido)
(Yury gave similar responses, so (a) I'll just respond here, and (b)
it's encouraging that you both responded so quickly with the same
message)
On 5 May 2015 at 20:44, Brett Cannon
That's not to say that everything needs to be beginner-friendly, but it *does* mean that it's hard for the wider Python community to meaningfully comment, or evaluate or sanity-check the design. We're left with a sense of "trust us, it makes sense if you need it, everyone else can ignore it".
Watch David Beazley's talk from PyCon this year and you can watch him basically re-implement asyncio on stage in under 45 minutes. It's not as complicated as it seems when you realize there is an event loop driving everything (which people have been leaving out of the conversation since it doesn't tie into the syntax directly).
I'll watch that - it should be fun. But I have seen things like that before, and I've got an idea how to write an event loop. You're right that it's easy to lose track of the fundamentally simple idea in all the complex discussions. To me that feels like a peculiar failure of the abstraction, in that in some circumstances it makes things feel *more* complex than they are :-)
Personally, I feel as if PEP 492 is looking a little premature - maybe the focus should be on making asyncio more accessible first, and *then* adding syntax.
I think this ties the concept of adding syntax to Python to make coroutine-based programming easier too much to asyncio; the latter is just an implementation of the former. This PEP doesn't require asyncio beyond the fact that will be what provides the initial event loop in the stdlib.
It's very hard to separate coroutines from asyncio, because there's no other example (not even a toy one) to reason about. It would probably be helpful to have a concrete example of a basic event loop that did *nothing* but schedule tasks. No IO waiting or similar, just scheduling. I have a gut feeling that event loops are more than just asyncio, but without examples to point to it's hard to keep a focus on that fact. And even harder to isolate "what is an event loop mechanism" from "what is asyncio specific". For example, asyncio.BaseEventLoop has a create_connection method. That's *obviously* not a fundamental aspect of a generic event loop, But call_soon (presumably) is. Having a documented "basic event loop" interface would probably help emphasise the idea than event loops don't have to be asyncio. (Actually, what *is* the minimal event loop interface that is needed for the various task/future mechanisms to work, independently of asyncio? And what features of an event loop etc are needed for the PEP, if it's being used outside of asyncio?) I guess the other canonical event loop use case is GUI system message dispatchers.
You can argue that the syntax is needed to help make async more accessible - but if that's the case then the terminology debates and confusion are clear evidence that it's not succeeding in that goal.
Perhaps, but arguing about the nitty-gritty details of something doesn't automatically lead to a clearer understanding of the higher level concept. Discussing how turning a steering wheel in a car might help you grasp how cars turn, but it isn't a requirement to get "turn the wheel left to make the car go left".
Fair point. If only I could avoid driving into walls :-)
Of course, that's based on my perception of one of the goals of the PEP as being "make coroutines and asyncio more accessible", If the actual goals are different, my conclusion is invalid.
I think the goal is "make coroutines easier to use" and does not directly relate to asyncio.
OK. But in that case, some examples using a non-asyncio toy "just schedule tasks" event loop might help.
Well, twisted always had defer_to_thread. Asyncio has run_in_executor, but that seems to be callback-based rather than coroutine-based?
Yep.
... and so you can't use it with async/await?
Many people use requests for their web access. There are good reasons for this. Are you saying that until someone steps up and writes an async implementation of requests, I have to make a choice - requests or asyncio?
I believe so; you need something to implement __await__. This is true in any language that implements co-routines.
Unfortunately, I can't see myself choosing asyncio in that situation. Which again means that asyncio becomes "something that the average user can't use". Which in turn further entrenches it as a specialist-only tool.
You forgot to append "... yet" to that statement. Just because something isn't available out of the box without some effort to support doesn't mean it will never happen, else there would be absolutely no Python 3 users out there.
Fair point. Yuri mentioned aiohttp, as well. The one difference between this and Python 2/3, is that here you *have* to have two separate implementations. There's no equivalent of a "shared source" async and synchronous implementation of requests. So the typical "please support Python 3" issue that motivates projects to move forward doesn't exist in the same way. It's not to say that there won't be async versions of important libraries, it's just hard to see how the dynamics will work. I can't see myself raising an issue on cx_Oracle saying "please add asyncio support", and I don't know who else I would ask...
Co-routine-based asynchronous programming is new to Python, so as a community we don't have it as something everyone learns over time. If you don't come from an area that supports it then it will be foreign to you and not make sense without someone giving you a good tutorial on it. But considering C#, Dart, and Ecmascript 6 (will) have co-routine support -- and those are just the languages I can name off the top of my head -- using the exact same keywords suggests to me that it isn't *that* difficult of a topic to teach people. This is just one of those PEPs where you have to trust the people with experience in the area are making good design decisions for those of us who aren't in a position to contribute directly without more experience in the domain.
That's also a fair point, and it seems to me that there *is* reasonably general feeling that the experts can be trusted on the basic principles. There's also a huge amount of bikeshedding, but that's pretty much inevitable :-) But I do think that unless someone does something to offer some non-asyncio examples of coroutine-based asynchronous programming in Python, the link in people's minds between async and asyncio will become more and more entrenched. While asyncio is the only real event loop implementation, saying "async can be used for things other than asyncio" is a rather theoretical point. Is there anyone who feels they could write a stripped down but working example of a valid Python event loop *without* the asyncio aspects? Or is that what David Beazley's talk does? (I got the impression from what you said that he was aiming at async IO rather than just a non-IO event loop). Can asyncio.Future and asyncio.Task be reused with such an event loop, or would those need to be reimplemented as well? Writing your own event loop seems like a plausible exercise. Writing your own version of the whole task/future/coroutine/queue/synchronisation mechanisms seems like a lot to expect. And the event loop policy mechanism says that it works with loops that implement asyncio.BaseEventLoop (which as noted includes things like create_connection, etc). Paul
On 5 May 2015 at 21:38, Guido van Rossum
Jumping in to correct one fact.
On Tue, May 5, 2015 at 12:44 PM, Brett Cannon
wrote: On Tue, May 5, 2015 at 3:14 PM Paul Moore
wrote: Well, twisted always had defer_to_thread. Asyncio has run_in_executor, but that seems to be callback-based rather than coroutine-based?
Yep.
The run_in_executor call is not callback-based -- the confusion probably stems from the name of the function argument ('callback'). It actually returns a Future representing the result (or error) of an operation, where the operation is represented by the function argument. So if you have e.g. a function
def factorial(n): return 1 if n <= 0 else n*factorial(n-1)
you can run it in an executor from your async(io) code like this:
loop = asyncio.get_event_loop() result = yield from loop.run_in_executor(factorial, 100)
(In a PEP 492 coroutine substitute await for yield from.)
Thanks, that's an important correction. Given that, run_in_executor is the link to blocking calls that I was searching for. And yes, the "callback" terminology does make this far from obvious, unfortunately. As does the point at which it's introduced (before futures have been described) and the fact that it says "this method is a coroutine" rather than "this method returns a Future"[1]. Paul [1] I'm still struggling to understand the terminology, so if those two statements are equivalent, that's not yet obvious to me.
On Tue, May 5, 2015 at 1:39 PM, Paul Moore
It's very hard to separate coroutines from asyncio, because there's no other example (not even a toy one) to reason about.
What about Greg Ewing's example? http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yf_current/Exa... -- --Guido van Rossum (python.org/~guido)
On 2015-05-05 4:39 PM, Paul Moore wrote:
Is there anyone who feels they could write a stripped down but working example of a valid Python event loop*without* the asyncio aspects? Or is that what David Beazley's talk does? Yes, in David's talk, where he starts to use 'yield from' you can simply use new coroutines.
Yury
On Tue May 5 21:44:26 CEST 2015,Brett Cannon wrote:
It's not as complicated as it seems when you realize there is an event loop driving everything (which people have been leaving out of the conversation since it doesn't tie into the syntax directly).
Another reason people don't realize it is that the PEP goes out of its way to avoid saying so. I understand that you (and Yuri) don't want to tie the PEP too tightly to the specific event loop implementation in asyncio.events.AbstractEventLoop, but ... that particular conflation isn't really what people are confused about. "coroutines" often brings up thoughts of independent tasks. Yuri may well know that "(Python has asymmetric coroutines, that's it)", but others have posted that this was a surprise -- and the people posting here have far more python experience than most readers will. Anyone deeply involved enough to recognize that this PEP is only about (1) a particular type of co-routine -- a subset even of prior python usage (2) used for a particular purpose (3) coordinated via an external scheduler will already know that they can substitute other event loops. Proposed second paragraph of the abstract: This PEP assumes that the asynchronous tasks are scheduled and coordinated by an Event Loop similar to that of stdlib module asyncio.events.AbstractEventLoop. While the PEP is not tied to any specific Event Loop implementation, it is relevant only to the kind of coroutine that uses "yield" as a signal to the scheduler, indicating that the coroutine will be waiting until an event (such as IO) is completed. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ
On Tue, May 5, 2015 at 1:44 PM, Paul Moore
[Guido]
The run_in_executor call is not callback-based -- the confusion probably stems from the name of the function argument ('callback'). It actually returns a Future representing the result (or error) of an operation, where the operation is represented by the function argument. So if you have e.g. a function
def factorial(n): return 1 if n <= 0 else n*factorial(n-1)
you can run it in an executor from your async(io) code like this:
loop = asyncio.get_event_loop() result = yield from loop.run_in_executor(factorial, 100)
(In a PEP 492 coroutine substitute await for yield from.)
Thanks, that's an important correction. Given that, run_in_executor is the link to blocking calls that I was searching for. And yes, the "callback" terminology does make this far from obvious, unfortunately. As does the point at which it's introduced (before futures have been described) and the fact that it says "this method is a coroutine" rather than "this method returns a Future"[1].
Paul
[1] I'm still struggling to understand the terminology, so if those two statements are equivalent, that's not yet obvious to me.
I apologize for the confusing documentation. We need more help from qualified tech writers! Writing PEP 3156 was a huge undertaking for me; after that I was exhausted and did not want to take on writing the end user documentation as well, so it was left unfinished. :-( In PEP 3156 (asyncio package) there are really three separate concepts: - Future, which is a specific class (of which Task is a subclass); - coroutine, by which in this context is meant a generator object obtained by calling a generator function decorated with @asyncio.coroutine and written to conform to the asyncio protocol for coroutines (i.e. don't use bare yield, only use yield from, and the latter always with either a Future or a coroutine as argument); - either of the above, which is actually the most common requirement -- most asyncio functions that support one also support the other, and either is allowable as the argument to `yield from`. In the implementation we so often flipped between Future and coroutine that I imagine sometimes the implementation and docs differ; also, we don't have a good short name for "either of the above" so we end up using one or the other as a shorthand. *Unless* you want to attach callbacks, inspect the result or exception, or cancel it (all of which require a Future), your code shouldn't be concerned about the difference -- you should just use `res = yield from func(args)` and use try/except to catch exceptions if you care. And if you do need a Future, you can call the function asyncio.async() on it (which in PEP 492 is renamed to ensure_future()). In the PEP 492 world, these concepts map as follows: - Future translates to "something with an __await__ method" (and asyncio Futures are trivially made compliant by defining Future.__await__ as an alias for Future.__iter__); - "asyncio coroutine" maps to "PEP 492 coroutine object" (either defined with `async def` or a generator decorated with @types.coroutine -- note that @asyncio.coroutine incorporates the latter); - "either of the above" maps to "awaitable". -- --Guido van Rossum (python.org/~guido)
On 5 May 2015 at 21:57, Guido van Rossum
On Tue, May 5, 2015 at 1:39 PM, Paul Moore
wrote: It's very hard to separate coroutines from asyncio, because there's no other example (not even a toy one) to reason about.
What about Greg Ewing's example? http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yf_current/Exa...
That doesn't cover any of the higher level abstractions like tasks or futures (at least not by those names or with those interfaces). And I don't see where the PEP 492 additions would fit in (OK, "replace yield from with await" is part of it, but I don't see the rest). We may be talking at cross purposes here. There's a lot of asyncio that doesn't seem to me to be IO-related. Specifically the future and task abstractions. I view those as relevant to "coroutine programming in Python" because they are referenced in any discussion of coroutines (you yield from a future, for example). If you see them as purely asyncio related (and not directly usable from outside of an asyncio context) then that may explain some of my confusion (but at the cost of reducing the coroutine concept to something pretty trivial in the absence of a library that independently implements these concepts). In some ways I wish there had been an "asyncio" library that covered the areas that are fundamentally about IO multiplexing. And a separate library (just "async", maybe, although that's now a bad idea as it clashes with a keyword :-)) that covered generic event loop, task and synchronisation areas. But that's water under the bridge now. Paul
On 5 May 2015 at 22:12, Guido van Rossum
I apologize for the confusing documentation. We need more help from qualified tech writers! Writing PEP 3156 was a huge undertaking for me; after that I was exhausted and did not want to take on writing the end user documentation as well, so it was left unfinished. :-(
Fair enough. When I properly document one of my projects, *then* I'll think about complaining :-) These things happen.
In PEP 3156 (asyncio package) there are really three separate concepts:
- Future, which is a specific class (of which Task is a subclass);
- coroutine, by which in this context is meant a generator object obtained by calling a generator function decorated with @asyncio.coroutine and written to conform to the asyncio protocol for coroutines (i.e. don't use bare yield, only use yield from, and the latter always with either a Future or a coroutine as argument);
- either of the above, which is actually the most common requirement -- most asyncio functions that support one also support the other, and either is allowable as the argument to `yield from`.
In the implementation we so often flipped between Future and coroutine that I imagine sometimes the implementation and docs differ; also, we don't have a good short name for "either of the above" so we end up using one or the other as a shorthand.
OK, that makes a lot of sense.
*Unless* you want to attach callbacks, inspect the result or exception, or cancel it (all of which require a Future), your code shouldn't be concerned about the difference -- you should just use `res = yield from func(args)` and use try/except to catch exceptions if you care. And if you do need a Future, you can call the function asyncio.async() on it (which in PEP 492 is renamed to ensure_future()).
Again, makes sense. Although there are some bits of example code in the docs that call asyncio.async() on a coroutine and throw away the result (for example, https://docs.python.org/3/library/asyncio-task.html#example-future-with-run-...). That confuses me. Are you saying that async() modifies its (coroutine) argument to make it a Future? Rather than wrapping a coroutine in a Future, which gets returned?
In the PEP 492 world, these concepts map as follows:
- Future translates to "something with an __await__ method" (and asyncio Futures are trivially made compliant by defining Future.__await__ as an alias for Future.__iter__);
- "asyncio coroutine" maps to "PEP 492 coroutine object" (either defined with `async def` or a generator decorated with @types.coroutine -- note that @asyncio.coroutine incorporates the latter);
- "either of the above" maps to "awaitable".
OK. Although "future" is a nicer term than "something with an __await__ method" and the plethora of flavours of coroutine is not great. But given that the only term we'll need in common cases is "awaitable", it's still a net improvement. So in the PEP 492 world, there's no such thing as a Task outside of asyncio? Or, to put it another way, a Task is only relevant in an IO context (unless an alternative event loop library implemented a similar concept), and we should only be talking in terms of awaitables and futures (given concurrent.futures and asyncio, I doubt you're going to be able to stop people using "Future" for the generic term for "something with an __await__ method" at best, and quite possibly as equivalent to "awaitable", unfortunately). Paul
Tue May 5 21:48:36 CEST 2015, Yury Selivanov wrote:
As for terminology, I view this discussion differently. It's not about the technical details (Python has asymmetric coroutines, that's it), but rather on how to disambiguate coroutines implemented with generators and yield-from, from new 'async def' coroutines.
Not just "How?", but "Why?". Why do they *need* to be disambiguated? With the benefit of having recently read all that discussion (as opposed to just the PEP), my answer is ... uh ... that generators vs "async def" is NOT an important distinction. What matters (as best I can tell) is: "something using yield (or yield from) to mark execution context switches" vs "other kinds of callables, including those using yield to make an iterator" I'm not quite sure that the actual proposal even really separates them effectively, in part because the terminology keeps suggesting other distinctions instead. (The glossary does help; just not enough.) -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ
On 5 May 2015 at 22:09, Jim J. Jewett
Proposed second paragraph of the abstract:
This PEP assumes that the asynchronous tasks are scheduled and coordinated by an Event Loop similar to that of stdlib module asyncio.events.AbstractEventLoop. While the PEP is not tied to any specific Event Loop implementation, it is relevant only to the kind of coroutine that uses "yield" as a signal to the scheduler, indicating that the coroutine will be waiting until an event (such as IO) is completed.
+1. If that's not accurate then by all means correct any mistakes in it. But assuming it *is* accurate, it would help a lot. Paul
On 2015-05-05 5:31 PM, Jim J. Jewett wrote:
Tue May 5 21:48:36 CEST 2015, Yury Selivanov wrote:
As for terminology, I view this discussion differently. It's not about the technical details (Python has asymmetric coroutines, that's it), but rather on how to disambiguate coroutines implemented with generators and yield-from, from new 'async def' coroutines. Not just "How?", but "Why?".
Why do they*need* to be disambiguated?
To clearly show how one interacts with the other, to explain how backwards compatibility is implemented, and to better illustrate some additional (and necessary) restrictions we put on 'async def' coroutines. Otherwise, the PEP would be completely unreadable :) Yury
On Tue, May 5, 2015 at 2:29 PM, Paul Moore
On 5 May 2015 at 22:12, Guido van Rossum
wrote: I apologize for the confusing documentation. We need more help from qualified tech writers! Writing PEP 3156 was a huge undertaking for me; after that I was exhausted and did not want to take on writing the end user documentation as well, so it was left unfinished. :-(
Fair enough. When I properly document one of my projects, *then* I'll think about complaining :-) These things happen.
In PEP 3156 (asyncio package) there are really three separate concepts:
- Future, which is a specific class (of which Task is a subclass);
- coroutine, by which in this context is meant a generator object obtained by calling a generator function decorated with @asyncio.coroutine and written to conform to the asyncio protocol for coroutines (i.e. don't use bare yield, only use yield from, and the latter always with either a Future or a coroutine as argument);
- either of the above, which is actually the most common requirement -- most asyncio functions that support one also support the other, and either is allowable as the argument to `yield from`.
In the implementation we so often flipped between Future and coroutine that I imagine sometimes the implementation and docs differ; also, we don't have a good short name for "either of the above" so we end up using one or the other as a shorthand.
OK, that makes a lot of sense.
*Unless* you want to attach callbacks, inspect the result or exception, or cancel it (all of which require a Future), your code shouldn't be concerned about the difference -- you should just use `res = yield from func(args)` and use try/except to catch exceptions if you care. And if you do need a Future, you can call the function asyncio.async() on it (which in PEP 492 is renamed to ensure_future()).
Again, makes sense. Although there are some bits of example code in the docs that call asyncio.async() on a coroutine and throw away the result (for example,
https://docs.python.org/3/library/asyncio-task.html#example-future-with-run-... ). That confuses me. Are you saying that async() modifies its (coroutine) argument to make it a Future? Rather than wrapping a coroutine in a Future, which gets returned?
No, it wraps a coroutine (i.e. a generator) in a Task, but leaves a Future alone. I'm stumped why that example calls async() and then throws the result away. I suspect it won't work without it (or else Victor wouldn't have added the call) but the reason seems, um, deep. I think wrapping it in a Task enters the generator in the event loop's queue of runnables -- otherwise the generator may well be garbage-collected without ever running. Such complexity doesn't belong in such a simple example though.
In the PEP 492 world, these concepts map as follows:
- Future translates to "something with an __await__ method" (and asyncio Futures are trivially made compliant by defining Future.__await__ as an alias for Future.__iter__);
- "asyncio coroutine" maps to "PEP 492 coroutine object" (either defined with `async def` or a generator decorated with @types.coroutine -- note that @asyncio.coroutine incorporates the latter);
- "either of the above" maps to "awaitable".
OK. Although "future" is a nicer term than "something with an __await__ method" and the plethora of flavours of coroutine is not great. But given that the only term we'll need in common cases is "awaitable", it's still a net improvement.
So in the PEP 492 world, there's no such thing as a Task outside of asyncio? Or, to put it another way, a Task is only relevant in an IO context (unless an alternative event loop library implemented a similar concept), and we should only be talking in terms of awaitables and futures (given concurrent.futures and asyncio, I doubt you're going to be able to stop people using "Future" for the generic term for "something with an __await__ method" at best, and quite possibly as equivalent to "awaitable", unfortunately).
I'm not sure. But it's true that Futures and Tasks in asyncio serve the purpose of linking the event loop (whose basic functioning is callback-based) to coroutines (implemented by generators). The basic idea is that when some I/O completes the event loop will call a callback function registered for that particular I/O operation; the callback then is actually a bound method of a Future or Task that causes the latter to be marked as "complete" (i.e. having a result) which in turn will call other callbacks (registered with the Future using add_done_callback()); in the case of a Task (i.e. a special kind of Future that wraps a generator/coroutine) this will resume the coroutine. (Actually it may resume an entire stack of coroutines that are blocked waiting for each other at yield-from; in my spare time I'm working on an explanation of the machinery underlying yield, yield from and await that will explain this.) It's likely that you could write a much simpler event loop by assuming just coroutines (either the kind implemented by generators using yield from or the PEP 492 kind). The reason asyncio uses callbacks at the lower levels is the hope of fostering interoperability with Twisted and Tornado (and even gevent, which also has an event loop at the bottom of everything). -- --Guido van Rossum (python.org/~guido)
On May 5, 2015 2:14 PM, "Guido van Rossum"
In the PEP 492 world, these concepts map as follows:
- Future translates to "something with an __await__ method" (and asyncio
Futures are trivially made compliant by defining Future.__await__ as an alias for Future.__iter__);
- "asyncio coroutine" maps to "PEP 492 coroutine object" (either defined
with `async def` or a generator decorated with @types.coroutine -- note that @asyncio.coroutine incorporates the latter);
- "either of the above" maps to "awaitable".
Err, aren't the first and third definitions above identical? Surely we want to say: an async def function is a convenient shorthand for creating a custom awaitable (exactly like how generators are a convenient shorthand for creating custom iterators), and a Future is-an awaitable that also adds some extra methods. -n
On Tue, May 5, 2015 at 3:01 PM, Nathaniel Smith
In the PEP 492 world, these concepts map as follows:
- Future translates to "something with an __await__ method" (and asyncio
Futures are trivially made compliant by defining Future.__await__ as an alias for Future.__iter__);
- "asyncio coroutine" maps to "PEP 492 coroutine object" (either defined
with `async def` or a generator decorated with @types.coroutine -- note that @asyncio.coroutine incorporates the latter);
- "either of the above" maps to "awaitable".
Err, aren't the first and third definitions above identical?
Surely we want to say: an async def function is a convenient shorthand for creating a custom awaitable (exactly like how generators are a convenient shorthand for creating custom iterators), and a Future is-an awaitable that also adds some extra methods.
The current PEP 492 proposal does endow the object returned by calling an async function (let's call it a coroutine object) with an __await__ method. And there's a good reason for this -- the bytecode generated for await treats coroutine objects special, just like the bytecode generated for yield-from treats generator objects special. The special behavior they have in common is the presence of send() and throw() methods, which are used to allow send() and throw() calls on the outer generator to be passed into the inner generator with minimal fuss. (This is the reason why "yield from X" is *not* equivalent to "for x in X: yield x".) @Yury: I have a feeling the PEP could use more clarity here -- perhaps the section "Await Expression" should explain what the interepreter does for each type of awaitable? -- --Guido van Rossum (python.org/~guido)
Jim, On 2015-05-05 5:09 PM, Jim J. Jewett wrote:
On Tue May 5 21:44:26 CEST 2015,Brett Cannon wrote:
It's not as complicated as it seems when you realize there is an event loop driving everything (which people have been leaving out of the conversation since it doesn't tie into the syntax directly). [..] Proposed second paragraph of the abstract:
This PEP assumes that the asynchronous tasks are scheduled and coordinated by an Event Loop similar to that of stdlib module asyncio.events.AbstractEventLoop. While the PEP is not tied to any specific Event Loop implementation, it is relevant only to the kind of coroutine that uses "yield" as a signal to the scheduler, indicating that the coroutine will be waiting until an event (such as IO) is completed.
Thank you for this suggestion. I've added it to the PEP: https://hg.python.org/peps/rev/7ac132b24f1f Yury
Paul Moore wrote:
It would probably be helpful to have a concrete example of a basic event loop that did *nothing* but schedule tasks. No IO waiting or similar, just scheduling.
Take a look at the example I developed when working on the yield-from pep: http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yf_current/Exa... The first version of the event loop presented there does exactly that, just schedules tasks in a round- robin fashion. Then I gradually add more features to it.
Actually, what *is* the minimal event loop interface that is needed for the various task/future mechanisms to work, independently of asyncio?
I don't it's possible to answer that question, because there isn't a single answer. The minimal set of features that an event loop needs depends on what you want to achieve with it. Even the notion of "just schedules tasks" is ambiguous. What does "schedule" mean? Does it just mean round-robin switching between them, or should they be able to synchronise with each other in some way? Should it be possible for a task to suspend itself for an interval of real world time, or does that come under the heading of I/O (since you're waiting for an external event, i.e. the computer's clock reaching some time)? Etc.
And what features of an event loop etc are needed for the PEP, if it's being used outside of asyncio?)
I don't think *any* particular event loop features are needed. You can't pick out any of these features as being "core". For example, it would be possible to have an event loop that handled socket I/O but *didn't* do round-robin scheduling -- it could just keep on running the same task, even if it yielded, until it blocked waiting for an external event. Such a scheduler would probably be quite adequate for many use cases. It seems to me that the idea of "generator-based tasks managed by an event loop" is more of a design pattern than something you can write a detailed API specification for. Another problem with the "core" idea is that you can't start with an event loop that "just does scheduling" and then add on other features such as I/O *from the outside*. There has to be some point at which everything comes together, which means choosing something like select() or poll() or I/O completion queues, and build that into the heart of your event loop. At that point it's no longer something with a simple core. -- Greg
Paul Moore wrote:
What about Greg Ewing's example? http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yf_current/Exa...
That doesn't cover any of the higher level abstractions like tasks or futures (at least not by those names or with those interfaces).
Because a minimal event loop doesn't *need* those. In my little scheduler, a "task" is nothing more than a yield-frommable object sitting on a queue of things to be run. There is no need to wrap it in another object. And there's really no need for the concept of a "future" at all, except maybe at the boundary between generator-based async code and other things that are based on callbacks. Even then, a "future" is really just "an object that can be passed to yield-from". There is no need for a concrete Future class, it's just a protocol.
And I don't see where the PEP 492 additions would fit in (OK, "replace yield from with await" is part of it, but I don't see the rest).
That's really all there is to it. The rest is concerned with catching certain kinds of mistakes, and providing convenient syntax for some patterns of using 'await'.
There's a lot of asyncio that doesn't seem to me to be IO-related. Specifically the future and task abstractions. I view those as relevant to "coroutine programming in Python" because they are referenced in any discussion of coroutines (you yield from a future, for example).
Only because they've been elevated to prominence by asyncio and its documentation, which I regard as unfortunate. When Guido was designing asyncio, I tried very hard to dissuade him from giving futures such a central place in the model. I saw them as an unnecessary concept that would only clutter up people's thinking. Seeing all the confusion now, I'm more convinced than ever that I was right. :-(
In some ways I wish there had been an "asyncio" library that covered the areas that are fundamentally about IO multiplexing. And a separate library (just "async", maybe, although that's now a bad idea as it clashes with a keyword :-)) that covered generic event loop, task and synchronisation areas.
As I said before, I don't think it's really possible to factor an event loop into those kinds of parts. You may be able to factor the *API* that way, but any given implementation has to address all the parts at once. -- Greg
Hi Yury, On 05.05.2015 20:25, Yury Selivanov wrote:
We forget to address the major problems here. How can someone in a "sync" script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way.
asyncio and twisted answered these questions ;) The answer is that you have to write async implementations.
gevent has a different answer, but greenlents/stackless is something that will never be merged in CPython and other implementations.
I think monkeypatching and the gevent way is wrong. And explicit is better than implicit. I have to clear this. I meant how can we make this async stuff more accessible to the average sync user. Sometime even without writing or knowing how to write coroutines or other async stuff. Let me explain it with a deeper example (Some of it is related to Python 2 and twisted): I had the same problem for a server application using twisted. Provide easy interfaces to my users most not aware of async stuff. My solution was to write my own decorator similar to twisted's @inlineCallbacks. On top of it I added one more level to the decorator and distinguish if it was called from the main thread (also running the mainloop) and other threads. This object usable also as decorator is called "task" and has some utility methods. And returns a "deferred". (in asyncio this would be a Future) Resulting in code like: @task def myAsyncFunction(webaddress): result = yield fetch(webaddress) # only to show sleep example yield task.sleep(0.1) task.Return(result) Usable in a sync context (extra script thread): def do(): content = myAsyncFunction("http://xyz") or async context: @task def ado(): content = yield myAsyncFunction("http://xyz") The task decorator has functionality to check if something is called from the main thread (-> also a task and async) or it is called from another thread (-> sync or script). So this async interface is usable from both worlds. If someone operates async he/she must only know the task decorator and when to yield. If he/she uses it in sync mode nothing special has to be done. To allow all this the server starts the async main loop in the main thread and executes the script in an extra script thread. The user has every time his own thread, also for rpc stuff. The only way to switch into the main loop is to decorate a function as @task, every task is a coroutine and executed in the main thread (also thread of main loop). Benefit of all this: - Easy to write a async task it is marked as one and special stuff belongs to the task object. (task.Return is needed because we are in Python 2) - The normal user executes his stuff in his own thread and he/she can program in sync mode. No problem it is an extra thread and the main loop does not block. - A network client or other stuff has to be written only once, most time this can be a @task in the async world. But this should not care the end user. We don't have to implement all twice once for async and once for the sync world. -> Less overhead This is what I mean if I say we must address the bridging problem between the worlds. It think it is the wrong way to divide it in async and sync stuff and duplicate all networking libraries in the sync and async ones. For me the answer is to write one async netowrk library and use it in both, a sync script and in an async main loop. With an easy interface and not forcing the user to know this is an async library I have to do something special. And in future go one step further and use all this combined with PyParallel to solve the multiprocessing problem in Python. (Script in extra thread, mainloop in main thread, executed and managed via PyParallel avoiding the gil) But this is only a vision/dream of mine.
And for all this we still can't tell them "oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5". It does not and it is only most useful for networking stuff nothing more.
"networking stuff", and in particular, web, is a huge part of current Python usage. Please don't underestimate that.
I do not. But for most they want only use it as a client and the main concern for most is "I want to get this web page" and not "I will implement a web client have to do this async and get it". Regards, Wolfgang
On 6 May 2015 at 09:20, Greg Ewing
That doesn't cover any of the higher level abstractions like tasks or futures (at least not by those names or with those interfaces).
Because a minimal event loop doesn't *need* those.
It doesn't *need* them, but as abstractions they allow easier building of reusable higher-level libraries. You can write an event loop with nothing but coroutines, but to build reusable libraries on top of it, you need some common interfaces.
In my little scheduler, a "task" is nothing more than a yield-frommable object sitting on a queue of things to be run. There is no need to wrap it in another object.
And there's really no need for the concept of a "future" at all, except maybe at the boundary between generator-based async code and other things that are based on callbacks. Even then, a "future" is really just "an object that can be passed to yield-from". There is no need for a concrete Future class, it's just a protocol.
Agreed, you don't need a Future class, all you need is to agree what reusable code is allowed to do with the core objects you are passing around - that's how duck typing works. The objects *I* can see are futures (in a PEP 492 world, "awaitables" which may or may not be equivalent in terms of the operations you'd want to focus on) and the event loop itself. In your example, the event loop is implicit (as it's a singleton, you use global functions rather than methods on the loop object) but that's a minor detail.
And I don't see where the PEP 492 additions would fit in (OK, "replace yield from with await" is part of it, but I don't see the rest).
That's really all there is to it. The rest is concerned with catching certain kinds of mistakes, and providing convenient syntax for some patterns of using 'await'.
So, "things you can wait on" have one operation - "wait for a result". That's OK. You can create such things as coroutines, which is also fine. You may want to create such things explicitly (equivalent to generators vs __iter__) - maybe that's where __aiter__ comes in in PEP 492 and the Future class in asyncio. Again, all fine. You also need operations like "schedule a thing to run", which is the event loop "interface". Your sample has the following basic event loop methods that I can see: run, schedule, unschedule, and expire_timeslice (that last one may be an implementation detail, but the other 3 seem pretty basic). PEP 492 has nothing to say on the event loop side of things (something that became clear to me during this discussion).
There's a lot of asyncio that doesn't seem to me to be IO-related. Specifically the future and task abstractions. I view those as relevant to "coroutine programming in Python" because they are referenced in any discussion of coroutines (you yield from a future, for example).
Only because they've been elevated to prominence by asyncio and its documentation, which I regard as unfortunate.
When Guido was designing asyncio, I tried very hard to dissuade him from giving futures such a central place in the model. I saw them as an unnecessary concept that would only clutter up people's thinking. Seeing all the confusion now, I'm more convinced than ever that I was right. :-(
Futures seem to me to be (modulo a few details) what "awaitables" are in PEP 492. I can't see how you can meaningfully talk about event loops in a Python context without having *some* term for "things you wait for". Maybe Future wasn't a good name, and maybe the parallel with concurrent.futures.Future wasn't helpful (I think both things were fine, but you may not) but we had to have *some* way of talking about them, and of constructing standalone awaitables. PEP 492 has new, and hopefully better, ways, but I think that awaitables *have* to be central to any model where you wait for things... By the way, it feels to me like I'm now arguing in favour of PEP 492 with a reasonable understanding of what it "means". Assuming what I said above isn't complete rubbish, thanks to everyone who's helped me get to this point of understanding through this thread! (And if I haven't understood, that's my fault, and still thanks to everyone for their efforts :-)) Paul
Guido van Rossum wrote:
the bytecode generated for await treats coroutine objects special, just like the bytecode generated for yield-from treats generator objects special. The special behavior they have in common is the presence of send() and throw() methods,
I don't think that's quit accurate. Yield-from treats any object having send() and throw() methods the same way it treats a generator -- there's nothing special about the generator *type*. Presumably 'await' is the same. -- Greg
On Tue, May 5, 2015 at 1:39 PM, Paul Moore
It would probably be helpful to have a concrete example of a basic event loop that did *nothing* but schedule tasks. No IO waiting or similar, just scheduling. I have a gut feeling that event loops are more than just asyncio, but without examples to point to it's hard to keep a focus on that fact. And even harder to isolate "what is an event loop mechanism" from "what is asyncio specific". For example, asyncio.BaseEventLoop has a create_connection method. That's *obviously* not a fundamental aspect of a generic event loop, But call_soon (presumably) is. Having a documented "basic event loop" interface would probably help emphasise the idea than event loops don't have to be asyncio. (Actually, what *is* the minimal event loop interface that is needed for the various task/future mechanisms to work, independently of asyncio? And what features of an event loop etc are needed for the PEP, if it's being used outside of asyncio?)
Twisted has a pretty good taxonomy of event loop methods, in the interfaces at the bottom of this page: http://twistedmatrix.com/documents/15.1.0/core/howto/reactor-basics.html and the comparison matrix at http://twistedmatrix.com/documents/15.1.0/core/howto/choosing-reactor.html The asyncio event loops implement most of these (not the exact interfaces, but the same functionality). Tornado implements FDSet, Time, and part of Threads in the IOLoop itself, with the rest of the functionality coming from separate classes. (You may wonder then why Twisted and asyncio put everything in the core event loop? It's necessary to abstract over certain platform differences, which is one big reason why Tornado has poor support for Windows). -Ben
I guess the other canonical event loop use case is GUI system message dispatchers.
You can argue that the syntax is needed to help make async more accessible - but if that's the case then the terminology debates and confusion are clear evidence that it's not succeeding in that goal.
Perhaps, but arguing about the nitty-gritty details of something doesn't automatically lead to a clearer understanding of the higher level concept. Discussing how turning a steering wheel in a car might help you grasp how cars turn, but it isn't a requirement to get "turn the wheel left to make the car go left".
Fair point. If only I could avoid driving into walls :-)
Of course, that's based on my perception of one of the goals of the PEP as being "make coroutines and asyncio more accessible", If the actual goals are different, my conclusion is invalid.
I think the goal is "make coroutines easier to use" and does not directly relate to asyncio.
OK. But in that case, some examples using a non-asyncio toy "just schedule tasks" event loop might help.
Well, twisted always had defer_to_thread. Asyncio has run_in_executor, but that seems to be callback-based rather than coroutine-based?
Yep.
... and so you can't use it with async/await?
Many people use requests for their web access. There are good reasons for this. Are you saying that until someone steps up and writes an async implementation of requests, I have to make a choice - requests or asyncio?
I believe so; you need something to implement __await__. This is true in any language that implements co-routines.
Unfortunately, I can't see myself choosing asyncio in that situation. Which again means that asyncio becomes "something that the average user can't use". Which in turn further entrenches it as a specialist-only tool.
You forgot to append "... yet" to that statement. Just because something isn't available out of the box without some effort to support doesn't mean it will never happen, else there would be absolutely no Python 3 users out there.
Fair point. Yuri mentioned aiohttp, as well. The one difference between this and Python 2/3, is that here you *have* to have two separate implementations. There's no equivalent of a "shared source" async and synchronous implementation of requests. So the typical "please support Python 3" issue that motivates projects to move forward doesn't exist in the same way. It's not to say that there won't be async versions of important libraries, it's just hard to see how the dynamics will work. I can't see myself raising an issue on cx_Oracle saying "please add asyncio support", and I don't know who else I would ask...
Co-routine-based asynchronous programming is new to Python, so as a community we don't have it as something everyone learns over time. If you don't come from an area that supports it then it will be foreign to you and not make sense without someone giving you a good tutorial on it. But considering C#, Dart, and Ecmascript 6 (will) have co-routine support -- and those are just the languages I can name off the top of my head -- using the exact same keywords suggests to me that it isn't *that* difficult of a topic to teach people. This is just one of those PEPs where you have to trust the people with experience in the area are making good design decisions for those of us who aren't in a position to contribute directly without more experience in the domain.
That's also a fair point, and it seems to me that there *is* reasonably general feeling that the experts can be trusted on the basic principles. There's also a huge amount of bikeshedding, but that's pretty much inevitable :-)
But I do think that unless someone does something to offer some non-asyncio examples of coroutine-based asynchronous programming in Python, the link in people's minds between async and asyncio will become more and more entrenched. While asyncio is the only real event loop implementation, saying "async can be used for things other than asyncio" is a rather theoretical point.
Is there anyone who feels they could write a stripped down but working example of a valid Python event loop *without* the asyncio aspects? Or is that what David Beazley's talk does? (I got the impression from what you said that he was aiming at async IO rather than just a non-IO event loop). Can asyncio.Future and asyncio.Task be reused with such an event loop, or would those need to be reimplemented as well? Writing your own event loop seems like a plausible exercise. Writing your own version of the whole task/future/coroutine/queue/synchronisation mechanisms seems like a lot to expect. And the event loop policy mechanism says that it works with loops that implement asyncio.BaseEventLoop (which as noted includes things like create_connection, etc).
Paul _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ben%40bendarnell.com
participants (18)
-
Arnaud Delobelle
-
Ben Darnell
-
Brett Cannon
-
Ethan Furman
-
Glenn Linderman
-
Greg Ewing
-
Guido van Rossum
-
Gustavo Carneiro
-
Jim J. Jewett
-
Koos Zevenhoven
-
Nathaniel Smith
-
Paul Moore
-
Stefan Behnel
-
Steven D'Aprano
-
tds333@gmail.com
-
Wolfgang
-
Wolfgang Langner
-
Yury Selivanov