[ANN] async_generator v1.2 released
Hi all, I just released v1.2 of my async_generator package: https://pypi.org/pypi/async_generator/ This package makes it easy to write PEP 525-style "async generators", even on Python 3.5. It looks like: from async_generator import async_generator, yield_, yield_from_ @async_generator async def parse_json_separated_newlines_from_network(streamreader): async for line in streamreader: await yield_(json.loads(line)) Starting in v1.1 (released last week but not announced widely), the generators are now highly compatible with 3.6's native async generators, including full support for athrow / asend / aclose. They also fully support "yield from", which is otherwise unavailable even on Python 3.6. v1.2 fixed some edge cases in "yield from" -- but the headline feature is that we're now *so* compatible with 3.6's native async generators that you can now use our yield from support directly inside a native generator: from async_generator import yield_from_ async def f(): yield 2 yield 3 # yields [1, 2, 3, 4] async def g(): yield 1 await yield_from_(f()) yield 4 Have fun, -n -- Nathaniel J. Smith -- https://vorpus.org
Does this library offer any advantages over the same functionality in asyncio_extras? 15.11.2016, 08:52, Nathaniel Smith kirjoitti:
Hi all,
I just released v1.2 of my async_generator package:
https://pypi.org/pypi/async_generator/
This package makes it easy to write PEP 525-style "async generators", even on Python 3.5. It looks like:
from async_generator import async_generator, yield_, yield_from_
@async_generator async def parse_json_separated_newlines_from_network(streamreader): async for line in streamreader: await yield_(json.loads(line))
Starting in v1.1 (released last week but not announced widely), the generators are now highly compatible with 3.6's native async generators, including full support for athrow / asend / aclose. They also fully support "yield from", which is otherwise unavailable even on Python 3.6.
v1.2 fixed some edge cases in "yield from" -- but the headline feature is that we're now *so* compatible with 3.6's native async generators that you can now use our yield from support directly inside a native generator:
from async_generator import yield_from_
async def f(): yield 2 yield 3
# yields [1, 2, 3, 4] async def g(): yield 1 await yield_from_(f()) yield 4
Have fun, -n
On Nov 22, 2016 6:22 AM, "Alex Grönholm" <alex.gronholm@nextday.fi> wrote:
Does this library offer any advantages over the same functionality in
asyncio_extras? You would probably know better than me, but last time I looked, the generator code in asyncio_extras didn't support asend/athrow/aclose, didn't support yield from, and took shortcuts that assume that you're using asyncio as your coroutine runner. So I believe async_generator is more complete and more correct (in the sense that it sticks strictly to implementing the language level semantics for async generators without assuming any particular runner). Of course asyncio_extras has lots of other handy things in it too, while async_generator is very strictly focused on just this one thing. (Well, I threw in an implementation of aclosing too, since you can't really use async generators without it, but that's pretty trivial.) It might make sense for asyncio_extras to out-source generator handling to async_generator? Or not. -n
I'm not sure where asyncio_extras's async generator implementation assumes you're using asyncio. Could you elaborate on that? Native async generators don't support "yield from" either, so I didn't try to add that until there's a new PEP that defines its exact semantics. As for aclose/asend/athrow support, I will have to look into that. Could we synchronize the implementations so that they're compatible with each other? I would've preferred there to be only a single implementation (asyncio_extras was published first, but admittedly I didn't advertise it that well) but now it'd be best if we work together, wouldn't you agree? 22.11.2016, 22:32, Nathaniel Smith kirjoitti:
On Nov 22, 2016 6:22 AM, "Alex Grönholm" <alex.gronholm@nextday.fi <mailto:alex.gronholm@nextday.fi>> wrote:
Does this library offer any advantages over the same functionality
in asyncio_extras?
You would probably know better than me, but last time I looked, the generator code in asyncio_extras didn't support asend/athrow/aclose, didn't support yield from, and took shortcuts that assume that you're using asyncio as your coroutine runner. So I believe async_generator is more complete and more correct (in the sense that it sticks strictly to implementing the language level semantics for async generators without assuming any particular runner).
Of course asyncio_extras has lots of other handy things in it too, while async_generator is very strictly focused on just this one thing. (Well, I threw in an implementation of aclosing too, since you can't really use async generators without it, but that's pretty trivial.) It might make sense for asyncio_extras to out-source generator handling to async_generator? Or not.
-n
On Tue, Nov 22, 2016 at 2:22 PM, Alex Grönholm <alex.gronholm@nextday.fi> wrote:
I'm not sure where asyncio_extras's async generator implementation assumes you're using asyncio. Could you elaborate on that?
If I'm reading it right, it assumes that the only two things that might be yielded to the coroutine runner are either (a) the special yield wrapper, or (b) an awaitable object like an asyncio.Future. This works on asyncio, because that's all the asyncio runner supports, but it doesn't work with (for example) curio. async_generator (like native async generators) allows arbitrary objects to be yielded to the coroutine runner.
Native async generators don't support "yield from" either, so I didn't try to add that until there's a new PEP that defines its exact semantics.
Yeah, one of the motivations for async_generator has been to figure out what the semantics for native generators should be :-). The native async generators in 3.6 use the same implementation trick as async_generator (not entirely sure if Yury got it from me, or if it's an independent invention), and the yield from implementation is partly motivated as a proof of concept / testing ground for hopefully getting it into 3.7. The yield from semantics are pretty straightforward though, I think.
As for aclose/asend/athrow support, I will have to look into that.
Could we synchronize the implementations so that they're compatible with each other? I would've preferred there to be only a single implementation (asyncio_extras was published first, but admittedly I didn't advertise it that well) but now it'd be best if we work together, wouldn't you agree?
Makes sense to me :-). I think the obvious approach would be for async_extras to just depend on async_generator, since AFAICT async_generator is pretty much complete, and like you say, there's not much point in carrying around two copies of the same thing. But if you have another suggestion I'd be interested to hear it... -n
22.11.2016, 22:32, Nathaniel Smith kirjoitti:
On Nov 22, 2016 6:22 AM, "Alex Grönholm" <alex.gronholm@nextday.fi> wrote:
Does this library offer any advantages over the same functionality in asyncio_extras?
You would probably know better than me, but last time I looked, the generator code in asyncio_extras didn't support asend/athrow/aclose,
didn't
support yield from, and took shortcuts that assume that you're using asyncio as your coroutine runner. So I believe async_generator is more complete and more correct (in the sense that it sticks strictly to implementing the language level semantics for async generators without assuming any particular runner).
Of course asyncio_extras has lots of other handy things in it too, while async_generator is very strictly focused on just this one thing. (Well, I threw in an implementation of aclosing too, since you can't really use async generators without it, but that's pretty trivial.) It might make sense for asyncio_extras to out-source generator handling to async_generator? Or not.
-n
-- Nathaniel J. Smith -- https://vorpus.org
23.11.2016, 01:34, Nathaniel Smith kirjoitti:
On Tue, Nov 22, 2016 at 2:22 PM, Alex Grönholm <alex.gronholm@nextday.fi <mailto:alex.gronholm@nextday.fi>> wrote:
I'm not sure where asyncio_extras's async generator implementation assumes you're using asyncio. Could you elaborate on that?
If I'm reading it right, it assumes that the only two things that might be yielded to the coroutine runner are either (a) the special yield wrapper, or (b) an awaitable object like an asyncio.Future. This works on asyncio, because that's all the asyncio runner supports, but it doesn't work with (for example) curio. async_generator (like native async generators) allows arbitrary objects to be yielded to the coroutine runner.
You are misreading the code. It is in no way limited to what asyncio accepts. It doesn't even import asyncio in the asyncyield or generator modules. The only parts of the library that depend on PEP 3156 event loops are the ones that involve executors and threads.
Native async generators don't support "yield from" either, so I didn't try to add that until there's a new PEP that defines its exact semantics.
Yeah, one of the motivations for async_generator has been to figure out what the semantics for native generators should be :-). The native async generators in 3.6 use the same implementation trick as async_generator (not entirely sure if Yury got it from me, or if it's an independent invention), and the yield from implementation is partly motivated as a proof of concept / testing ground for hopefully getting it into 3.7. The yield from semantics are pretty straightforward though, I think.
As for aclose/asend/athrow support, I will have to look into that.
Could we synchronize the implementations so that they're compatible with each other? I would've preferred there to be only a single implementation (asyncio_extras was published first, but admittedly I didn't advertise it that well) but now it'd be best if we work together, wouldn't you agree?
Makes sense to me :-). I think the obvious approach would be for async_extras to just depend on async_generator, since AFAICT async_generator is pretty much complete, and like you say, there's not much point in carrying around two copies of the same thing. But if you have another suggestion I'd be interested to hear it...
I may consider that at some point, but I'm not yet comfortable with those extended capabilities. Meanwhile I will add any missing functionality (asend, athrow etc.) to mine. One thing I noticed is that there seems to be no way to detect async generator functions in your implementation. That is something I would want to have before switching.
-n
22.11.2016, 22:32, Nathaniel Smith kirjoitti:
On Nov 22, 2016 6:22 AM, "Alex Grönholm" <alex.gronholm@nextday.fi
<mailto:alex.gronholm@nextday.fi>> wrote:
Does this library offer any advantages over the same functionality in asyncio_extras?
You would probably know better than me, but last time I looked, the generator code in asyncio_extras didn't support asend/athrow/aclose, didn't support yield from, and took shortcuts that assume that you're using asyncio as your coroutine runner. So I believe async_generator is more complete and more correct (in the sense that it sticks strictly to implementing the language level semantics for async generators without assuming any particular runner).
Of course asyncio_extras has lots of other handy things in it too, while async_generator is very strictly focused on just this one thing. (Well, I threw in an implementation of aclosing too, since you can't really use async generators without it, but that's pretty trivial.) It might make sense for asyncio_extras to out-source generator handling to async_generator? Or not.
-n
-- Nathaniel J. Smith -- https://vorpus.org
On Nov 23, 2016 11:29 PM, "Alex Grönholm" <alex.gronholm@nextday.fi> wrote:
23.11.2016, 01:34, Nathaniel Smith kirjoitti:
On Tue, Nov 22, 2016 at 2:22 PM, Alex Grönholm <alex.gronholm@nextday.fi>
wrote:
I'm not sure where asyncio_extras's async generator implementation assumes you're using asyncio. Could you elaborate on that?
If I'm reading it right, it assumes that the only two things that might be yielded to the coroutine runner are either (a) the special yield wrapper, or (b) an awaitable object like an asyncio.Future. This works on asyncio, because that's all the asyncio runner supports, but it doesn't work with (for example) curio. async_generator (like native async generators) allows arbitrary objects to be yielded to the coroutine runner.
You are misreading the code. It is in no way limited to what asyncio accepts. It doesn't even import asyncio in the asyncyield or generator modules. The only parts of the library that depend on PEP 3156 event loops are the ones that involve executors and threads.
Native async generators don't support "yield from" either, so I didn't
to add that until there's a new PEP that defines its exact semantics.
Yeah, one of the motivations for async_generator has been to figure out what the semantics for native generators should be :-). The native async generators in 3.6 use the same implementation trick as async_generator (not entirely sure if Yury got it from me, or if it's an independent invention), and the yield from implementation is partly motivated as a proof of concept / testing ground for hopefully getting it into 3.7. The yield from semantics are pretty straightforward though, I think.
As for aclose/asend/athrow support, I will have to look into that.
Could we synchronize the implementations so that they're compatible with each other? I would've preferred there to be only a single implementation (asyncio_extras was published first, but admittedly I didn't advertise it that well) but now it'd be best if we work together, wouldn't you agree?
Makes sense to me :-). I think the obvious approach would be for async_extras to just depend on async_generator, since AFAICT async_generator is pretty much complete, and like you say, there's not much
I didn't say that it imported asyncio. I said that it assumes the only things that will be yielded are the things that asyncio yields. This is the line that I'm worried about: https://github.com/agronholm/asyncio_extras/blob/aec412e1b7034ca3cad386c381e... The code awaits the value yielded by the coroutine, but there's no guarantee that this value is awaitable. It's an arbitrary Python object representing a message sent to the coroutine runner. It turns out that asyncio only uses awaitable objects for its messages, so this code can get away with this on asyncio, but if you try using this code with curio then I'm pretty sure you're going to end up doing something like "await (3,)" and then blowing up. try point in carrying around two copies of the same thing. But if you have another suggestion I'd be interested to hear it...
I may consider that at some point, but I'm not yet comfortable with those
extended capabilities. Meanwhile I will add any missing functionality (asend, athrow etc.) to mine. You could just not import yield_from_ and then you don't have any extended capabilities... but up to you, obviously.
One thing I noticed is that there seems to be no way to detect async generator functions in your implementation. That is something I would want to have before switching.
Good point. That should be pretty trivial to add. -n
On Thu, Nov 24, 2016 at 1:23 PM, Nathaniel Smith <njs@pobox.com> wrote: [...]
One thing I noticed is that there seems to be no way to detect async generator functions in your implementation. That is something I would want to have before switching.
Good point. That should be pretty trivial to add.
Just pushed async_generator v1.3 to PyPI, with new isasyncgen, isasyncgenfunction, and (on 3.6) registration with collections.abc.AsyncGenerator. -n -- Nathaniel J. Smith -- https://vorpus.org
25.11.2016, 09:25, Nathaniel Smith kirjoitti:
On Thu, Nov 24, 2016 at 1:23 PM, Nathaniel Smith <njs@pobox.com> wrote: [...]
One thing I noticed is that there seems to be no way to detect async generator functions in your implementation. That is something I would want to have before switching. Good point. That should be pretty trivial to add. Just pushed async_generator v1.3 to PyPI, with new isasyncgen, isasyncgenfunction, and (on 3.6) registration with
collections.abc.AsyncGenerator. And you just *had* to make it incompatible with the async generators from asyncio_extras? "_is_async_gen_function" vs "_is_async_generator"? I thought we agreed on cooperating?
-n
On Thu, Nov 24, 2016 at 11:59 PM, Alex Grönholm <alex.gronholm@nextday.fi> wrote:
25.11.2016, 09:25, Nathaniel Smith kirjoitti:
On Thu, Nov 24, 2016 at 1:23 PM, Nathaniel Smith <njs@pobox.com> wrote: [...]
One thing I noticed is that there seems to be no way to detect async generator functions in your implementation. That is something I would want to have before switching.
Good point. That should be pretty trivial to add.
Just pushed async_generator v1.3 to PyPI, with new isasyncgen, isasyncgenfunction, and (on 3.6) registration with collections.abc.AsyncGenerator.
And you just *had* to make it incompatible with the async generators from asyncio_extras? "_is_async_gen_function" vs "_is_async_generator"? I thought we agreed on cooperating?
I started doing that, but... it's not an async generator, isasyncgen is a different inspect function... -n -- Nathaniel J. Smith -- https://vorpus.org
25.11.2016, 12:09, Nathaniel Smith kirjoitti:
On Thu, Nov 24, 2016 at 11:59 PM, Alex Grönholm <alex.gronholm@nextday.fi> wrote:
25.11.2016, 09:25, Nathaniel Smith kirjoitti:
On Thu, Nov 24, 2016 at 1:23 PM, Nathaniel Smith <njs@pobox.com> wrote: [...]
One thing I noticed is that there seems to be no way to detect async generator functions in your implementation. That is something I would want to have before switching. Good point. That should be pretty trivial to add. Just pushed async_generator v1.3 to PyPI, with new isasyncgen, isasyncgenfunction, and (on 3.6) registration with
collections.abc.AsyncGenerator. And you just *had* to make it incompatible with the async generators from asyncio_extras? "_is_async_gen_function" vs "_is_async_generator"? I thought we agreed on cooperating? I started doing that, but... it's not an async generator, isasyncgen is a different inspect function...
-n
It's an arbitrary string that will likely never be seen by anyone except for people working on the libraries. But it makes a world of difference in compatibility. I named it this way to be consistent with asyncio which marks yield-based coroutines with "_is_coroutine". So please reconsider.
On Fri, Nov 25, 2016 at 10:46 AM, Alex Grönholm <alex.gronholm@nextday.fi> wrote:
25.11.2016, 12:09, Nathaniel Smith kirjoitti:
On Thu, Nov 24, 2016 at 11:59 PM, Alex Grönholm <alex.gronholm@nextday.fi> wrote:
25.11.2016, 09:25, Nathaniel Smith kirjoitti:
On Thu, Nov 24, 2016 at 1:23 PM, Nathaniel Smith <njs@pobox.com> wrote: [...]
One thing I noticed is that there seems to be no way to detect async generator functions in your implementation. That is something I would want to have before switching.
Good point. That should be pretty trivial to add.
Just pushed async_generator v1.3 to PyPI, with new isasyncgen, isasyncgenfunction, and (on 3.6) registration with collections.abc.AsyncGenerator.
And you just *had* to make it incompatible with the async generators from asyncio_extras? "_is_async_gen_function" vs "_is_async_generator"? I thought we agreed on cooperating?
I started doing that, but... it's not an async generator, isasyncgen is a different inspect function...
-n
It's an arbitrary string that will likely never be seen by anyone except for people working on the libraries. But it makes a world of difference in compatibility. I named it this way to be consistent with asyncio which marks yield-based coroutines with "_is_coroutine". So please reconsider.
Sure, I guess it doesn't matter too much either way. Can we pause a moment to ask whether we really want the two async generator types to return true for each other's introspection function, though? Right now they aren't really interchangeable, and the only user for this kind of introspection I've seen is your contextmanager.py. It seems to use the inspect versions of isasyncgen and isasyncgenfunction, because it wants to know whether asend/athrow/aclose are supported. Right now, if it switched to using async_generator's isasyncgen/isasyncgenfunction, it would continue to work; but if I then switched async_generator's isasyncgen/isasyncgenfunction to detect asyncio_extras's generators, it would stop working again. Speaking of which, what do you want to do about isasyncgen? -n -- Nathaniel J. Smith -- https://vorpus.org
26.11.2016, 09:47, Nathaniel Smith kirjoitti:
On Fri, Nov 25, 2016 at 10:46 AM, Alex Grönholm <alex.gronholm@nextday.fi> wrote:
25.11.2016, 12:09, Nathaniel Smith kirjoitti:
On Thu, Nov 24, 2016 at 11:59 PM, Alex Grönholm <alex.gronholm@nextday.fi> wrote:
25.11.2016, 09:25, Nathaniel Smith kirjoitti:
On Thu, Nov 24, 2016 at 1:23 PM, Nathaniel Smith <njs@pobox.com> wrote: [...]
> One thing I noticed is that there seems to be no way to detect async > generator functions in your implementation. That is something I would > want > to have before switching. Good point. That should be pretty trivial to add. Just pushed async_generator v1.3 to PyPI, with new isasyncgen, isasyncgenfunction, and (on 3.6) registration with
collections.abc.AsyncGenerator. And you just *had* to make it incompatible with the async generators from asyncio_extras? "_is_async_gen_function" vs "_is_async_generator"? I thought we agreed on cooperating? I started doing that, but... it's not an async generator, isasyncgen is a different inspect function...
-n
It's an arbitrary string that will likely never be seen by anyone except for people working on the libraries. But it makes a world of difference in compatibility. I named it this way to be consistent with asyncio which marks yield-based coroutines with "_is_coroutine". So please reconsider. Sure, I guess it doesn't matter too much either way.
Can we pause a moment to ask whether we really want the two async generator types to return true for each other's introspection function, though? Right now they aren't really interchangeable, and the only user for this kind of introspection I've seen is your contextmanager.py. It seems to use the inspect versions of isasyncgen and isasyncgenfunction, because it wants to know whether asend/athrow/aclose are supported. Right now, if it switched to using async_generator's isasyncgen/isasyncgenfunction, it would continue to work; but if I then switched async_generator's isasyncgen/isasyncgenfunction to detect asyncio_extras's generators, it would stop working again.
Speaking of which, what do you want to do about isasyncgen?
-n
I've created a new branch in asyncio_extras which delegates the implementation of async generators to async_generator. All tests pass with 100% combined coverage. Before I merge this into master, I would like to implement some changes in your library: * Set the default value for yield_() argument to None, as in my yield_async() * Add type hints to methods and functions * Look into implementing missing attributes which neither lib had before (if technically possible): ag_frame, ag_running, ag_code, __name__, __qualname__ I hope these changes aren't too invasive. They should be fully backwards compatible.
On Nov 26, 2016 2:07 AM, "Alex Grönholm" <alex.gronholm@nextday.fi> wrote:
26.11.2016, 09:47, Nathaniel Smith kirjoitti:
On Fri, Nov 25, 2016 at 10:46 AM, Alex Grönholm <alex.gronholm@nextday.fi>
wrote:
25.11.2016, 12:09, Nathaniel Smith kirjoitti:
On Thu, Nov 24, 2016 at 11:59 PM, Alex Grönholm <
alex.gronholm@nextday.fi> wrote:
25.11.2016, 09:25, Nathaniel Smith kirjoitti:
On Thu, Nov 24, 2016 at 1:23 PM, Nathaniel Smith <njs@pobox.com>
wrote: [...]
>> >> One thing I noticed is that there seems to be no way to detect async generator functions in your implementation. That is something I would want to have before switching. > > Good point. That should be pretty trivial to add.
Just pushed async_generator v1.3 to PyPI, with new isasyncgen, isasyncgenfunction, and (on 3.6) registration with
collections.abc.AsyncGenerator. >>>>> >>>>> And you just *had* to make it incompatible with the async generators from asyncio_extras? "_is_async_gen_function" vs "_is_async_generator"? I thought we agreed on cooperating? >>>> >>>> I started doing that, but... it's not an async generator, isasyncgen is a different inspect function... -n >>> >>> It's an arbitrary string that will likely never be seen by anyone except for people working on the libraries. But it makes a world of difference in compatibility. I named it this way to be consistent with asyncio which marks yield-based coroutines with "_is_coroutine". So please reconsider. >> >> Sure, I guess it doesn't matter too much either way. Can we pause a moment to ask whether we really want the two async generator types to return true for each other's introspection function, though? Right now they aren't really interchangeable, and the only user for this kind of introspection I've seen is your contextmanager.py. It seems to use the inspect versions of isasyncgen and isasyncgenfunction, because it wants to know whether asend/athrow/aclose are supported. Right now, if it switched to using async_generator's isasyncgen/isasyncgenfunction, it would continue to work; but if I then switched async_generator's isasyncgen/isasyncgenfunction to detect asyncio_extras's generators, it would stop working again. Speaking of which, what do you want to do about isasyncgen? -n > > I've created a new branch in asyncio_extras which delegates the implementation of async generators to async_generator. All tests pass with 100% combined coverage.
Cool!
Before I merge this into master, I would like to implement some changes in your library: Set the default value for yield_() argument to None, as in my yield_async()
Makes sense.
Add type hints to methods and functions
I haven't had a chance to play with these much, but no objections.
Look into implementing missing attributes which neither lib had before (if technically possible): ag_frame, ag_running, ag_code, __name__, __qualname__
__name__ and __qualname__ are obviously a good idea. The others I'm not so sure about -- I'd worry that anyone who's reaching that deep into the internals is not going to be satisfied by any half-way attempts at compatibility, and it seems unlikely we can achieve detailed compatibility at that level. But maybe it's not that bad; I haven't looked at them in detail. Do you have any particular use cases in mind for them? -n
26.11.2016, 17:42, Nathaniel Smith kirjoitti:
On Nov 26, 2016 2:07 AM, "Alex Grönholm" <alex.gronholm@nextday.fi <mailto:alex.gronholm@nextday.fi>> wrote:
26.11.2016, 09:47, Nathaniel Smith kirjoitti:
On Fri, Nov 25, 2016 at 10:46 AM, Alex Grönholm
<alex.gronholm@nextday.fi <mailto:alex.gronholm@nextday.fi>> wrote:
25.11.2016, 12:09, Nathaniel Smith kirjoitti:
On Thu, Nov 24, 2016 at 11:59 PM, Alex Grönholm
<alex.gronholm@nextday.fi <mailto:alex.gronholm@nextday.fi>> wrote:
25.11.2016, 09:25, Nathaniel Smith kirjoitti: > > On Thu, Nov 24, 2016 at 1:23 PM, Nathaniel Smith
<njs@pobox.com <mailto:njs@pobox.com>> wrote: [...]
>>> >>> One thing I noticed is that there seems to be no way to detect async generator functions in your implementation. That is something I would want to have before switching. >> >> Good point. That should be pretty trivial to add. > > Just pushed async_generator v1.3 to PyPI, with new isasyncgen, isasyncgenfunction, and (on 3.6) registration with
collections.abc.AsyncGenerator. >>>>> >>>>> And you just *had* to make it incompatible with the async generators from asyncio_extras? "_is_async_gen_function" vs "_is_async_generator"? I thought we agreed on cooperating? >>>> >>>> I started doing that, but... it's not an async generator, isasyncgen is a different inspect function... -n >>> >>> It's an arbitrary string that will likely never be seen by anyone except for people working on the libraries. But it makes a world of difference in compatibility. I named it this way to be consistent with asyncio which marks yield-based coroutines with "_is_coroutine". So please reconsider. >> >> Sure, I guess it doesn't matter too much either way. Can we pause a moment to ask whether we really want the two async generator types to return true for each other's introspection function, though? Right now they aren't really interchangeable, and the only user for this kind of introspection I've seen is your contextmanager.py. It seems to use the inspect versions of isasyncgen and isasyncgenfunction, because it wants to know whether asend/athrow/aclose are supported. Right now, if it switched to using async_generator's isasyncgen/isasyncgenfunction, it would continue to work; but if I then switched async_generator's isasyncgen/isasyncgenfunction to detect asyncio_extras's generators, it would stop working again. Speaking of which, what do you want to do about isasyncgen? -n > > I've created a new branch in asyncio_extras which delegates the implementation of async generators to async_generator. All tests pass with 100% combined coverage.
Cool!
Before I merge this into master, I would like to implement some changes in your library: Set the default value for yield_() argument to None, as in my yield_async()
Makes sense.
Add type hints to methods and functions
I haven't had a chance to play with these much, but no objections.
Look into implementing missing attributes which neither lib had before (if technically possible): ag_frame, ag_running, ag_code, __name__, __qualname__
__name__ and __qualname__ are obviously a good idea. The others I'm not so sure about -- I'd worry that anyone who's reaching that deep into the internals is not going to be satisfied by any half-way attempts at compatibility, and it seems unlikely we can achieve detailed compatibility at that level. But maybe it's not that bad; I haven't looked at them in detail. Do you have any particular use cases in mind for them?
I don't have a particular use case in mind, I'm just looking for completeness. They're right in the spec and I believe simple getters should do just fine (coroutines should have matching attributes which can be used directly). I'll send you a PR when I get around to it.
-n
24.11.2016, 23:23, Nathaniel Smith kirjoitti:
On Nov 23, 2016 11:29 PM, "Alex Grönholm" <alex.gronholm@nextday.fi <mailto:alex.gronholm@nextday.fi>> wrote:
23.11.2016, 01:34, Nathaniel Smith kirjoitti:
On Tue, Nov 22, 2016 at 2:22 PM, Alex Grönholm
<alex.gronholm@nextday.fi <mailto:alex.gronholm@nextday.fi>> wrote:
I'm not sure where asyncio_extras's async generator implementation assumes you're using asyncio. Could you elaborate on that?
If I'm reading it right, it assumes that the only two things that might be yielded to the coroutine runner are either (a) the special yield wrapper, or (b) an awaitable object like an asyncio.Future. This works on asyncio, because that's all the asyncio runner supports, but it doesn't work with (for example) curio. async_generator (like native async generators) allows arbitrary objects to be yielded to the coroutine runner.
You are misreading the code. It is in no way limited to what asyncio accepts. It doesn't even import asyncio in the asyncyield or generator modules. The only parts of the library that depend on PEP 3156 event loops are the ones that involve executors and threads.
I didn't say that it imported asyncio. I said that it assumes the only things that will be yielded are the things that asyncio yields. This is the line that I'm worried about:
https://github.com/agronholm/asyncio_extras/blob/aec412e1b7034ca3cad386c381e...
The code awaits the value yielded by the coroutine, but there's no guarantee that this value is awaitable. It's an arbitrary Python object representing a message sent to the coroutine runner. It turns out that asyncio only uses awaitable objects for its messages, so this code can get away with this on asyncio, but if you try using this code with curio then I'm pretty sure you're going to end up doing something like "await (3,)" and then blowing up.
PEP 492 clearly states the following: It is aTypeErrorto pass anything other than an/awaitable/object to anawaitexpression. That (3,) is not an awaitable, so the example is invalid. That said, I will re-examine this part of the implementation and correct it if necessary. So far I just haven't encountered anything that would produce an error.
Native async generators don't support "yield from" either, so I didn't try to add that until there's a new PEP that defines its exact semantics.
Yeah, one of the motivations for async_generator has been to figure out what the semantics for native generators should be :-). The native async generators in 3.6 use the same implementation trick as async_generator (not entirely sure if Yury got it from me, or if it's an independent invention), and the yield from implementation is partly motivated as a proof of concept / testing ground for hopefully getting it into 3.7. The yield from semantics are pretty straightforward though, I think.
As for aclose/asend/athrow support, I will have to look into that.
Could we synchronize the implementations so that they're compatible with each other? I would've preferred there to be only a single implementation (asyncio_extras was published first, but admittedly I didn't advertise it that well) but now it'd be best if we work together, wouldn't you agree?
Makes sense to me :-). I think the obvious approach would be for async_extras to just depend on async_generator, since AFAICT async_generator is pretty much complete, and like you say, there's not much point in carrying around two copies of the same thing. But if you have another suggestion I'd be interested to hear it...
I may consider that at some point, but I'm not yet comfortable with those extended capabilities. Meanwhile I will add any missing functionality (asend, athrow etc.) to mine.
You could just not import yield_from_ and then you don't have any extended capabilities... but up to you, obviously.
One thing I noticed is that there seems to be no way to detect async generator functions in your implementation. That is something I would want to have before switching.
Good point. That should be pretty trivial to add.
-n
On Fri, Nov 25, 2016 at 12:03 AM, Alex Grönholm <alex.gronholm@nextday.fi> wrote:
24.11.2016, 23:23, Nathaniel Smith kirjoitti:
On Nov 23, 2016 11:29 PM, "Alex Grönholm" <alex.gronholm@nextday.fi> wrote:
23.11.2016, 01:34, Nathaniel Smith kirjoitti:
On Tue, Nov 22, 2016 at 2:22 PM, Alex Grönholm <alex.gronholm@nextday.fi> wrote:
I'm not sure where asyncio_extras's async generator implementation assumes you're using asyncio. Could you elaborate on that?
If I'm reading it right, it assumes that the only two things that might be yielded to the coroutine runner are either (a) the special yield wrapper, or (b) an awaitable object like an asyncio.Future. This works on asyncio, because that's all the asyncio runner supports, but it doesn't work with (for example) curio. async_generator (like native async generators) allows arbitrary objects to be yielded to the coroutine runner.
You are misreading the code. It is in no way limited to what asyncio accepts. It doesn't even import asyncio in the asyncyield or generator modules. The only parts of the library that depend on PEP 3156 event loops are the ones that involve executors and threads.
I didn't say that it imported asyncio. I said that it assumes the only things that will be yielded are the things that asyncio yields. This is the line that I'm worried about:
https://github.com/agronholm/asyncio_extras/blob/aec412e1b7034ca3cad386c381e...
The code awaits the value yielded by the coroutine, but there's no guarantee that this value is awaitable. It's an arbitrary Python object representing a message sent to the coroutine runner. It turns out that asyncio only uses awaitable objects for its messages, so this code can get away with this on asyncio, but if you try using this code with curio then I'm pretty sure you're going to end up doing something like "await (3,)" and then blowing up.
PEP 492 clearly states the following:
It is a TypeError to pass anything other than an awaitable object to an await expression.
That (3,) is not an awaitable, so the example is invalid. That said, I will re-examine this part of the implementation and correct it if necessary.
I feel like I'm running out of ideas for how to explain this, and starting to just repeat myself :-(. There is nothing that says the return value from coro.__next__() must be awaitable. Coroutines can yield arbitrary objects. (3,) is not awaitable, but it's a perfectly valid thing to be yielded from a coroutine.
So far I just haven't encountered anything that would produce an error.
That's because you've only tested on asyncio, not curio. I promise! -n -- Nathaniel J. Smith -- https://vorpus.org
participants (2)
-
Alex Grönholm -
Nathaniel Smith