Hi all!
I am excited about seeing what's going on with asyncio and PEP492 etc. I really like that Python is becoming more suitable for the increasing amount of async code and that the distinction between async functions and generators is increasing.
In addition, however, I would also like to see the async functions and methods come even closer to regular functions and methods. This is something that is keeping me from using asyncio at the moment even if I would like to. Below I'll try to explain what and why, and a little bit of how. If it is not clear, please ask :)
Motivation:
One of the best things about asyncio and coroutines/async functions is that you can write asynchronous code as if it were synchronous, the difference in many places being just the use of "await" ("yield from") when calling something that may end up doing IO (somewhere down the function call chain) and that the code is run from an event loop.
When writing a package that does IO, you have the option to make it either synchronous or asynchronous. Regardless of the choice, the code will look roughly the same. But what if you want to be able to do both? Should you maintain two versions, one with "async" and "await" everywhere and one without?
Besides the keywords "async" and "await", async code of course differs from synchronous code by the functions/coroutines that are used for IO at the end of the function call chain. Here, I mean the end (close to) where the "yield" expressions are hidden in the async versions. At the other end of the calling chain, async code needs the event loop and associated framework (almost always asyncio?) which hides all the async scheduling fanciness etc. I'm not sure about the terminology, but I will use "L end" and "Y end" to refer to the two ends here. (L for event Loop; Y for Yield)
The Y and L ends need to be compatible with each other for the code to work. While asyncio and the standard library might provide both ends in many cases, there can also be situations where a package would want to work with different combinations of L and Y end, or completely without an event loop, i.e. synchronously.
In a very simple example, one might want to wrap different implementations of sleep() in a function that would pick the right one depending on the context. Perhaps something like this:
async def any_sleep(seconds): if __async__.framework is None: time.sleep(1) elif __async__.framework is asyncio: await asyncio.sleep(1) else: raise RuntimeError("Was called with an unsupported async framework.")
[You could of course replace sleep() with socket IO or whatever, but sleep is nice and simple. Also, a larger library would probably have a whole chain of async functions and methods before calling something like this]
But if await is only allowed inside "async def", then how can any_sleep() be conveniently run in non-async code? Also, there is nothing like __async__.framework. Below, I describe what I think a potential solution might look like.
Potential solution:
This is simplified version; for instance, as "awaitables", I consider only async function objects here. I describe the idea in three parts:
(1) next(...):
Add a keyword argument "async_framework" (or whatever) to next(...) with a default value of None. When an async framework, typically asyncio, starts an async function object (coroutine) with a call to next(...), it would do something like next(coro, async_framework = asyncio). Here, asyncio could of course be replaced with any object that identifies the framework. This information would then be somehow attached to the async function object.
(2) __async__.framework or something similar:
Add something like __async__ that has an attribute such as .framework that allows the code inside the async function to access the information passed to next(...) by the framework (L end) using the keyword argument of next [see (1)].
(3) Generalized "await":
[When the world is ready:] Allow using "await" anywhere, not just within async functions. Inside async functions, the behavior of "await" would be the same as in PEP492, with the addition that it would somehow propagate the __async__.framework value to the awaited coroutine. Outside async functions, "await" would do roughly the same as this function:
def await(async_func_obj): try: next(async_func_obj) # same as next(async_func_obj, async_framework = None) except StopIteration as si: return si.value raise RuntimeError("The function does not support synchronous execution")
(This function would, of course, work in Python 3.4, but it would be mostly useless because the async functions would not know that they are being called in a 'synchronous program'. IIUC, this function would be valid even with PEP492, but having this as a function would be ugly in the long run.)
Some random thoughts:
With this addition to Python, one could write libraries that work both async and non-async. When await is not inside async def, one would expect it to potentially do blocking IO, just like an await inside async def would suggest that there is a yield/suspend somewhere in there.
For testing, I tried to see if there is a reasonable way to make a hack with __async__.framework that could be set by next(), but did not find an obvious way. For instance, coro.gi_frame.f_locals is read-only, I believe.
An alternative to this approach could be that await would implicitly start a temporary event loop for running the coroutine, but how would it know which event loop? This might also have a huge performance overhead.
Relation to PEP492:
This of course still needs more thinking, but I wanted to post it here now in case there is desire to prepare for something like this already in PEP492. It is not completely clear if/how this would need to affect PEP492, but some things come to mind. For example, this could potentially remove the need for __aenter__, __aiter__, etc. or even "async for" and "async with". If __aenter__ is defined as "async def", then a with statement would do an "await" on it, and the context manager would have __async__.framework (or whatever it would be called) available, for determining what behavior is appropriate.
Was this clear enough to understand which problem(s) this would be solving and how? I'd be happy to hear about any thoughts on this :).
Best regards, Koos
Hi Koos,
2015-05-05 15:55 GMT+02:00 Koos Zevenhoven koos.zevenhoven@aalto.fi:
With this addition to Python, one could write libraries that work both async and non-async. When await is not inside async def, one would expect it to potentially do blocking IO, just like an await inside async def would suggest that there is a yield/suspend somewhere in there.
To be honest with you, I'd this type of ideas back in my mind, but for now, I've no suggestion to avoid end-developer nor low-developer nightmares.
For example, we may detect if it's async or not if you have: result = await response.payload() or result = response.payload() The issue I see with that and certainly already explained during PEP492 discussions, is that it will be difficult for the developer to spot where he is forgotten await keyword, because he won't have errors.
Moreover, in the use cases where async is less efficient that sync, it should be interesting to be possible, maybe with a context manager to define a block of code where all await are in fact sync (without to use event loop). But, even if a talentuous low-developper find a solution to implement this idea, because I'm not sure it's technically possible, in fact it will more easier even for end-developers to use the sync library version of this need.
FYI, I've made an yocto library for my company where I need to be sync for some use cases and async for some other use cases. For the sync and async public API where the business logic behind most functions are identical, I've followed the same pattern as in Python-LDAP: http://www.python-ldap.org/doc/html/ldap.html#sending-ldap-requests I've postfixed all sync functions by "_s".
For a more complex library, it may possible to have two differents classes with explicit names.
At least to me, it's enough to work efficiently, explicit is better than
Ludovic Gasc (GMLudo) http://www.gmludo.eu/
On 2015-05-05 17:57, Ludovic Gasc wrote: >
For example, we may detect if it's async or not if you have: result = await response.payload() or result = response.payload() The issue I see with that and certainly already explained during PEP492 discussions, is that it will be difficult for the developer to spot where he is forgotten await keyword, because he won't have errors.
Thank you for your email!
I've been following quite a bit of the PEP492 discussions, but not sure if I have missed something. If there is something about await outside async def that goes further than "It is a SyntaxError to use await outside of an async def function (like it is a SyntaxError to use yield outside of def function.)", which is directly from the PEP, I've missed that. A link or pointer would be helpful.
In any case, I think I understand the problem you are referring to, but is that any different from forgetting a postfix "_s" in the approach you mention below?
Moreover, in the use cases where async is less efficient that sync, it should be interesting to be possible, maybe with a context manager to define a block of code where all await are in fact sync (without to use event loop). But, even if a talentuous low-developper find a solution to implement this idea, because I'm not sure it's technically possible, in fact it will more easier even for end-developers to use the sync library version of this need.
Surely that is possible, although may of course be hard to implement :). I think this is related to this earlier suggestion by Joshua Bartlett (which I do like):
https://mail.python.org/pipermail/python-ideas/2013-January/018519.html
However, I don't think it solves this problem. It would just become a more verbose version of what I suggested.
>
FYI, I've made an yocto library for my company where I need to be sync for some use cases and async for some other use cases. For the sync and async public API where the business logic behind most functions are identical, I've followed the same pattern as in Python-LDAP: http://www.python-ldap.org/doc/html/ldap.html#sending-ldap-requests I've postfixed all sync functions by "_s".
For a more complex library, it may possible to have two differents classes with explicit names.
At least to me, it's enough to work efficiently, explicit is better than implicit ;-)
In my mind, this is not at all about explicit vs. implicit. It is mostly about letting the coroutines know what kind of context they are being run from. Anyway, I'm pretty sure there are plenty of people in the Python community who don't think efficiency is enough, but that is a matter of personal preference. I want everything, and that's why I'm using Python ;).
-- Koos
It seems like it might be a lot easier to approach this from the other end: Is it possible to write a decorator that takes an async coroutine function, strips out all the awaits, and returns a regular sync function? If so, all you need to do is write everything as async, and then users can "from spam import sync as spam" or "from spam import async as spam" (where async just imports all the real functions, while sync imports them and calls the decorator on all of them). That also avoids the need to have all the looking up the event loop, switching between different code branches, etc. inside every function at runtime. (Not that it matters for the performance of sleep(1), but it might matter for the performance of other functions—and, more importantly, it might make the implementation of those functions simpler and easier to debug through.)
On Tuesday, May 5, 2015 7:01 AM, Koos Zevenhoven
<koos.zevenhoven@aalto.fi> wrote:
Hi all!
I am excited about seeing what's going on with asyncio and PEP492 etc. I really like that Python is becoming more suitable for the increasing amount of async code and that the distinction between async functions and generators is increasing.
In addition, however, I would also like to see the async functions and methods come even closer to regular functions and methods. This is something that is keeping me from using asyncio at the moment even if I would like to. Below I'll try to explain what and why, and a little bit of how. If it is not clear, please ask :)
Motivation:
One of the best things about asyncio and coroutines/async functions is that you can write asynchronous code as if it were synchronous, the difference in many places being just the use of "await" ("yield from") when calling something that may end up doing IO (somewhere down the function call chain) and that the code is run from an event loop.
When writing a package that does IO, you have the option to make it either synchronous or asynchronous. Regardless of the choice, the code will look roughly the same. But what if you want to be able to do both? Should you maintain two versions, one with "async" and "await" everywhere and one without?
Besides the keywords "async" and "await", async code of course differs from synchronous code by the functions/coroutines that are used for IO at the end of the function call chain. Here, I mean the end (close to) where the "yield" expressions are hidden in the async versions. At the other end of the calling chain, async code needs the event loop and associated framework (almost always asyncio?) which hides all the async scheduling fanciness etc. I'm not sure about the terminology, but I will use "L end" and "Y end" to refer to the two ends here. (L for event Loop; Y for Yield)
The Y and L ends need to be compatible with each other for the code to work. While asyncio and the standard library might provide both ends in many cases, there can also be situations where a package would want to work with different combinations of L and Y end, or completely without an event loop, i.e. synchronously.
In a very simple example, one might want to wrap different implementations of sleep() in a function that would pick the right one depending on the context. Perhaps something like this:
async def any_sleep(seconds): if __async__.framework is None: time.sleep(1) elif __async__.framework is asyncio: await asyncio.sleep(1) else: raise RuntimeError("Was called with an unsupported async framework.")
[You could of course replace sleep() with socket IO or whatever, but sleep is nice and simple. Also, a larger library would probably have a whole chain of async functions and methods before calling something like this]
But if await is only allowed inside "async def", then how can any_sleep() be conveniently run in non-async code? Also, there is nothing like __async__.framework. Below, I describe what I think a potential solution might look like.
Potential solution:
This is simplified version; for instance, as "awaitables", I consider only async function objects here. I describe the idea in three parts:
(1) next(...):
Add a keyword argument "async_framework" (or whatever) to next(...) with a default value of None. When an async framework, typically asyncio, starts an async function object (coroutine) with a call to next(...), it would do something like next(coro, async_framework = asyncio). Here, asyncio could of course be replaced with any object that identifies the framework. This information would then be somehow attached to the async function object.
(2) __async__.framework or something similar:
Add something like __async__ that has an attribute such as .framework that allows the code inside the async function to access the information passed to next(...) by the framework (L end) using the keyword argument of next [see (1)].
(3) Generalized "await":
[When the world is ready:] Allow using "await" anywhere, not just within async functions. Inside async functions, the behavior of "await" would be the same as in PEP492, with the addition that it would somehow propagate the __async__.framework value to the awaited coroutine. Outside async functions, "await" would do roughly the same as this function:
def await(async_func_obj): try: next(async_func_obj) # same as next(async_func_obj, async_framework = None) except StopIteration as si: return si.value raise RuntimeError("The function does not support synchronous execution")
(This function would, of course, work in Python 3.4, but it would be mostly useless because the async functions would not know that they are being called in a 'synchronous program'. IIUC, this function would be valid even with PEP492, but having this as a function would be ugly in the long run.)
Some random thoughts:
With this addition to Python, one could write libraries that work both async and non-async. When await is not inside async def, one would expect it to potentially do blocking IO, just like an await inside async def would suggest that there is a yield/suspend somewhere in there.
For testing, I tried to see if there is a reasonable way to make a hack with __async__.framework that could be set by next(), but did not find an obvious way. For instance, coro.gi_frame.f_locals is read-only, I believe.
An alternative to this approach could be that await would implicitly start a temporary event loop for running the coroutine, but how would it know which event loop? This might also have a huge performance overhead.
Relation to PEP492:
This of course still needs more thinking, but I wanted to post it here now in case there is desire to prepare for something like this already in PEP492. It is not completely clear if/how this would need to affect PEP492, but some things come to mind. For example, this could potentially remove the need for __aenter__, __aiter__, etc. or even "async for" and "async with". If __aenter__ is defined as "async def", then a with statement would do an "await" on it, and the context manager would have __async__.framework (or whatever it would be called) available, for determining what behavior is appropriate.
Was this clear enough to understand which problem(s) this would be solving and how? I'd be happy to hear about any thoughts on this :).
Best regards, Koos
Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Quick notes:
On Tue, May 5, 2015 at 11:00 AM, Andrew Barnert via Python-ideas python-ideas@python.org wrote:
It seems like it might be a lot easier to approach this from the other end: Is it possible to write a decorator that takes an async coroutine function, strips out all the awaits, and returns a regular sync function? If so, all you need to do is write everything as async, and then users can "from spam import sync as spam" or "from spam import async as spam" (where async just imports all the real functions, while sync imports them and calls the decorator on all of them).
That also avoids the need to have all the looking up the event loop, switching between different code branches, etc. inside every function at runtime. (Not that it matters for the performance of sleep(1), but it might matter for the performance of other functions—and, more importantly, it might make the implementation of those functions simpler and easier to debug through.)
On Tuesday, May 5, 2015 7:01 AM, Koos Zevenhoven koos.zevenhoven@aalto.fi wrote:
Hi all!
I am excited about seeing what's going on with asyncio and PEP492 etc. I really like that Python is becoming more suitable for the increasing amount of async code and that the distinction between async functions and generators is increasing.
In addition, however, I would also like to see the async functions and methods come even closer to regular functions and methods. This is something that is keeping me from using asyncio at the moment even if I would like to. Below I'll try to explain what and why, and a little bit of how. If it is not clear, please ask :)
Motivation:
One of the best things about asyncio and coroutines/async functions is that you can write asynchronous code as if it were synchronous, the difference in many places being just the use of "await" ("yield from") when calling something that may end up doing IO (somewhere down the function call chain) and that the code is run from an event loop.
When writing a package that does IO, you have the option to make it either synchronous or asynchronous. Regardless of the choice, the code will look roughly the same. But what if you want to be able to do both? Should you maintain two versions, one with "async" and "await" everywhere and one without?
Besides the keywords "async" and "await", async code of course differs from synchronous code by the functions/coroutines that are used for IO at the end of the function call chain. Here, I mean the end (close to) where the "yield" expressions are hidden in the async versions. At the other end of the calling chain, async code needs the event loop and associated framework (almost always asyncio?) which hides all the async scheduling fanciness etc. I'm not sure about the terminology, but I will use "L end" and "Y end" to refer to the two ends here. (L for event Loop; Y for Yield)
The Y and L ends need to be compatible with each other for the code to work. While asyncio and the standard library might provide both ends in many cases, there can also be situations where a package would want to work with different combinations of L and Y end, or completely without an event loop, i.e. synchronously.
In a very simple example, one might want to wrap different implementations of sleep() in a function that would pick the right one depending on the context. Perhaps something like this:
async def any_sleep(seconds): if __async__.framework is None: time.sleep(1) elif __async__.framework is asyncio: await asyncio.sleep(1) else: raise RuntimeError("Was called with an unsupported async framework.")
[You could of course replace sleep() with socket IO or whatever, but sleep is nice and simple. Also, a larger library would probably have a whole chain of async functions and methods before calling something like this]
But if await is only allowed inside "async def", then how can any_sleep() be conveniently run in non-async code? Also, there is nothing like __async__.framework. Below, I describe what I think a potential solution might look like.
Potential solution:
This is simplified version; for instance, as "awaitables", I consider only async function objects here. I describe the idea in three parts:
(1) next(...):
Add a keyword argument "async_framework" (or whatever) to next(...) with a default value of None. When an async framework, typically asyncio, starts an async function object (coroutine) with a call to next(...), it would do something like next(coro, async_framework = asyncio). Here, asyncio could of course be replaced with any object that identifies the framework. This information would then be somehow attached to the async function object.
(2) __async__.framework or something similar:
Add something like __async__ that has an attribute such as .framework that allows the code inside the async function to access the information passed to next(...) by the framework (L end) using the keyword argument of next [see (1)].
(3) Generalized "await":
[When the world is ready:] Allow using "await" anywhere, not just within async functions. Inside async functions, the behavior of "await" would be the same as in PEP492, with the addition that it would somehow propagate the __async__.framework value to the awaited coroutine. Outside async functions, "await" would do roughly the same as this function:
def await(async_func_obj): try: next(async_func_obj) # same as next(async_func_obj, async_framework = None) except StopIteration as si: return si.value raise RuntimeError("The function does not support synchronous execution")
(This function would, of course, work in Python 3.4, but it would be mostly useless because the async functions would not know that they are being called in a 'synchronous program'. IIUC, this function would be valid even with PEP492, but having this as a function would be ugly in the long run.)
Some random thoughts:
With this addition to Python, one could write libraries that work both async and non-async. When await is not inside async def, one would expect it to potentially do blocking IO, just like an await inside async def would suggest that there is a yield/suspend somewhere in there.
For testing, I tried to see if there is a reasonable way to make a hack with __async__.framework that could be set by next(), but did not find an obvious way. For instance, coro.gi_frame.f_locals is read-only, I believe.
An alternative to this approach could be that await would implicitly start a temporary event loop for running the coroutine, but how would it know which event loop? This might also have a huge performance overhead.
Relation to PEP492:
This of course still needs more thinking, but I wanted to post it here now in case there is desire to prepare for something like this already in PEP492. It is not completely clear if/how this would need to affect PEP492, but some things come to mind. For example, this could potentially remove the need for __aenter__, __aiter__, etc. or even "async for" and "async with". If __aenter__ is defined as "async def", then a with statement would do an "await" on it, and the context manager would have __async__.framework (or whatever it would be called) available, for determining what behavior is appropriate.
Was this clear enough to understand which problem(s) this would be solving and how? I'd be happy to hear about any thoughts on this :).
Best regards, Koos
Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido)
On 5/5/2015 2:48 PM, Guido van Rossum wrote:
Quick notes:
If we're going through all of the effort to elevate await and async def to syntax, then can't the interpreter also be aware if it's running an event loop? Then, if we are running an event loop, await becomes "yield from", using the event loop. But if we're not running an event loop, then await becomes a blocking wait, using some version of run_until_complete, whether really from asyncio or baked into the interpreter.
This way, I can write my library code as being async, but it's still usable from non-async code (although it would need to be called with await, of course).
I'll admit I haven't thought this all the way through, and I'm still reading through PEP 492. But if I can write my async code as if it were blocking using await, why can't it really be blocking, too?
Eric.
No, we can't, because the async/await are interpreted by the compiler, while the presence of an event loop is a condition of the runtime.
On Tue, May 5, 2015 at 2:03 PM, Eric V. Smith eric@trueblade.com wrote:
On 5/5/2015 2:48 PM, Guido van Rossum wrote:
Quick notes:
If we're going through all of the effort to elevate await and async def to syntax, then can't the interpreter also be aware if it's running an event loop? Then, if we are running an event loop, await becomes "yield from", using the event loop. But if we're not running an event loop, then await becomes a blocking wait, using some version of run_until_complete, whether really from asyncio or baked into the interpreter.
This way, I can write my library code as being async, but it's still usable from non-async code (although it would need to be called with await, of course).
I'll admit I haven't thought this all the way through, and I'm still reading through PEP 492. But if I can write my async code as if it were blocking using await, why can't it really be blocking, too?
Eric.
Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido)
Hi Guido and Andrew,
Thank you for your prompt responses!
On 5.5.2015 21:48, Guido van Rossum wrote:
Quick notes:
And since there is asyncio in the standard library, I would assume there typically is no reason to do that either(?) However, as a side effect of my proposal, there would still be a way to use an if statement to pick the right async code to match the framework, along with matching the non-async version :).
Speaking of side effects, I think the same "__async__" variable might also naturally provide this:
https://mail.python.org/pipermail/python-ideas/2015-April/033152.html
By the way, if I understand your first note, it might be the same as my "The Y and L ends need to be compatible with each other for the code to work." Sorry about the terminology. I hope the explanations of Y and L are somewhat understandable.
This is what is keeping me from using asyncio. Ignoring performance overhead, if in any synchronous script (or interactive prompt or ipython notebook) all calls to my library would look like that, I will happily use my 2.7 version that uses threads. Well, I admit that the part about "happily" is not completely true in my case.
Instead, I would be quite happy typing "await <function_call>", since awaiting the function call (to finish/return a value) is exactly what I would be doing, regardless of whether there is an event loop or not.
Regarding my proposal, there would still be a way for libraries to provide this functionality, if desired :).
Please see also the comments below.
On Tue, May 5, 2015 at 11:00 AM, Andrew Barnert via Python-ideas
<python-ideas@python.org mailto:python-ideas@python.org> wrote:
It seems like it might be a lot easier to approach this from the
other end: Is it possible to write a decorator that takes an async
coroutine function, strips out all the awaits, and returns a
regular sync function? If so, all you need to do is write
everything as async, and then users can "from spam import sync as
spam" or "from spam import async as spam" (where async just
imports all the real functions, while sync imports them and calls
the decorator on all of them).
Interesting idea. If this is possible, it would solve part of the issue, but the "Y end" (sorry) of the chain may still need to be done by hand.
>
That also avoids the need to have all the
looking up the event
loop, switching between different code branches, etc. inside every
function at runtime. (Not that it matters for the performance of
sleep(1), but it might matter for the performance of other
functions—and, more importantly, it might make the implementation
of those functions simpler and easier to debug through.)
This could indeed save some if statements at runtime.
Note that the if statements would not be inside every function, but only in the ones that do the actual IO. For instance, some 3rd-party library might use wrappers around socket send and socket recv to choose between sync and async versions, and that might be all the IO it needs to build several layers of async code. Even better, had someone taken the time to provide these if statements inside the standard library, the whole 3rd-party async library would just magically work also in synchronous code :).
Best regards, Koos
>
On Tuesday, May 5, 2015 7:01 AM, Koos
Zevenhoven
<koos.zevenhoven@aalto.fi <mailto:koos.zevenhoven@aalto.fi>> wrote:
Hi all!
I am excited about seeing what's going on with asyncio and
PEP492 etc. I
really like that Python is becoming more suitable for the
increasing
amount of async code and that the distinction between async
functions
and generators is increasing.
In addition, however, I would also like to see the async
functions and
methods come even closer to regular functions and methods.
This is
something that is keeping me from using asyncio at the moment
even if I
would like to. Below I'll try to explain what and why, and a
little bit
of how. If it is not clear, please ask :)
Motivation:
One of the best things about asyncio and coroutines/async
functions is
that you can write asynchronous code as if it were
synchronous, the
difference in many places being just the use of "await"
("yield from")
when calling something that may end up doing IO (somewhere
down the
function call chain) and that the code is run from an event loop.
When writing a package that does IO, you have the option to
make it
either synchronous or asynchronous. Regardless of the choice,
the code
will look roughly the same. But what if you want to be able to
do both?
Should you maintain two versions, one with "async" and "await"
everywhere and one without?
Besides the keywords "async" and "await", async code of course
differs
from synchronous code by the functions/coroutines that are
used for IO
at the end of the function call chain. Here, I mean the end
(close to)
where the "yield" expressions are hidden in the async
versions. At the
other end of the calling chain, async code needs the event
loop and
associated framework (almost always asyncio?) which hides all
the async
scheduling fanciness etc. I'm not sure about the terminology,
but I will
use "L end" and "Y end" to refer to the two ends here. (L for
event
Loop; Y for Yield)
The Y and L ends need to be compatible with each other for the
code to
work. While asyncio and the standard library might provide
both ends in
many cases, there can also be situations where a package would
want to
work with different combinations of L and Y end, or completely
without
an event loop, i.e. synchronously.
In a very simple example, one might want to wrap different
implementations of sleep() in a function that would pick the
right one
depending on the context. Perhaps something like this:
async def any_sleep(seconds):
if __async__.framework is None:
time.sleep(1)
elif __async__.framework is asyncio:
await asyncio.sleep(1)
else:
raise RuntimeError("Was called with an unsupported
async
framework.")
[You could of course replace sleep() with socket IO or
whatever, but
sleep is nice and simple. Also, a larger library would
probably have a
whole chain of async functions and methods before calling
something like
this]
But if await is only allowed inside "async def", then how can
any_sleep() be conveniently run in non-async code? Also, there is
nothing like __async__.framework. Below, I describe what I
think a
potential solution might look like.
Potential solution:
This is simplified version; for instance, as "awaitables", I
consider
only async function objects here. I describe the idea in three
parts:
(1) next(...):
Add a keyword argument "async_framework" (or whatever) to
next(...) with
a default value of None. When an async framework, typically
asyncio,
starts an async function object (coroutine) with a call to
next(...), it
would do something like next(coro, async_framework = asyncio).
Here,
asyncio could of course be replaced with any object that
identifies the
framework. This information would then be somehow attached to
the async
function object.
(2) __async__.framework or something similar:
Add something like __async__ that has an attribute such as
.framework
that allows the code inside the async function to access the
information
passed to next(...) by the framework (L end) using the keyword
argument
of next [see (1)].
(3) Generalized "await":
[When the world is ready:] Allow using "await" anywhere, not
just within
async functions. Inside async functions, the behavior of
"await" would
be the same as in PEP492, with the addition that it would somehow
propagate the __async__.framework value to the awaited coroutine.
Outside async functions, "await" would do roughly the same as
this function:
def await(async_func_obj):
try:
next(async_func_obj) # same as next(async_func_obj,
async_framework = None)
except StopIteration as si:
return si.value
raise RuntimeError("The function does not support
synchronous
execution")
(This function would, of course, work in Python 3.4, but it
would be
mostly useless because the async functions would not know that
they are
being called in a 'synchronous program'. IIUC, this *function*
would be
valid even with PEP492, but having this as a function would be
ugly in
the long run.)
Some random thoughts:
With this addition to Python, one could write libraries that
work both
async and non-async. When await is not inside async def, one
would
expect it to potentially do blocking IO, just like an await
inside async
def would suggest that there is a yield/suspend somewhere in
there.
For testing, I tried to see if there is a reasonable way to
make a hack
with __async__.framework that could be set by next(), but did
not find
an obvious way. For instance, coro.gi_frame.f_locals is
read-only, I
believe.
An alternative to this approach could be that await would
implicitly
start a temporary event loop for running the coroutine, but
how would it
know which event loop? This might also have a huge performance
overhead.
Relation to PEP492:
This of course still needs more thinking, but I wanted to post
it here
now in case there is desire to prepare for something like this
already
in PEP492. It is not completely clear if/how this would need
to affect
PEP492, but some things come to mind. For example, this could
potentially remove the need for __aenter__, __aiter__, etc. or
even
"async for" and "async with". If __aenter__ is defined as
"async def",
then a with statement would do an "await" on it, and the
context manager
would have __async__.framework (or whatever it would be called)
available, for determining what behavior is appropriate.
Was this clear enough to understand which problem(s) this
would be
solving and how? I'd be happy to hear about any thoughts on
this :).
Best regards,
Koos
_______________________________________________
Python-ideas mailing list
Python-ideas@python.org <mailto:Python-ideas@python.org>
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________
Python-ideas mailing list
Python-ideas@python.org <mailto:Python-ideas@python.org>
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido http://python.org/%7Eguido)