await outside async function could map to asyncio.run ?
Hi all, Currently, you can not use await outside an async function, the following code: async def lol(): return 'bar' def test(): return await lol() print(test()) Will fail with: SyntaxError: 'await' outside async function Of course, you can use asyncio.run and then it works fine: import asyncio async def lol(): return 'bar' def test(): return asyncio.run(lol()) print(test()) Why not make using await do asyncio.run "behind the scenes" when called outside async function ? Thank you in advance for your replies -- ∞
See some of the previous discussions on asyncio reentrancy. https://mail.python.org/pipermail/async-sig/2019-March/thread.html I do think there is some case for non rentrency and nested loop where what you define here would block an outer loop, but most people suggesting what you ask actually want re-entrency, which is not possible there. In your case test() would fail in a context where there is already a running loo. Also Explicit is better than implicit. -- Matthias On Fri, 12 Jun 2020 at 13:50, J. Pic <jpic@yourlabs.org> wrote:
Hi all,
Currently, you can not use await outside an async function, the following code:
async def lol(): return 'bar'
def test(): return await lol()
print(test())
Will fail with: SyntaxError: 'await' outside async function
Of course, you can use asyncio.run and then it works fine:
import asyncio
async def lol(): return 'bar'
def test(): return asyncio.run(lol())
print(test())
Why not make using await do asyncio.run "behind the scenes" when called outside async function ?
Thank you in advance for your replies
-- ∞ _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/LOCYSY... Code of Conduct: http://python.org/psf/codeofconduct/
I do think there is some case for non rentrency and nested loop where what you define here would block an outer loop, but most people suggesting what you ask actually want re-entrency, which is not possible there.
Do you mean that it's not possible to implement this at the syntax level because we don't know until runtime if this is being call from a loop (async calling sync code) ?
Also Explicit is better than implicit.
I'm not sure about the limits of this statement, I'm pretty happy to call foo.bar instead of foo.__getattribute__('bar'), but I'm still grateful for __getattribute__ that I can override. But of course, if there were two competing __getattribute__ implementation then the language should force me to choose one of them explicitly.
Do you mean that it's not possible to implement this at the syntax level because we don't know until runtime if this is being call from a loop (async calling sync code) ?
No. I'm saying you should clarify the semantics you want if you have if your sync `test()` is called from within an async function and an async eventloop that already have tasks. 1) Do you allow switches to tasks that are "outside" of test, or 2) Does your inner loop block all the outer loop tasks. -- M On Fri, 12 Jun 2020 at 14:51, J. Pic <jpic@yourlabs.org> wrote:
I do think there is some case for non rentrency and nested loop where what you define here would block an outer loop, but most people suggesting what you ask actually want re-entrency, which is not possible there.
Do you mean that it's not possible to implement this at the syntax level because we don't know until runtime if this is being call from a loop (async calling sync code) ?
Also Explicit is better than implicit.
I'm not sure about the limits of this statement, I'm pretty happy to call foo.bar instead of foo.__getattribute__('bar'), but I'm still grateful for __getattribute__ that I can override. But of course, if there were two competing __getattribute__ implementation then the language should force me to choose one of them explicitly.
On 2020-06-12 5:47 p.m., J. Pic wrote:
Hi all,
Currently, you can not use await outside an async function, the following code:
async def lol(): return 'bar'
def test(): return await lol()
print(test())
Will fail with: SyntaxError: 'await' outside async function
Of course, you can use asyncio.run and then it works fine:
import asyncio
async def lol(): return 'bar'
def test(): return asyncio.run(lol())
print(test())
Why not make using await do asyncio.run "behind the scenes" when called outside async function ?
Thank you in advance for your replies
What if we extend awaitables to be optionally "runnable"? An await outside async def, would map to awaitable.run(). In the specific case of coroutines, this would also automatically propagate down. E.g. async def foo(): await bar() def baz() await foo() A call to baz() would thus create a coroutine foo(), and then run() it. By my proposed semantics, the await bar() would also call run() on the result of bar()! In particular, this could allow one to turn any arbitrary async coroutine into a blocking subroutine, as long as it supports this run() protocol. This would be great for refactoring existing blocking code into async - just use stuff as if it's sync, then when you're done, switch up your function with async def, and change relevant calls into it into "await" calls. Rinse and repeat until you're ready to add a top-level event loop. Optionally, this would also allow us to deprecate blocking APIs (like open(), etc) and asyncio, and merge them into one TBD API. Perhaps as part of Python 4. (e.g. turn existing open() into await open(), particularly in non-async-def. I *think* this could be done automatically as part of some sort of "3to4", even.)
-- ∞
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/LOCYSY... Code of Conduct: http://python.org/psf/codeofconduct/
An await outside async def, would map to awaitable.run().
A call to baz() would thus create a coroutine foo(), and then run() it.
-1. The `await` expression already has a well-defined purpose of suspending the coroutine it's used within until the awaitable is completed. I think this change in behavior would be rather confusing, and results in minimal (if any) practical benefit since you can already accomplish the main intended behavior of effectively running coroutines as blocking subroutines (see next section).
In particular, this could allow one to turn any arbitrary async coroutine into a blocking subroutine, as long as it supports this run() protocol. This would be great for refactoring existing blocking code into async - just use stuff as if it's sync, then when you're done, switch up your function with async def, and change relevant calls into it into "await" calls. Rinse and repeat until you're ready to add a top-level event loop.
This can already be effectively accomplished by gradually converting the synchronous code into coroutines and running them individually with asyncio's loop.run_until_complete(coro()) (which blocks until finished) from a subroutine while the other parts of the code are still transitioning to async. However, the bulk of the effort is considering the architecture of the program, specifically analyzing the parts that need to be completed sequentially vs those that can be completed concurrently, and then figuring out exactly when/where the results are needed. So, I don't think that making an await expression outside of a coroutine function an alias to an awaitable.run() and effectively doing the same thing as loop.run_until_complete() would save substantial development time, and it adds additional unneeded complexity to the await expression for little to no practical benefit. Also, I very much appreciate knowing that when I see "await" used, I'm certain it's being called within a coroutine and more importantly, the expected behavior. Having to check whether the function is awaited in is a coroutine or a subroutine to determine the behavior could prove to be quite cumbersome in non-trivial programs. A significant part of why await was added in the first place was to avoid the ambiguity present with "yield from". In general, I think there's a significant advantage in having a singular purpose for a keyword, rather than changing based on where it's used.
Optionally, this would also allow us to deprecate blocking APIs (like open(), etc
Huge -1 for deprecating the existing blocking APIs. Although converting some IO-bound functions to asynchronous equivalents can yield significant performance benefits (when structured optimally), it's not always worthwhile to do so. Particularly in situations where the underlying OS call doesn't have asynchronous support (frequently the case for async file system operations), or when the intended use case for the program wouldn't benefit from the performance improvements enough to justify the added complexity and development time. So blocking IO-bound functions still have a significant purpose, and should definitely __not__ be deprecated. Even if there was a working automatic conversion process to place an "await" before each of the calls, it would still cause a lot of breakage and add additional noise to the code that wouldn't provide additional useful context. Also, consider the amount of times that various Python tutorials and textbooks use open() that would all have to be rewritten because it no longer works without "await" before it. Realistically speaking, I don't think there's any chance of something as commonly used as open() being deprecated, regardless how far out the removal would be. Even if/when async file system support at the OS level becomes the mainstream (which it currently isn't), I still don't think it would be worth breaking open().
and asyncio, and merge them into one TBD API. Perhaps as part of Python 4.
Even if I agreed with the main proposal, I would still be against this part. Stdlib module deprecation is a very long process that has to be strongly justified, considering the amount of breakage that would occur from its eventual removal. So even with the assumption that the proposal would provide some real benefit in the transition from sync to async (which I disagree with), I don't think this would come even close to justifying a full deprecation of asyncio into a separate "TBD API". Thanks for sharing the idea Soni, but I would personally be opposed to every component of the proposal. On Fri, Jun 12, 2020 at 7:20 PM Soni L. <fakedme+py@gmail.com> wrote:
On 2020-06-12 5:47 p.m., J. Pic wrote:
Hi all,
Currently, you can not use await outside an async function, the following code:
async def lol(): return 'bar'
def test(): return await lol()
print(test())
Will fail with: SyntaxError: 'await' outside async function
Of course, you can use asyncio.run and then it works fine:
import asyncio
async def lol(): return 'bar'
def test(): return asyncio.run(lol())
print(test())
Why not make using await do asyncio.run "behind the scenes" when called outside async function ?
Thank you in advance for your replies
What if we extend awaitables to be optionally "runnable"?
An await outside async def, would map to awaitable.run(). In the specific case of coroutines, this would also automatically propagate down. E.g.
async def foo(): await bar()
def baz() await foo()
A call to baz() would thus create a coroutine foo(), and then run() it. By my proposed semantics, the await bar() would also call run() on the result of bar()!
In particular, this could allow one to turn any arbitrary async coroutine into a blocking subroutine, as long as it supports this run() protocol. This would be great for refactoring existing blocking code into async - just use stuff as if it's sync, then when you're done, switch up your function with async def, and change relevant calls into it into "await" calls. Rinse and repeat until you're ready to add a top-level event loop.
Optionally, this would also allow us to deprecate blocking APIs (like open(), etc) and asyncio, and merge them into one TBD API. Perhaps as part of Python 4. (e.g. turn existing open() into await open(), particularly in non-async-def. I *think* this could be done automatically as part of some sort of "3to4", even.)
-- ∞
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.orghttps://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/LOCYSY... Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/Y73VH5... Code of Conduct: http://python.org/psf/codeofconduct/
On 2020-06-13 2:32 a.m., Kyle Stanley wrote:
An await outside async def, would map to awaitable.run().
A call to baz() would thus create a coroutine foo(), and then run() it.
-1. The `await` expression already has a well-defined purpose of suspending the coroutine it's used within until the awaitable is completed. I think this change in behavior would be rather confusing, and results in minimal (if any) practical benefit since you can already accomplish the main intended behavior of effectively running coroutines as blocking subroutines (see next section).
It'd also let you force blocking mode by calling .run() directly, whereas "await" would let the context make the decision for you.
In particular, this could allow one to turn any arbitrary async coroutine into a blocking subroutine, as long as it supports this run() protocol. This would be great for refactoring existing blocking code into async - just use stuff as if it's sync, then when you're done, switch up your function with async def, and change relevant calls into it into "await" calls. Rinse and repeat until you're ready to add a top-level event loop.
This can already be effectively accomplished by gradually converting the synchronous code into coroutines and running them individually with asyncio's loop.run_until_complete(coro()) (which blocks until finished) from a subroutine while the other parts of the code are still transitioning to async. However, the bulk of the effort is considering the architecture of the program, specifically analyzing the parts that need to be completed sequentially vs those that can be completed concurrently, and then figuring out exactly when/where the results are needed. So, I don't think that making an await expression outside of a coroutine function an alias to an awaitable.run() and effectively doing the same thing as loop.run_until_complete() would save substantial development time, and it adds additional unneeded complexity to the await expression for little to no practical benefit.
Also, I very much appreciate knowing that when I see "await" used, I'm certain it's being called within a coroutine and more importantly, the expected behavior. Having to check whether the function is awaited in is a coroutine or a subroutine to determine the behavior could prove to be quite cumbersome in non-trivial programs. A significant part of why await was added in the first place was to avoid the ambiguity present with "yield from". In general, I think there's a significant advantage in having a singular purpose for a keyword, rather than changing based on where it's used.
There's a huge difference between "yield from" and "await". Additionally, what if I'm not using asyncio?
Optionally, this would also allow us to deprecate blocking APIs (like open(), etc
Huge -1 for deprecating the existing blocking APIs. Although converting some IO-bound functions to asynchronous equivalents can yield significant performance benefits (when structured optimally), it's not always worthwhile to do so. Particularly in situations where the underlying OS call doesn't have asynchronous support (frequently the case for async file system operations), or when the intended use case for the program wouldn't benefit from the performance improvements enough to justify the added complexity and development time. So blocking IO-bound functions still have a significant purpose, and should definitely __not__ be deprecated. Even if there was a working automatic conversion process to place an "await" before each of the calls, it would still cause a lot of breakage and add additional noise to the code that wouldn't provide additional useful context.
Also, consider the amount of times that various Python tutorials and textbooks use open() that would all have to be rewritten because it no longer works without "await" before it. Realistically speaking, I don't think there's any chance of something as commonly used as open() being deprecated, regardless how far out the removal would be. Even if/when async file system support at the OS level becomes the mainstream (which it currently isn't), I still don't think it would be worth breaking open().
As I said previously, you can still guarantee blocking behaviour by explicitly calling .run(), instead of using await. There is a lot of friction in refactoring to async/await, including asyncio being suboptimal and the fact that you can't really call async functions (but in blocking mode) from non-async code.
and asyncio, and merge them into one TBD API. Perhaps as part of Python 4.
Even if I agreed with the main proposal, I would still be against this part. Stdlib module deprecation is a very long process that has to be strongly justified, considering the amount of breakage that would occur from its eventual removal. So even with the assumption that the proposal would provide some real benefit in the transition from sync to async (which I disagree with), I don't think this would come even close to justifying a full deprecation of asyncio into a separate "TBD API".
Thanks for sharing the idea Soni, but I would personally be opposed to every component of the proposal.
What's the point of having async, if it has to be such a massive pain to refactor my existing code into? Some of the issues caused by only partially switching to async, namely that you have to go out of your way to call an async function from non-async code, feel to me like unnecessary friction. I understand and love having 2 calling conventions, but, heck even having coroutines carry a run() method would be better than what we have today. And then there's the issue of nested event loops. It just makes things even more problematic in hybrid codebases. :/ Consider my proposal the best of both worlds: you get all the benefits of async/await, and all the benefits of blocking APIs, and you get to choose whether to use autodetection or explicitly force the latter. (you could also force the existence of a nested event loop, but I don't think that'd be a good idea.)
On Fri, Jun 12, 2020 at 7:20 PM Soni L. <fakedme+py@gmail.com <mailto:fakedme%2Bpy@gmail.com>> wrote:
On 2020-06-12 5:47 p.m., J. Pic wrote:
Hi all,
Currently, you can not use await outside an async function, the following code:
async def lol(): return 'bar'
def test(): return await lol()
print(test())
Will fail with: SyntaxError: 'await' outside async function
Of course, you can use asyncio.run and then it works fine:
import asyncio
async def lol(): return 'bar'
def test(): return asyncio.run(lol())
print(test())
Why not make using await do asyncio.run "behind the scenes" when called outside async function ?
Thank you in advance for your replies
What if we extend awaitables to be optionally "runnable"?
An await outside async def, would map to awaitable.run(). In the specific case of coroutines, this would also automatically propagate down. E.g.
async def foo(): await bar()
def baz() await foo()
A call to baz() would thus create a coroutine foo(), and then run() it. By my proposed semantics, the await bar() would also call run() on the result of bar()!
In particular, this could allow one to turn any arbitrary async coroutine into a blocking subroutine, as long as it supports this run() protocol. This would be great for refactoring existing blocking code into async - just use stuff as if it's sync, then when you're done, switch up your function with async def, and change relevant calls into it into "await" calls. Rinse and repeat until you're ready to add a top-level event loop.
Optionally, this would also allow us to deprecate blocking APIs (like open(), etc) and asyncio, and merge them into one TBD API. Perhaps as part of Python 4. (e.g. turn existing open() into await open(), particularly in non-async-def. I *think* this could be done automatically as part of some sort of "3to4", even.)
-- ∞
_______________________________________________ Python-ideas mailing list --python-ideas@python.org <mailto:python-ideas@python.org> To unsubscribe send an email topython-ideas-leave@python.org <mailto:python-ideas-leave@python.org> https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived athttps://mail.python.org/archives/list/python-ideas@python.org/message/LOCYSY... Code of Conduct:http://python.org/psf/codeofconduct/
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org <mailto:python-ideas@python.org> To unsubscribe send an email to python-ideas-leave@python.org <mailto:python-ideas-leave@python.org> https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/Y73VH5... Code of Conduct: http://python.org/psf/codeofconduct/
On 13/06/20 8:47 am, J. Pic wrote:
Why not make using await do asyncio.run "behind the scenes" when called outside async function ?
That would directly tie the await syntax to the asyncio module. This would not be a good idea. Currently there is nothing special about asyncio -- it's just one of many possible frameworks built on async/await. This proposal would elevate it to a very special status, making it effectively part of the language rather than just a module in the stdlib. -- Greg
Well correct me if I'm wrong (I did a bit of homework), but threads are hard to get right because they may switch at any time. When we do async instead of threads, it's because we want task switch on blocking operations only. The thing that is still not quite clear to me, and really I must apologize because I know you kind people have gone through quite some effort to clarify this for me, and I'm feeling pretty bad enough about it, I don't want to be the vampire of the list, I welcome all criticism on my behaviour even off list, as some of you know, I'm just a script-kiddie that became a script-daddy (don't worry I'm not going to apologize all the time, at least now it's done for this list !), anyway, given that: - a blocking call is caused by sleeps and io, - we want blocking calls to cause a task switch, - await serves to signal the coder of a potential task switch. A coder *should* know if a function is going to do a sleep or a blocking call, if i'm calling git.push() for example, there's definitely going to be some subprocess, networking, fs work etc. As such, I feel like I'm *already* supposed to know that this implies blocking calls, and that a task switch may occur if calling git.push() in an async job, otherwise would I really be doing a good job at being a developer ? Some have said that sending people who don't get so much satisfaction (getting some, but not "so much") writing async code to just use threading, wouldn't gevent be the closest alternative to suggest ? Thanks a heap for all your replies and for keeping the list open, Have a great Sunday ;)
On Sun, Jun 14, 2020 at 10:38 PM J. Pic <jpic@yourlabs.org> wrote:
Well correct me if I'm wrong (I did a bit of homework), but threads are hard to get right because they may switch at any time.
When we do async instead of threads, it's because we want task switch on blocking operations only.
That's an oversimplification, but yes. If you have code like this: limit = 10 def spam(some,stuff): global limit if limit: limit -= 1 spam(more,stuff) then yes, you are extremely at risk of multiple threads messing you up, because between "if limit" and "limit -= 1", another thread could run the same code. Doing the same thing with an async function and "await spam(more,stuff)" wouldn't have this risk, because you know for sure that there can be no context switch outside of that "await" keyword. But the problem here isn't threads. The problem is mutable globals, and not having any sort of lock to keep it safe. Threads aren't inherently dangerous in the sense that everything will suddenly break the instant you create a second thread; what happens is that vulnerable code can work in a single-threaded situation without ever being triggered. So async/await is easier to get right because it behaves like single-threaded code but with well-defined context switches, but multi-threaded code isn't THAT hard to get right. Both async/await and Python threads are vulnerable to the problem that a *spinning* thread will prevent other tasks from running, but with async/await, ANY big job will block, whereas with threads, it has to be something keeping the CPU busy while holding the GIL. So threads are safer against that sort of thing, while being vulnerable to a different sort of thing (a thing which, IMO, is a coding error anyway). ChrisA
participants (6)
-
Chris Angelico -
Greg Ewing -
J. Pic -
Kyle Stanley -
Matthias Bussonnier -
Soni L.