Re: New syntax for decorators that are compatible with both normal and async functions
On Feb 10, 2020, at 02:34, Bar Harel <bharel@barharel.com> wrote:
While I'm not this idea can totally work, I do think we need a method of closing the gap between asyc coroutines and normal functions.
As of today, async applications need to create an entirely different codebase, sometimes just by copy and paste, while changing function definitions to async and function calls to await.
This creates a lot of work, from porting code to maintaining both variants, to just having code duplication all around.
It’s a well-known problem that async is “contagious”: once you turn one function into a coroutine, you have to turn the whole call tree async up to some point where you can either block or daemonize a coroutine. Together with the fact that async operations have not just different syntax but different names and APIs than their sync equivalents, this makes it hard to port code between the two paradigms. But C# and every other language that’s borrowed the idea has the same problem, and as far as I know, nobody’s thought of a good answer yet. Just saying “this is bad, so we should do something” doesn’t get us anywhere unless that something is doable and solves the problem and we know what it is. And I don’t know if that’s the case here. The OP proposed something that can only work by impossible magic, and you suggested it would be nice if we had something actually implementable that was just as easy to use, and it definitely would be nice, but unless someone has an idea for that something… (That definitely doesn’t mean it isn’t worth pursuing. Just because nobody else has thought of it yet doesn’t mean it’s impossible, after all.) I think breaking down decorators into smaller building blocks that can be composed might be promising. If you have a 30 different decorators that all want to do something (that doesn’t block) before calling the wrapper function or coro, you can write a single `@before` that does have to duplicate its code and then 30 decorators that use `before` and don’t need any duplication. I’m not sure how far you can get with that. Certainly not all the way to the full generality of arbitrary-code decorators (although I don’t think even the OP’s magic would get use there—you still can’t put any code into the decorator that might be blocking except for the wrapped function, especially since usually the difference isn’t just a matter of await or not, but of calling a different method on a different object…). But maybe far enough to be useful? I don’t know. Another possible avenue is to drop back a level and write wrappers that work on futures. Async futures and concurrent futures duck type as lowest-common-denominator futures, and maybe some of the helpers you want can be built that way without duplicating their code? Meanwhile, for porting large existing threaded or synchronous codebases, you may be better off using the gevent model, where awaiting is internal and invisible. You can find out pretty quickly whether all of your libraries are compatible or not. If they are, you won’t get all of the advantages (e.g., with await it’s often obvious when you can remove a mutex or other sync object; with gevent, because it looks nearly identical to preemptive threading, if that mutex was necessary in the preemptive version, it still looks necessary in the gevent version even though it isn’t, unless you think it through carefully), but only having to rewrite 2% of your code instead of 60% may be worth it anyway. Just because asyncio is usually the best model for new code doesn’t mean it’s the always best model for porting all code.
I wonder why async needed to be like it is. await foo(...) could have turned into something like foo.__await__(...) and had foo.__call__ just run the function anyway until actual yielding is necessary (at which point, raise RuntimeError). oh well, can't change that now I guess. :/ On 2020-02-10 2:49 p.m., Andrew Barnert via Python-ideas wrote:
On Feb 10, 2020, at 02:34, Bar Harel <bharel@barharel.com> wrote:
While I'm not this idea can totally work, I do think we need a method of closing the gap between asyc coroutines and normal functions.
As of today, async applications need to create an entirely different codebase, sometimes just by copy and paste, while changing function definitions to async and function calls to await.
This creates a lot of work, from porting code to maintaining both variants, to just having code duplication all around.
It’s a well-known problem that async is “contagious”: once you turn one function into a coroutine, you have to turn the whole call tree async up to some point where you can either block or daemonize a coroutine. Together with the fact that async operations have not just different syntax but different names and APIs than their sync equivalents, this makes it hard to port code between the two paradigms.
But C# and every other language that’s borrowed the idea has the same problem, and as far as I know, nobody’s thought of a good answer yet. Just saying “this is bad, so we should do something” doesn’t get us anywhere unless that something is doable and solves the problem and we know what it is. And I don’t know if that’s the case here. The OP proposed something that can only work by impossible magic, and you suggested it would be nice if we had something actually implementable that was just as easy to use, and it definitely would be nice, but unless someone has an idea for that something… (That definitely doesn’t mean it isn’t worth pursuing. Just because nobody else has thought of it yet doesn’t mean it’s impossible, after all.)
I think breaking down decorators into smaller building blocks that can be composed might be promising. If you have a 30 different decorators that all want to do something (that doesn’t block) before calling the wrapper function or coro, you can write a single `@before` that does have to duplicate its code and then 30 decorators that use `before` and don’t need any duplication. I’m not sure how far you can get with that. Certainly not all the way to the full generality of arbitrary-code decorators (although I don’t think even the OP’s magic would get use there—you still can’t put any code into the decorator that might be blocking except for the wrapped function, especially since usually the difference isn’t just a matter of await or not, but of calling a different method on a different object…). But maybe far enough to be useful? I don’t know.
Another possible avenue is to drop back a level and write wrappers that work on futures. Async futures and concurrent futures duck type as lowest-common-denominator futures, and maybe some of the helpers you want can be built that way without duplicating their code?
Meanwhile, for porting large existing threaded or synchronous codebases, you may be better off using the gevent model, where awaiting is internal and invisible. You can find out pretty quickly whether all of your libraries are compatible or not. If they are, you won’t get all of the advantages (e.g., with await it’s often obvious when you can remove a mutex or other sync object; with gevent, because it looks nearly identical to preemptive threading, if that mutex was necessary in the preemptive version, it still looks necessary in the gevent version even though it isn’t, unless you think it through carefully), but only having to rewrite 2% of your code instead of 60% may be worth it anyway. Just because asyncio is usually the best model for new code doesn’t mean it’s the always best model for porting all code.
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/P2VO5L... Code of Conduct: http://python.org/psf/codeofconduct/
On Mon, Feb 10, 2020 at 9:50 AM Andrew Barnert via Python-ideas <python-ideas@python.org> wrote:
It’s a well-known problem that async is “contagious”: [...]
But C# and every other language that’s borrowed the idea has the same problem, and as far as I know, nobody’s thought of a good answer yet.
Threads don't have that problem: you can use non-thread-aware code with callbacks in your threaded program if you do your own locking. Haskell (GHC) doesn't have that problem: it has fibers that use a programming interface like C#/Python threads, but they're multiplexed by user-mode code within a single OS thread. 16-bit Windows didn't have that problem. Stackless and greenlet don't have that problem. It's a problem that can be solved by just doing the obvious thing, the thing that Python already did with threads: don't define a novel syntax for coroutines, but instead use the syntax that already existed. Async/await syntax is a static type system with two types, may-yield and will-not-yield. There's no provision for writing generic/polymorphic code over those types, so you have to write everything twice. Like any static type system it has some benefits, but I don't think it's worth the cost, especially in Python, which has always eschewed mandatory static typing. I don't know how to fix Python now that it's gone so thoroughly down this path (starting with the yield keyword 18 years ago). But it's not a problem that ever needed to exist. Coroutines aren't that hard. -- Ben
On 2020-02-11 4:33 a.m., Ben Rudiak-Gould wrote:
On Mon, Feb 10, 2020 at 9:50 AM Andrew Barnert via Python-ideas <python-ideas@python.org> wrote:
It’s a well-known problem that async is “contagious”: [...]
But C# and every other language that’s borrowed the idea has the same problem, and as far as I know, nobody’s thought of a good answer yet.
Threads don't have that problem: you can use non-thread-aware code with callbacks in your threaded program if you do your own locking. Haskell (GHC) doesn't have that problem: it has fibers that use a programming interface like C#/Python threads, but they're multiplexed by user-mode code within a single OS thread. 16-bit Windows didn't have that problem. Stackless and greenlet don't have that problem.
It's a problem that can be solved by just doing the obvious thing, the thing that Python already did with threads: don't define a novel syntax for coroutines, but instead use the syntax that already existed.
Async/await syntax is a static type system with two types, may-yield and will-not-yield. There's no provision for writing generic/polymorphic code over those types, so you have to write everything twice. Like any static type system it has some benefits, but I don't think it's worth the cost, especially in Python, which has always eschewed mandatory static typing.
I don't know how to fix Python now that it's gone so thoroughly down this path (starting with the yield keyword 18 years ago). But it's not a problem that ever needed to exist. Coroutines aren't that hard.
I would suggest having an "await in" operator. Or an "autowait" function: def autowait(value): if is_awaitable(value): return value async def force_await(): return value return force_await() This would allow one to write "async-ready" code while calling non-async functions.
-- Ben _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/ZDQIZ5... Code of Conduct: http://python.org/psf/codeofconduct/
The idea on the OP can be fully implemented with ~10 lines of code in a decorator - which could be further factored out into a decorator-for-decorators: ``` import asyncio, inspect def hybrid_decorator(func): coro = inspect.iscoroutinefunction(func) if coro: async def inner(*args, **kw): return await func(*args, **kw) else: def inner(*args, **kw): return func(*args, **kw) def wrapper(*args, **kw): ... value = inner(*args, **kw) ... return value if coro: wrapper = asyncio.coroutine(wrapper) return wrapper ``` The thing is that just in Python 3.8 the `@courotine` decorator has been deprecated - not sure if there is another "in language" way of changing an already defined, no "await" in body, function to a coroutine function. One way or another, this approach seems quite feasible for whoever wants to make use of such a feature with no need for new syntax (but maybe need for having an "undeprecated" way of doing what `@coroutine` does) Here is the approach factored out to a "decorator decorator" that does the dirty job: ``` def hybrid_async(deco): @wraps(deco) def wrapper(func): coro = inspect.iscoroutinefunction(func) if coro: async def inner(*args, **kw): return await func(*args, **kw) else: inner = func wrapped = wraps(func)(deco(inner)) if coro: wrapped = asyncio.coroutine(wrapped) return wrapped return wrapper @hybrid_async def log_run(func): @wraps(func) def wrapper(*args, **kw): print("start", flush=True) v = func(*args, **kw) print("stop", flush=True) return v return wrapper ``` On Tue, 11 Feb 2020 at 11:59, Soni L. <fakedme+py@gmail.com> wrote:
On Mon, Feb 10, 2020 at 9:50 AM Andrew Barnert via Python-ideas <python-ideas@python.org> wrote:
It’s a well-known problem that async is “contagious”: [...]
But C# and every other language that’s borrowed the idea has the same
On 2020-02-11 4:33 a.m., Ben Rudiak-Gould wrote: problem, and as far as I know, nobody’s thought of a good answer yet.
Threads don't have that problem: you can use non-thread-aware code with callbacks in your threaded program if you do your own locking. Haskell (GHC) doesn't have that problem: it has fibers that use a programming interface like C#/Python threads, but they're multiplexed by user-mode code within a single OS thread. 16-bit Windows didn't have that problem. Stackless and greenlet don't have that problem.
It's a problem that can be solved by just doing the obvious thing, the thing that Python already did with threads: don't define a novel syntax for coroutines, but instead use the syntax that already existed.
Async/await syntax is a static type system with two types, may-yield and will-not-yield. There's no provision for writing generic/polymorphic code over those types, so you have to write everything twice. Like any static type system it has some benefits, but I don't think it's worth the cost, especially in Python, which has always eschewed mandatory static typing.
I don't know how to fix Python now that it's gone so thoroughly down this path (starting with the yield keyword 18 years ago). But it's not a problem that ever needed to exist. Coroutines aren't that hard.
I would suggest having an "await in" operator. Or an "autowait" function:
def autowait(value): if is_awaitable(value): return value async def force_await(): return value return force_await()
This would allow one to write "async-ready" code while calling non-async functions.
-- Ben _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/ZDQIZ5... Code of Conduct: http://python.org/psf/codeofconduct/
Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/P4SHOV... Code of Conduct: http://python.org/psf/codeofconduct/
On 2/11/2020 2:33 AM, Ben Rudiak-Gould wrote:
It’s a well-known problem that async is “contagious”: [...]
But C# and every other language that’s borrowed the idea has the same problem, and as far as I know, nobody’s thought of a good answer yet. Threads don't have that problem: you can use non-thread-aware code with callbacks in your threaded program if you do your own locking. Haskell (GHC) doesn't have that problem: it has fibers that use a
On Mon, Feb 10, 2020 at 9:50 AM Andrew Barnert via Python-ideas <python-ideas@python.org> wrote: programming interface like C#/Python threads, but they're multiplexed by user-mode code within a single OS thread. 16-bit Windows didn't have that problem. Stackless and greenlet don't have that problem.
It's a problem that can be solved by just doing the obvious thing, the thing that Python already did with threads: don't define a novel syntax for coroutines, but instead use the syntax that already existed.
Async/await syntax is a static type system with two types, may-yield and will-not-yield. There's no provision for writing generic/polymorphic code over those types, so you have to write everything twice. Like any static type system it has some benefits, but I don't think it's worth the cost, especially in Python, which has always eschewed mandatory static typing.
I don't know how to fix Python now that it's gone so thoroughly down this path (starting with the yield keyword 18 years ago). But it's not a problem that ever needed to exist. Coroutines aren't that hard.
One of the goals of the async/await syntax is to make it clear where control can yield, and therefor where you need to be concerned about mutating state. You don't have this explicitness with threads or with a hypothetical syntax that would hide these yield points, so you'd need locks. In my experience, incorrect locking is the source of many hard-to-find bugs with threads. Eric Eric
On Feb 11, 2020, at 14:02, Eric V. Smith <eric@trueblade.com> wrote:
One of the goals of the async/await syntax is to make it clear where control can yield, and therefor where you need to be concerned about mutating state. You don't have this explicitness with threads or with a hypothetical syntax that would hide these yield points, so you'd need locks. In my experience, incorrect locking is the source of many hard-to-find bugs with threads.
My own experience converting a smallish gevent project to asyncio was that we only found a few errors—but a ton of unnecessary locks that were the cause of a lot of wasted context switching and other overhead (and debugging annoyance). And I’ve had similar experiences with Windows fibers and longjmp fiberish things in C. That was enough to convince me that the benefits of mandatorily marking switch points isn’t just unnecessary static typing, but a good use of it. If there is a problem with asyncio (and similar libraries, both in Python and elsewhere), it’s that it forces you to rethink more than just where you can block. For example, you can’t just turn a csock.recv(BUFSIZE) into an await csock.recv(BUFSIZE) because there is no asyncsocket.socket that’s just like socket.socket except its methods are coroutines. But most people aren’t writing most of their code at the lowest level of abstraction; usually you end up, e.g., building a stack of protocol objects that look very similar between twisted and asyncio, even if the lowest level code that you rarely look at is very different. So I think it’s not really a problem for most people.
participants (5)
-
Andrew Barnert -
Ben Rudiak-Gould -
Eric V. Smith -
Joao S. O. Bueno -
Soni L.