After years of playing with asyncio, I'm still having a harder time using it than any other async architecture around. There are a lot of different reasons for it, but this mail want to address one particular one: The event loop and policy can be tweaked at any time, by anyone. Now, it's hard enough to have to deal, manually, with a low-level event loop. But having it exposed that much, and it being that flexible means any code can just do whatever it wants with it, and make a mess. Several things in particular, comes to mind: - Changing the event loop policy - Changing the event loop - Spawning a new loop - Starting the loop - Stopping the loop - Closing the loop Now, if you want to make any serious project with it, you currently have to guard against all of those, especially if you want to have proper cleanup code, good error message and a decent debugging experience. I tried to do it for one year, and currently, it's very hard. You have a lot of checks to make, redundantly in a lot of places. Some things can only be done by providing a custom event policy/loop yourself, and, of course, expecting (aka documenting and praying) that it's used. For a lot of things, when it breaks, the people that haven't read the doc in depth will have a hard time to understand the problem after the fact. Sometimes, it's just that your code use somebody else code that is not here to read your doc anymore. Now you have to check their code to understand what they are doing that breaks your expectations about the loop / policy or workflow. Barring the creating of an entire higher level framework that everybody will agree on using and that makes messing up way harder, we can improve this situation by adding hooks to those events. I hence propose to add: - asyncio.on_change_policy(cb:Callable[[EventLoopPolicy, EventLoopPolicy], EventLoopPolicy]) - asyncio.on_set_event_loop(cb:Callable[[EventLoop, EventLoop], EventLoop]) - asyncio.on_create_event_loop(cb:Callable[[EventLoop], EventLoop]) - EventLoop.on_start(cb:Callable[EventLoop]) - EventLoop.on_stop(cb:Awaitable[EventLoop]) - EventLoop.on_close(cb:Callable[EventLoop]) - EventLoop.on_set_debug_mode(cb:Callable[[loop]]) This would allow to implement safer, more robust and easier to debug code. E.G: - you can raise a warning stating that if somebody changes the event policy, it must inherit from your custom one or deal with disabled features - you can raise an exception on loop swap and forbid it, saying that your small script doesn't support it yet so that it's easy to understand the limit of your code - you can hook on the event loop life cycle to automatically get on board, or run clean up code, starting logging, warn that you were supposed to start the loop yourself, etc

Hi Yuri,

I actually want to propose to reduce policies API surface by deprecating and then removing "set_child_watcher()" methods. Besides, there are many other higher priority To-Do items for asyncio in 3.8, like implementing Trio's nursery-like objects and cancellation scopes or fixing tracebacks in Tasks.

That said, the above is my "IMO". And in your email you haven't actually provided clear scenarios that could be solved by adding "event loop hooks" to asyncio. So I have a few questions for you:

- Do you have real-life examples of libraries that abuse policies in some weird ways?

It's not abuse, it's just regular use of a documented feature, but a very potent feature with deep consequences.

- Are those libraries popular?

aiothttp comes to mind. But the fact is, I think people explicitly avoid using policies and custom loops because they know that they can be swapped and you can't make it a hard requirement since they can be swapped under your node.

- What's the actual problem they try to solve by using policies?

e.g: https://github.com/aio-libs/aiohttp/blob/53828229a13dc72137f430abc7a0c469678...

- What problem are you trying to solve in your code that uses policies?

I have tried to create an abstraction that creates an uvloop if it exists, or a regular loop otherwise, or integrated it to the twisted reactor or the trio event loop if it's loaded. I want to abstract that from the user, so I tried to put that in a policy. But that's dangerous since it can be changed at any time, so I gave up on it and made it explicit. Of course, if the user misses that in the doc (hopefully, it's an company internal code so they should be trained), it will be a bummer to debug. Another example is the proof of concept of nurseries we talked about on twitter: https://0bin.net/paste/V5KyhAg-2i5EOyoK#dzBvhdCVeFy8Q2xNcxXyqwtyQFgkxlKI3u5Q... Yet another one, with a custom loop this time: I want to provide a fail fast mode for code using loop.run_forever. The goal is to make it crashes when an uncaught exception occurs instead of just log it in the console to ease debugging in a local machine. Of course it would be best to use run_until_complete instead but you don't always get to choose. So we set a task factory on the loop. But, of course, you lose everything if something changes the loop on the fly, which my code has no idea has happened. Another use case is to just log that the loop / policy has changed in debug mode. It's always something I want to know anyway, because it has consequences on my entire program since those are pretty fundamental components.

- Why do you think this isn't a documentation/tutorial issue?> - Can you list 2-3 clear examples where having hooks would benefit an average asyncio user?

The best documentation is the one you don't need to write. But if I'm being honest, I'd like to have it despite having a good warning in the documentation. When you run Django's manage.py runserver, it will check your code for a lot of common issues and raise a warning or an exception, letting you know what to do. With those hooks, I could check if a policy or a loop is changed, and if some things in my code depends on a policy or a loop, I can do the same and raise a warning or an exception. This benefit the users because they get the info exactly in context, and not just rely on them reading the doc, understanding it, remembering it and applying it. And that benefits the framework writers because that's less support, and less bug reports to deal with. Using asyncio is scary and mysterious enough for a lot of people, so I want to make the experience as natural as possible. I don't want people to have to read my doc and learn what event loops and policies are for basic usage. It's too much. But on the other hand, I do want them to be able to debug a problem if the loop or policy is swapped.

Thank you, Yury

Thank you too

Twisted's reactor API has some lifecycle hooks: https://twistedmatrix.com/documents/18.4.0/api/twisted.internet.interfaces.I... My impression is that this is actually pretty awkward for twisted/asyncio interoperability, because if you're trying to use a twisted library on top of an asyncio loop then there's no reliable way to implement these methods. And twisted uses these internally for things like managing its thread pool. There is some subtlety though because in twisted, reactors can't transition from the stopped state back into the running state, which implies an invariant where the start and shutdown hooks can be called at most once. Anyway, I'm not a twisted expert, but wanted to flag this so you all know that if you're talking about adding lifecycle hooks then you know to go talk to them and get the details. (Trio does have some sorta-kinda analogous functionality. Specifically it has a concept of "system tasks" that are automatically cancelled when the main task exits, so they have a chance to do any cleanup at that point. But trio's lifecycle model is so different that I'm not sure how helpful this is.) -n On Tue, Jun 5, 2018, 05:48 Michel Desmoulin <desmoulinmichel@gmail.com> wrote:

After years of playing with asyncio, I'm still having a harder time using it than any other async architecture around. There are a lot of different reasons for it, but this mail want to address one particular one:

The event loop and policy can be tweaked at any time, by anyone.

Now, it's hard enough to have to deal, manually, with a low-level event loop. But having it exposed that much, and it being that flexible means any code can just do whatever it wants with it, and make a mess.

Several things in particular, comes to mind:

- Changing the event loop policy - Changing the event loop - Spawning a new loop - Starting the loop - Stopping the loop - Closing the loop

Now, if you want to make any serious project with it, you currently have to guard against all of those, especially if you want to have proper cleanup code, good error message and a decent debugging experience.

I tried to do it for one year, and currently, it's very hard. You have a lot of checks to make, redundantly in a lot of places. Some things can only be done by providing a custom event policy/loop yourself, and, of course, expecting (aka documenting and praying) that it's used.

For a lot of things, when it breaks, the people that haven't read the doc in depth will have a hard time to understand the problem after the fact.

Sometimes, it's just that your code use somebody else code that is not here to read your doc anymore. Now you have to check their code to understand what they are doing that breaks your expectations about the loop / policy or workflow.

Barring the creating of an entire higher level framework that everybody will agree on using and that makes messing up way harder, we can improve this situation by adding hooks to those events.

I hence propose to add:

- asyncio.on_change_policy(cb:Callable[[EventLoopPolicy, EventLoopPolicy], EventLoopPolicy])

- asyncio.on_set_event_loop(cb:Callable[[EventLoop, EventLoop], EventLoop])

- asyncio.on_create_event_loop(cb:Callable[[EventLoop], EventLoop])

- EventLoop.on_start(cb:Callable[EventLoop])

- EventLoop.on_stop(cb:Awaitable[EventLoop])

- EventLoop.on_close(cb:Callable[EventLoop])

- EventLoop.on_set_debug_mode(cb:Callable[[loop]])

This would allow to implement safer, more robust and easier to debug code. E.G:

- you can raise a warning stating that if somebody changes the event policy, it must inherit from your custom one or deal with disabled features

- you can raise an exception on loop swap and forbid it, saying that your small script doesn't support it yet so that it's easy to understand the limit of your code

- you can hook on the event loop life cycle to automatically get on board, or run clean up code, starting logging, warn that you were supposed to start the loop yourself, etc _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

After years of playing with asyncio, I'm still having a harder time using it than any other async architecture around. There are a lot of different reasons for it, but this mail want to address one particular one: The event loop and policy can be tweaked at any time, by anyone. Now, it's hard enough to have to deal, manually, with a low-level event loop. But having it exposed that much, and it being that flexible means any code can just do whatever it wants with it, and make a mess. Several things in particular, comes to mind: - Changing the event loop policy - Changing the event loop - Spawning a new loop - Starting the loop - Stopping the loop - Closing the loop Now, if you want to make any serious project with it, you currently have to guard against all of those, especially if you want to have proper cleanup code, good error message and a decent debugging experience. I tried to do it for one year, and currently, it's very hard. You have a lot of checks to make, redundantly in a lot of places. Some things can only be done by providing a custom event policy/loop yourself, and, of course, expecting (aka documenting and praying) that it's used. For a lot of things, when it breaks, the people that haven't read the doc in depth will have a hard time to understand the problem after the fact. Sometimes, it's just that your code use somebody else code that is not here to read your doc anymore. Now you have to check their code to understand what they are doing that breaks your expectations about the loop / policy or workflow. Barring the creating of an entire higher level framework that everybody will agree on using and that makes messing up way harder, we can improve this situation by adding hooks to those events. I hence propose to add: - asyncio.on_change_policy(cb:Callable[[EventLoopPolicy, EventLoopPolicy], EventLoopPolicy]) - asyncio.on_set_event_loop(cb:Callable[[EventLoop, EventLoop], EventLoop]) - asyncio.on_create_event_loop(cb:Callable[[EventLoop], EventLoop]) - EventLoop.on_start(cb:Callable[EventLoop]) - EventLoop.on_stop(cb:Awaitable[EventLoop]) - EventLoop.on_close(cb:Callable[EventLoop]) - EventLoop.on_set_debug_mode(cb:Callable[[loop]]) This would allow to implement safer, more robust and easier to debug code. E.G: - you can raise a warning stating that if somebody changes the event policy, it must inherit from your custom one or deal with disabled features - you can raise an exception on loop swap and forbid it, saying that your small script doesn't support it yet so that it's easy to understand the limit of your code - you can hook on the event loop life cycle to automatically get on board, or run clean up code, starting logging, warn that you were supposed to start the loop yourself, etc

Hi Yuri,

I actually want to propose to reduce policies API surface by deprecating and then removing "set_child_watcher()" methods. Besides, there are many other higher priority To-Do items for asyncio in 3.8, like implementing Trio's nursery-like objects and cancellation scopes or fixing tracebacks in Tasks.

That said, the above is my "IMO". And in your email you haven't actually provided clear scenarios that could be solved by adding "event loop hooks" to asyncio. So I have a few questions for you:

- Do you have real-life examples of libraries that abuse policies in some weird ways?

It's not abuse, it's just regular use of a documented feature, but a very potent feature with deep consequences.

- Are those libraries popular?

aiothttp comes to mind. But the fact is, I think people explicitly avoid using policies and custom loops because they know that they can be swapped and you can't make it a hard requirement since they can be swapped under your node.

- What's the actual problem they try to solve by using policies?

e.g: https://github.com/aio-libs/aiohttp/blob/53828229a13dc72137f430abc7a0c469678...

- What problem are you trying to solve in your code that uses policies?

I have tried to create an abstraction that creates an uvloop if it exists, or a regular loop otherwise, or integrated it to the twisted reactor or the trio event loop if it's loaded. I want to abstract that from the user, so I tried to put that in a policy. But that's dangerous since it can be changed at any time, so I gave up on it and made it explicit. Of course, if the user misses that in the doc (hopefully, it's an company internal code so they should be trained), it will be a bummer to debug. Another example is the proof of concept of nurseries we talked about on twitter: https://0bin.net/paste/V5KyhAg-2i5EOyoK#dzBvhdCVeFy8Q2xNcxXyqwtyQFgkxlKI3u5Q... Yet another one, with a custom loop this time: I want to provide a fail fast mode for code using loop.run_forever. The goal is to make it crashes when an uncaught exception occurs instead of just log it in the console to ease debugging in a local machine. Of course it would be best to use run_until_complete instead but you don't always get to choose. So we set a task factory on the loop. But, of course, you lose everything if something changes the loop on the fly, which my code has no idea has happened. Another use case is to just log that the loop / policy has changed in debug mode. It's always something I want to know anyway, because it has consequences on my entire program since those are pretty fundamental components.

- Why do you think this isn't a documentation/tutorial issue?> - Can you list 2-3 clear examples where having hooks would benefit an average asyncio user?

The best documentation is the one you don't need to write. But if I'm being honest, I'd like to have it despite having a good warning in the documentation. When you run Django's manage.py runserver, it will check your code for a lot of common issues and raise a warning or an exception, letting you know what to do. With those hooks, I could check if a policy or a loop is changed, and if some things in my code depends on a policy or a loop, I can do the same and raise a warning or an exception. This benefit the users because they get the info exactly in context, and not just rely on them reading the doc, understanding it, remembering it and applying it. And that benefits the framework writers because that's less support, and less bug reports to deal with. Using asyncio is scary and mysterious enough for a lot of people, so I want to make the experience as natural as possible. I don't want people to have to read my doc and learn what event loops and policies are for basic usage. It's too much. But on the other hand, I do want them to be able to debug a problem if the loop or policy is swapped.

Thank you, Yury

Thank you too

Twisted's reactor API has some lifecycle hooks: https://twistedmatrix.com/documents/18.4.0/api/twisted.internet.interfaces.I... My impression is that this is actually pretty awkward for twisted/asyncio interoperability, because if you're trying to use a twisted library on top of an asyncio loop then there's no reliable way to implement these methods. And twisted uses these internally for things like managing its thread pool. There is some subtlety though because in twisted, reactors can't transition from the stopped state back into the running state, which implies an invariant where the start and shutdown hooks can be called at most once. Anyway, I'm not a twisted expert, but wanted to flag this so you all know that if you're talking about adding lifecycle hooks then you know to go talk to them and get the details. (Trio does have some sorta-kinda analogous functionality. Specifically it has a concept of "system tasks" that are automatically cancelled when the main task exits, so they have a chance to do any cleanup at that point. But trio's lifecycle model is so different that I'm not sure how helpful this is.) -n On Tue, Jun 5, 2018, 05:48 Michel Desmoulin <desmoulinmichel@gmail.com> wrote:

After years of playing with asyncio, I'm still having a harder time using it than any other async architecture around. There are a lot of different reasons for it, but this mail want to address one particular one:

The event loop and policy can be tweaked at any time, by anyone.

Now, it's hard enough to have to deal, manually, with a low-level event loop. But having it exposed that much, and it being that flexible means any code can just do whatever it wants with it, and make a mess.

Several things in particular, comes to mind:

- Changing the event loop policy - Changing the event loop - Spawning a new loop - Starting the loop - Stopping the loop - Closing the loop

Now, if you want to make any serious project with it, you currently have to guard against all of those, especially if you want to have proper cleanup code, good error message and a decent debugging experience.

I tried to do it for one year, and currently, it's very hard. You have a lot of checks to make, redundantly in a lot of places. Some things can only be done by providing a custom event policy/loop yourself, and, of course, expecting (aka documenting and praying) that it's used.

For a lot of things, when it breaks, the people that haven't read the doc in depth will have a hard time to understand the problem after the fact.

Sometimes, it's just that your code use somebody else code that is not here to read your doc anymore. Now you have to check their code to understand what they are doing that breaks your expectations about the loop / policy or workflow.

Barring the creating of an entire higher level framework that everybody will agree on using and that makes messing up way harder, we can improve this situation by adding hooks to those events.

I hence propose to add:

- asyncio.on_change_policy(cb:Callable[[EventLoopPolicy, EventLoopPolicy], EventLoopPolicy])

- asyncio.on_set_event_loop(cb:Callable[[EventLoop, EventLoop], EventLoop])

- asyncio.on_create_event_loop(cb:Callable[[EventLoop], EventLoop])

- EventLoop.on_start(cb:Callable[EventLoop])

- EventLoop.on_stop(cb:Awaitable[EventLoop])

- EventLoop.on_close(cb:Callable[EventLoop])

- EventLoop.on_set_debug_mode(cb:Callable[[loop]])

This would allow to implement safer, more robust and easier to debug code. E.G:

- you can raise a warning stating that if somebody changes the event policy, it must inherit from your custom one or deal with disabled features

- you can raise an exception on loop swap and forbid it, saying that your small script doesn't support it yet so that it's easy to understand the limit of your code

- you can hook on the event loop life cycle to automatically get on board, or run clean up code, starting logging, warn that you were supposed to start the loop yourself, etc _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/