Re: [Python-ideas] Add hooks to asyncio lifecycle
Le 09/06/2018 à 12:33, Andrew Svetlov a écrit :
If we consistently apply the idea of hook for internal python structure modification too many things should be changed. Import machinery, tracemalloc, profilers/tracers, name it. If your code (I still don't see the real-life example) wants to check a policy change -- just do it.
# on initialization policy = asyncio.get_event_loop_policy()
# somewhere in code if policy is not asyncio.get_event_loop_policy(): raise RuntimeError("Policy was changed")
You need to repeat this check everywhere because nothing guaranty a code hasn't change it after the check. While a call back allow you to catch it every time. You can set either raise, warn, monkey patch, disable features, etc.
On Sat, 9 Jun 2018 at 14:31, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
Le 09/06/2018 à 12:33, Andrew Svetlov a écrit :
If we consistently apply the idea of hook for internal python structure modification too many things should be changed. Import machinery, tracemalloc, profilers/tracers, name it. If your code (I still don't see the real-life example) wants to check a policy change -- just do it.
# on initialization policy = asyncio.get_event_loop_policy()
# somewhere in code if policy is not asyncio.get_event_loop_policy(): raise RuntimeError("Policy was changed")
You need to repeat this check everywhere because nothing guaranty a code hasn't change it after the check.
While a call back allow you to catch it every time.
You can set either raise, warn, monkey patch, disable features, etc.
IMHO, it is not any framework's job to check for this. It is a programmer error. You don't need to babysit programmers so much. Not if the cost is adding new APIs. I agree with Andrew, if we open this precedent, next thing we know, Python has to provide callbacks for any internal state changes. Why not callbacks for when modules are imported? Etc. etc. It leads to API noise. Doesn't seem to be justified in this case, since I would guess almost all applications only change event loop policy once, during startup, and never again. -- Gustavo J. A. M. Carneiro Gambit Research "The universe is always one step beyond logic." -- Frank Herbert
IMHO, it is not any framework's job to check for this. It is a programmer error.
Not clearing the memory is a programmer error either but a gc helps. Not closing a file either but using `with` help. Not passing the proper type is a programmer error but we have type hints. We even have assert that we can be disabled to check that the arguments of a function match a test when debugging. You don't need to babysit programmers so much. Helping to debug is not babysitting. It's a important part of the user experience, and it's especially needed on asyncio, one of the hardest part of the stdlib to code with. Not if the cost is adding new APIs. The cost of bugs is greater than the cost of API. The current API gives no sane way to prevent the bug. You can't expect anyone to check if the policy / loop / task factory has changed every single time in the code you depend on it.
I agree with Andrew, if we open this precedent, next thing we know, Python has to provide callbacks for any internal state changes.
It's not internal, it's a public API. Why not
callbacks for when modules are imported?
Good example. We have import hooks to run code every time a module is imported : https://www.python.org/dev/peps/pep-0302/#specification-part-2-registering-h... Etc. etc. It leads to API
noise.
Every features is either useful or noise. I argue that this one is useful. Doesn't seem to be justified in this case, since I would guess
almost all applications only change event loop policy once, during startup, and never again.
Yes, but when is start up ? E.G: I'm currently working on a large QT api on an old 2.7 code base. It has threads to avoid blocking the UI, qt events loops of course and the tornado events loop providing a web API. The orchestration of that very complex app requires careful initialization for a start up time of about 7s, and we had to code some wrapper and document it as the only entry point to make sure that if a junior breaks things the error message is very clear. RuntimeError('The event loop should not be set when...') is way easier to debug than an obscure down the road error on some data structure that is incorrectly accessed. What I'm proposing is to make that easy to implement by just letting anyone put a check in there. Overriding policy, loops or tasks factories are usually down for critical parts of the system. The errors emerging from a bug in there are very cryptic. Asyncio design made the choice to expose very low level things. You literally don't have this problem in languages like JS because nobody can change those. Now it's here, it's a footgun, and it would be nice to provide a way to put it in a holster.
On 10 June 2018 at 07:59, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
What I'm proposing is to make that easy to implement by just letting anyone put a check in there.
Overriding policy, loops or tasks factories are usually down for critical parts of the system. The errors emerging from a bug in there are very cryptic.
Asyncio design made the choice to expose very low level things. You literally don't have this problem in languages like JS because nobody can change those.
Now it's here, it's a footgun, and it would be nice to provide a way to put it in a holster.
With the API need framed that way, perhaps all that asyncio is currently missing is an "asyncio.lock_policy(unlock_token, err_callback)" API such that your application can declare that initialisation is completed and no further event loop policy changes should be allowed? (The "unlock_token" would be an arbitrary app-provided object that must also be passed to the corresponding "unlock_policy" call - that way libraries couldn't unlock the policy after the application locks it, since they won't have a reference to the app-specific unlock token). Adding further callback hooks for more events seems like it will just push the problem back another level, and you'll have the potential for conflicts between callbacks registered with the new hooks, and an even harder to understand overall system. By contrast, the above would be amenable to doing something like: 1. Per-process setup code establishes a particular event loop policy, and then locks it 2. Until the policy gets unlocked again, attempts to change it will call the err_callback (so the app can raise a custom access denied exception) 3. get_event_loop(), set_event_loop(), and new_event_loop() are all already managed by the event loop policy, so shouldn't need new hooks 4. stop(), close(), set_debug(), set_task_factory(), etc are all already managed by the event loop (and hence by the event loop policy), so shouldn't need new hooks Right now, the weak link in that chain is that there's no way for the application to keep a library from switching out the event policy with a new one, and subsequently bypassing all of the app's control over how it expects event loops to be managed. Given a way to close that loophole, the application should already have the ability to enforce everything else that it wants to enforce via the existing event loop and event loop policy APIs. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Policy locking is a viable idea at first glance. On Sun, Jun 10, 2018 at 8:05 AM Nick Coghlan <ncoghlan@gmail.com> wrote:
On 10 June 2018 at 07:59, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
What I'm proposing is to make that easy to implement by just letting anyone put a check in there.
Overriding policy, loops or tasks factories are usually down for critical parts of the system. The errors emerging from a bug in there are very cryptic.
Asyncio design made the choice to expose very low level things. You literally don't have this problem in languages like JS because nobody can change those.
Now it's here, it's a footgun, and it would be nice to provide a way to put it in a holster.
With the API need framed that way, perhaps all that asyncio is currently missing is an "asyncio.lock_policy(unlock_token, err_callback)" API such that your application can declare that initialisation is completed and no further event loop policy changes should be allowed?
(The "unlock_token" would be an arbitrary app-provided object that must also be passed to the corresponding "unlock_policy" call - that way libraries couldn't unlock the policy after the application locks it, since they won't have a reference to the app-specific unlock token).
Adding further callback hooks for more events seems like it will just push the problem back another level, and you'll have the potential for conflicts between callbacks registered with the new hooks, and an even harder to understand overall system.
By contrast, the above would be amenable to doing something like:
1. Per-process setup code establishes a particular event loop policy, and then locks it 2. Until the policy gets unlocked again, attempts to change it will call the err_callback (so the app can raise a custom access denied exception) 3. get_event_loop(), set_event_loop(), and new_event_loop() are all already managed by the event loop policy, so shouldn't need new hooks 4. stop(), close(), set_debug(), set_task_factory(), etc are all already managed by the event loop (and hence by the event loop policy), so shouldn't need new hooks
Right now, the weak link in that chain is that there's no way for the application to keep a library from switching out the event policy with a new one, and subsequently bypassing all of the app's control over how it expects event loops to be managed. Given a way to close that loophole, the application should already have the ability to enforce everything else that it wants to enforce via the existing event loop and event loop policy APIs.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- Thanks, Andrew Svetlov
I like it. First, it solves the issue for policies, and let people decide how they want to deal with the problem (drop the lib, subclass the policy/factory, etc). But it also solves the problem for loops, because loops are set by the task factory, and so you can easily check somebody is changing your loop from you locked policy and do whatever you want. This also solves the problem of: - task factories - event loop life cycle hooks Indeed, if somebody needs those, he/she can implement a custom loop, which can be safe guarded by the policy, which is locked. It doesn't have the drawback of my proposal of being overly general, and is quite simple to implement. But it does let people get creative with the stack.
I want to abstract that from the user, so I tried to put that in a policy. But that's dangerous since it can be changed at any time, so I gave up on it and made it explicit. Of course, if the user misses that in the doc (hopefully, it's an company internal code so they should be trained), it will be a bummer to debug.
I still don't get it... If you have a framework you presumably have an entry point. Why can't you set up your policy in that entrypoint? Why would a user attempt to change the policy at runtime (you haven't listed examples of libraries that do this)? I see a lot of "I want to protect users from ..." arguments but I haven't yet seen "this and that happened in production and we haven't been able to debug what happened for a while". Do you handle cases when people install a blocking logging handler in their async application? Do you handle cases when a malfunctioning sys.excepthook is installed? What about cases when users accidentally import gevent somewhere in their asyncio application and it monkeypatches the 'socket' module (this is a real horror story, by the way)? My point is that there are so many things that users can do that will break any framework, be it asyncio or django or trio. This sounds like "if something can happen it will happen" kind of thing, but I haven't yet seen good examples of real code that suffers from non-locked policies. Using the nurseries example doesn't count, as this is something that we want to have as a builtin functionality in 3.8. Locking policies can lead to more predictable user experience; OTOH what happens if, say, aiohttp decides to lock its policy to use uvloop and thus make it impossible for its users to use tokio or some other loop implementation? Yury On Mon, Jun 11, 2018 at 4:23 AM Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
I like it.
First, it solves the issue for policies, and let people decide how they want to deal with the problem (drop the lib, subclass the policy/factory, etc).
But it also solves the problem for loops, because loops are set by the task factory, and so you can easily check somebody is changing your loop from you locked policy and do whatever you want.
This also solves the problem of:
- task factories - event loop life cycle hooks
Indeed, if somebody needs those, he/she can implement a custom loop, which can be safe guarded by the policy, which is locked.
It doesn't have the drawback of my proposal of being overly general, and is quite simple to implement. But it does let people get creative with the stack.
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- Yury
In my mind aiohttp doesn't depend on On Mon, Jun 11, 2018, 19:24 Yury Selivanov <yselivanov.ml@gmail.com> wrote:
I want to abstract that from the user, so I tried to put that in a policy. But that's dangerous since it can be changed at any time, so I gave up on it and made it explicit. Of course, if the user misses that in the doc (hopefully, it's an company internal code so they should be trained), it will be a bummer to debug.
I still don't get it... If you have a framework you presumably have an entry point. Why can't you set up your policy in that entrypoint? Why would a user attempt to change the policy at runtime (you haven't listed examples of libraries that do this)? I see a lot of "I want to protect users from ..." arguments but I haven't yet seen "this and that happened in production and we haven't been able to debug what happened for a while".
Do you handle cases when people install a blocking logging handler in their async application? Do you handle cases when a malfunctioning sys.excepthook is installed? What about cases when users accidentally import gevent somewhere in their asyncio application and it monkeypatches the 'socket' module (this is a real horror story, by the way)? My point is that there are so many things that users can do that will break any framework, be it asyncio or django or trio.
This sounds like "if something can happen it will happen" kind of thing, but I haven't yet seen good examples of real code that suffers from non-locked policies. Using the nurseries example doesn't count, as this is something that we want to have as a builtin functionality in 3.8.
Locking policies can lead to more predictable user experience; OTOH what happens if, say, aiohttp decides to lock its policy to use uvloop and thus make it impossible for its users to use tokio or some other loop implementation?
Yury
On Mon, Jun 11, 2018 at 4:23 AM Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
I like it.
First, it solves the issue for policies, and let people decide how they want to deal with the problem (drop the lib, subclass the policy/factory, etc).
But it also solves the problem for loops, because loops are set by the task factory, and so you can easily check somebody is changing your loop from you locked policy and do whatever you want.
This also solves the problem of:
- task factories - event loop life cycle hooks
Indeed, if somebody needs those, he/she can implement a custom loop, which can be safe guarded by the policy, which is locked.
It doesn't have the drawback of my proposal of being overly general, and is quite simple to implement. But it does let people get creative with the stack.
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- Yury _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- Thanks, Andrew Svetlov
Sorry, smartphone is not my preferred tool. aiohttp doesn't depend on event loop implementation but uses public API only. aiohttp test suite allows to check against asyncio, uvloop, and tokio but it is another story. On Mon, Jun 11, 2018 at 7:35 PM Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
In my mind aiohttp doesn't depend on
On Mon, Jun 11, 2018, 19:24 Yury Selivanov <yselivanov.ml@gmail.com> wrote:
I want to abstract that from the user, so I tried to put that in a policy. But that's dangerous since it can be changed at any time, so I gave up on it and made it explicit. Of course, if the user misses that in the doc (hopefully, it's an company internal code so they should be trained), it will be a bummer to debug.
I still don't get it... If you have a framework you presumably have an entry point. Why can't you set up your policy in that entrypoint? Why would a user attempt to change the policy at runtime (you haven't listed examples of libraries that do this)? I see a lot of "I want to protect users from ..." arguments but I haven't yet seen "this and that happened in production and we haven't been able to debug what happened for a while".
Do you handle cases when people install a blocking logging handler in their async application? Do you handle cases when a malfunctioning sys.excepthook is installed? What about cases when users accidentally import gevent somewhere in their asyncio application and it monkeypatches the 'socket' module (this is a real horror story, by the way)? My point is that there are so many things that users can do that will break any framework, be it asyncio or django or trio.
This sounds like "if something can happen it will happen" kind of thing, but I haven't yet seen good examples of real code that suffers from non-locked policies. Using the nurseries example doesn't count, as this is something that we want to have as a builtin functionality in 3.8.
Locking policies can lead to more predictable user experience; OTOH what happens if, say, aiohttp decides to lock its policy to use uvloop and thus make it impossible for its users to use tokio or some other loop implementation?
Yury
On Mon, Jun 11, 2018 at 4:23 AM Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
I like it.
First, it solves the issue for policies, and let people decide how they want to deal with the problem (drop the lib, subclass the policy/factory, etc).
But it also solves the problem for loops, because loops are set by the task factory, and so you can easily check somebody is changing your loop from you locked policy and do whatever you want.
This also solves the problem of:
- task factories - event loop life cycle hooks
Indeed, if somebody needs those, he/she can implement a custom loop, which can be safe guarded by the policy, which is locked.
It doesn't have the drawback of my proposal of being overly general, and is quite simple to implement. But it does let people get creative with the stack.
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- Yury _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- Thanks, Andrew Svetlov
-- Thanks, Andrew Svetlov
aiohttp doesn't depend on event loop implementation but uses public API only.
Yeah, I understand. I was using it as an example of what happens if a popular library like aiohttp decides to lock the policy for whatever reason. To add to my point: event loop policies should only be used to inject a custom event loop implementation, like uvloop. They shouldn't be used to add some framework- or library-specific functionality. That's why I think that locking policies does not make a lot of sense. Yury On Mon, Jun 11, 2018 at 12:40 PM Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
Sorry, smartphone is not my preferred tool. aiohttp doesn't depend on event loop implementation but uses public API only. aiohttp test suite allows to check against asyncio, uvloop, and tokio but it is another story.
On Mon, Jun 11, 2018 at 7:35 PM Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
In my mind aiohttp doesn't depend on
On Mon, Jun 11, 2018, 19:24 Yury Selivanov <yselivanov.ml@gmail.com> wrote:
I want to abstract that from the user, so I tried to put that in a policy. But that's dangerous since it can be changed at any time, so I gave up on it and made it explicit. Of course, if the user misses that in the doc (hopefully, it's an company internal code so they should be trained), it will be a bummer to debug.
I still don't get it... If you have a framework you presumably have an entry point. Why can't you set up your policy in that entrypoint? Why would a user attempt to change the policy at runtime (you haven't listed examples of libraries that do this)? I see a lot of "I want to protect users from ..." arguments but I haven't yet seen "this and that happened in production and we haven't been able to debug what happened for a while".
Do you handle cases when people install a blocking logging handler in their async application? Do you handle cases when a malfunctioning sys.excepthook is installed? What about cases when users accidentally import gevent somewhere in their asyncio application and it monkeypatches the 'socket' module (this is a real horror story, by the way)? My point is that there are so many things that users can do that will break any framework, be it asyncio or django or trio.
This sounds like "if something can happen it will happen" kind of thing, but I haven't yet seen good examples of real code that suffers from non-locked policies. Using the nurseries example doesn't count, as this is something that we want to have as a builtin functionality in 3.8.
Locking policies can lead to more predictable user experience; OTOH what happens if, say, aiohttp decides to lock its policy to use uvloop and thus make it impossible for its users to use tokio or some other loop implementation?
Yury
On Mon, Jun 11, 2018 at 4:23 AM Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
I like it.
First, it solves the issue for policies, and let people decide how they want to deal with the problem (drop the lib, subclass the policy/factory, etc).
But it also solves the problem for loops, because loops are set by the task factory, and so you can easily check somebody is changing your loop from you locked policy and do whatever you want.
This also solves the problem of:
- task factories - event loop life cycle hooks
Indeed, if somebody needs those, he/she can implement a custom loop, which can be safe guarded by the policy, which is locked.
It doesn't have the drawback of my proposal of being overly general, and is quite simple to implement. But it does let people get creative with the stack.
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- Yury _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- Thanks, Andrew Svetlov
-- Thanks, Andrew Svetlov
-- Yury
Well, if we need something -- locking is better than hooks. But yes, we need a real life example. While the example is absent let's postpone the solution. On Mon, Jun 11, 2018 at 7:50 PM Yury Selivanov <yselivanov.ml@gmail.com> wrote:
aiohttp doesn't depend on event loop implementation but uses public API only.
Yeah, I understand. I was using it as an example of what happens if a popular library like aiohttp decides to lock the policy for whatever reason.
To add to my point: event loop policies should only be used to inject a custom event loop implementation, like uvloop. They shouldn't be used to add some framework- or library-specific functionality. That's why I think that locking policies does not make a lot of sense.
Yury On Mon, Jun 11, 2018 at 12:40 PM Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
Sorry, smartphone is not my preferred tool. aiohttp doesn't depend on event loop implementation but uses public API
aiohttp test suite allows to check against asyncio, uvloop, and tokio but it is another story.
On Mon, Jun 11, 2018 at 7:35 PM Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
In my mind aiohttp doesn't depend on
On Mon, Jun 11, 2018, 19:24 Yury Selivanov <yselivanov.ml@gmail.com>
wrote:
I want to abstract that from the user, so I tried to put that in a policy. But that's dangerous since it can be changed at any time, so I gave up on it and made it explicit. Of course, if the user misses that in the doc (hopefully, it's an company internal code so they should be trained), it will be a bummer to debug.
I still don't get it... If you have a framework you presumably have an entry point. Why can't you set up your policy in that entrypoint? Why would a user attempt to change the policy at runtime (you haven't listed examples of libraries that do this)? I see a lot of "I want to protect users from ..." arguments but I haven't yet seen "this and that happened in production and we haven't been able to debug what happened for a while".
Do you handle cases when people install a blocking logging handler in their async application? Do you handle cases when a malfunctioning sys.excepthook is installed? What about cases when users accidentally import gevent somewhere in their asyncio application and it monkeypatches the 'socket' module (this is a real horror story, by the way)? My point is that there are so many things that users can do that will break any framework, be it asyncio or django or trio.
This sounds like "if something can happen it will happen" kind of thing, but I haven't yet seen good examples of real code that suffers from non-locked policies. Using the nurseries example doesn't count, as this is something that we want to have as a builtin functionality in 3.8.
Locking policies can lead to more predictable user experience; OTOH what happens if, say, aiohttp decides to lock its policy to use uvloop and thus make it impossible for its users to use tokio or some other loop implementation?
Yury
On Mon, Jun 11, 2018 at 4:23 AM Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
I like it.
First, it solves the issue for policies, and let people decide how
want to deal with the problem (drop the lib, subclass the policy/factory, etc).
But it also solves the problem for loops, because loops are set by
only. they the
task factory, and so you can easily check somebody is changing your loop from you locked policy and do whatever you want.
This also solves the problem of:
- task factories - event loop life cycle hooks
Indeed, if somebody needs those, he/she can implement a custom loop, which can be safe guarded by the policy, which is locked.
It doesn't have the drawback of my proposal of being overly general, and is quite simple to implement. But it does let people get creative with the stack.
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- Yury _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- Thanks, Andrew Svetlov
-- Thanks, Andrew Svetlov
-- Yury
-- Thanks, Andrew Svetlov
I still don't get it... If you have a framework you presumably have an entry point. Why can't you set up your policy in that entrypoint? Why would a user attempt to change the policy at runtime (you haven't listed examples of libraries that do this)?
Not at run time. But several libraries have several entry points, that you must use in proper order, and setup correctly. Locking helps with finding when you do an improper setup without reading all documentation for all libs completely to make sure you didn't miss a corner case. It will also help project owners because people will come to the bug tracker saying "I have LockedPolicyError" when I do that, which is a lot easier to solve than "I have an AttributeError on a None object". I see a lot of "I want to
protect users from ..." arguments but I haven't yet seen "this and that happened in production and we haven't been able to debug what happened for a while".
It's like with asyncio.get_event_loop. We haven't seen the problem for a a long time. Then eventually, people starting to use several loops in several threads, and they said that it caused a problem. It's the same here. I'm advising that we don't wait for people to have the problem to solve it.
Do you handle cases when people install a blocking logging handler in their async application?
We can't prevent that without a great cost. It's terribly complicated problem, especially since logging is blocking and thread safe by design. There is not even good documentation on the best practice on using logging with asyncio. Most people have no idea how to do it (although in my experience, most dev use the logging module incorrectly outside of asyncio too). Twisted rewrote an entire logging system to solve that problem. Locking the policy is not a great cost. And it's about creating a user friendly API. Do you handle cases when a malfunctioning
sys.excepthook is installed?
When I override sys.excepthook, I take great care of backuping the old hook, and calling it in my hook. And I really wish the stdlib had something built in to do that cleanly, because I'm pretty sure most lib overriding sys.excepthook all do that in a different way, if they do it at all. Every time I use something to help with stack trace (coloring, auto logging, etc), I have to read the source code to check they are compatible. I should not have to do that. What about cases when users accidentally
import gevent somewhere in their asyncio application and it monkeypatches the 'socket' module (this is a real horror story, by the way)?
Monkey patching is not officially supported by anything, anywhere, so this argument is moot. But it opens currently another door for my argument: currently the only way to make sure nobody erase our policy is to monkey patch the function to set the policy. Do we really want monkey patch to be the only solution to this problem ? My point is that there are so many things that users can do
that will break any framework, be it asyncio or django or trio.
Yes, eventually the question is how easily and cleanly can we provide a solution to avoid the problem or help to debug it. Django is a good example. I run sanity checks on common errors on your model on startup to ease your life as a dev, and the life on the maintainers and their bug tracker.
This sounds like "if something can happen it will happen" kind of thing,
The reason I bring get_event_loop on the table is that I knew years ago when I read that source code that it would come back to bite us. And I said nothing because I assume the core devs would have though of that and that I was mistaken. But even if I have spoken up then, my experience with Python-idea is that most of the time, people would have told me "no". They would have told me that "nobody is going to do that". I'm getting used to it. It took me a lot of time of "path should inherit from strings" to finally have enough people involved that we get a solution to the problem (which ended up being __fspath__). At the beginning, the answer was "no, there is no problem". So basically, now I speak up, knowing that people will say "no". But at least I said it. Maybe in 2 years we will go back to that an implement it, or another solution to this problem. but I haven't yet seen good examples of real code that suffers
from non-locked policies. Using the nurseries example doesn't count, as this is something that we want to have as a builtin functionality in 3.8.
So in we will be able to use that in 2 years.
Locking policies can lead to more predictable user experience; OTOH what happens if, say, aiohttp decides to lock its policy to use uvloop and thus make it impossible for its users to use tokio or some other loop implementation?
That would be very, very good. It would make explicit that aiohttp is: - using a custom policy - assuming it's the only one - not allowing another lib to use one - rely on it to work - has no mechanism to allow to cohabit This would mean user would learn very quickly and easily about those issues, report it to the bug tracker if they matter and allow a debate and a solution. It's not the case for aiohttp though. Another benefit is that this is a universal process. Once we get locking in place, all libs will quickly realize if they do something wrong. While right now, we just can't know. Technically, Go and JS already lock those... by not providing access to any of the machinery. So they don't have the problem. I wish the event loop never had a Python API in the first place, so that the only way to extend it would be in C. But this ship has sailed. Trio nursery is a good example of defensive programming too : can you prove that a lib is calling ensure_future() without proper managing it's lifecycle ? Yes but it requires a lot of work. Can people use ensure_future correctly ? Yes but we know some will mess up. I certainly did. So it's easier to provide a proper way to do things. Locking the policy is providing a clean and easy way to do things.
But even if I have spoken up then, my experience with Python-idea is
On Tue, Jun 12, 2018 at 5:34 AM Michel Desmoulin <desmoulinmichel@gmail.com> wrote: [..] that most of the time, people would have told me "no". They would have told me that "nobody is going to do that".
I'm getting used to it. It took me a lot of time of "path should inherit from strings" to finally have enough people involved that we get a solution to the problem (which ended up being __fspath__). At the beginning, the answer was "no, there is no problem".
So basically, now I speak up, knowing that people will say "no". But at least I said it. Maybe in 2 years we will go back to that an implement it, or another solution to this problem.
[..]
Locking the policy is providing a clean and easy way to do things.
Michel, I know you're an avid asyncio user. I've seen your kind comments about asyncio and uvloop on reddit, twitter, etc. So I really don't want to discourage you from posting here and proposing ideas. But arguments like 'people would have told me "no" ... I'm getting used to it.' aren't helpful. I've repeatedly asked you in this thread to provide a couple of good and easy to follow examples so that we can make an informed decision about whether we should add policy locking mechanism or not. Example: We want to create library "A" because it will work only with event loop "B". Checking the event loop once in A's APIs doesn't work [because ...]. Another example: We use asyncio in production and we see that people change policies at runtime all the time; we need to guard against that somehow. Unless you can clearly demonstrate that locking solves real world problems we are not going to implement it, because get_event_loop() and policies are *already* too complicated. You say that Django performs a bunch of sanity checks, but I don't see a good explanation of why you can't just call `isinstance(asyncio.get_event_loop_policy())` in your framework/application. I also keep repeating that libraries, in general, shouldn't be hardcoded to work with some specific policies, and they should not provide alternative policies unless they provide an alternative event loop implementation. And I don't want to add locking to somehow encourage libraries to use policies. I use a bunch of asyncio libraries in my code and none of them (except aiohttp) uses policies. And aiohttp provides policies only to help you run HTTP servers under gunicorn etc, it's up to the user if they want to use them or not. I'm currently prototyping nurseries/cancel scopes for asyncio and I don't need to use policies to make them work. So I honestly don't see a clear case to add locking so far. Maybe having a PEP would allow us to build a compelling case for adding policies locking. Yury
participants (5)
-
Andrew Svetlov -
Gustavo Carneiro -
Michel Desmoulin -
Nick Coghlan -
Yury Selivanov