[Python-ideas] Add hooks to asyncio lifecycle

Michel Desmoulin desmoulinmichel at gmail.com
Wed Jun 6 06:40:25 EDT 2018


Hi Yuri,

> 
> I actually want to propose to reduce policies API surface by
> deprecating and then removing "set_child_watcher()" methods.  Besides,
> there are many other higher priority To-Do items for asyncio in 3.8,
> like implementing Trio's nursery-like objects and cancellation scopes
> or fixing tracebacks in Tasks.
> 
> That said, the above is my "IMO".  And in your email you haven't
> actually provided clear scenarios that could be solved by adding
> "event loop hooks" to asyncio.  So I have a few questions for you:
> 
> - Do you have real-life examples of libraries that abuse policies in
> some weird ways?

It's not abuse, it's just regular use of a documented feature, but a
very potent feature with deep consequences.

> - Are those libraries popular?

aiothttp comes to mind. But the fact is, I think people explicitly avoid
using policies and custom loops because they know that they can be
swapped and you can't make it a hard requirement since they can be
swapped under your node.

> - What's the actual problem they try to solve by using policies?

e.g:
https://github.com/aio-libs/aiohttp/blob/53828229a13dc72137f430abc7a0c469678f3dd6/aiohttp/worker.py

> - What problem are you trying to solve in your code that uses policies?

I have tried to create an abstraction that creates an uvloop if it
exists, or a regular loop otherwise, or integrated it to the twisted
reactor or the trio event loop if it's loaded.

I want to abstract that from the user, so I tried to put that in a
policy. But that's dangerous since it can be changed at any time, so I
gave up on it and made it explicit. Of course, if the user misses that
in the doc (hopefully, it's an company internal code so they should be
trained), it will be a bummer to debug.

Another example is the proof of concept of nurseries we talked about on
twitter:

https://0bin.net/paste/V5KyhAg-2i5EOyoK#dzBvhdCVeFy8Q2xNcxXyqwtyQFgkxlKI3u5QG0buIcT

Yet another one, with a custom loop this time:

I want to provide a fail fast mode for code using loop.run_forever. The
goal is to make it crashes when an uncaught exception occurs instead of
just log it in the console to ease debugging in a local machine.

Of course it would be best to use run_until_complete instead but you
don't always get to choose.

So we set a task factory on the loop. But, of course, you lose
everything if something changes the loop on the fly, which my code has
no idea has happened.

Another use case is to just log that the loop / policy has changed in
debug mode. It's always something I want to know anyway, because it has
consequences on my entire program since those are pretty fundamental
components.

> - Why do you think this isn't a documentation/tutorial issue?> - Can you list 2-3 clear examples where having hooks would benefit an
> average asyncio user?

The best documentation is the one you don't need to write. But if I'm
being honest, I'd like to have it despite having a good warning in the
documentation.

When you run Django's manage.py runserver, it will check your code for a
lot of common issues and raise a warning or an exception, letting you
know what to do.

With those hooks, I could check if a policy or a loop is changed, and if
some things in my code depends on a policy or a loop, I can do the same
and raise a warning or an exception.

This benefit the users because they get the info exactly in context, and
not just rely on them reading the doc, understanding it, remembering it
and applying it.

And that benefits the framework writers because that's less support, and
less bug reports to deal with.

Using asyncio is scary and mysterious enough for a lot of people, so I
want to make the experience as natural as possible. I don't want people
to have to read my doc and learn what event loops and policies are for
basic usage. It's too much. But on the other hand, I do want them to be
able to debug a problem if the loop or policy is swapped.

> 
> Thank you,
> Yury

Thank you too


More information about the Python-ideas mailing list