[Async-sig] Asyncio loop instrumentation

[Pau Freixes]

Hi Yuri,

Its good to know that we are on the same page regarding the lack of a
feature that should be a must. Since Asyncio has become stable and
widely used by many organizations - such as us [1], the needs of tools
that allow us to instrumentalize asynchronous code that runs on top of
Asyncio have increased.

A good example is how some changes in Aiohttp were implemented [2] -
disclaimer, I'm the author of this code part - to allow the developers
to gather more information about how the HTTP calls perform at both
layers, application, and protocol.

This proposal, just a POC, goes in the same direction and tries to
mitigate this lack for the event loop. The related work regarding the
`load` method is conjunctural but helps to understand why this feature
is such important.

I still believe that we can start to fill the gap for Python 3.7, if
finally the window time to implement it gets closed before all work is
done at least we will have some work done.

I still have some questions to be answered that might help to focus
this work in the right way. Few of them as a proof of the rationale.
Perhaps, how much coupled has to be this feature to the AbstractLoop
making it a specification for other loop implementations. And others
purely technical. But, it's true that we must go further with this
questions if we believe that we can take advantage of all of this
effort.

Regards,

[1] https://medium.com/@SkyscannerEng/running-aiohttp-at-scale-2656b7a83a09
[2] https://github.com/aio-libs/aiohttp/pull/2429

On Sun, Dec 31, 2017 at 8:12 PM, Yury Selivanov <yselivanov at gmail.com> wrote:
> When PEP 567 is accepted, I plan to implement advanced instrumentation in uvloop, to monitor basically all io/callback/loop events. I'm still -1 to do this in asyncio at least in 3.7, because i'd like us to have some time to experiment with such instrumentation in real production code (preferably at scale)
>
> Yury
>
> Sent from my iPhone
>
>> On Dec 31, 2017, at 10:02 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>
>> On Sun, 31 Dec 2017 18:32:21 +0100
>> Pau Freixes <pfreixes at gmail.com> wrote:
>>>
>>> These new implementation of the load method - remember that it returns
>>> a load factor between 0.0 and 1.0 that inform you about how bussy is
>>> your loop -
>>
>> What does it mean exactly? Is it the ratio of CPU time over wall clock
>> time?
>>
>> Depending on your needs, the `psutil` library (*) and/or the new
>> `time.thread_time` function (**) may also help.
>>
>> (*) https://psutil.readthedocs.io/en/latest/
>> (**) https://docs.python.org/3.7/library/time.html#time.thread_time
>>
>>> For this proposal [4], POC, I've preferred make a reduced list of events:
>>>
>>> * `loop_start` : Executed when the loop starts for the first time.
>>> * `tick_start` : Executed when a new loop tick is started.
>>> * `io_start` : Executed when a new IO process starts.
>>> * `io_end` : Executed when the IO process ends.
>>> * `tick_end` : Executed when the loop tick ends.
>>> * `loop_stop` : Executed when the loop stops.
>>
>> What do you call a "IO process" in this context?
>>
>> Regards
>>
>> Antoine.
>>
>>
>> _______________________________________________
>> Async-sig mailing list
>> Async-sig at python.org
>> https://mail.python.org/mailman/listinfo/async-sig
>> Code of Conduct: https://www.python.org/psf/codeofconduct/
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/



-- 
--pau