Mailman 3 Asyncio loop instrumentation - Async-sig

31 Dec 2017

      Hi, foks

First of all, I hope that you have had a good 2017 and I wish for your
the best for 2018.

This email is the continuation of a plan B of the first proposal [1]
to articulate a way to measure
the load of the Asyncio loop. The main objections with the first
implementation were focused on
the technical debt that the implementation imposed, taking into
account that the feature was
definitely out of the main scope of the Asyncio loop goal.

Nathaniel proposed a plan B based on implement some kind of
instrumentalization that will allow
developers to implement features such as the load one. I put off the
plan for a while having, wrongly, feeling that an implementation of
the loop wired with the proper events will impact with the loop
performance. Far away from the reality, the suggested implementation
in terms of performance penalty is almost negligible, at least for
what I considered the happy path which means that there are no
instruments listening for these events.

These new implementation of the load method - remember that it returns
a load factor between 0.0 and 1.0 that inform you about how bussy is
your loop - based on an instrument can be checked with the following
snippet:

async def coro(loop, idx):
        await asyncio.sleep(idx % 10)
        if load() > 0.9:
                return False
        start = loop.time()
        while loop.time() - start < 0.02:
                pass
        return True

async def run(loop, n):
        tasks = [coro(loop, i) for i in range(n)]
        results = await asyncio.gather(*tasks)
        abandoned = len([r for r in results if not r])
        print("Load reached for {} coros/seq: {}, abandoned
{}/{}".format(n/10, load(), abandoned))

async def main(loop):
        await run(loop, 100)

loop = asyncio.get_event_loop()
loop.add_instrument(LoadInstrument)
loop.run_until_complete(main(loop))

The `LoadInstrument` [2] meets the contract of the LoopInstrument[3]
that allow it to listen the proper
loop signals that will be used to calculate the load of the loop.

For this proposal [4], POC, I've preferred make a reduced list of events:

* `loop_start` : Executed when the loop starts for the first time.
* `tick_start` : Executed when a new loop tick is started.
* `io_start` : Executed when a new IO process starts.
* `io_end` : Executed when the IO process ends.
* `tick_end` : Executed when the loop tick ends.
* `loop_stop` : Executed when the loop stops.

The idea of giving just this short list of events try to avoid over
complicate third loops implementations, implementing the minimum set
of events that a typical reactor has to implement.

I would like to gather your feedback for this new approximation, and
if you believe that it might be interesting which are the next steps
that must be done.

Cheers,

[1] https://mail.python.org/pipermail/async-sig/2017-August/000382.html
[2] https://github.com/pfreixes/asyncio_load_instrument/blob/master/asyncio_load...
[3] https://github.com/pfreixes/cpython/blob/asyncio_loop_instrumentation/Lib/as...
[4] https://github.com/pfreixes/cpython/commit/adc3ba46979394997c40aa89178b47244...

-- 
--pau

Asyncio loop instrumentation

Pau Freixes

Antoine Pitrou

Yury Selivanov

Pau Freixes

Pau Freixes

INADA Naoki

Pau Freixes

Yury Selivanov

tags

participants (4)