[Async-sig] async testing question

[Chris Jerdonek]

On Mon, Jul 3, 2017 at 10:39 AM, Dima Tisnek <dimaqq at gmail.com> wrote:
> I'd say mock 2nd to `await time.sleep(1); assert False, "should not
> happen"` with the earlier just in case test harness or code under test
> is broken.
>
> The tricky part is how to cancel your library function at the right
> time (i.e. not too early).

So I wound up trying a combination of Dima and Nathaniel's suggestion
of mocking / hooking the second function do_more() to cancel the
"parent" task, and then just waiting for the cancellation to occur.
You can see the result of my efforts here (it is to test the fix of a
bug in the websockets library):

https://github.com/aaugustin/websockets/pull/194

The test is definitely more complicated than I'd like. And it has a
couple asyncio.sleep(0.1)'s that would be nice to get rid of to make
the test faster and eliminate flakiness. Dima is right that one tricky
thing is how not to call cancel() too early. (That is the reason for
one of my sleep(0.1)'s.)

I could see tools or patterns being useful here.

--Chris

>
> You could, perhaps, mock 1st call to
> `ensure_future(async_cancel_task())` but imagine that code under test
> gets changed to:
>
> async to_be_tested():
>   await first()
>   logging.debug("...")  # you don't expect event loop interaction
> here, but what if?
>   await second()
>   await third()
>
> If it's all right for your test to fail on such a change, then fine :)
> If you consider that unexpected breakage, then I dunno what you can do :P
>
> On 1 July 2017 at 22:06, Chris Jerdonek <chris.jerdonek at gmail.com> wrote:
>> On Sat, Jul 1, 2017 at 3:49 AM, Dima Tisnek <dimaqq at gmail.com> wrote:
>>> Hi Chris,
>>>
>>> This specific test is easy to write (mock first to return a resolved future,
>>> 2nd to block and 3rd to assert False)
>>
>> Saying it's easy doesn't necessarily help the questioner. :)
>>
>> Issues around combinatorics I understand. It's more the mechanics of
>> the basic testing pattern I'd like advice on.
>>
>> For example, if I mock the second function to be blocking, how do I
>> invoke the higher-level function in a way so I can continue at the
>> point where the second function blocks? And without introducing
>> brittleness or relying on implementation details of the event loop?
>>
>> (By the way, it seems you wouldn't want to mock the third function in
>> cases like if the proper handling of task.cancel() depends on the
>> behavior of the third function, for example if CancelledError is being
>> caught.)
>>
>> --Chris
>>
>>>
>>> OTOH complexity of the general case is unbounded and generally exponential.
>>> It's akin to testing multithreaded code.
>>> (There's an academic publication from Microsoft where they built a runtime
>>> that would run each test really many times, where scheduler is rigged to
>>> order runnable tasks differently on each run. I hope someone rewrites this
>>> for asyncio)
>>>
>>> Certainty [better] tools are needed, and ultimately it's a tradeoff between
>>> sane/understable/maintainable tests and testing deeper/more corner cases.
>>>
>>> Just my 2c...
>>>
>>> On Jul 1, 2017 12:11, "Chris Jerdonek" <chris.jerdonek at gmail.com> wrote:
>>>>
>>>> I have a question about testing async code.
>>>>
>>>> Say I have a coroutine:
>>>>
>>>>     async def do_things():
>>>>         await do_something()
>>>>         await do_more()
>>>>         await do_even_more()
>>>>
>>>> And future:
>>>>
>>>>     task = ensure_future(do_things())
>>>>
>>>> Is there a way to write a test case to check that task.cancel() would
>>>> behave correctly if, say, do_things() is waiting at the line
>>>> do_more()?
>>>>
>>>> In real life, this situation can happen if a function like the
>>>> following is called, and an exception happens in one of the given
>>>> tasks.  One of the tasks in the "pending" list could be at the line
>>>> do_more().
>>>>
>>>>     done, pending = await asyncio.wait(tasks,
>>>>                          return_when=asyncio.FIRST_EXCEPTION)
>>>>
>>>> But in a testing situation, you don't necessarily have control over
>>>> where each task ends up when FIRST_EXCEPTION occurs.
>>>>
>>>> Thanks,
>>>> --Chris
>>>> _______________________________________________
>>>> Async-sig mailing list
>>>> Async-sig at python.org
>>>> https://mail.python.org/mailman/listinfo/async-sig
>>>> Code of Conduct: https://www.python.org/psf/codeofconduct/