async testing question
I have a question about testing async code. Say I have a coroutine: async def do_things(): await do_something() await do_more() await do_even_more() And future: task = ensure_future(do_things()) Is there a way to write a test case to check that task.cancel() would behave correctly if, say, do_things() is waiting at the line do_more()? In real life, this situation can happen if a function like the following is called, and an exception happens in one of the given tasks. One of the tasks in the "pending" list could be at the line do_more(). done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_EXCEPTION) But in a testing situation, you don't necessarily have control over where each task ends up when FIRST_EXCEPTION occurs. Thanks, --Chris
Hi Chris, This specific test is easy to write (mock first to return a resolved future, 2nd to block and 3rd to assert False) OTOH complexity of the general case is unbounded and generally exponential. It's akin to testing multithreaded code. (There's an academic publication from Microsoft where they built a runtime that would run each test really many times, where scheduler is rigged to order runnable tasks differently on each run. I hope someone rewrites this for asyncio) Certainty [better] tools are needed, and ultimately it's a tradeoff between sane/understable/maintainable tests and testing deeper/more corner cases. Just my 2c... On Jul 1, 2017 12:11, "Chris Jerdonek" <chris.jerdonek@gmail.com> wrote:
I have a question about testing async code.
Say I have a coroutine:
async def do_things(): await do_something() await do_more() await do_even_more()
And future:
task = ensure_future(do_things())
Is there a way to write a test case to check that task.cancel() would behave correctly if, say, do_things() is waiting at the line do_more()?
In real life, this situation can happen if a function like the following is called, and an exception happens in one of the given tasks. One of the tasks in the "pending" list could be at the line do_more().
done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_EXCEPTION)
But in a testing situation, you don't necessarily have control over where each task ends up when FIRST_EXCEPTION occurs.
Thanks, --Chris _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
On Jul 1, 2017, at 6:49 AM, Dima Tisnek <dimaqq@gmail.com> wrote:
There's an academic publication from Microsoft where they built a runtime that would run each test really many times, where scheduler is rigged to order runnable tasks differently on each run. I hope someone rewrites this for asyncio
Do you have a link to the publication? Yury
GAMBIT: Effective Unit Testing for Concurrency Libraries https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/gambit-p... There are related publications, but I'm pretty sure that's the right research group. On 1 July 2017 at 13:15, Yury Selivanov <yselivanov@gmail.com> wrote:
On Jul 1, 2017, at 6:49 AM, Dima Tisnek <dimaqq@gmail.com> wrote:
There's an academic publication from Microsoft where they built a runtime that would run each test really many times, where scheduler is rigged to order runnable tasks differently on each run. I hope someone rewrites this for asyncio
Do you have a link to the publication?
Yury
On Sat, Jul 1, 2017 at 3:49 AM, Dima Tisnek <dimaqq@gmail.com> wrote:
Hi Chris,
This specific test is easy to write (mock first to return a resolved future, 2nd to block and 3rd to assert False)
Saying it's easy doesn't necessarily help the questioner. :) Issues around combinatorics I understand. It's more the mechanics of the basic testing pattern I'd like advice on. For example, if I mock the second function to be blocking, how do I invoke the higher-level function in a way so I can continue at the point where the second function blocks? And without introducing brittleness or relying on implementation details of the event loop? (By the way, it seems you wouldn't want to mock the third function in cases like if the proper handling of task.cancel() depends on the behavior of the third function, for example if CancelledError is being caught.) --Chris
OTOH complexity of the general case is unbounded and generally exponential. It's akin to testing multithreaded code. (There's an academic publication from Microsoft where they built a runtime that would run each test really many times, where scheduler is rigged to order runnable tasks differently on each run. I hope someone rewrites this for asyncio)
Certainty [better] tools are needed, and ultimately it's a tradeoff between sane/understable/maintainable tests and testing deeper/more corner cases.
Just my 2c...
On Jul 1, 2017 12:11, "Chris Jerdonek" <chris.jerdonek@gmail.com> wrote:
I have a question about testing async code.
Say I have a coroutine:
async def do_things(): await do_something() await do_more() await do_even_more()
And future:
task = ensure_future(do_things())
Is there a way to write a test case to check that task.cancel() would behave correctly if, say, do_things() is waiting at the line do_more()?
In real life, this situation can happen if a function like the following is called, and an exception happens in one of the given tasks. One of the tasks in the "pending" list could be at the line do_more().
done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_EXCEPTION)
But in a testing situation, you don't necessarily have control over where each task ends up when FIRST_EXCEPTION occurs.
Thanks, --Chris _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
I'd say mock 2nd to `await time.sleep(1); assert False, "should not happen"` with the earlier just in case test harness or code under test is broken. The tricky part is how to cancel your library function at the right time (i.e. not too early). You could, perhaps, mock 1st call to `ensure_future(async_cancel_task())` but imagine that code under test gets changed to: async to_be_tested(): await first() logging.debug("...") # you don't expect event loop interaction here, but what if? await second() await third() If it's all right for your test to fail on such a change, then fine :) If you consider that unexpected breakage, then I dunno what you can do :P On 1 July 2017 at 22:06, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Sat, Jul 1, 2017 at 3:49 AM, Dima Tisnek <dimaqq@gmail.com> wrote:
Hi Chris,
This specific test is easy to write (mock first to return a resolved future, 2nd to block and 3rd to assert False)
Saying it's easy doesn't necessarily help the questioner. :)
Issues around combinatorics I understand. It's more the mechanics of the basic testing pattern I'd like advice on.
For example, if I mock the second function to be blocking, how do I invoke the higher-level function in a way so I can continue at the point where the second function blocks? And without introducing brittleness or relying on implementation details of the event loop?
(By the way, it seems you wouldn't want to mock the third function in cases like if the proper handling of task.cancel() depends on the behavior of the third function, for example if CancelledError is being caught.)
--Chris
OTOH complexity of the general case is unbounded and generally exponential. It's akin to testing multithreaded code. (There's an academic publication from Microsoft where they built a runtime that would run each test really many times, where scheduler is rigged to order runnable tasks differently on each run. I hope someone rewrites this for asyncio)
Certainty [better] tools are needed, and ultimately it's a tradeoff between sane/understable/maintainable tests and testing deeper/more corner cases.
Just my 2c...
On Jul 1, 2017 12:11, "Chris Jerdonek" <chris.jerdonek@gmail.com> wrote:
I have a question about testing async code.
Say I have a coroutine:
async def do_things(): await do_something() await do_more() await do_even_more()
And future:
task = ensure_future(do_things())
Is there a way to write a test case to check that task.cancel() would behave correctly if, say, do_things() is waiting at the line do_more()?
In real life, this situation can happen if a function like the following is called, and an exception happens in one of the given tasks. One of the tasks in the "pending" list could be at the line do_more().
done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_EXCEPTION)
But in a testing situation, you don't necessarily have control over where each task ends up when FIRST_EXCEPTION occurs.
Thanks, --Chris _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
On Mon, Jul 3, 2017 at 10:39 AM, Dima Tisnek <dimaqq@gmail.com> wrote:
I'd say mock 2nd to `await time.sleep(1); assert False, "should not happen"` with the earlier just in case test harness or code under test is broken.
The tricky part is how to cancel your library function at the right time (i.e. not too early).
So I wound up trying a combination of Dima and Nathaniel's suggestion of mocking / hooking the second function do_more() to cancel the "parent" task, and then just waiting for the cancellation to occur. You can see the result of my efforts here (it is to test the fix of a bug in the websockets library): https://github.com/aaugustin/websockets/pull/194 The test is definitely more complicated than I'd like. And it has a couple asyncio.sleep(0.1)'s that would be nice to get rid of to make the test faster and eliminate flakiness. Dima is right that one tricky thing is how not to call cancel() too early. (That is the reason for one of my sleep(0.1)'s.) I could see tools or patterns being useful here. --Chris
You could, perhaps, mock 1st call to `ensure_future(async_cancel_task())` but imagine that code under test gets changed to:
async to_be_tested(): await first() logging.debug("...") # you don't expect event loop interaction here, but what if? await second() await third()
If it's all right for your test to fail on such a change, then fine :) If you consider that unexpected breakage, then I dunno what you can do :P
On 1 July 2017 at 22:06, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Sat, Jul 1, 2017 at 3:49 AM, Dima Tisnek <dimaqq@gmail.com> wrote:
Hi Chris,
This specific test is easy to write (mock first to return a resolved future, 2nd to block and 3rd to assert False)
Saying it's easy doesn't necessarily help the questioner. :)
Issues around combinatorics I understand. It's more the mechanics of the basic testing pattern I'd like advice on.
For example, if I mock the second function to be blocking, how do I invoke the higher-level function in a way so I can continue at the point where the second function blocks? And without introducing brittleness or relying on implementation details of the event loop?
(By the way, it seems you wouldn't want to mock the third function in cases like if the proper handling of task.cancel() depends on the behavior of the third function, for example if CancelledError is being caught.)
--Chris
OTOH complexity of the general case is unbounded and generally exponential. It's akin to testing multithreaded code. (There's an academic publication from Microsoft where they built a runtime that would run each test really many times, where scheduler is rigged to order runnable tasks differently on each run. I hope someone rewrites this for asyncio)
Certainty [better] tools are needed, and ultimately it's a tradeoff between sane/understable/maintainable tests and testing deeper/more corner cases.
Just my 2c...
On Jul 1, 2017 12:11, "Chris Jerdonek" <chris.jerdonek@gmail.com> wrote:
I have a question about testing async code.
Say I have a coroutine:
async def do_things(): await do_something() await do_more() await do_even_more()
And future:
task = ensure_future(do_things())
Is there a way to write a test case to check that task.cancel() would behave correctly if, say, do_things() is waiting at the line do_more()?
In real life, this situation can happen if a function like the following is called, and an exception happens in one of the given tasks. One of the tasks in the "pending" list could be at the line do_more().
done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_EXCEPTION)
But in a testing situation, you don't necessarily have control over where each task ends up when FIRST_EXCEPTION occurs.
Thanks, --Chris _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
[Pulling in comments that were added to a different thread] On Tue, Jul 4, 2017 at 3:03 AM, Dima Tisnek <dimaqq@gmail.com> wrote:
Come to think of it, what sane tests need is a custom event loop or clever mocks around asyncio.sleep, asyncio.Condition.wait, etc. So that code under test never sleeps. ... In any case you should not have to add delays in your mocks or fixtures to hack specific order of task execution by the event loop.
Regarding guaranteeing a certain execution order, and going back to an earlier question of mine, is there a way to introspect a task to find out the name of the function it is currently waiting on? It seems like such a function could go a long way towards guaranteeing a required ordering, and without having to introduce sleeps, etc. Inside a mock, you would be able to wait exactly until needed conditions are satisfied. I was experimenting with task.get_stack() [1], and it seems you can get the line number of where a task is waiting. But using the line number would be more brittle than using the function name. --Chris [1] https://docs.python.org/3/library/asyncio-task.html#asyncio.Task.get_stack
On Jul 1, 2017 3:11 AM, "Chris Jerdonek" <chris.jerdonek@gmail.com> wrote: I have a question about testing async code. Say I have a coroutine: async def do_things(): await do_something() await do_more() await do_even_more() And future: task = ensure_future(do_things()) Is there a way to write a test case to check that task.cancel() would behave correctly if, say, do_things() is waiting at the line do_more()? One possibility for handling this case with a minimum of mocking would be to hook do_more so that it calls task.cancel and then calls the regular do_more. Beyond that it depends on what the actual functions are, I guess. If do_more naturally blocks under some conditions then you might be able to set up those conditions and then call cancel. Or you could try experimenting with tests that call sleep(0) a fixed number of times before issuing the cancel, and repeat with different iteration counts to find different cancel points. (This would benefit from some kind of collaboration with the scheduler, but even a simple hack like this will probably get you more coverage than you had before. It does assume that your test never actually sleeps though.) -n
On Sat, Jul 1, 2017 at 1:42 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Jul 1, 2017 3:11 AM, "Chris Jerdonek" <chris.jerdonek@gmail.com> wrote: Is there a way to write a test case to check that task.cancel() would behave correctly if, say, do_things() is waiting at the line do_more()?
One possibility for handling this case with a minimum of mocking would be to hook do_more so that it calls task.cancel and then calls the regular do_more.
Beyond that it depends on what the actual functions are, I guess. If do_more naturally blocks under some conditions then you might be able to set up those conditions and then call cancel. Or you could try experimenting with tests that call sleep(0) a fixed number of times before issuing the cancel, and repeat with different iteration counts to find different cancel points.
Thanks, Nathaniel. The following would be overkill in my case, but your suggestion makes me wonder if it would make sense for there to be testing tools that have functions to do things like "run the event loop until <this future> is at <this line of code>." Do such things exist? This is a little bit related to what Dima was saying about tools. --Chris
For asyncio, you can write your test functions as coroutines if you use pytest-asyncio. You can even write test fixtures using coroutines. Mocking coroutine functions can be done using asynctest, although I've found that library a bit buggy. Chris Jerdonek kirjoitti 02.07.2017 klo 00:00:
On Sat, Jul 1, 2017 at 1:42 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Jul 1, 2017 3:11 AM, "Chris Jerdonek" <chris.jerdonek@gmail.com> wrote: Is there a way to write a test case to check that task.cancel() would behave correctly if, say, do_things() is waiting at the line do_more()?
One possibility for handling this case with a minimum of mocking would be to hook do_more so that it calls task.cancel and then calls the regular do_more.
Beyond that it depends on what the actual functions are, I guess. If do_more naturally blocks under some conditions then you might be able to set up those conditions and then call cancel. Or you could try experimenting with tests that call sleep(0) a fixed number of times before issuing the cancel, and repeat with different iteration counts to find different cancel points. Thanks, Nathaniel. The following would be overkill in my case, but your suggestion makes me wonder if it would make sense for there to be testing tools that have functions to do things like "run the event loop until <this future> is at <this line of code>." Do such things exist? This is a little bit related to what Dima was saying about tools.
--Chris _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
Hi, asynctest provides a asynctest.TestCase class, inheriting unittest.TestCase. It supports coroutines as test cases and adds a few other useful features like checking that there are no scheduled callbacks left(see http://asynctest.readthedocs.io/en/latest/asynctest.case.html#asynctest.case...) or ClockedTestCase which allows to to control time in the test. Sorry for the bugs left in asynctest, I try to add as many tests as possible to covers all cases, but I'm having a hard time keeping the library compatible with unittest. A few remarks though: * keeping the behavior of asynctest in sync with unittest sometimes leads to unexpected behaviors making asynctest look more buggy than it is (at least I hope so...), * some libraries are hard to mock correctly because they use advanced features, for instance, aiohttp uses its own coroutine type, I still don't know what can be done with those cases, * I'm also thinking about removing support of python 3.4 (@coroutine decorator, etc) as it's a lot of work. Unfortunately, I only have a few hours here and there to work on my free time, as I'm not sponsored by my employer anymore. And, as many other open-source libraries out there, I don't receive a lot of feedback nor help :) Thanks for using (or at least trying to use) asynctest! Martin 2017-07-04 9:38 GMT+02:00 Alex Grönholm <alex.gronholm@nextday.fi>:
For asyncio, you can write your test functions as coroutines if you use pytest-asyncio. You can even write test fixtures using coroutines. Mocking coroutine functions can be done using asynctest, although I've found that library a bit buggy.
Chris Jerdonek kirjoitti 02.07.2017 klo 00:00:
On Sat, Jul 1, 2017 at 1:42 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Jul 1, 2017 3:11 AM, "Chris Jerdonek" <chris.jerdonek@gmail.com> wrote: Is there a way to write a test case to check that task.cancel() would behave correctly if, say, do_things() is waiting at the line do_more()?
One possibility for handling this case with a minimum of mocking would be to hook do_more so that it calls task.cancel and then calls the regular do_more.
Beyond that it depends on what the actual functions are, I guess. If do_more naturally blocks under some conditions then you might be able to set up those conditions and then call cancel. Or you could try experimenting with tests that call sleep(0) a fixed number of times before issuing the cancel, and repeat with different iteration counts to find different cancel points.
Thanks, Nathaniel. The following would be overkill in my case, but your suggestion makes me wonder if it would make sense for there to be testing tools that have functions to do things like "run the event loop until <this future> is at <this line of code>." Do such things exist? This is a little bit related to what Dima was saying about tools.
--Chris _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
_______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
-- Martin <http://www.martiusweb.net> Richard www.martiusweb.net
participants (6)
-
Alex Grönholm -
Chris Jerdonek -
Dima Tisnek -
Martin Richard -
Nathaniel Smith -
Yury Selivanov