how to fail on exceptions in other threads and destructors
Hello, I would like to make sure that a test fails when it results in an uncaught exception - even if the exception happens in a destructor or in a separate thread. What's the best way to do this? My idea is to modify sys.excepthook such that it keeps a list of all the exceptions encountered during a test run. Then, at the end of every test, I could check this list and raise an exception if there were any calls to sys.excepthook. However, I'm not quite sure how to do this properly without having to request an explicit fixture in every test. Is there a way to have initialization and finalization code run for every test function? And what kind of exception would I need to raise in the finalization code for py.test to recognize that it is the test that failed (rather than a bug in the finalization code)? Thanks, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C
Hi Nikolaus, On Sat, Jun 01, 2013 at 14:41 -0700, Nikolaus Rath wrote:
Hello,
I would like to make sure that a test fails when it results in an uncaught exception - even if the exception happens in a destructor or in a separate thread.
What's the best way to do this?
My idea is to modify sys.excepthook such that it keeps a list of all the exceptions encountered during a test run. Then, at the end of every test, I could check this list and raise an exception if there were any calls to sys.excepthook.
However, I'm not quite sure how to do this properly without having to request an explicit fixture in every test. Is there a way to have initialization and finalization code run for every test function? And what kind of exception would I need to raise in the finalization code for py.test to recognize that it is the test that failed (rather than a bug in the finalization code)?
You could use an "autouse" fixture which gets called without naming the fixture in test/fixture function arguments. But you'd register a finalizer and this would indeed not fail the actual test but the teardown. To fail the actual test, you might look into implementing the pytest_pyfunc_call() hook. It is responsible for calling a python test function. You would want to wrap the existing hook implementation and add your exception-detection logic like this: # content of conftest.py in your testing dir import pytest @pytest.mark.tryfirst def pytest_pyfunc_call(__multicall__): ret = __multicall__.execute() # if this raises we don't care # perform extra checks and raise if there is a problem return ret # return whatever our wrapped hook impl returns The ``__multicall__`` bit is for interacting with other hook implementations so that we don't have to replicate their logic. You could also use the "pytest_runtest_call" hook if you also have doctests or other non-function types of tests. cheers, holger
Thanks,
-Nikolaus
-- »Time flies like an arrow, fruit flies like a Banana.«
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C _______________________________________________ Pytest-dev mailing list Pytest-dev@python.org http://mail.python.org/mailman/listinfo/pytest-dev
On 06/03/2013 02:42 AM, holger krekel wrote:
Hi Nikolaus,
On Sat, Jun 01, 2013 at 14:41 -0700, Nikolaus Rath wrote:
Hello,
I would like to make sure that a test fails when it results in an uncaught exception - even if the exception happens in a destructor or in a separate thread.
What's the best way to do this?
My idea is to modify sys.excepthook such that it keeps a list of all the exceptions encountered during a test run. Then, at the end of every test, I could check this list and raise an exception if there were any calls to sys.excepthook.
However, I'm not quite sure how to do this properly without having to request an explicit fixture in every test. Is there a way to have initialization and finalization code run for every test function? And what kind of exception would I need to raise in the finalization code for py.test to recognize that it is the test that failed (rather than a bug in the finalization code)?
You could use an "autouse" fixture which gets called without naming the fixture in test/fixture function arguments.
Ah, that's useful. Thanks for the pointer!
To fail the actual test, you might look into implementing the pytest_pyfunc_call() hook. It is responsible for calling a python test function. You would want to wrap the existing hook implementation and add your exception-detection logic like this:
# content of conftest.py in your testing dir
import pytest
@pytest.mark.tryfirst def pytest_pyfunc_call(__multicall__): ret = __multicall__.execute() # if this raises we don't care # perform extra checks and raise if there is a problem return ret # return whatever our wrapped hook impl returns
The ``__multicall__`` bit is for interacting with other hook implementations so that we don't have to replicate their logic. You could also use the "pytest_runtest_call" hook if you also have doctests or other non-function types of tests.
Could you elaborate on the difference between pytest_pyfunc_call and pytest_runtest_call? The former doesn't seem to be mentioned anywhere in the docs. (Some of my tests are unittest.TestCases.) I'm also not sure about the tryfirst decorator. Why is that needed? Do I also need it when using pytest_runtest_call? As a side note, I just tried to set up something based on an autouse fixture and found that sys.excepthook is ignored both for threads and destructors. There is a workaround for threads (http://bugs.python.org/issue1230540), but __del__ seems to be a hopeless case -- I believe it prints directly to stderr from C code. So what I'll try instead is to make my extra tests check if a test resulted in any (or maybe any suspicious) output to stderr. Best, Nikolaus
participants (2)
-
holger krekel -
Nikolaus Rath