[pytest-dev] how to fail on exceptions in other threads and destructors

[Nikolaus Rath]

On 06/03/2013 02:42 AM, holger krekel wrote:
> Hi Nikolaus,
> 
> On Sat, Jun 01, 2013 at 14:41 -0700, Nikolaus Rath wrote:
>> Hello,
>>
>> I would like to make sure that a test fails when it results in an
>> uncaught exception - even if the exception happens in a destructor or in
>> a separate thread.
>>
>> What's the best way to do this?
>>
>> My idea is to modify sys.excepthook such that it keeps a list of all the
>> exceptions encountered during a test run. Then, at the end of every
>> test, I could check this list and raise an exception if there were any
>> calls to sys.excepthook.
>>
>> However, I'm not quite sure how to do this properly without having to
>> request an explicit fixture in every test. Is there a way to have
>> initialization and finalization code run for every test function? And
>> what kind of exception would I need to raise in the finalization code
>> for py.test to recognize that it is the test that failed (rather than a
>> bug in the finalization code)?
> 
> You could use an "autouse" fixture which gets called without naming
> the fixture in test/fixture function arguments. 

Ah, that's useful. Thanks for the pointer!

> To fail the actual test, you might look into implementing the
> pytest_pyfunc_call() hook.  It is responsible for calling a python test
> function.  You would want to wrap the existing hook implementation and
> add your exception-detection logic like this:
> 
>     # content of conftest.py in your testing dir
> 
>     import pytest
> 
>     @pytest.mark.tryfirst
>     def pytest_pyfunc_call(__multicall__):
>         ret = __multicall__.execute()  # if this raises we don't care
>         # perform extra checks and raise if there is a problem
>         return ret   # return whatever our wrapped hook impl returns
> 
> The ``__multicall__`` bit is for interacting with other hook
> implementations so that we don't have to replicate their logic.  
> You could also use the "pytest_runtest_call" hook if you 
> also have doctests or other non-function types of tests.


Could you elaborate on the difference between pytest_pyfunc_call and
pytest_runtest_call? The former doesn't seem to be mentioned anywhere in
the docs. (Some of my tests are unittest.TestCases.)

I'm also not sure about the tryfirst decorator. Why is that needed? Do I
also need it when using pytest_runtest_call?


As a side note, I just tried to set up something based on an autouse
fixture and found that sys.excepthook is ignored both for threads and
destructors. There is a workaround for threads
(http://bugs.python.org/issue1230540), but __del__ seems to be a
hopeless case -- I believe it prints directly to stderr from C code. So
what I'll try instead is to make my extra tests check if a test resulted
in any (or maybe any suspicious) output to stderr.



Best,
Nikolaus