Mailman 3 June 2013 - pytest-dev

Re: [pytest-dev] [pytest-rerunfailures] pytest-rerunfailures not using fixtures on reruns (#10)
by Leah Klearman July 16, 2013

July 16, 2013

Hi Holger and other py.test mavens, Bob has reported a problem with my py.test plugin pytest-rerunfailures [1] not re-running the setup before rerunning the test. Looking at my code, it has pytest_runtest_protocol() [2] looping on _pytest.runner.runtestprotocol() [3], which in turn runs the setup, the test, and the teardown. [1] https://github.com/klrmn/pytest-rerunfailures [2] https://github.com/klrmn/pytest-rerunfailures/blob/master/rerunfailures/plu… [3] https://bitbucket.org/hpk42/pytest/src/fdc28ac2029f1c0be1dac4991c2f1b014c39… I haven't taken the hours needed to get my head fully into py.test plugin development mode, but I'm not sure I can implement a fix at my layer. I'm hoping someone here will have some insight. Thanks, -Leah On Sun, Apr 14, 2013 at 6:29 PM, Bob Silverberg <notifications(a)github.com>wrote: > I just verified this behaviour myself with a simple test [1]. I see it > with both funcargs and fixtures, but I'm not sure if it has to do with the > plugin, or the way py.test works. It does inject the value into the test > method, but it doesn't rerun the fixture, so it seems like it is caching > the first run of the fixture and using that on subsequent runs. > > I'm not sure if this is something that the plugin can have any effect on, > or if it's just the way fixtures work. It is specified for this fixture > that it is scope='function', and perhaps py.test makes that happen by > checking the function name, which is, of course, the same for each run. I > did try removing the scope argument from the fixture but that had no > effect. > > Do you have any thoughts about this, @klrmn <https://github.com/klrmn>? > > [1] https://gist.github.com/bobsilverberg/5385035 > > — > Reply to this email directly or view it on GitHub<https://github.com/klrmn/pytest-rerunfailures/issues/10#issuecomment-163636…> > . >

4 8

parametrize and ids
by Brianna Laugher June 6, 2013

June 6, 2013

Hi, I was wondering if anyone has ideas about 'nicer' ways to specify IDs for tests when using the parametrize mark. Example: my parametrize might look like this: @py.test.mark.parametrize(('wx', 'expectedCoverage', 'expectedTrend'), [ (['ra2', 'ra2', 'ra4', 'ra4'], 'Isol', None), (['sh2', 'sh2', 'sh4', 'sh4'], 'Isol', 'increasing'), (['ts2', 'ts2', 'ts4', 'ts4'], 'Isol', None), (['ra2', 'ra2', 'ra2', None], 'Isol', None), (['sh2', 'sh2', 'sh2', None], 'Isol', 'clearing'), (['ts2', 'ts2', 'ts2', None], 'Isol', None), ([None, None, 'ra2', 'ra2'], 'Isol', None), ([None, None, 'sh2', 'sh2'], 'Isol', 'developing'), ([None, None, 'ts2', 'ts2'], 'Isol', None), ]) which results in IDs like this: <Function 'test_noTrendsStartingWithChc[.30-Isol-increasing]'> <Function 'test_noTrendsStartingWithChc[.33-Isol-.35]'> <Function 'test_noTrendsStartingWithChc[.36-Isol-.38]'> <Function 'test_noTrendsStartingWithChc[.39-Isol-clearing]'> <Function 'test_noTrendsStartingWithChc[.42-Isol-.44]'> <Function 'test_noTrendsStartingWithChc[.45-Isol-.47]'> <Function 'test_noTrendsStartingWithChc[.48-Isol-developing]'> <Function 'test_noTrendsStartingWithChc[.51-Isol-.53]'> I can add IDs (I just learned last week! :)) like this: @py.test.mark.parametrize(('wx', 'expectedCoverage', 'expectedTrend'), [ (['ra2', 'ra2', 'ra4', 'ra4'], 'Isol', None), (['sh2', 'sh2', 'sh4', 'sh4'], 'Isol', 'increasing'), (['ts2', 'ts2', 'ts4', 'ts4'], 'Isol', None), (['ra2', 'ra2', 'ra2', None], 'Isol', None), (['sh2', 'sh2', 'sh2', None], 'Isol', 'clearing'), (['ts2', 'ts2', 'ts2', None], 'Isol', None), ([None, None, 'ra2', 'ra2'], 'Isol', None), ([None, None, 'sh2', 'sh2'], 'Isol', 'developing'), ([None, None, 'ts2', 'ts2'], 'Isol', None), ], ids=['IsolRA increasing', 'IsolSH increasing', 'IsolTS increasing', 'IsolRA clearing', 'IsolSH clearing', 'IsolTS clearing', 'IsolRA developing', 'IsolSH developing', 'IsolTS developing', ]) but there are a couple of things I don't like about this: * takes a lot of lines * ID is separate from the test data, bad for maintenance, if the test data gets changed the test name could easily be overlooked when it should be updated too What I would really like is if a string in the tuples could represent the name. Maybe there could be a "magic" fieldname (id, testid, _pytestid ?) that could be stripped out from the data and used as the id. e.g. @py.test.mark.parametrize(('id', 'wx', 'expectedCoverage', 'expectedTrend'), [ ('ChcRA increasing', ['ra1', 'ra1', 'ra4', 'ra4'], 'Wide', None), ('ChcSH increasing', ['sh1', 'sh1', 'sh4', 'sh4'], 'Wide', None), ('ChcTS increasing', ['ts1', 'ts1', 'ts4', 'ts4'], 'Wide', None), ]) def test_noTrendsStartingWithChc(wx, expectedCoverage, expectedTrend): # blah This would be really nice. Another alternative could be if you could specify a function that would take the test data and form a string for the test id from it. An example here might be @py.test.mark.parametrize(('wx', 'expectedCoverage', 'expectedTrend'), [ (['ra1', 'ra1', 'ra4', 'ra4'], 'Wide', None), (['sh1', 'sh1', 'sh4', 'sh4'], 'Wide', None), (['ts1', 'ts1', 'ts4', 'ts4'], 'Wide', None), ], idFn=lambda (wx, cov, trend): '{!s}{} {!s}'.format(wx[0], cov, trend)) However this is quite a bit more messy, and not as flexible as specifying your own string. I would be interested to hear what people think about this, as well as if there are other ways of specifying ids that I have overlooked. (I know I could lay out the data the way I want and use pytest_generate_tests to slice it up for metafunc.parametrize, but I actually prefer explicitly writing out each test case as it tends to be a lot simpler for other developers and future-me to grok.) thanks, Brianna -- They've just been waiting in a mountain for the right moment: http://modernthings.org/

2 3

Re: [pytest-dev] how to fail on exceptions in other threads and destructors
by Nikolaus Rath June 5, 2013

June 5, 2013

On 06/01/2013 06:42 PM, Adam Goucher wrote: >> I would like to make sure that a test fails when it results in an >> uncaught exception - even if the exception happens in a destructor or in >> a separate thread. >> >> What's the best way to do this? >> >> My idea is to modify sys.excepthook such that it keeps a list of all the >> exceptions encountered during a test run. Then, at the end of every >> test, I could check this list and raise an exception if there were any >> calls to sys.excepthook. >> >> However, I'm not quite sure how to do this properly without having to >> request an explicit fixture in every test. Is there a way to have >> initialization and finalization code run for every test function? And >> what kind of exception would I need to raise in the finalization code >> for py.test to recognize that it is the test that failed (rather than a >> bug in the finalization code)? > > IIRC, I think what you want to do is look at the pytest_runtest_makereport hook. It gets called at each part of the execution flow. You would then look at call.when to determine when the thing that went wrong, went wrong. > > https://github.com/Element-34/Py.Saunter-Examples/blob/master/picpaste/webd… is kinda an example. Though in my case I ignore the setup and teardown bits. That looks close to what I want to do, thanks for the link! However, I have some trouble understanding what's actually happening here: Am I correct that, despite its name, the pytest_runtest_makereport hook is called not just after the test has been teared down, but also after setup and calling? Or is it called before setup, calling and teardown? Also, what exactly is the __multicall__ argument? According to http://pytest.org/latest/plugins.html#py-test-hook-reference, pytest_runtest_makereport accepts just two (item, call) parameters...? Best, Nikolaus

2 1

pytest-cache 1.0: new --ff option to first run previous failures
by holger krekel June 4, 2013

June 4, 2013

Hi, i just release pytest-cache-1.0 to pypi which allows to re-run failures from previous test runs and provides a mechanism for other plugins to store data across test session (used e.g. by pytest-pep8). See https://pypi.python.org/pypi/pytest-cache for some docs. The main news is the "--ff" option which runs all tests but first the failures from the last run. Thanks to Jack Riches for the patch. Full Changelog below. cheers, holger 1.0 ---------------------------------------------- - merged "--ff" (failedfirst) option to run all tests but run the last-failed ones first. Thanks Jack Riches. - fix issue6 - --clearcache doesn't bail out if .cache doesn't exist, thanks longlho - fix issue4: consider setup time skip of a previous failure as no longer failing - ensure --cache output is sorted

1 0

Trouble accessing captured stderr
by Nikolaus Rath June 4, 2013

June 4, 2013

Hello, I have a problem accessing the captured stderr. I have set up the following test: $ ls mytest/ conftest.py test_one.py $ cat mytest/conftest.py import pytest @pytest.fixture(autouse=True) def add_stderr_check(request, capsys): def check_stderr(): stderr = capsys.readouterr()[1] if 'exception' in stderr.lower(): raise AssertionError('Suspicious output to stderr') request.addfinalizer(check_stderr) $ cat mytest/test_one.py import threading def test_thread(): def fail(): raise RuntimeError('Do not call me!') t = threading.Thread(target=fail) t.start() t.join() def test_del(): d = ImmortalObject() assert d.value == 2 del d class ImmortalObject: def __init__(self): self.value = 2 def __del__(self): raise RuntimeError('You cannot kill me!') As long as I call the tests individually, they behave exactly as expected: > $ py.test mytest/ -v -k test_thread > =========================================== test session starts ============================================ > platform linux -- Python 3.3.2 -- pytest-2.3.5 -- /usr/bin/python3.3 > collected 2 items > > mytest/test_one.py:3: test_thread PASSED > mytest/test_one.py:3: test_thread ERROR > > ================================================== ERRORS ================================================== > _____________________________________ ERROR at teardown of test_thread _____________________________________ > > def check_stderr(): > stderr = capsys.readouterr()[1] > if 'exception' in stderr.lower(): >> raise AssertionError('Suspicious output to stderr') > E AssertionError: Suspicious output to stderr > > mytest/conftest.py:8: AssertionError > --------------------------------------------- Captured stderr ---------------------------------------------- > Exception in thread Thread-1: > Traceback (most recent call last): > File "/usr/lib/python3.3/threading.py", line 637, in _bootstrap_inner > self.run() > File "/usr/lib/python3.3/threading.py", line 594, in run > self._target(*self._args, **self._kwargs) > File "/home/nikratio/in-progress/s3ql/mytest/test_one.py", line 5, in fail > raise RuntimeError('Do not call me!') > RuntimeError: Do not call me! > > ================================== 1 tests deselected by '-ktest_thread' =================================== > ============================= 1 passed, 1 deselected, 1 error in 0.01 seconds ============================== and: > [1] nikratio@vostro:~/in-progress/s3ql$ py.test mytest/ -v -k test_del > =========================================== test session starts ============================================ > platform linux -- Python 3.3.2 -- pytest-2.3.5 -- /usr/bin/python3.3 > collected 2 items > > mytest/test_one.py:10: test_del PASSED > mytest/test_one.py:10: test_del ERROR > > ================================================== ERRORS ================================================== > ______________________________________ ERROR at teardown of test_del _______________________________________ > > def check_stderr(): > stderr = capsys.readouterr()[1] > if 'exception' in stderr.lower(): >> raise AssertionError('Suspicious output to stderr') > E AssertionError: Suspicious output to stderr > > mytest/conftest.py:8: AssertionError > --------------------------------------------- Captured stderr ---------------------------------------------- > Exception RuntimeError: RuntimeError('You cannot kill me!',) in <bound method ImmortalObject.__del__ of <test_one.ImmortalObject object at 0x7f9fb945fb90>> ignored > ==================================== 1 tests deselected by '-ktest_del' ==================================== > ============================= 1 passed, 1 deselected, 1 error in 0.01 seconds ============================== However, if I attempt to run both tests, I get a very long and (to me) confusing error: > $ py.test mytest/ -v > =========================================== test session starts ============================================ > platform linux -- Python 3.3.2 -- pytest-2.3.5 -- /usr/bin/python3.3 > collected 2 items > > mytest/test_one.py:3: test_thread PASSED > mytest/test_one.py:3: test_thread ERROR > mytest/test_one.py:10: test_del FAILED > > ================================================== ERRORS ================================================== > _____________________________________ ERROR at teardown of test_thread _____________________________________ > > def check_stderr(): > stderr = capsys.readouterr()[1] > if 'exception' in stderr.lower(): >> raise AssertionError('Suspicious output to stderr') > E AssertionError: Suspicious output to stderr > > mytest/conftest.py:8: AssertionError > --------------------------------------------- Captured stderr ---------------------------------------------- > Exception in thread Thread-1: > Traceback (most recent call last): > File "/usr/lib/python3.3/threading.py", line 637, in _bootstrap_inner > self.run() > File "/usr/lib/python3.3/threading.py", line 594, in run > self._target(*self._args, **self._kwargs) > File "/home/nikratio/in-progress/s3ql/mytest/test_one.py", line 5, in fail > raise RuntimeError('Do not call me!') > RuntimeError: Do not call me! > > ================================================= FAILURES ================================================= > _________________________________________________ test_del _________________________________________________ > > self = <CallInfo when='call' exception: 'CaptureFixture' object has no attribute 'capture'> > func = <function call_runtest_hook.<locals>.<lambda> at 0x7f413dfc5950>, when = 'call' > > def __init__(self, func, when): > #: context of invocation: one of "setup", "call", > #: "teardown", "memocollect" > self.when = when > self.start = time() > try: > try: >> self.result = func() > > ../../.local/lib/python3.3/site-packages/_pytest/runner.py:129: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >> return CallInfo(lambda: ihook(item=item, **kwds), when=when) > > ../../.local/lib/python3.3/site-packages/_pytest/runner.py:116: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > kwargs = {'item': <Function 'test_del'>} > plugins = [<Session 's3ql'>, <_pytest.core.PluginManager object at 0x7f413eb6dfd0>, <module '_pytest.config' from '/home/nikrati...in.py'>, <module '_pytest.terminal' from '/home/nikratio/.local/lib/python3.3/site-packages/_pytest/terminal.py'>, ...] > > def call_matching_hooks(**kwargs): > plugins = self.config._getmatchingplugins(self.fspath) >> return hookmethod.pcall(plugins, **kwargs) > > ../../.local/lib/python3.3/site-packages/_pytest/main.py:159: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > self = <HookCaller 'pytest_runtest_call'> > plugins = [<Session 's3ql'>, <_pytest.core.PluginManager object at 0x7f413eb6dfd0>, <module '_pytest.config' from '/home/nikrati...in.py'>, <module '_pytest.terminal' from '/home/nikratio/.local/lib/python3.3/site-packages/_pytest/terminal.py'>, ...] > kwargs = {'item': <Function 'test_del'>} > methods = [<function pytest_runtest_call at 0x7f413e9314d0>, <function pdbitem at 0x7f413e9544d0>, <bound method CaptureManager.pytest_runtest_call of <_pytest.capture.CaptureManager object at 0x7f413eb77710>>] > > def pcall(self, plugins, **kwargs): > methods = self.hookrelay._pm.listattr(self.name, plugins=plugins) >> return self._docall(methods, kwargs) > > ../../.local/lib/python3.3/site-packages/_pytest/core.py:445: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > self = <HookCaller 'pytest_runtest_call'> > methods = [<function pytest_runtest_call at 0x7f413e9314d0>, <function pdbitem at 0x7f413e9544d0>, <bound method CaptureManager.pytest_runtest_call of <_pytest.capture.CaptureManager object at 0x7f413eb77710>>] > kwargs = {'item': <Function 'test_del'>} > > def _docall(self, methods, kwargs): > self.trace(self.name, kwargs) > self.trace.root.indent += 1 > mc = MultiCall(methods, kwargs, firstresult=self.firstresult) > try: >> res = mc.execute() > > ../../.local/lib/python3.3/site-packages/_pytest/core.py:452: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > self = <MultiCall 0 results, 2 meths, kwargs={'item': <Function 'test_del'>}> > > def execute(self): > while self.methods: > method = self.methods.pop() > kwargs = self.getkwargs(method) >> res = method(**kwargs) > > ../../.local/lib/python3.3/site-packages/_pytest/core.py:370: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > self = <_pytest.capture.CaptureManager object at 0x7f413eb77710>, item = <Function 'test_del'> > > @pytest.mark.tryfirst > def pytest_runtest_call(self, item): > self.resumecapture_item(item) >> self.activate_funcargs(item) > > ../../.local/lib/python3.3/site-packages/_pytest/capture.py:158: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > self = <_pytest.capture.CaptureManager object at 0x7f413eb77710>, pyfuncitem = <Function 'test_del'> > > def activate_funcargs(self, pyfuncitem): > funcargs = getattr(pyfuncitem, "funcargs", None) > if funcargs is not None: > for name, capfuncarg in funcargs.items(): > if name in ('capsys', 'capfd'): > assert not hasattr(self, '_capturing_funcarg') > self._capturing_funcarg = capfuncarg >> capfuncarg._start() > > ../../.local/lib/python3.3/site-packages/_pytest/capture.py:128: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > self = <_pytest.capture.CaptureFixture object at 0x7f413ea47f10> > > def _start(self): >> self.capture.startall() > E AttributeError: 'CaptureFixture' object has no attribute 'capture' > > ../../.local/lib/python3.3/site-packages/_pytest/capture.py:209: AttributeError > =============================== 1 failed, 1 passed, 1 error in 0.03 seconds ================================ Can someone explain what's happening here? Best, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C

2 1

how to fail on exceptions in other threads and destructors
by Nikolaus Rath June 3, 2013

June 3, 2013

Hello, I would like to make sure that a test fails when it results in an uncaught exception - even if the exception happens in a destructor or in a separate thread. What's the best way to do this? My idea is to modify sys.excepthook such that it keeps a list of all the exceptions encountered during a test run. Then, at the end of every test, I could check this list and raise an exception if there were any calls to sys.excepthook. However, I'm not quite sure how to do this properly without having to request an explicit fixture in every test. Is there a way to have initialization and finalization code run for every test function? And what kind of exception would I need to raise in the finalization code for py.test to recognize that it is the test that failed (rather than a bug in the finalization code)? Thanks, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C

2 2