py.test doesn't detect/warn on duplicate tests with parametrize
Hi, In our distributed tests we have been having intermittent failures on one particular parametrized test, with the failure always being on the same test case - failing in pytest_pyfunc_call in the look-up of a funcarg. This funcarg is used in a very large number of our tests (ie otherwise seems reliable) and has function scope. I was just looking into it (traceback below) and I suddenly realised that the case was duplicated. Our test has 135 entries in parametrize so obviously someone (probably me!) just copy-pasted and accidentally left one line the same. It seems to me it might be appropriate for py.test to warn about this, or even give a collection error? What do people think? Secondly does it seem like this might somehow be related to the intermittent test failure? It is baffling me. thanks, Brianna ../pytest-2.3.5-py2.7.egg/_pytest/runner.py:129: in __init__
self.result = func()
../pytest-2.3.5-py2.7.egg/_pytest/runner.py:116: in <lambda>
return CallInfo(lambda: ihook(item=item, **kwds), when=when) ../pytest-2.3.5-py2.7.egg/_pytest/main.py:159: in call_matching_hooks return hookmethod.pcall(plugins, **kwargs) ../pytest-2.3.5-py2.7.egg/_pytest/core.py:445: in pcall return self._docall(methods, kwargs) ../pytest-2.3.5-py2.7.egg/_pytest/core.py:452: in _docall res = mc.execute() ../pytest-2.3.5-py2.7.egg/_pytest/core.py:370: in execute res = method(**kwargs) ../pytest-2.3.5-py2.7.egg/_pytest/runner.py:86: in pytest_runtest_call item.runtest() ../pytest-2.3.5-py2.7.egg/_pytest/python.py:982: in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) ../pytest-2.3.5-py2.7.egg/_pytest/main.py:159: in call_matching_hooks return hookmethod.pcall(plugins, **kwargs) ../pytest-2.3.5-py2.7.egg/_pytest/core.py:445: in pcall return self._docall(methods, kwargs) ../pytest-2.3.5-py2.7.egg/_pytest/core.py:452: in _docall res = mc.execute() ../pytest-2.3.5-py2.7.egg/_pytest/core.py:370: in execute res = method(**kwargs) ../pytest-2.3.5-py2.7.egg/_pytest/python.py:152: in pytest_pyfunc_call testargs[arg] = funcargs[arg] E KeyError: 'dbs'
-- They've just been waiting in a mountain for the right moment: http://modernthings.org/
Hi Brianna, was in vacation and then didn't have an immediate answer to offer ... On Mon, Aug 19, 2013 at 16:51 +1000, Brianna Laugher wrote:
Hi,
In our distributed tests we have been having intermittent failures on one particular parametrized test, with the failure always being on the same test case - failing in pytest_pyfunc_call in the look-up of a funcarg. This funcarg is used in a very large number of our tests (ie otherwise seems reliable) and has function scope.
I was just looking into it (traceback below) and I suddenly realised that the case was duplicated. Our test has 135 entries in parametrize so obviously someone (probably me!) just copy-pasted and accidentally left one line the same.
It seems to me it might be appropriate for py.test to warn about this, or even give a collection error? What do people think?
Maybe a warning would make sense. There are also plans to internally switch to manage parametrization by indexes into the parametrization values (currently the ID is generated from the string representation of values for numbers/strings/floats or some internal numbering for other types). With such a scheme, two identical parametrizations wouldn't be considered wrong per se.
Secondly does it seem like this might somehow be related to the intermittent test failure? It is baffling me.
It's hard to say, i am afraid. I am still unhappy with the finalization mechanics and it's quite possible something is going wrong there related to your situation. best, holger
participants (2)
-
Brianna Laugher -
holger krekel