
I've noticed a bug which is most probably the fault of zc.recipe.testrunner, and as I'm not sure where to report bugs against this buildout recipe, I'll drop a note here. The problem occurs when checking test coverage using the test runner's --coverage option. I've run bin/test --coverage on a suite of doc tests, and it gave me some 70% of coverage for a particular code file. After adding a test for one of the uncovered pieces of code, I got less than 70% coverage for the same file, everything else being equal. I found that now, one method was reported as uncovered that definitely executed in the test run (as evidenced by putting in a pdb call, for example). I could even reproduce the behaviour. Maybe it helps to know that when I asked about it on #zope3-dev, agroszer noted that the same happened to him, and usually in functional tests. -- Thomas

On Tue, Feb 12, 2008 at 02:28:21PM +0100, Thomas Lotze wrote:
I've noticed a bug which is most probably the fault of zc.recipe.testrunner, and as I'm not sure where to report bugs against this buildout recipe, I'll drop a note here.
The problem occurs when checking test coverage using the test runner's --coverage option. I've run bin/test --coverage on a suite of doc tests, and it gave me some 70% of coverage for a particular code file. After adding a test for one of the uncovered pieces of code, I got less than 70% coverage for the same file, everything else being equal.
It's not a problem with the recipe, it's a problem with zope.testing, or, more fundamentally, with Python. The sys.settrace hook used to implement coverage tracing is non-recursive. zope.testing tests its own ability to do coverage tracing, and since the test is run in the same Python process as the test runner, the side effect of the test's calling sys.settrace is that the runner is no longer able to trace coverage. There are other ways to reset the sys.settrace hook. My favourite is class SomeClass(object): def __iter__(self): ... def __len__(self): return len(list(self)) list(obj) cals obj.__len__ as an optimization. This results in an infinite recursive loop, but then list(obj) traps that, hides it, and does the brute-force list conversion. As a side effect of sys.settrace, the stack overflow RuntimeError happens inside the tracing hook and Python removes it for misbehaving. Tracing stops from that point on. Python's doctest also had side effects that used to disable tracing. Marius Gedminas -- You can't have megalomania. *I* have megalomania. -- Joe Bednorz

Marius Gedminas wrote:
It's not a problem with the recipe, it's a problem with zope.testing, or,
Sorry, that should have been obvious to me. No idea what made me blame it on the recipe...
more fundamentally, with Python. The sys.settrace hook used to implement coverage tracing is non-recursive.
zope.testing tests its own ability to do coverage tracing, and since the test is run in the same Python process as the test runner, the side effect of the test's calling sys.settrace is that the runner is no longer able to trace coverage.
Ah, this makes some sense since the function that was reported uncovered after the change is tested later in the same doc test.
list(obj) cals obj.__len__ as an optimization. This results in an infinite recursive loop, but then list(obj) traps that, hides it, and does the brute-force list conversion. As a side effect of sys.settrace, the stack overflow RuntimeError happens inside the tracing hook and Python removes it for misbehaving. Tracing stops from that point on.
That's what I call gory details...
Python's doctest also had side effects that used to disable tracing.
OK, so it's no longer a subject for this list at least. -- Thomas
participants (2)
-
Marius Gedminas
-
Thomas Lotze