[Twisted-Python] Spurious failure on winxp32-py2.7
While running tests I see that the following tests fails randomly... it it was failing from some time twisted.words.test.test_xpath.XPathTests.test_position =============================================================================== [FAIL] Traceback (most recent call last): File "C:\buildslave\twisted\winxp32-py2.7\Twisted\twisted\trial\_dist\test\test_disttrial.py", line 371, in test_runUntilFailure self.assertEqual(5, len(called)) File "C:\buildslave\twisted\winxp32-py2.7\Twisted\twisted\trial\_synctest.py", line 447, in assertEqual % (msg, pformat(first), pformat(second))) twisted.trial.unittest.FailTest: not equal: a = 5 b = 2 ---------- On my local computer (Ubuntu 14.04) the test is always skipped. Does anyone has any idea why the test is not skipped ? I would like to have stable results from this tests as with the current state I always have to double check if a failure on XP is a new one or is just this bad tests. Thanks! -- Adi Roiban
On 02:19 pm, adi@roiban.ro wrote:
While running tests I see that the following tests fails randomly... it it was failing from some time
twisted.words.test.test_xpath.XPathTests.test_position
This is an unrelated test. I'm not sure if this is a simple copy/paste mistake or if you're mis-reading trial output. This is the test before the test that failed. The test that failed is named at the bottom of the failure report: twisted.trial._dist.test.test_disttrial.DistTrialRunnerTestCase.test_runUntilFailure This test has intermittently failed on Windows since disttrial was merged into trunk. The ticket tracking this problem is https://twistedmatrix.com/trac/ticket/6027
=============================================================================== [FAIL] Traceback (most recent call last): File "C:\buildslave\twisted\winxp32-py2.7\Twisted\twisted\trial\_dist\test\test_disttrial.py", line 371, in test_runUntilFailure self.assertEqual(5, len(called)) File "C:\buildslave\twisted\winxp32-py2.7\Twisted\twisted\trial\_synctest.py", line 447, in assertEqual % (msg, pformat(first), pformat(second))) twisted.trial.unittest.FailTest: not equal: a = 5 b = 2
----------
On my local computer (Ubuntu 14.04) the test is always skipped.
The skip report should tell you why it's skipped. Presumably you're missing one of the dependencies - though off the top of my head I'm not sure what third-party dependencies this code has.
Does anyone has any idea why the test is not skipped ?
The default state of a test should *not* be to skip. It should be to run. A skipped test is a useless test. Jean-Paul
I would like to have stable results from this tests as with the current state I always have to double check if a failure on XP is a new one or is just this bad tests.
Thanks! -- Adi Roiban
On Jan 25, 2015, at 7:28 AM, exarkun@twistedmatrix.com wrote:
This test has intermittently failed on Windows since disttrial was merged into trunk.
And, as you yourself pointed out on the ticket, sometimes on Linux too. I still have absolutely no clue why this happens, by the way. If anyone else could be prompted to investigate I would be much obliged
The default state of a test should *not* be to skip. It should be to run. A skipped test is a useless test.
As a matter of general principle, the only reason a test should ever be skipped within Twisted is if an optional dependency is not available. And, just to be clear, the presence of "TODO'd" tests is pretty much just a bug, fix old ones and never add new ones :-). -glyph
On 26 January 2015 at 04:00, Glyph <glyph@twistedmatrix.com> wrote:
On Jan 25, 2015, at 7:28 AM, exarkun@twistedmatrix.com wrote:
This test has intermittently failed on Windows since disttrial was merged into trunk.
And, as you yourself pointed out on the ticket, sometimes on Linux too.
I still have absolutely no clue why this happens, by the way. If anyone else could be prompted to investigate I would be much obliged
I have updated my findings here: https://twistedmatrix.com/trac/ticket/6027
The default state of a test should *not* be to skip. It should be to run. A skipped test is a useless test.
As a matter of general principle, the only reason a test should ever be skipped within Twisted is if an optional dependency is not available. And, just to be clear, the presence of "TODO'd" tests is pretty much just a bug, fix old ones and never add new ones :-).
I think that these todo are just another way of creating a ticket ... with a promise to annoy you forever Would it help if I extract them into proper Trac tickes and clean the code? -------- Maybe update docs and stress that todo tests should not be committed ... or never mention about such thing as TODO markers, I just write tests and expect them to fail without marking them as todo. http://twistedmatrix.com/documents/current/core/development/policy/test-stan... (not sure why some docs are on wiki and others in the repo) And maybe also add a twistedchecker check and check that no new tests are commited with todo. -- Adi Roiban
On Jan 26, 2015, at 7:56 AM, Adi Roiban <adi@roiban.ro> wrote:
On 26 January 2015 at 04:00, Glyph <glyph@twistedmatrix.com> wrote:
On Jan 25, 2015, at 7:28 AM, exarkun@twistedmatrix.com wrote:
This test has intermittently failed on Windows since disttrial was merged into trunk.
And, as you yourself pointed out on the ticket, sometimes on Linux too.
I still have absolutely no clue why this happens, by the way. If anyone else could be prompted to investigate I would be much obliged
I have updated my findings here: https://twistedmatrix.com/trac/ticket/6027 <https://twistedmatrix.com/trac/ticket/6027>
Thanks for looking into it.
The default state of a test should *not* be to skip. It should be to run. A skipped test is a useless test.
As a matter of general principle, the only reason a test should ever be skipped within Twisted is if an optional dependency is not available. And, just to be clear, the presence of "TODO'd" tests is pretty much just a bug, fix old ones and never add new ones :-).
I think that these todo are just another way of creating a ticket ... with a promise to annoy you forever
And to annoy every contributor ever who works on something unrelated :).
Would it help if I extract them into proper Trac tickes and clean the code?
Some of them are already filed, but... yes, having a ticket for every one would be helpful. In fact, we might want to have two tickets for each one: one "delete this todo'd test" ticket, and one "actually fix this issue" ticket, since we don't need to keep hanging on to the tests in the repo themselves while we wait for someone to take care of the issues.
--------
Maybe update docs and stress that todo tests should not be committed ... or never mention about such thing as TODO markers, I just write tests and expect them to fail without marking them as todo.
http://twistedmatrix.com/documents/current/core/development/policy/test-stan...
(not sure why some docs are on wiki and others in the repo)
There's no really good reason, that's just where things are. Probably everything should move into the repo.
And maybe also add a twistedchecker check and check that no new tests are commited with todo.
If you wanted to add that it would definitely be helpful. -glyph
participants (4)
-
Adi Roiban -
exarkun＠twistedmatrix.com -
Glyph -
Glyph Lefkowitz