[Twisted-Python] "_wait" is blowing up?!
File "/usr/lib/python2.5/site-packages/twisted/trial/unittest.py", line 763, in run self._wait(d) File "/usr/lib/python2.5/site-packages/twisted/trial/unittest.py", line 901, in _wait reactor.run() File "/usr/lib/python2.5/site-packages/twisted/internet/posixbase.py", Hi, We have a pretty big tests suite that uses trial to test all sorts of twisted things. We are pretty careful to do the right thing in our tests and leave the reactor clean....but recently we began seeing a transient really wierd bug when we run our test suite. Some characteristics of the problem: 1. The problem only appears sometimes - seems random 2. The problem shows up in different test methods 3. Sometimes it hangs the tests suite for a long time - eventually twisted continues though Here is the traceback we get. line 220, in run self.mainLoop() File "/usr/lib/python2.5/site-packages/twisted/internet/posixbase.py", line 228, in mainLoop self.runUntilCurrent() --- <exception caught here> --- File "/usr/lib/python2.5/site-packages/twisted/internet/base.py", line 533, in runUntilCurrent f(*a, **kw) File "/usr/lib/python2.5/site-packages/twisted/internet/defer.py", line 261, in errback self._startRunCallbacks(fail) File "/usr/lib/python2.5/site-packages/twisted/internet/defer.py", line 290, in _startRunCallbacks raise AlreadyCalledError twisted.internet.defer.AlreadyCalledError: ERROR It appears that something is going on in the deadly _wait. Any thoughts on how we might fix this? Thanks Brian
Hi Brian, On 06:25 am, ellisonbg.net@gmail.com wrote:
Some characteristics of the problem:
1. The problem only appears sometimes - seems random 2. The problem shows up in different test methods 3. Sometimes it hangs the tests suite for a long time - eventually twisted continues though
Here is the traceback we get. (...) It appears that something is going on in the deadly _wait. Any thoughts on how we might fix this?
I realize that this problem is difficult to track down, but I'm not sure what to tell you without more specific information about how your code is interacting with trial. Is your code available somewhere? Do you reliably see this error eventually if you use trial --until- failure? Exactly what version of Twisted are you using? What reactor do you typically run your tests under? Not sure that I'd be able to tell you what's going on even with an answer to all of these questions, but they'd be a start :).
Is your code available somewhere?
yes, the code is the IPython mainline, on launchpad: $ bzr branch lp:ipython Then, simply $ trial IPython Will show the problem. You will also need foolcap to run the test suite.
Do you reliably see this error eventually if you use trial --until- failure?
It shows up as an Error rather than a Failure. But yes, it still shows up.
Exactly what version of Twisted are you using?
Latest stable.
What reactor do you typically run your tests under?
Default on all platforms.
Not sure that I'd be able to tell you what's going on even with an answer to all of these questions, but they'd be a start :).
Thanks, I know this type of thing is tough to debug. Any clever tricks I could use to track it down? Thanks Brian
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Thu, 28 Aug 2008 11:52:02 -0700, Brian Granger <ellisonbg.net@gmail.com> wrote:
Is your code available somewhere?
yes, the code is the IPython mainline, on launchpad:
$ bzr branch lp:ipython
Then, simply
$ trial IPython
Will show the problem. You will also need foolcap to run the test suite.
Do you reliably see this error eventually if you use trial --until- failure?
It shows up as an Error rather than a Failure. But yes, it still shows up.
Exactly what version of Twisted are you using?
Latest stable.
What reactor do you typically run your tests under?
Default on all platforms.
Not sure that I'd be able to tell you what's going on even with an answer to all of these questions, but they'd be a start :).
Thanks, I know this type of thing is tough to debug. Any clever tricks I could use to track it down?
You can try running with trial --debug (possibly also --nopm). This will cause Deferred debugging to be enabled, which will cause the call site of each Deferred's creation to be saved. Then you can inspect this to learn which Deferred is being called back multiple times. Once you learn this, further debugging approaches may become apparent (like looking at all the call sites which can call back a Deferred created there, and figuring out how more than one of them can be executed). Jean-Paul
By random chance I found the problem in this case. Thanks for the suggestions though! Cheers, Brian On Thu, Aug 28, 2008 at 12:11 PM, Jean-Paul Calderone <exarkun@divmod.com> wrote:
On Thu, 28 Aug 2008 11:52:02 -0700, Brian Granger <ellisonbg.net@gmail.com> wrote:
Is your code available somewhere?
yes, the code is the IPython mainline, on launchpad:
$ bzr branch lp:ipython
Then, simply
$ trial IPython
Will show the problem. You will also need foolcap to run the test suite.
Do you reliably see this error eventually if you use trial --until- failure?
It shows up as an Error rather than a Failure. But yes, it still shows up.
Exactly what version of Twisted are you using?
Latest stable.
What reactor do you typically run your tests under?
Default on all platforms.
Not sure that I'd be able to tell you what's going on even with an answer to all of these questions, but they'd be a start :).
Thanks, I know this type of thing is tough to debug. Any clever tricks I could use to track it down?
You can try running with trial --debug (possibly also --nopm). This will cause Deferred debugging to be enabled, which will cause the call site of each Deferred's creation to be saved. Then you can inspect this to learn which Deferred is being called back multiple times. Once you learn this, further debugging approaches may become apparent (like looking at all the call sites which can call back a Deferred created there, and figuring out how more than one of them can be executed).
Jean-Paul
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (3)
-
Brian Granger -
glyph＠divmod.com -
Jean-Paul Calderone