Re: [Twisted-Python] Moderately-long running processes timing out in Twisted
I looked at the adbapi code and it appears the ConnectionLost exception is coming from there, not from cx_Oracle. At least that's what is in the error log: 2008-09-05 12:25:48-0400 [HTTPChannel,9003,127.0.0.1] adbapi closing: cx_Oracle 2008-09-05 12:25:48-0400 [HTTPChannel,9003,127.0.0.1] Connection lost. 2008-09-05 12:25:48-0400 [-] Unhandled Error Traceback (most recent call last): File "/opt/Python25/lib/python2.5/threading.py", line 440, in run self.__target(*self.__args, **self.__kwargs) File "/opt/python/lib/python2.5/site-packages/twisted/python/threadpool.py", line 161, in _worker context.call(ctx, function, *args, **kwargs) File "/opt/python/lib/python2.5/site-packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/opt/python/lib/python2.5/site-packages/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) --- <exception caught here> --- File "/opt/python/lib/python2.5/site-packages/twisted/internet/threads.py", line 24, in _putResultInDeferred result = f(*args, **kwargs) File "/opt/python/lib/python2.5/site-packages/twisted/enterprise/adbapi.py", line 380, in _runInteraction conn.rollback() File "/opt/python/lib/python2.5/site-packages/twisted/enterprise/adbapi.py", line 65, in rollback raise ConnectionLost() twisted.enterprise.adbapi.ConnectionLost: I don't know what --- <exception caught here> --- means. I'm guessing there's some sort of issue with Oracle but our DBA didn't see anything obvious in his logs. Is there a way for me to tell what exactly the "exception caught here" is? Steve ----- Original Message ---- From: L. Daniel Burr <ldanielburr@mac.com> To: Twisted general discussion <twisted-python@twistedmatrix.com> Sent: Friday, September 5, 2008 1:15:40 PM Subject: Re: [Twisted-Python] Moderately-long running processes timing out in Twisted Hi Steve, On Fri, 05 Sep 2008 11:51:15 -0500, Steve Lewis <spiritualmechanic@yahoo.com> wrote:
We're trying to use twisted as a sort of bus, a centralized place to manage all of our SOA enpoints (mostly through XML-RPC). We have a few services that access the Oracle DB directly (adbapi). In production we're getting intermittent ConnectionLost exceptions from cx_Oracle. Most of these are queries that take a couple of seconds.
Hmm, it is kind of hard to say what the problem could be, without seeing the code that is using adbapi and cx_Oracle. When you say you are receiving ConnectionLost exceptions from cx_Oracle, are they *really* from cx_Oracle, i.e., "ORA-03135 - Connection lost...", or are they just the result of adbapi calling its connectionLost method?
Has anyone seen this kind of behavior before? What we're probably going to do in the short term is just remove twisted from the equation for these methods that seem to be timing out. But we're relatively new to twisted and I wonder if we're doing something that's obviously wrong.
I've used cx_Oracle with twisted many times, with many different workloads, including stored procedures with OUT parameters, and it has always worked as expected. If you are able to provide some code, this list might be able to provide you with more prescriptive advice. Hope this helps, L. Daniel Burr _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Fri, 5 Sep 2008 11:06:25 -0700 (PDT), Steve Lewis <spiritualmechanic@yahoo.com> wrote:
I looked at the adbapi code and it appears the ConnectionLost exception is coming from there, not from cx_Oracle. At least that's what is in the error log:
2008-09-05 12:25:48-0400 [HTTPChannel,9003,127.0.0.1] adbapi closing: cx_Oracle 2008-09-05 12:25:48-0400 [HTTPChannel,9003,127.0.0.1] Connection lost. 2008-09-05 12:25:48-0400 [-] Unhandled Error Traceback (most recent call last): File "/opt/Python25/lib/python2.5/threading.py", line 440, in run self.__target(*self.__args, **self.__kwargs) File "/opt/python/lib/python2.5/site-packages/twisted/python/threadpool.py", line 161, in _worker context.call(ctx, function, *args, **kwargs) File "/opt/python/lib/python2.5/site-packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/opt/python/lib/python2.5/site-packages/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) --- <exception caught here> --- File "/opt/python/lib/python2.5/site-packages/twisted/internet/threads.py", line 24, in _putResultInDeferred result = f(*args, **kwargs) File "/opt/python/lib/python2.5/site-packages/twisted/enterprise/adbapi.py", line 380, in _runInteraction conn.rollback() File "/opt/python/lib/python2.5/site-packages/twisted/enterprise/adbapi.py", line 65, in rollback raise ConnectionLost() twisted.enterprise.adbapi.ConnectionLost:
I don't know what
--- <exception caught here> ---
means. I'm guessing there's some sort of issue with Oracle but our DBA didn't see anything obvious in his logs. Is there a way for me to tell what exactly the "exception caught here" is?
"exception caught here" is telling you where in the call stack the exception was caught. Frames above it weren't escaped by the exception. Frames below it were. I think you're seeing an example of poor error reporting in adbapi. This has recently been improved and the improvements will be included in the next release of Twisted. If you can somehow test a recent trunk revision of Twisted, you might find that it reports a more specific problem which points to the actual cause of the failure. What this does indicate, I think, is that you had a transaction which could not be committed for some reason. adbapi is rolling back the transaction before reporting the reason the commit failed. However, the ConnectionLost exception, which is boring and not really relevant to anything you're doing, is happening before the real exception can be reported. Jean-Paul
It seems that you have been a victim of at least one recently-fixed bug: http://twistedmatrix.com/trac/ticket/3244 and at least one long-standing one: http://twistedmatrix.com/trac/ticket/1753 As you can see from the tone of my (unfortunate) comment on the latter ticket, I don't think a lot of people use Oracle with Twisted. However, the issue is simple to hack around temporarily: set your ConnectionPool's "good_sql" attribute to 'select 1 from dual' as itamar suggests on #1753. Then, please submit a patch, and ideally a buildslave with a copy of oracle installed so that we can maintain support for it :). On 06:06 pm, spiritualmechanic@yahoo.com wrote:
I looked at the adbapi code and it appears the ConnectionLost exception is coming from there, not from cx_Oracle. At least that's what is in the error log:
2008-09-05 12:25:48-0400 [HTTPChannel,9003,127.0.0.1] adbapi closing: cx_Oracle 2008-09-05 12:25:48-0400 [HTTPChannel,9003,127.0.0.1] Connection lost. 2008-09-05 12:25:48-0400 [-] Unhandled Error Traceback (most recent call last): File "/opt/Python25/lib/python2.5/threading.py", line 440, in run self.__target(*self.__args, **self.__kwargs) File "/opt/python/lib/python2.5/site- packages/twisted/python/threadpool.py", line 161, in _worker context.call(ctx, function, *args, **kwargs) File "/opt/python/lib/python2.5/site- packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/opt/python/lib/python2.5/site- packages/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) --- <exception caught here> --- File "/opt/python/lib/python2.5/site- packages/twisted/internet/threads.py", line 24, in _putResultInDeferred result = f(*args, **kwargs) File "/opt/python/lib/python2.5/site- packages/twisted/enterprise/adbapi.py", line 380, in _runInteraction conn.rollback() File "/opt/python/lib/python2.5/site- packages/twisted/enterprise/adbapi.py", line 65, in rollback raise ConnectionLost() twisted.enterprise.adbapi.ConnectionLost:
I don't know what
--- <exception caught here> ---
means. I'm guessing there's some sort of issue with Oracle but our DBA didn't see anything obvious in his logs. Is there a way for me to tell what exactly the "exception caught here" is?
Steve
----- Original Message ---- From: L. Daniel Burr <ldanielburr@mac.com> To: Twisted general discussion <twisted-python@twistedmatrix.com> Sent: Friday, September 5, 2008 1:15:40 PM Subject: Re: [Twisted-Python] Moderately-long running processes timing out in Twisted
Hi Steve,
On Fri, 05 Sep 2008 11:51:15 -0500, Steve Lewis <spiritualmechanic@yahoo.com> wrote:
We're trying to use twisted as a sort of bus, a centralized place to manage all of our SOA enpoints (mostly through XML-RPC). We have a few services that access the Oracle DB directly (adbapi). In production we're getting intermittent ConnectionLost exceptions from cx_Oracle. Most of these are queries that take a couple of seconds.
Hmm, it is kind of hard to say what the problem could be, without seeing the code that is using adbapi and cx_Oracle.
When you say you are receiving ConnectionLost exceptions from cx_Oracle, are they *really* from cx_Oracle, i.e., "ORA-03135 - Connection lost...", or are they just the result of adbapi calling its connectionLost method?
Has anyone seen this kind of behavior before? What we're probably going to do in the short term is just remove twisted from the equation for these methods that seem to be timing out. But we're relatively new to twisted and I wonder if we're doing something that's obviously wrong.
I've used cx_Oracle with twisted many times, with many different workloads, including stored procedures with OUT parameters, and it has always worked as expected. If you are able to provide some code, this list might be able to provide you with more prescriptive advice.
Hope this helps,
L. Daniel Burr
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (3)
-
glyph＠divmod.com -
Jean-Paul Calderone -
Steve Lewis