Re: [Twisted-Python] Moderately-long running processes timing out in Twisted
We're using good_sql already. We're doing our best to move away from Oracle, honest! ;) Steve ----- Original Message ---- From: "glyph@divmod.com" <glyph@divmod.com> To: Twisted general discussion <twisted-python@twistedmatrix.com> Sent: Friday, September 5, 2008 2:22:08 PM Subject: Re: [Twisted-Python] Moderately-long running processes timing out in Twisted It seems that you have been a victim of at least one recently-fixed bug: http://twistedmatrix.com/trac/ticket/3244 and at least one long-standing one: http://twistedmatrix.com/trac/ticket/1753 As you can see from the tone of my (unfortunate) comment on the latter ticket, I don't think a lot of people use Oracle with Twisted. However, the issue is simple to hack around temporarily: set your ConnectionPool's "good_sql" attribute to 'select 1 from dual' as itamar suggests on #1753. Then, please submit a patch, and ideally a buildslave with a copy of oracle installed so that we can maintain support for it :). On 06:06 pm, spiritualmechanic@yahoo.com wrote:
I looked at the adbapi code and it appears the ConnectionLost exception is coming from there, not from cx_Oracle. At least that's what is in the error log:
2008-09-05 12:25:48-0400 [HTTPChannel,9003,127.0.0.1] adbapi closing: cx_Oracle 2008-09-05 12:25:48-0400 [HTTPChannel,9003,127.0.0.1] Connection lost. 2008-09-05 12:25:48-0400 [-] Unhandled Error Traceback (most recent call last): File "/opt/Python25/lib/python2.5/threading.py", line 440, in run self.__target(*self.__args, **self.__kwargs) File "/opt/python/lib/python2.5/site- packages/twisted/python/threadpool.py", line 161, in _worker context.call(ctx, function, *args, **kwargs) File "/opt/python/lib/python2.5/site- packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/opt/python/lib/python2.5/site- packages/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) --- <exception caught here> --- File "/opt/python/lib/python2.5/site- packages/twisted/internet/threads.py", line 24, in _putResultInDeferred result = f(*args, **kwargs) File "/opt/python/lib/python2.5/site- packages/twisted/enterprise/adbapi.py", line 380, in _runInteraction conn.rollback() File "/opt/python/lib/python2.5/site- packages/twisted/enterprise/adbapi.py", line 65, in rollback raise ConnectionLost() twisted.enterprise.adbapi.ConnectionLost:
I don't know what
--- <exception caught here> ---
means. I'm guessing there's some sort of issue with Oracle but our DBA didn't see anything obvious in his logs. Is there a way for me to tell what exactly the "exception caught here" is?
Steve
----- Original Message ---- From: L. Daniel Burr <ldanielburr@mac.com> To: Twisted general discussion <twisted-python@twistedmatrix.com> Sent: Friday, September 5, 2008 1:15:40 PM Subject: Re: [Twisted-Python] Moderately-long running processes timing out in Twisted
Hi Steve,
On Fri, 05 Sep 2008 11:51:15 -0500, Steve Lewis <spiritualmechanic@yahoo.com> wrote:
We're trying to use twisted as a sort of bus, a centralized place to manage all of our SOA enpoints (mostly through XML-RPC). We have a few services that access the Oracle DB directly (adbapi). In production we're getting intermittent ConnectionLost exceptions from cx_Oracle. Most of these are queries that take a couple of seconds.
Hmm, it is kind of hard to say what the problem could be, without seeing the code that is using adbapi and cx_Oracle.
When you say you are receiving ConnectionLost exceptions from cx_Oracle, are they *really* from cx_Oracle, i.e., "ORA-03135 - Connection lost...", or are they just the result of adbapi calling its connectionLost method?
Has anyone seen this kind of behavior before? What we're probably going to do in the short term is just remove twisted from the equation for these methods that seem to be timing out. But we're relatively new to twisted and I wonder if we're doing something that's obviously wrong.
I've used cx_Oracle with twisted many times, with many different workloads, including stored procedures with OUT parameters, and it has always worked as expected. If you are able to provide some code, this list might be able to provide you with more prescriptive advice.
Hope this helps,
L. Daniel Burr
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 06:31 pm, spiritualmechanic@yahoo.com wrote:
We're using good_sql already. We're doing our best to move away from Oracle, honest! ;) Steve
(lots of stuff snipped) Hmm. I was guessing about the most likely cause since the other (fixed) bug inhibits error reporting. Do you get a more useful traceback out of twisted trunk@HEAD? I wouldn't advocate using that in production, but the error-logging bugfix would tell you at least what's really going wrong.
On Fri, Sep 5, 2008 at 3:33 PM, <glyph@divmod.com> wrote:
On 06:31 pm, spiritualmechanic@yahoo.com wrote:
We're using good_sql already. We're doing our best to move away from Oracle, honest! ;) Steve
(lots of stuff snipped)
Hmm. I was guessing about the most likely cause since the other (fixed) bug inhibits error reporting.
Do you get a more useful traceback out of twisted trunk@HEAD? I wouldn't advocate using that in production, but the error-logging bugfix would tell you at least what's really going wrong.
This could be way off base, but i used to see this on stale connections being utilized again in the next xmlrpc job, basically timing out. (mysqldb- Mysql Server has gone away). Using db.ping() could keep them alive, or testing the connection/recreating before executing a new xmlrpc job. do these exceptions occur when they are running not through the xmlrpc. Jeff
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (3)
-
glyph@divmod.com
-
Jeff Dyke
-
Steve Lewis