Mailman 3 October 2010 - Twisted

[Twisted-Python] Twisted protocol instances : a puzzle
by Marcus.CM Oct. 18, 2010

Oct. 18, 2010

Hi, This might seems like a trivial question or at least i would have thought so but it has eluded me for some time on how to do this elegantly. In twisted mail, since the only part where we instantiate an instance of account is under : def requestAvatar(self, avatarId, mind, *interfaces): account = xxinterface() # instantiate our account How do i find out if a particular connection is already dead in the given "account" class? For smtp for example, "mind" is None and avatarId is just the id of the login user. Even if this is pop3 or imap, the same problem arise how do i know if the connection of the particular "Server" instance is dead. Thanks for looking. Marcus.

3 5

Re: [Twisted-Python] connectionLost never reached after calling loseConnection: stuck in CLOSE_WAIT forever
by Stefano Debenedetti Oct. 17, 2010

Oct. 17, 2010

Hello Glyph, thanks for your reply and suggestions. I don't have a self-contained sample yet but at least I managed to reproduce it reliably on my installation and after a few more experiments I think I am narrowing this down, please read below. > On Oct 16, 2010, at 11:22 AM, Stefano Debenedetti wrote: > >> Does this sound familiar in any way? Any suggestions off top of head >> while I try to come up with a self-contained sample which is >> reliably reproducing the issue I'm seeing happening only "sometimes >> and quite seldom[TM]"? > > I can't recall having seen this exact issue in the past, but as you've described it it sounds like you may have discovered a Twisted bug. I'm looking forward to your example. > > I do have a few questions: > > * What version of Twisted are you using? > * Have you tried a more recent version? Trunk? I'm using 10.1.0. I haven't tested on trunk because I see basically no difference in internet/abstract.py and internet/tcp.py but if you really think I should I will give trunk a try. > * What reactor are you using? > * Have you tried a different reactor? Same behavior with select, poll and epoll. > * What platform/OS are you on? What version? > * Have you tried a different platform? I am using debian lenny with a self-compiled 2.6.35.2 kernel. /etc/debian_version says: 5.0.5 > I am also curious whether changing > > proto.transport.loseConnection() > > to > reactor.callLater(0, proto.transport.loseConnection) > > makes any difference to your example. I tried this and it didn't make any difference. Using a 1 second delay didn't improve things either. What did make a difference was to comment this line, the problem never happens without it: to.transport.registerProducer(_from.transport, True) Next test I did was to try registering the producer as non-streaming: to.transport.registerProducer(_from.transport, False) This also fixes the problem but it causes an exception to be printed in the log once per set of A, B and C connections: Traceback (most recent call last): File "/home/lala/lib/python/twisted/python/log.py", line 84, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/home/lala/lib/python/twisted/python/log.py", line 69, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/home/lala/lib/python/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/home/lala/lib/python/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) --- <exception caught here> --- File "/home/lala/lib/python/twisted/internet/pollreactor.py", line 184, in _doReadOrWrite why = selectable.doWrite() File "/home/lala/lib/python/twisted/internet/tcp.py", line 428, in doWrite result = abstract.FileDescriptor.doWrite(self) File "/home/lala/lib/python/twisted/internet/abstract.py", line 145, in doWrite self.producer.resumeProducing() File "/home/lala/lib/python/twisted/internet/abstract.py", line 339, in resumeProducing assert self.connected and not self.disconnecting exceptions.AssertionError: This lead me to change the following lines in the doWrite code in internet/abstract.py: if self.disconnecting: # But if I was previously asked to let the connection die, do # so. return self._postLoseConnection() elif self.producer is not None and ((not self.streamingProducer) or self.producerPaused): # tell them to supply some more. self.producerPaused = 0 self.producer.resumeProducing() #elif self.disconnecting: # # But if I was previously asked to let the connection die, do # # so. # return self._postLoseConnection() Basically this just inverts the order of checks: first check if disconnecting, then check if a producer should be unpaused. This makes the above traceback disappear and still fixes my CLOSE_WAIT problem. But using a non-streaming producer makes my app consume a lot more memory so I reverted back my code to register the producer as streaming: to.transport.registerProducer(_from.transport, True) Now the CLOSE_WAIT issue is gone, no traceback appears in the log and my app consumes the same memory as before. Victory? I will still try to come up with a self-contained sample which reproduces the CLOSE_WAIT problem but in the meanwhile I would like to ask if the above-mentioned change to the doWrite definition in internet/abstract.py is likely to destroy the universe in the near future or if it actually sounds like a good idea. > Thanks, and good luck, > > -glyph Thanks a lot for your help! ciao ste

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Oct. 17, 2010

Oct. 17, 2010

1 0

[Twisted-Python] connectionLost never reached after calling loseConnection: stuck in CLOSE_WAIT forever
by Stefano Debenedetti Oct. 17, 2010

Oct. 17, 2010

Hi, I have a similar problem as described in this old mail [1] except that I am not using threads at all. Basically my app is a server that accepts connections from a client (connection A), when A is established it opens connections B and C to another server and forwards data from A to B and from C back to A. Here is how data is forwarded: def forward(_from, to): _from.dataReceived = to.transport.write to.transport.registerProducer(_from.transport, True) def loseConnection(proto, onlost=lambda *args: None): proto.connectionLost = onlost proto.transport.unregisterProducer() proto.transport.loseConnection() as soon as connectionMade has been called on connection A, B and C: forward(A_protocol, B_protocol) forward(C_protocol, A_protocol) as soon as it decides its job is done and wants to tear down all connections: loseConnection(B_protocol) loseConnection(C_protocol, lambda *a: loseConnection(A_protocol)) >From the debug logs in my code I am sure that the loseConnection function is always called with the A_protocol as argument, yet sometimes and totally unpredictably as far as I can tell connectionLost is not called on some A_protocol instances and corresponding A connections get stuck in CLOSE_WAIT state. Another hint is that netstat shows 1 byte in the Recv-Q for those connections (and 0 in the Send-Q). >From another mail [2] in that old thread I got this snippet """ Your issue with CLOSE_WAIT sockets could be due to registering producers which have no further data to produce. """ This sounds like it could be my case, even if as you can see I am always unregistering producers from transports before I call loseConnection on them. Does this sound familiar in any way? Any suggestions off top of head while I try to come up with a self-contained sample which is reliably reproducing the issue I'm seeing happening only "sometimes and quite seldom[TM]"? Thanks, ciao ste [1] http://twistedmatrix.com/pipermail/twisted-python/2008-June/017853.html http://twistedmatrix.com/pipermail/twisted-python/2008-June/017855.html

2 1

[Twisted-Python] Twisted Reverse Proxy
by disappearedng Oct. 17, 2010

Oct. 17, 2010

Hello everyone, I am deploying twisted as a web server for my site. I am looking into possibilities of reverse proxying. I have the following code right now hooked up to my reactor for django. I am using comet, and I realize that I absolutely must use port 80 hence I am looking into possibilities of reverse proxying. On this site, I found the following example: # Django setup sys.path.append("shoout_web") os.environ['DJANGO_SETTINGS_MODULE'] = 'shoout_web.settings' def wrapper_WSGIRootWrapper(): # Build the wrapper first generic = WSGIHandler() def HandlerWrapper(environ, start_response): environ['engine'] = engine return generic(environ, start_response) # Thread and Allowing Ctrl-C to get you out cleanly: pool = threadpool.ThreadPool() pool.start() reactor.addSystemEventTrigger('after', 'shutdown', pool.stop) return wsgi.WSGIResource(reactor, pool, HandlerWrapper) WSGIRoot = wrapper_WSGIRootWrapper() # Reverse Proxy class Simple(Resource): isLeaf = False def getChild(self, name, request): if name == "orbited": print "orbited" return proxy.ReverseProxyResource('localhost', 12345, "/"+name) else: return WSGIRoot.getChildWithDefault(name, request) # Attaching proxy + django log_dir = './.log' if not os.path.exists(log_dir): os.makedirs(log_dir) reactor.listenTCP(DJANGO_PORT, server.Site(Simple(), logPath=os.path.join(log_dir, '.django.log'))) My trouble is I don't really know what to fill in in the else part of that second code part. I looked at text_proxy on twisted-src and there weren't substantial examples for this. Any help?

2 1

[Twisted-Python] connectionLost never reached after calling loseConnection: stuck in CLOSE_WAIT forever
by Stefano Debenedetti Oct. 16, 2010

Oct. 16, 2010

Apologies for the double posting, I am resending this because the original message appears to have triggered some mbox format handling bug somewhere and it appears truncated at the first content line starting with "From" here: http://twistedmatrix.com/pipermail/twisted-python/2010-October/023029.html sorry about this ste -------- Messaggio originale -------- Oggetto: connectionLost never reached after calling loseConnection: stuck in CLOSE_WAIT forever Data: Sat, 16 Oct 2010 17:22:56 +0200 Mittente: Stefano Debenedetti <ste(a)demaledetti.net> A: twisted-python(a)twistedmatrix.com Hi, I have a similar problem as described in this old mail [1] except that I am not using threads at all. Basically my app is a server that accepts connections from a client (connection A), when A is established it opens connections B and C to another server and forwards data from A to B and from C back to A. Here is how data is forwarded: def forward(_from, to): _from.dataReceived = to.transport.write to.transport.registerProducer(_from.transport, True) def loseConnection(proto, onlost=lambda *args: None): proto.connectionLost = onlost proto.transport.unregisterProducer() proto.transport.loseConnection() as soon as connectionMade has been called on connection A, B and C: forward(A_protocol, B_protocol) forward(C_protocol, A_protocol) as soon as it decides its job is done and wants to tear down all connections: loseConnection(B_protocol) loseConnection(C_protocol, lambda *a: loseConnection(A_protocol)) The debug logs in my code assure me that the loseConnection function is always called with the A_protocol as argument, yet sometimes and totally unpredictably as far as I can tell connectionLost is not called on some A_protocol instances and corresponding A connections get stuck in CLOSE_WAIT state. Another hint is that netstat shows 1 byte in the Recv-Q for those connections (and 0 in the Send-Q). In another mail [2] in that old thread I got this snippet """ Your issue with CLOSE_WAIT sockets could be due to registering producers which have no further data to produce. """ This sounds like it could be my case, even if as you can see I am always unregistering producers from transports before I call loseConnection on them. Does this sound familiar in any way? Any suggestions off top of head while I try to come up with a self-contained sample which is reliably reproducing the issue I'm seeing happening only "sometimes and quite seldom[TM]"? Thanks, ciao ste [1] http://twistedmatrix.com/pipermail/twisted-python/2008-June/017853.html [2] http://twistedmatrix.com/pipermail/twisted-python/2008-June/017855.html

1 0

[Twisted-Python] problem using imap mail client
by jim gardener Oct. 15, 2010

Oct. 15, 2010

hi, I installed twisted on ubuntu lucid .When I checked the version ,it shows Version('twisted', 10, 0, 0) I tried to run imap4client.py giving it hostname='imap.gmail.com' and entered my username and password..When I run the program I am getting "twisted.internet.error.TCPTimedOutError: TCP connection timed out" error. I was able to login and read my gmail using python imaplib though..I am wondering why it doesn't work with twisted thanks jim

5 6

[Twisted-Python] State of "insults"?
by Nathan Oct. 14, 2010

Oct. 14, 2010

JP, what's the current state of the "insults" module you wrote? I'm looking to bolt an ncurses-like client interface onto one of my projects, and I didn't want to reinvent the wheel if it had already been done, so some diligent searching turned up the "insults" module you wrote. There's almost no examples, and the API documentation seems to consist of only method names (which are all very reasonably named), but not much in the way of actual descriptions. Is this a stable, portable module that I can expect to work on OS X and Linux, or is it a buggy work-in-progress (or something in between)? Searching the list archives turned up almost no discussion about it since 2007. The biggest benefits I see with ncurses are that it's got tons of documentation and I know it will work wherever Python works. The benefits I see with insults is that it is designed to work with Twisted and seems to have a decent selection of widgets, so I wouldn't have to implement too many basic widgets. ~ Nathan

2 2

[Twisted-Python] Missing self.wakeUp() in epoll reactor's callLater
by CARSON Oct. 12, 2010

Oct. 12, 2010

I noticed that callLater in the epollReactor does not actually wake up the reactor; the code for the threaded select reactor, however, does do this via a call to self.wakeUp. It seems like epoll reactor should be patched so that call later ends with self.wakeUp - does this seem correct?

2 1

[Twisted-Python] I am confused on transport.write and transport.writeSomeData
by koder chen Oct. 11, 2010

Oct. 11, 2010

It seems that, when use "self.transport.write" in the protocol implementation, it will buffer the content for a long time before it actually send it out, especially when the size of content is big, is there a 'flush' like method that I can do the flush manually? And when use 'self.transport.writeSomeData', if the size of content to be sent is large, it will not be able to send anything. What is the strategy to use it?

2 2