Mailman 3 February 2013 - Twisted

[Twisted-Python] Unable to write to "stuck" TCP client connections
by Wenxiang Wu Feb. 26, 2013

Feb. 26, 2013

Hi, I am encountering a weird bug, where some TCP client connections get into a state where the server is able to read data sent from the client, but not able to send any data with transport.write(). With some help from the #twitsed IRC channel, I was able to gather the following information regarding the bug. While I'm still unable to provide steps to reporduce this bug, I am able to reliably find clients who are in this state. I am running 24 instances of the twisted server (Epoll reactor) running on Ubuntu, with a peak traffic of >130k users. At any instance, there are < 20 TCP connections stuck in this state. Here is some information about the bug: 1. transport.write() does not send anything down the socket 2. transport.doWrite() will send all the data that has been buffered up, and then stop sending any new data. 3. transport.writeSomeData() will send data 3. reactor.getWriters() will return a list of transports that are all stuck in this state, and the writers will remain in this list. 4. Calling reactor.removeWriter(transport) will "unstuck" the transport and data gets streamed once again. 5. A small number of clients will receive data for a while, and return to this stuck state. Most return to normal once reactor.removeWriter() is called. 6. Based on the suggestion from IRC user _habnabit, I used strace after removing the writer, here is the output: (4:52:09 PM) thewrongboy: epoll_ctl(3, EPOLL_CTL_MOD, 6504, {EPOLLIN, {u32=6504, u64=22205092589476200}}) = 0 (4:52:09 PM) thewrongboy: epoll_ctl(3, EPOLL_CTL_MOD, 6504, {EPOLLIN|EPOLLOUT, {u32=6504, u64=22205092589476200}}) = 0 (4:52:09 PM) thewrongboy: epoll_ctl(3, EPOLL_CTL_MOD, 6504, {EPOLLIN, {u32=6504, u64=22205092589476200}}) = 0 (4:52:09 PM) thewrongboy: epoll_ctl(3, EPOLL_CTL_MOD, 6504, {EPOLLIN|EPOLLOUT, {u32=6504, u64=22205092589476200}}) = 0 (4:52:09 PM) thewrongboy: epoll_ctl(3, EPOLL_CTL_MOD, 6504, {EPOLLIN, {u32=6504, u64=22205092589476200}}) = 0 For now, I am using a LoopingCall to check and remove transports that are stuck in getWriters(). I am using Twisted 12.3.0 on Ubuntu 12.04 - 3.2.0-35-generic #55-Ubuntu SMP Wed Dec 5 17:42:16 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux. Has anyone else experienced this weird problem? I'd love to provide more information regarding this bug.

5 8

[Twisted-Python] combine local and remote calls in perspective brokers
by Sergey Gerasimov Feb. 26, 2013

Feb. 26, 2013

Hello to all! I'm using PB as communication protocol for components of my system. Components can be located locally on the only one machine or in distributed fashion. It depends on the config file. At the moment all communications are implemented as callRemote calls. I would like to keep this unified way of communication but make inter-machine calls as efficient as possible. What is the best way to do that?

4 4

[Twisted-Python] Inline callbacks: generating the same Deferred at multiple places
by Pierre Jaury Feb. 25, 2013

Feb. 25, 2013

Hey everyone, First of all, I am a french developer mostly dealing with security matters, I have been a Twisted enthusiast for a couple of years now and have been hacking around with Twisted for the very last couple of months. I am aware that Deferred-related classes are very stable nowadays and that such a change does not stand half a chance to get anywhere. Anyway, Twisted behavior kind of annoys me when it comes to inline callback management. Indeed, the gotResult function used when generating Deferred objects from a function decorated with inlineCallbacks currently looks like: def gotResult(r): if waiting[0]: waiting[0] = False waiting[1] = r else: _inlineCallbacks(r, g, deferred) So far the function does not return anything explicitely, thus following callbacks get a very nice `None` as first argument. This is not so annoying when using inline callbacks with very simple and straightforward cases as advised by Twisted documentation. Anyway, when generating the same Deferred object from multiple inlineCallbacks-decorated functions, one would expect every `yield` statement to return the same value. As a matter of fact, this is not the case (the first generation of the Deferred object will return its actual current state but later generations will return `None`). Because a Deferred object generation from an inlineCallbacks-decorated function does not allow for anything but waiting for the Deferred object to fire and returning its state, it would probably not harm to forward this state to the next callback as well. Something like: def gotResult(r): if waiting[0]: waiting[0] = False waiting[1] = r else: _inlineCallbacks(r, g, deferred) return r would probably do the trick and allow for complex scenarios where the same Deferred object is generated multiple times by different functions before being fired. As I am not an accomplished Twisted hacker, I cannot say if this kind of change would harm or break any code out there. Yet it seems to me that according to inlineCallbacks behavior, it is very unlikely to have any consequence at all but improving (extending at least) inlineCallbacks behavior. Am I so wrong? (I would very much like someone to point out the one huge detail I completely forgot before I make this change by monkey patching and submit a feature request :) Regards, Pierre. -- Pierre Jaury <pierre(a)jaury.eu> Weblog - http://kaiyou.fr GPG ID - E804FB60

3 4

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Feb. 24, 2013

Feb. 24, 2013

1 0

[Twisted-Python] Building a TCP server, having issues
by Adam Heller Feb. 18, 2013

Feb. 18, 2013

Guys, I'll try to be as exact as I can be with my question, but it may end up being pretty broad. (I'm also relatively new to python, please go easy on my code although suggestions are of course welcome.) The issue I'm running into: When a user connects to the TCP server, they are displayed a welcome menu of things they can do. I'm simply catching the 'data' in an if statement to determine which menu item was picked. The problem is that some of the menu items require further input. I've had to use a series of bools to determine if a certain menu option was previously picked that will then directly send the data into the function / class. Is there an easier way to go about this? My current method seems a bit backwards and over complicated. I'm thinking I missed something in the documentation. Anyway, here's the code: https://infotomb.com/q4293 // Pass: twisted (it was part of a challenge, so to speak, don't take too much out of context) You can see I had trouble with next() and dh(). I'm now trying to move this TCP 'framework' over to another project (the above was already submitted), and I'm still running into the same issues. Thoughts? -A

4 4

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Feb. 17, 2013

Feb. 17, 2013

1 0

[Twisted-Python] TaskStopped error, unsure of the cause and solution
by Paul Wiseman Feb. 12, 2013

Feb. 12, 2013

Hey, I'm getting the following error in my logs (lots and lots), but only after the server has been running for some time, I'm not sure what the cause is or what I can do to remedy it. This is what I see in the logs: 2012-12-20 17:07:17+0000 [HTTP11ClientProtocol,client] Unhandled Error Traceback (most recent call last): Failure: twisted.web._newclient.RequestTransmissionFailed: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>] 2012-12-20 17:07:17+0000 [HTTP11ClientProtocol,client] Unexpected exception from twisted.web.client.FileBodyProducer.stopProducing Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/endpoints.py", line 99, in connectionLost return self._wrappedProtocol.connectionLost(reason) File "/usr/local/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/web/_newclient.py", line 859, in dispatcher return func(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/web/_newclient.py", line 1458, in _connectionLost_TRANSMITTING self._currentRequest.stopWriting() File "/usr/local/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/web/_newclient.py", line 760, in stopWriting _callAppFunction(self.bodyProducer.stopProducing) --- <exception caught here> --- File "/usr/local/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/web/_newclient.py", line 191, in _callAppFunction function() File "/usr/local/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/web/client.py", line 760, in stopProducing self._task.stop() File "/usr/local/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/task.py", line 459, in stop self._checkFinish() File "/usr/local/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/task.py", line 469, in _checkFinish raise self._completionState twisted.internet.task.TaskStopped: Just loads of those, they always seem to come through in pairs (the first twisted.internet.error.ConnectionDone, and then the twisted.internet.task.TaskStopped as above) I'm just doing PUT requests with twisted.web.client.Agent and using a twisted.web.client.FileBodyProducer to provide the content. I'm doing the PUT while I still have the request from a render_GET, so I thought maybe it's caused if the user disconnects while I'm doing the PUT? The idea is I redirect them to the file I PUT with request.redirect, which seems to work fine (until I get this error and it doesn't) The reason I do a PUT and redirect, instead of just writing back to the request is that I check if the file is in a cache first, I then only PUT if it's not there- then redirect afterwards. Any clues or ideas of what is happening would be hugely helpful thanks!! Paul

3 6

[Twisted-Python] Passing parameters to Adapter
by Fabian Sturm Feb. 12, 2013

Feb. 12, 2013

Dear twisted developers, I have a sshdaemon based on sshsimpleserver.py which works great. But I now want to pass a command line argument to the EchoProtocol. How can I do this? In this case I would like to pass the 'options.test' option to my protocol. if __name__ == '__main__': parser = optparse.OptionParser() parser.add_option('-p', '--port', action = 'store', type = 'int', dest = 'port', default = 1235, help = 'server port') parser.add_option('-t', '--test', action = 'store', type = 'string', dest = 'test', default = '123') (options, args) = parser.parse_args() components.registerAdapter(ExampleSession, ExampleAvatar, session.ISession) [...] reactor.listenTCP(options.port, ExampleFactory()) reactor.run() Since the session instance is created by the factory, I can't seem to be able to pass additional args to e.g. the constructor. Thanks a lot! Fabian ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.

1 0

[Twisted-Python] imap google mail client
by Luca Sbardella Feb. 11, 2013

Feb. 11, 2013

Hello, first time here for me. I'm writing a little application to access gmail using the IMAP4 library. I'm using the connection string strport='ssl:host=imap.gmail.com:port=993' but it doesn't connect (timeout failure). The code is the mail_client function in this module https://github.com/quantmind/pulsar/blob/handshake/examples/webmail/manage.… What am I doing wrong? Thanks Luca

3 5

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Feb. 10, 2013

Feb. 10, 2013

1 0