Mailman 3 February 2010 - Twisted

[Twisted-Python] [ANNOUNCE] Twisted 10.0.0pre1 is now released
by Jonathan Lange 24 Feb '10

24 Feb '10

Live from PyCon Atlanta, I'm pleased to herald the approaching footsteps of the 10.0 release. Tarballs for the first Twisted 10.0.0 pre-release are now available at: http://people.canonical.com/~jml/Twisted/ This release is the first release ever with the new NEWS building system, which turns out to be utterly fantastic. We're also using this release to actually hammer out a release process. You can find the draft at: http://twistedmatrix.com/trac/wiki/ReleaseProcess Please feel free to update it with questions, thoughts, corrections and advice. Thanks, jml

8 15

[Twisted-Python] Reactor events help request ..
by Watson, David 24 Feb '10

24 Feb '10

Hi Folks .. I'm not entirely a newbie a python and twisted, but I have come to the point where I need a bit more expertise on this topic, so I am hoping that I have come to the right place .. What I'm hoping, that you folks can help out with a minor problem that I am having with the twistd reactor class and/or the callLater function I have been using a reactor.callLater function to schedule an event that happens in the reactor to send out a heartbeat type message at regular intervals .. This works great ! However, it appears to me that the self.sendLine(msg) and or the self.transport.write(msg) writing functions are blocked functions waiting for a reactor event to occur at a future scheduled time before acutally flushing out the internal write buffers .. I am currently using the twistd protocol factory to create a custom protocol to communicate between a server and client, which up until now has not been a real concern, but I am trying to improve the preformance and turn around time between the client and server and this blocking operation is a major source of unneeded delay .. My problem is that I have scheduled heartbeat events that I want to send to the server .. as well as the non scheduled events that I want to go to the server imediately .. I have been scouring the various python/twistd document sites, but to no avial, on all of the sites that I have seen, the method used have all done the same thing , by re-scheduling something into the future with a reactor.callLater () function, which causes a delayed write to occur .. not a problem for something that is 1 or 2 second s in the furture, but it is a serious problem when the next event is 120 seconds away .. I have tried using a self.transport.write(msg) function, but the actual communication still seems to be buffered and stuck waiting for the scheduled reactor event .. What I think I need is something similar to self.transport.flush() which would flush out the write buffers on demand, but does not seen to exist .. I am hoping that someone has faced the same/similar stituation and give me some guidance and/or to point me in the correct direction .. Thanks for any help you can offer .. Cheers Dave Watson

2 2

Re: [Twisted-Python] Sending jpeg data over TCP/IP
by gary clark 24 Feb '10

24 Feb '10

The correct way I think is to use base64 and just one connection after more research. Good to research. Thanks, Garyc --- On Sun, 2/14/10, gary clark <burslem2001(a)yahoo.com> wrote: > From: gary clark <burslem2001(a)yahoo.com> > Subject: Re: [Twisted-Python] Sending jpeg data over TCP/IP > To: "Twisted general discussion" <twisted-python(a)twistedmatrix.com> > Date: Sunday, February 14, 2010, 12:18 PM > > hey Alexandra, > > The only reason why I suggested another server would be to > distribute the > load on the system. Essentially the files that are intended > to be sent over will consume alot of processing since .jpeg > etc are humongous. > Yes two connections would be required. Well you could use > the MD5 sum, I will not, essentially your after a unique > identifier that represents the file your sending. What I > intend to do is simply embed the filename and unique > identifier as a header. Hence I do not not need to recompute > the MD5 at the receiving end, which to be honest seems > overkill. > > Anyway good luck. KISS for software is the best approach. > > Thanks, > Garyc > > > --- On Sun, 2/14/10, Alexandre Quessy <alexandre(a)quessy.net> > wrote: > > > From: Alexandre Quessy <alexandre(a)quessy.net> > > Subject: Re: [Twisted-Python] Sending jpeg data over > TCP/IP > > To: "Twisted general discussion" <twisted-python(a)twistedmatrix.com> > > Date: Sunday, February 14, 2010, 11:10 AM > > Hello again everyone, > > Maybe using two senders/receivers would help. The > control > > protocol, > > which can use XML or JSON, or whatever, would identify > the > > files by > > their md5 sum? The file transfert protocol would > detect the > > header of > > each file to separate them. > > > > a > > > > gary clark wrote: > > > There are probably several way to accomplish this > I > > just needed to think about it a wee bit longer. One > way > > would be to prepend an identifier to the file,strip > the > > header from the raw data on reception and then save > the > > image. I dont think its complicated. I may need a > seperate > > server to handle the files though. > > > > > > Thanks, > > > Garyc > > > > > > --- On Sat, 2/13/10, Maarten ter Huurne <maarten(a)treewalker.org> > > wrote: > > > > > >> From: Maarten ter Huurne <maarten(a)treewalker.org> > > >> Subject: Re: [Twisted-Python] Sending jpeg > data > > over TCP/IP > > >> To: "Twisted general discussion" <twisted-python(a)twistedmatrix.com> > > >> Date: Saturday, February 13, 2010, 6:07 PM > > >> On Sunday 14 February 2010, Alexandre > > >> Quessy wrote: > > >> > > >>> This said, sending them using a > programmer's > > solution > > >> - not a sysadmin > > >>> solution - would be closer to my own > skills, > > so I am > > >> interested in > > >>> knowing if this could be suitable. I > think, > > though, > > >> that it would be > > >>> faster to use a transfert protocol that > would > > be > > >> implemented in C, not > > >>> Python. Am I wrong? > > >> I would suggest to implement it in Python > first > > and then > > >> benchmark it. Maybe > > >> the simplest implementation is already fast > > enough. Maybe > > >> the bottleneck is > > >> the network or the disk you're writing to; in > that > > case you > > >> would be better > > >> off upgrading your switches or buying an SSD > > instead of > > >> writing C code. > > >> > > >> Bye, > > >> Maarten > > >> > > >> > _______________________________________________ > > >> Twisted-Python mailing list > > >> Twisted-Python(a)twistedmatrix.com > > >> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > >> > > > > > > > > > _______________________________________________ > > > Twisted-Python mailing list > > > Twisted-Python(a)twistedmatrix.com > > > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > > > > > > _______________________________________________ > > Twisted-Python mailing list > > Twisted-Python(a)twistedmatrix.com > > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > >

3 2

Re: [Twisted-Python] twisted.protocol.sip
by Landreville 24 Feb '10

24 Feb '10

On Tue, Feb 23, 2010 at 4:42 PM, Lorenzo Mainardi <lormayna(a)gmail.com> wrote: > Landreville ha scritto: >> On Tue, Feb 23, 2010 at 10:28 AM, Lorenzo Mainardi <lormayna(a)gmail.com> wrote: >>> 2010/2/23 Landreville <landreville(a)deadtreepages.com>: >>> >>>> I'll put it up somewhere so you can see it, >>>> but I'm fairly new at twisted so someone else might say it is the >>>> wrong approach. >>>> >>> Hi, I'm start doing the same, so we can merge the works. >>> At the moment I don't have so much code, but you can put in google >>> code or somewhere (If it's possible I prefer to use mercurial instead >>> of SVN or GIT). >>> >> >> I've got it up on launchpad at: https://launchpad.net/sipper > > Hi download the branch and subscribed the mailing list. > I'll try to write some code. > I'm just using this as a SIP client, not as a proxy server. What are you making with SIP?

1 0

Re: [Twisted-Python] twisted.protocol.sip
by Landreville 23 Feb '10

23 Feb '10

On Tue, Feb 23, 2010 at 10:28 AM, Lorenzo Mainardi <lormayna(a)gmail.com> wrote: > 2010/2/23 Landreville <landreville(a)deadtreepages.com>: > >> I'll put it up somewhere so you can see it, >> but I'm fairly new at twisted so someone else might say it is the >> wrong approach. >> > > Hi, I'm start doing the same, so we can merge the works. > At the moment I don't have so much code, but you can put in google > code or somewhere (If it's possible I prefer to use mercurial instead > of SVN or GIT). > I've got it up on launchpad at: https://launchpad.net/sipper I don't have time right now to explain it all, but the reactor is started in SIPPhone.py and thats where the registration is started as well.

1 0

[Twisted-Python] twisted.protocol.sip
by Lorenzo Mainardi 23 Feb '10

23 Feb '10

Hello, I'm trying to extend the class twisted.protocols.sip.MessageParser for create my parser. I had read the documentation and I found this: "Shouldn't be connected to actual transport.". What does it means? I can't use that binding it to a socket? I found also that many other object in twisted.protocols.sip are not complete and/or not working (some of that return everytime NotImplementedYet). Is it correct or I'm doing some mistakes? -- LORENZO MAINARDI Email: lormayna(a)gmail.com Linux Registered User: 461615 Key Fingerprint: AC63 5C15 562F 71AF C853 4D4A C03F 75EB 52F4 A0D0

2 1

Re: [Twisted-Python] Try/catching yielded Exceptions and getting proper tracebacks
by Terry Jones 23 Feb '10

23 Feb '10

Hi Paul >>>>> "Paul" == Paul Goins <general(a)vultaire.net> writes: You're only printing the exception, not a full traceback, so you don't see much. I tend to write what you're doing as follows: from twisted.python import log @defer.inlineCallbacks def xmlrpc_dosomething(self): d = self._do_something_else() d.addErrback(log.err) result = yield d defer.returnValue(result) If you try that you'll see a full traceback. The above lets log.err handle the failure that comes back via the errback on the deferred you get from _do_something_else, and log.err knows how to get the full traceback. I don't know if it's clear, but whenever you call an inlineCallbacks decorated method/func, you get a deferred back (unless you happen to mistakenly use inlineCallbacks to decorate something that's not a generator). You can add errbacks to that deferred, just like any other, and if you're making the call from inside another inlineCallbacks decorated function, you can just do as above: add call/errbacks to the deferred, and then yield it. Terry

2 1

[Twisted-Python] Looking for companies doing Twisted
by Nicolas Dietrich 22 Feb '10

22 Feb '10

Hi there, I'm not sure if this is the right place to ask, so sorry for the spam in advance. Anyway: I'm looking for companies which might be interested in developing a booking system for an emerging Germany-based long-distance railway undertaking during this year. This will be a distributed system, so Twisted might be a good solution for this. Please approach me directly for any details. Thankful for all hints, Nicolas

1 0

[Twisted-Python] Try/catching yielded Exceptions and getting proper tracebacks
by Paul Goins 22 Feb '10

22 Feb '10

I'm frequently using the inlineCallbacks idiom for writing code, but am often finding myself frustrated with rather counterintuitive tracebacks. Let's say I'm writing an XML-RPC handler, and even if an exception occurs, I want it to return something to the user and dump the exception to the log file. However, if an exception occurs when I'm using the inlineCallbacks decorator, I sometimes get lacking tracebacks. Maybe I'll have the line number of the function that yielded the exception, but I'll have no detailed information as to where the exception really came from. For example, in the example I've pasted at the end of this mail, I get a traceback as follows: Traceback (most recent call last): File "exceptions.py", line 13, in xmlrpc_dosomething result = yield self._do_something_else() Exception: die die die!!! However, the exception happens within _do_something_else on line 20. I have a simple complete example copy/pasted below. Is there a better way to accomplish this? Thanks. - Paul Goins ----------------------------------- from twisted.web.xmlrpc import XMLRPC from twisted.web.server import Site from twisted.web.resource import Resource from twisted.internet import reactor, defer import sys, traceback class MyApp(XMLRPC): @defer.inlineCallbacks def xmlrpc_dosomething(self): try: result = yield self._do_something_else() defer.returnValue(result) except Exception: print >> sys.stderr, traceback.format_exc() @defer.inlineCallbacks def _do_something_else(self): yield raise Exception("die die die!!!") @defer.inlineCallbacks def cause_trouble(app): val = yield app.xmlrpc_dosomething() print val reactor.stop() if __name__ == "__main__": app = MyApp() r = Resource() r.putChild("RPC2", app) s = Site(r) reactor.listenTCP(8123, s) reactor.callLater(0.2, cause_trouble, app) reactor.run()

1 0

[Twisted-Python] Many connections and TIME_WAIT
by Donal McMullan 22 Feb '10

22 Feb '10

I've been prototyping a client that connects to thousands of servers and calls some method. It's not real important to me at this stage whether that's via xmlrpc, perspective broker, or something else. What seems to happen on the client machine is that each network connection that gets opened and then closed goes into a TIME_WAIT state, and eventually there are so many connections in that state that it's impossible to create any more. I'm keeping an eye on the output of netstat -an | wc -l Initially I've got 569 entries there. When I run my test client, that ramps up really quickly and peaks at about 2824. At that point, the client reports a callRemoteFailure: callRemoteFailure [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion: Connection lost. Increasing the file descriptor limits doesn't seem to have any effect on this. Is there an established Twisted sanctioned canonical way to free up this resource? Or am I doing something wrong? I'm looking into tweaking SO_REUSEADDR and SO_LINGER - that sound sane? Just tapping the lazywebs to see if anyone's already seen this in the wild. Thanks guys Donal

2 2