Mailman 3 March 2012 - Twisted

[Twisted-Python] How to Improve Twisted efficiency
by Peng Xiao -X (penxiao - Digital China at Cisco) 13 Mar '12

13 Mar '12

Hi experts, I wrote a TCP client program used twisted to receive lots of data from the server and decode them for analysis. But the efficiency is ugly. I used “profile” and “pstats” to analysis and found selectreactor.py is time consuming. what is selectreactor.py used for? How can I improve that? Very thanks. 2012-03-13 17:46:10+0800 [-] Tue Mar 13 17:46:10 2012 astar_prof.txt 2012-03-13 17:46:10+0800 [-] 2012-03-13 17:46:10+0800 [-] 9111 function calls (9070 primitive calls) in 119.166 CPU seconds 2012-03-13 17:46:10+0800 [-] 2012-03-13 17:46:10+0800 [-] Ordered by: internal time 2012-03-13 17:46:10+0800 [-] List reduced from 496 to 15 due to restriction <15> 2012-03-13 17:46:10+0800 [-] 2012-03-13 17:46:10+0800 [-] ncalls tottime percall cumtime percall filename:lineno(function) 2012-03-13 17:46:10+0800 [-] 171 119.092 0.696 119.094 0.696 :0(select) 2012-03-13 17:46:10+0800 [-] 156 0.005 0.000 0.007 0.000 http.py:225(datetimeToLogString) 2012-03-13 17:46:10+0800 [-] 4 0.005 0.001 0.007 0.002 protocol.py:131(dataReceived) 2012-03-13 17:46:10+0800 [-] 171 0.004 0.000 0.029 0.000 base.py:718(runUntilCurrent) 2012-03-13 17:46:10+0800 [-] 324 0.004 0.000 0.004 0.000 :0(append) 2012-03-13 17:46:10+0800 [-] 160 0.004 0.000 0.004 0.000 :0(heappop) 2012-03-13 17:46:10+0800 [-] 171 0.004 0.000 119.114 0.697 selectreactor.py:93(doSelect) 2012-03-13 17:46:10+0800 [-] 164 0.003 0.000 0.003 0.000 :0(heappush) 2012-03-13 17:46:10+0800 [-] 342 0.003 0.000 0.006 0.000 base.py:699(_insertNewDelayedCalls) 2012-03-13 17:46:10+0800 [-] 342 0.003 0.000 0.003 0.000 :0(keys) 2012-03-13 17:46:10+0800 [-] 358 0.002 0.000 0.002 0.000 :0(fileno) 2012-03-13 17:46:10+0800 [-] 163 0.002 0.000 0.005 0.000 base.py:648(callLater) 2012-03-13 17:46:10+0800 [-] 19/1 0.002 0.000 0.003 0.003 xml2dict.py:16(_parse_node) 2012-03-13 17:46:10+0800 [-] 686 0.002 0.000 0.002 0.000 :0(time) 2012-03-13 17:46:10+0800 [-] 51 0.002 0.000 0.003 0.000 :0(encode) 2012-03-13 17:46:10+0800 [-] Best Regards, Xiao Peng(肖鹏) Cisco Systems GSP-Beijing (China) Office: +86 10 8507 7181 Mobile: +86 18600560077(preferred)

2 1

[Twisted-Python] Distributed system programming
by nivedita datta 11 Mar '12

11 Mar '12

Hi, I have the following query "Can programming for distributed system done using twisted? If yes, kindly provide some resource to get started. If not, please suggest any existing module for the same on python." Regards, Nivedita

2 2

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 11 Mar '12

11 Mar '12

1 0

[Twisted-Python] ampoule & cx_Freeze
by gelin yan 09 Mar '12

09 Mar '12

Hi All Have your guys used cx_Freeze & ampoule? I did try to make a standalone executable app with ampoule; however, the program failed to execute at the early stage of starting. I noticed an exception raised on utils.py: p = FilePath(__main__.__file__) AttributeError: 'module' object has no attribute __file__. I guess once the main file is tuned into the executable one, __file__ will be removed. It looks like ampoule need this attribute. Any idea? Thanks gelin yan

2 2

[Twisted-Python] Twisted TCP Window Full due to time-consuming fuction
by Peng Xiao -X (penxiao - Digital China at Cisco) 07 Mar '12

07 Mar '12

Hi experts I have a program written by twisted to receive data from the server. But the speed is a little slow. The problem happened in the code below def dataReceived(self, data): """ Appends newly received data to the receive buffer, and then attempts to parse as many messages as possible. """ # Buffer possibly incomplete data first self.receiveBuffer += data # Attempt to parse as many messages as possible while(self.parseBuffer()): pass the function of parseBuffer will return false if the self.receiveBuffer has no message, and then the twisted will receive data from buffer again. Because the function of parseBuffer is a litter time consuming, so the receive buffer is full sometimes. Could there anybody help me out? Thanks a lot. Best Regards, Xiao peng

2 1

[Twisted-Python] Question : PyNotify call from twist daemon
by bino oetomo 06 Mar '12

06 Mar '12

Dear All ... I have a Serial service in my twisted script Inside my 'class ArduinoReceiver(LineReceiver):' , I have a function like : ---START---- def setdisplay(self, dispmode): global mynotify self.ntf_title = 'TOMBOL' self.ntf_body = 'Coba aja' try : mynotify.close() except : pass print 'Try to display button notification %s' % (dispmode) mynotify = pynotify.Notification(self.ntf_title, self.ntf_body) print 'MYNOTIFY ', str(mynotify) ----STOP--- When I try to run my tac file , and do something that will call that 'setdisplay' function , there is no notification poped-up But from the log file .. I got 2012-03-06 08:47:51+0000 [-] MYNOTIFY <pynotify.Notification object at 0x96a5b6c (NotifyNotification at 0x9665d90)> I made a test using shiped example : ---START--- #!/usr/bin/env python import pygtk pygtk.require('2.0') import pynotify import sys if __name__ == '__main__': if not pynotify.init("Markup"): sys.exit(1) n = pynotify.Notification("Summary", "Some bold, underlined, italic, " + "<a href='http://www.google.com'>linked</a> text") if not n.show(): print "Failed to send notification" sys.exit(1) ---STOP---- It work just fine. My system is : Ubuntu maveric, with just Xorg and OpenBox Kindly please give me any of your enlightment to fix this problem Sincerely -bino-

2 4

Re: [Twisted-Python] Twisted/PyPy rocks!
by Itamar Turner-Trauring 04 Mar '12

04 Mar '12

> Couple of points: > > * Have been able to use poll reactor on Linux, but epoll doesnt work .. is > that expected? Maybe because the epoll reactor implementation uses Twisted > own C wrapper instead of the - only > recently - available one in Python standard lib? That was fixed in trunk recently, you could try that. > * I will test PyPy with kqueue on FreeBSD .. > > * Are there regular buildslaves running trial on PyPy? Does that make > sense? Hopefully soon, we do want PyPy to be a fully supported official platform.

4 6

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 04 Mar '12

04 Mar '12

1 0

[Twisted-Python] Twisted+PyPy+kqueue => rocks^2!
by Tobias Oberstein 04 Mar '12

04 Mar '12

It's flying;) I've added kqueue support to PyPy and got it running with Twisted/PyPy on FreeBSD. https://bitbucket.org/pypy/pypy/pull-request/29/finish-kqueue-support Tested with Twisted trunk ('cause of #1918 kqueue support in Twisted) and hammered with a WebSocket testsuite. Twisted trial not yet .. Marketing blurb: "Twisted/PyPy's best supported platform is now BSD (less Linux, since of #3114)" Sorry, couldn't resist saying that;) \Tobias

1 0

[Twisted-Python] Twisted reactor cpu usage
by Ilya Biin 03 Mar '12

03 Mar '12

I'm using epoll reactor and sample HTTP Proxy realization. From another process i'm executing ~30 HTTP requests per second and twisted process is consuming 10% cpu The profiler output is: 937706 function calls (934675 primitive calls) in 59.988 CPU seconds Ordered by: cumulative time List reduced from 312 to 15 due to restriction <15> ncalls tottime percall cumtime percall filename:lineno(function) 1 0.000 0.000 59.988 59.988 base.py:1167(run) 1 0.076 0.076 59.988 59.988 base.py:1172(mainLoop) 12697 55.456 0.004 59.253 0.005 epollreactor.py:169(doPoll) 13553 0.086 0.000 3.797 0.000 log.py:71(callWithLogger) 13553 0.096 0.000 3.691 0.000 log.py:66(callWithContext) 13553 0.047 0.000 3.481 0.000 context.py:117(callWithContext) 13553 0.067 0.000 3.426 0.000 context.py:61(callWithContext) 13552 0.095 0.000 3.359 0.000 posixbase.py:544(_doReadOrWrite) 5454 0.097 0.000 1.736 0.000 tcp.py:182(doRead) 5453 0.167 0.000 1.639 0.000 basic.py:543(dataReceived) 12697 0.095 0.000 0.587 0.000 base.py:762(runUntilCurrent) 4453 0.025 0.000 0.574 0.000 http.py:495(rawDataReceived) 6000 0.036 0.000 0.547 0.000 http.py:1515(lineReceived) 2000 0.029 0.000 0.454 0.000 tcp.py:371(doConnect) 5098 0.104 0.000 0.428 0.000 abstract.py:212(doWrite) 937706 function calls (934675 primitive calls) in 59.988 CPU seconds Ordered by: internal time List reduced from 312 to 15 due to restriction <15> ncalls tottime percall cumtime percall filename:lineno(function) 12697 55.456 0.004 59.253 0.005 epollreactor.py:169(doPoll) 59101 0.277 0.000 0.277 0.000 <string>:1(fileno) 5098 0.252 0.000 0.252 0.000 tcp.py:212(writeSomeData) 24453 0.198 0.000 0.198 0.000 __init__.py:1230(getEffectiveLevel) 5453 0.167 0.000 1.639 0.000 basic.py:543(dataReceived) 5098 0.104 0.000 0.428 0.000 abstract.py:212(doWrite) 5454 0.097 0.000 1.736 0.000 tcp.py:182(doRead) 13553 0.096 0.000 3.691 0.000 log.py:66(callWithContext) 12697 0.095 0.000 0.587 0.000 base.py:762(runUntilCurrent) 13552 0.095 0.000 3.359 0.000 posixbase.py:544(_doReadOrWrite) 13553 0.086 0.000 3.797 0.000 log.py:71(callWithLogger) 1 0.076 0.076 59.988 59.988 base.py:1172(mainLoop) 16556 0.074 0.000 0.074 0.000 context.py:86(getContext) 17451 0.073 0.000 0.094 0.000 epollreactor.py:71(_add) 17098 0.068 0.000 0.085 0.000 epollreactor.py:112(_remove) What is wrong? Why twisted perfomance is so poor here? -- Ilya Biin

4 10