Mailman 3 March 2011 - Twisted

[Twisted-Python] Benchmark of Python WSGI servers
by Michael Thompson March 19, 2011

March 19, 2011

>From the guys who brought you async socket benchmark, http://nichol.as/asynchronous-servers-in-python, comes Python WSGI benchmark http://nichol.as/benchmark-of-python-web-servers. Is twisted coming out of this so badly because they are using the default reactor, as opposed to epoll? Perhaps the default reactor should be the best available rather than the lowest common denominator. Michael

4 6

[Twisted-Python] Best strategies for pb Referenceables running long methods from callRemote
by Charles Solar March 19, 2011

March 19, 2011

I am using PB to run remote methods in a testing system at my company. The code works very well but breaks down when I start running multiple tests at once. I have tracked this down to overflowing the thread pool on the remote machines. I am wondering if anyone might have better suggestions for running long methods from a remote method. I coded up a sample of what I am seeing here: http://pastebin.com/rBPp20Ms Basically I have 1 server that calls remote_execute on many clients on a remote server. This remote_execute method starts a new method using threads.deferToThread and returns the defer to make the server's callRemote defer wait until the remote long method end. What I do in those methods is run test code that waits, blocks, sleeps, and all sorts of nasty things that make the thread take a while. In the example code I simply sleep for 20 seconds. The problem I see with this code specifically is that I run out of threads on the pool and even though I wanted all execute methods to run at the same time, I see 10 run, then 10 more, then 10 more.. etc. The testing depends on all these methods being run at the same time as they run mechanisms that depend on each other and need everyone running. When I overflow the thread pool some methods do not run until other methods stop, which makes the whole test fail. I am not holding the GIL or blocking the reactor, which was the first thing I checked. Setting reactor.suggestThreadPoolSize(50) does help, but I do not think its the best solution, and does not work very well on our slow and older machines. Any feedback is appreciated Charles

2 5

Re: [Twisted-Python] [Twisted-commits] r31216 - skip glib and gtk reactors in the new test
by Itamar Turner-Trauring March 17, 2011

March 17, 2011

On Wed, 2011-03-16 at 20:55 -0600, pahan(a)wolfwood.twistedmatrix.com wrote: > + > + skippedReactors = ["Glib2Reactor", "Gtk2Reactor"] > + reactorClassName = reactor.__class__.__name__ > + if reactorClassName in skippedReactors and platform.isWindows(): > + raise SkipTest( > + "This test is broken on gtk/glib under Windows.") > + Shouldn't that be a todo, rather than a skip? Or maybe file a ticket.

3 2

[Twisted-Python] Unicode exceptions
by Temchenko Grigory March 17, 2011

March 17, 2011

Hey there, On my production server I have periodically happening unicode error but not on my desktop. It appears in logs: 2011-03-17 13:14:53+0000 [GameProtocol,941,95.78.43.17] <unicode instance at 0x9e304a0 with str error: Traceback (most recent call last): File "/usr/local/lib/python2.6/dist-packages/twisted/python/reflect.py", line 546, in _safeFormat return formatter(o) UnicodeEncodeError: 'ascii' codec can't encode characters in position 21-26: ordinal not in range(128) > It doesn't affect any logic in application but it's annoying in logs. The server runs under Ubuntu 10.10 Server, Python 2.6.5, Twisted 10.2.0. The desktop is Ubuntu 10.10 Desktop, Python 2.6.5, Twisted 10.2.0. Locales are the same: $ locale LANG=en_US.UTF-8 LANGUAGE=en_US.UTF-8 LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL=en_US.UTF-8 What do I have to check to get it fixed?

1 0

[Twisted-Python] sample twistorm code?
by Jean-Daniel March 17, 2011

March 17, 2011

Hello, I writing a script parsing a large log file and shoveling it into Postgresql. 1. I would like to read some codes examples using the twisted + storm integration, any idea of where to look? Also. I was thinking about modeling a line of logs with a namedtuple, because it is handy to create one without having to write a long and dumb __init__ which take every constructor argument and one by one assign them to the instance member of the same name. 2. Is the namedtuple compatible with storm objects? Namedtuple members are properties and I am not sure it plays well with the storm machinery. Thank you for your help, jdb

2 1

[Twisted-Python] LoopingCalls and unclean reactor
by Brad Milne March 16, 2011

March 16, 2011

Hi I have a series of MultiService objects, with child Services. Some of these services are TCPServers, for example, and others are my own objects (extending from Service). In the instance that I have a Service which controls a LoopingCall, I am getting intermittent 'unclean reactor' errors during tests. I feel I might be missing some handling of deferreds, perhaps. (using 8.2.0 - looking to migrate to 10.2.0 soon) Here is an example of the approach being used: class AdapterQueue(service.MultiService): def startService(self): service.Service.startService(self) self._looping_controller = LoopingCall(self._action) d = self._looping_controller.start(self.delay, False) d.addErrback(self._errorInScheduler) def stopService(self): service.Service.stopService(self) d = self._looping_controller.deferred self._looping_controller.stop() return d And an example error (happens about 1/3 of the time): DirtyReactorAggregateError: Reactor was unclean. DelayedCalls: (set twisted.internet.base.DelayedCall.debug = True to debug) <DelayedCall 30800112 [0.0189974308014s] called=0 cancelled=0 LoopingCall<0.033333333333333333>(AdapterQueue._action, *(), **{})() traceback at creation: File "C:\Python26\lib\threading.py", line 497, in __bootstrap self.__bootstrap_inner() File "C:\Python26\lib\threading.py", line 525, in __bootstrap_inner self.run() File "C:\Python26\lib\threading.py", line 477, in run self.__target(*self.__args, **self.__kwargs) File "D:\dev\eggs\nose-0.11.3-py2.6.egg\nose\twistedtools.py", line 57, in <lambda> installSignalHandlers=False)) File "D:\dev\eggs\twisted-8.2.0-py2.6-win32.egg\twisted\internet\base.py", line 1128, in run self.mainLoop() File "D:\dev\eggs\twisted-8.2.0-py2.6-win32.egg\twisted\internet\base.py", line 1137, in mainLoop self.runUntilCurrent() File "D:\dev\eggs\twisted-8.2.0-py2.6-win32.egg\twisted\internet\base.py", line 757, in runUntilCurrent call.func(*call.args, **call.kw) File "D:\dev\eggs\twisted-8.2.0-py2.6-win32.egg\twisted\internet\task.py", line 115, in __call__ d.addCallback(cb) File "D:\dev\eggs\twisted-8.2.0-py2.6-win32.egg\twisted\internet\defer.py", line 195, in addCallback callbackKeywords=kw) File "D:\dev\eggs\twisted-8.2.0-py2.6-win32.egg\twisted\internet\defer.py", line 186, in addCallbacks self._runCallbacks() File "D:\dev\eggs\twisted-8.2.0-py2.6-win32.egg\twisted\internet\defer.py", line 328, in _runCallbacks self.result = callback(self.result, *args, **kw) File "D:\dev\eggs\twisted-8.2.0-py2.6-win32.egg\twisted\internet\task.py", line 103, in cb self._reschedule() File "D:\dev\eggs\twisted-8.2.0-py2.6-win32.egg\twisted\internet\task.py", line 139, in _reschedule self.call = self.clock.callLater(nextTime - currentTime, self) > Many thanks Brad

2 2

[Twisted-Python] AMP cred stuff
by Stephen Waterbury March 15, 2011

March 15, 2011

Hey Glyph, Jp, and twisted sprinters, Sorry I couldn't stay longer -- I stretched my "travel day" as far as possible, but have to get back to my real job now (on which I'm under the gun atm). Hope I can sprint longer next time -- I had a lot of fun, and the energy in the room is amazing! I'll attach the code to a bug tonight (I'll create one if there isn't one). I hope I can carve out time to add Kerberos (in another issue, of course) soon -- selfishly, because I really need it for my app at work. Gotta get back to work now -- you guys all rock! Cheers, Steve

2 1

[Twisted-Python] UDP Logging Server
by SIC FS LIST March 13, 2011

March 13, 2011

Hello, I am trying to write a UDP based logging server. Generically speaking it looks somewhat like syslog except I needed a bit more flexibility that syslog can provide (or at least that I think it can provide). What I'm trying to accomplish is: -- receive UDP packet -- parse UDP packet -- write output to a log file -- have the log files rotated on a periodic basis So far I have a "working" implementation ... but I'm noticing that if I do the following: -- log when a message is received -- that for that message it "might" show up in the file a pretty lengthy period of time later The actual UDP protocol: class VocsLogger(DatagramProtocol): def datagramReceived(self, data, (host, port)): _proc_msg(self.transport, data, (host, port))._new_msg().addCallback(handler) The _proc_msg class: class _proc_msg: def __init__(self, sck, data, (host, port)): self._sck = sck self._data = data self._host = host self._port = port def _new_msg(self): d, _ = LogMsg().ParseSocketMsg(self._data) if d.type.upper() == DISKINFO[0]: DISKINFO[1].write(d.ToString() + "\n%s\n" % (LOG_DELIM)) elif d.type.upper() == LOADAVG[0]: LOADAVG[1].write(d.ToString() + "\n%s\n" % (LOG_DELIM)) elif d.type.upper() == MEMINFO[0]: MEMINFO[1].write(d.ToString() + "\n%s\n" % (LOG_DELIM)) elif d.type.upper() == NETDEV[0]: NETDEV[1].write(d.ToString() + "\n%s\n" % (LOG_DELIM)) elif d.type.upper() == PSAUX[0]: PSAUX[1].write(d.ToString() + "\n%s\n" % (LOG_DELIM)) elif d.type.upper() == WHOINFO[0]: WHOINFO[1].write(d.ToString() + "\n%s\n" % (LOG_DELIM)) else: DEFAULT[1].write(d.ToString() + "\n%s\n" % (LOG_DELIM)) And I have a log rotate service that looks like this: class LogRotateService(TimerService): def __init__(self): TimerService.__init__(self, 60 * 5, LogRotate) And then I'm using twistd to actually make it work: LogRotate() application = service.Application("vocs-logger") rotateLogService = LogRotateService() rotateLogService.setServiceParent(application) loggerService = internet.UDPServer(int(config['port']), VocsLogger(), interface=config['host']) loggerService.setServiceParent(application) I'm probably approaching this the wrong way and not sure if I sure really worry about deferring the actual process of writing to the log file or if there was a better way altogether. Thanks for any guidance. SDR

3 6

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com March 13, 2011

March 13, 2011

1 0

[Twisted-Python] Failing tests in trunk
by Facundo Batista March 12, 2011

March 12, 2011

I branched trunk from the LP repository. Running the tests all together ("trial twisted"), consistently makes me fail the following 12: twisted.conch.test.test_cftp.ListingTests.test_newFile twisted.conch.test.test_cftp.ListingTests.test_newSingleDigitDayOfMonth twisted.conch.test.test_cftp.ListingTests.test_oldFile twisted.conch.test.test_cftp.ListingTests.test_oldSingleDigitDayOfMonth twisted.mail.test.test_imap.NewFetchTestCase.testFetchAll twisted.mail.test.test_imap.NewFetchTestCase.testFetchAllUID twisted.mail.test.test_imap.NewFetchTestCase.testFetchFast twisted.mail.test.test_imap.NewFetchTestCase.testFetchFastUID twisted.mail.test.test_imap.NewFetchTestCase.testFetchFull twisted.mail.test.test_imap.NewFetchTestCase.testFetchFullUID twisted.mail.test.test_imap.NewFetchTestCase.testFetchInternalDate twisted.mail.test.test_imap.NewFetchTestCase.testFetchInternalDateUID However, if I run the tests doing "trial twisted.mail" or "trial twisted.conch", they never fail. I still didn't debug this, but I wanted to see if somebody experienced something similar to this. Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/

3 6