Mailman 3 November 2010 - Twisted

[Twisted-Python] pb.Cacheable doc question
by Stephen Waterbury Nov. 6, 2010

Nov. 6, 2010

First, the "PB Copyable: Passing Complex Types" doc is *great* and the examples are excellent -- my compliments to all who contributed! My question is about the pb.Cacheable section (http://twistedmatrix.com/documents/current/core/howto/pb-copyable.html#auto9) -- specifically the first sentence: 'Sometimes the object you want to send to the remote process is big and slow. "big" means it takes a lot of data (storage, network bandwidth, processing) to represent its state. "slow" means that state doesn't change very frequently.' I would think that the product of its size and its rate of change is the applicable metric -- i.e.: the bigger the object is *or* the faster it changes (not the slower), the more applicable Cacheable is, no? Steve

8 19

[Twisted-Python] Catching all the exceptions in a Twisted program
by Facundo Batista Nov. 6, 2010

Nov. 6, 2010

Hello! I've just subscribed to the list, but I'm using Twisted from a while ago. I need to do something, and wasn't able to find out how to do it. Well, I *did* find out, but it's a horrible hack, and actually depends on something that may be a bug, so it's worth asking here about this :) In a project I'm doing [0], I want to catch all problems (unhandled exceptions), log them and show them in stderr after a bit massaging. My first approach was to hook me in sys.excepthook, and it worked on some cases. But I saw some cases where it didn't [1]. Specifically, when the exception was inside a deferred, it didn't call excepthook. Why not? I started to check the code, and found the following snippet, at the end of Deferred._run_callbacks, in defer.py: if isinstance(self.result, failure.Failure): self.result.cleanFailure() if self._debugInfo is None: self._debugInfo = DebugInfo() self._debugInfo.failResult = self.result >From this, I have two questions: 1. The rest of the code only instantiates and uses DebugInfo "if self.debug". Why this part of the code doesn't care about the self.debug flag and always uses it? Is it intended and DebugInfo() not being used for debug is a misname left there for historical reasons? Should I open a bug about this? 2. I did the following (ugly) hack to achieve my results: import twisted.internet.defer # ugliest thing I saw in a *long* time class MyDebugInfo(object): def _error_happened(self, failure): msg = failure.getTraceback() _expose_exception(msg) failResult = property(fset=_error_happened) twisted.internet.defer.DebugInfo = MyDebugInfo Is there a better way to do this? Actually, this is working correctly, but I don't know if it's the right way to do it, and don't want to depend of DebugInfo() being called always in this case. Thank you very much! [0] http://launchpad.net/magicicada [1] http://bugs.launchpad.net/magicicada/+bug/665681 -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/

2 2

[Twisted-Python] [newb] another process question
by Neal Becker Nov. 6, 2010

Nov. 6, 2010

Using this code: ... def processEnded(self, reason): print "processEnded, task %s, status %s" % (self.task, reason.value.exitCode,) Here is what I get from a normally terminated process processEnded, task ['sleep', '4']31662, status 0 But now I kill the subprocess, with SIGTERM, and I get: processEnded, task ['sleep', '15']31663, status None This seems to disagree with the twisted process doc. Instead of an exitCode, I seem to have gotten 'None' .processEnded(status): This is called when the child process has been reaped, and receives information about the process' exit status. The status is passed in the form of a Failure instance, created with a .value that either holds a ProcessDone object if the process terminated normally (it died of natural causes instead of receiving a signal, and if the exit code was 0), or a ProcessTerminated object (with an .exitCode attribute) if something went wrong. This scheme may seem a bit weird, but I trust that it proves useful when dealing with exceptions that occur in asynchronous code.

2 1

[Twisted-Python] [newb] daemon question
by Neal Becker Nov. 5, 2010

Nov. 5, 2010

I have put together a simple twisted server using xmlrpc and process. I want to run this server as a unix daemon. Specifically, detach from controlling terminal, and redirect stdout, stderr to a log file. What is the easiest approach, and where to look for documentation on this?

4 3

[Twisted-Python] Inconsistency in ReconnectingClientFactory
by Brandl, Albert Nov. 3, 2010

Nov. 3, 2010

Hi! The ReconnectingClientFactory class allows to define the instance variable maxDelay. Its documentation is "Maximum number of seconds between connection attempts." maxDelay is used in the retry() method to bound self.delay. But this happens in the _first_ step of the calculation of self.delay. Afterwards, the bounded value is modified again by applying random.normalvariate to allow for the jitter. This can result in a value that is considerably bigger than maxDelay. Is there any reason that maxDelay is not used for bounding the final result of the calculation of self.delay? If not, the calculation should be fixed - I'd be happy to file an issue ;-). Best regards, Albert -- Albert Brandl Weiermayer Solutions GmbH | Abteistraße 12, A-4813 Altmünster phone: +43 (0) 720 70 30 14 | fax: +43 (0) 7612 20 3 56 web: http://www.weiermayer.com

2 1

[Twisted-Python] Twisted with callInThread
by Mike Zupan Nov. 2, 2010

Nov. 2, 2010

I stripped down my app but this should give you an example of what i'm doing def run_app(f): p = Popen(['/usr/bin/app'],stdout=PIPE) while True: o = p.stdout.readline() if o == '' and p.poll() != None: break reactor.callFromThread(f, o) class Echo(Protocol): def connectionMade(self): reactor.callInThread(run_app, self.appDataReceived) def dataReceived(self, data): data = data.strip() if data == "getmore": print "getmore" def appDataReceived(self, u): print u def main(): factory = Factory() factory.protocol = Echo reactor.listenTCP(3646, factory) reactor.run() if __name__ == "__main__": main() I have an app which I want to connect in and run a app that continually spits out data to stdout. Right now my app works but the issue is when the client exits the socket connection the /usr/bin/app still continues to run. The more socket connections made the more this app is still running. Is there anyway from the Echo Procool to kill the run_app() function?

2 1

[Twisted-Python] Immediate removal of cfreactor
by exarkun＠twistedmatrix.com Nov. 1, 2010

Nov. 1, 2010

Hello all, As reported in <http://twistedmatrix.com/trac/ticket/4652>, some problems have arisen with cfreactor. First, recent versions of Pyrex reject the Pyrex source for the support modules. This makes future development on cfreactor difficult (although Glyph's branch does resolve this problem - or at least appears to). Second, the C source files do not build with clang. This seems likely to become more important on future versions of OS X (the only platform where this code matters). Even OS X 10.6 makes clang easily available (according to an Ars Technica article, "Clang is the recommended compiler, and the focus of all of Apple's future efforts." <http://arstechnica.com/apple/reviews/2009/08/mac-os-x-10-6.ars/9>) Finally, when the cfreactor build fails, distutils prematurely exits, preventing the rest of the build from completing. This point makes this a serious problem for Twisted as a whole, rather than an obscure platform quirk that can be ignored without wide-ranging consequences. Apple has announced OS X 10.7 for summer 2011. Presumably at some point prior to this they will freeze on a version of Twisted to package. The best outcome for Twisted can probably be attained by ensuring this is a version of Twisted which does not have the build issues reported in #4652. Rather than leaving this to chance, I propose that we remove cfreactor and all supporting code ourselves, immediately (ie, in 10.2 - hi therve, go faster next time ;). I think the relatively small userbase of cfreactor (I don't know of any projects using it, nor does Google codesearch) and the fact that the version of cfreactor in trunk segfaults partway through running the test suite, combined with the potentially time-sensitive nature of this fix, merits an exception to the normal backwards policy. Rather than removing it, we could fix it, but the fact that no one seems to be using it makes me think that no one with the expertise and interest will step up to so. If we're going to do this, it would be good to do it in the next few days (say, by Tuesday) so as not to unduly further delay the 10.2 release process. Thoughts? Jean-Paul

3 4

Re: [Twisted-Python] connectionLost never reached after calling loseConnection: stuck in CLOSE_WAIT forever
by Stefano Debenedetti Nov. 1, 2010

Nov. 1, 2010

Hello Jean-Paul, thanks for tracking this down, you rock! I promise that when I'll have payed all by debts I'll buy one of those posters of Exarkun to hang on my wall! JP wrote: > After a few runs, I managed to reproduce the problem. I instrumented the reactor with some extra logging and test_producer.py with a manhole server. > The sequence of events appears to be something like this: > > OneA has a producer of OneE > OneA has a consumer of OneB > At some point OneB gives up and tells OneA to stopProducing (loseConnection) > OneA.loseConnection stops the reactor from reading OneA and starts it writing > OneA.doWrite happens > it finds the send buffer empty > it finds a registered producer (OneE) and resumes it > OneE never produces any more bytes > OneE loses its connection at some point and unregisters itself from OneA > OneA takes note that it has no more producer, but does nothing about it > > So the bug is likely that FileDescriptor.unregisterProducer doesn't do anything special when disconnecting=True. > > You should be able to reproduce this very simply by setting up a transport-producer/consumer pair, calling loseConnection on the transport, then unregistering the producer. > > This all sounds somewhat familiar, but I don't see an existing ticket for it, so maybe that's my imagination. > > Jean-Paul Following your indications I attached a minimal example to a new ticket: http://twistedmatrix.com/trac/ticket/4719 Some additional info: * problem occurs only if more than 64KB of data are written to the transport before its consumer calls stopProducing on it * problem occurs only if some time passes before its producer unregisters itself from the transport Thanks again for your help! :) ste

5 8