Mailman 3 December 2005 - Twisted

[Twisted-Python] Twisted Python 0.7.0 Prerelease Testing
by Glyph Lefkowitz 06 Aug '21

06 Aug '21

hi there, folks: I'd really like to release 0.7.0 but I would like it to be at least a little bit tested before I do so. Could those of you with CVS trees check everything out and see if it performs as advertised? Deeper bugs than that will have to wait for the next release, but I'd at least like to know if it works for someone other than me. Thanks. ______ __ __ _____ _ _ | ____ | \_/ |_____] |_____| |_____| |_____ | | | | @ t w i s t e d m a t r i x . c o m http://www.twistedmatrix.com/~glyph/

4 5

[Twisted-Python] Get IP of connecting client in XMLRPC server
by Remy C. Cool 28 Apr '13

28 Apr '13

Hello, On 18 May 2004, the following message was posted to this mailinglist: Jp Calderone exarkun at divmod.com wrote: >Daniel Newton wrote: > > I have a simple XML-PRC server similar to the example below: > > from twisted.web import xmlrpc, server > > class Example(xmlrpc.XMLRPC): > """An example object to be published.""" > > def xmlrpc_add(self, a, b): > """Return sum of arguments.""" > return a + b def xmlrpc_whatIsMyAddress(self): return self.transport.getPeer().host > > if __name__ == '__main__': > from twisted.internet import reactor > r = Example() > reactor.listenTCP(7080, server.Site(r)) > reactor.run() > > I want to be able to get the address of the client that calls the xmlrpc > method can anyone help me with this? This solution didn't work because 'transport' isn't a property of the Example class. I'm currently in the process of changing from a customized SimpleXMLRPCServer to a twisted XMLRPC server solution and I need to insert the client IP into the attributes passed to the called xmlrpc method. Anyone who knows the answer and is willing to share the info? Regards, Remy

3 3

[Twisted-Python] Problem: spawnProcess() never ending when installSignalHandlers=False
by Matt Feifarek 04 Feb '07

04 Feb '07

Hello. I'm using Twisted 1.3.0 on python2.4 on an Ubuntu-Hoary x86 Linux box. I'm using twisted and spawnProcess() with a Queue structure to chain processes that need to be called (specifically, imagemagick commands). I use the "processEnded" event/method of t.i.p.ProcessProtocol to go from one task to the next. Twisted shares a python process with a multi-threaded web-app-kit (Webware) so it has to be run in a thread. This worked well on other older versions of twisted and older versions of python; also on linux. I've tracked down the problem to be twisted itself (or my incorrect use of it) rather than my code. This example illustrates it: ------------------------------------------------- from twisted.internet import reactor from twisted.internet.protocol import ProcessProtocol class MyProt( ProcessProtocol ): def processEnded(self, reason): print "ended.", reason executable = '/bin/touch' args = ( '/tmp/foo.txt', ) args = (executable,) + tuple(args) reactor.spawnProcess( MyProt(), executable, args ) reactor.callLater( 5, reactor.stop ) #reactor.run() reactor.run(installSignalHandlers=False) ------------------------------------------------- As is, this code never runs the "processEnded()" method of MyProt(). If I change the reactor.run() parameter to leave the signal handlers on, it works. Note that I need to leave the signal handlers turned off because I can't run the reactor in the main thread; Webware needs that. I also know that the subprocess is giving up its file descriptors; I can see that if I put more print statements into MyProt(). And I know that the subprocess is working; /tmp/foo.txt is there. It seems that somehow the reactor is using signals to notice when a process has ended; this makes a certain ammount of sense intuitively, though I don't understand the nitty-gritty of processes and signals at all. And more importantly, this has worked before on other combinations of twisted/python. Is this a bug? If so, has it been fixed? If not, what should I do to detect when a spawned process is finished? Thanks very much. Twisted rocks.

6 14

[Twisted-Python] Status of wxPython support in Twisted 2.1: use wxreactor
by Itamar Shtull-Trauring 02 Jul '06

02 Jul '06

wxreactor in Twisted 2.1 uses threadedselectreactor. This means that wx apps ought to use wxreactor, not TSR directly; there are some issues with vanilla TSR wx integration that are also present in wxreactor, but will be fixed in future versions of wxreactor. What's broken (there may be more): 1. reactor shutdown events aren't handled correctly (e.g. Deferred returned from during-shutdown event). 2. Signal handling for e.g. Ctrl-C is caught by wx, not twisted. In order to get these fixed I need your help. I have a version of wxreactor that *probably* fixes these in Subversion trunk. You can try it out by running doc/core/examples/wxdemo.py. Hitting Ctrl-C, choosing Exit from the menu or closing the menu all should have the same effect - the application logs "shutting down in 0.3 seconds...", "2", "1", "0" then exits. In addition, "Hello world" should continue to be printed while the dialog box is open or you are holding down a menu. Please try this out, and report success or failure at <http://twistedmatrix.com/bugs/issue1235>. I'm especially interested to hear if it works on Windows or Mac OS X. Please include your OS and version of wxPython/wxWidgets when submitting reports.

3 7

[Twisted-Python] Securing a custom protocol
by Brian Granger 08 Jan '06

08 Jan '06

Hi, I have written a custom line based protocol using twisted. I need to add authentication and encryption to my protocol and I am wondering what the best route is. Here are some options that I see: 1. Use SSL for encryption and cred for user authentication 2. Use SSH through conch As I understand it, both of these approaches can do authentication and encryption of any protocol. What are the advantages and disadvantages or each? Which is more robust and supported in twisted? Are there other approaches in twisted that I am missing? Some design constraints I have: 1. I need the protocol to be as fast as possible - low latency and high throughput. 2. I need to be able to run the protocol without encryption when speed is more important than security. Thanks! Brian

2 2

[Twisted-Python] help with refcounts and memleaks
by Andrea Arcangeli 07 Jan '06

07 Jan '06

Hello, I was just shoked today when I noticed this: ------------------- import sys class A(object): y = None def x(self): pass def __del__(self): print 'deleted' a = A() print sys.getrefcount(a) if 1: a.y = a.x print sys.getrefcount(a) del a ------------------- I understood the cross references memleaks well, like "x.y = y; y.x= x; del x,y", but I didn't imagine that "a.y = a.x" would be enough to generate a memleak. "a.y = a.x" isn't referencing another structure, it's referencing itself only. Infact if I do this the memleak goes away!! ------------------- import sys class A(object): def x(self): pass y = x def __del__(self): print 'deleted' a = A() print sys.getrefcount(a) a.x() a.y() print a.x, a.y del a ------------------- Now the fact a static field doesn't generate a reference but a dynamic one does is quite confusing to me and it also opened a can of worms in my code. I can handle that now that I know about it, but I wonder what people recommends to solve memleaks of this kind. I'd also like to know how other languages like ruby and java behave in terms of self-references of objects. Can't the language understand it's a self reference, and in turn it's the same as an integer or a string, like it already does when the member is initialized statically? Infact can't the language be smart enough to even understand when two cross referenced objects lost visibility from all points of view, and drop both objects even if they hold a reference on each other? I understand this is a lot more complicated but wouldn't it be possible in theory? What does the garbage collection of other languages like ruby and java, the same as python or more advanced? So far my python programs never really cared to released memory (so my not full understanding of python refcounts wasn't a problem), but now since I'm dealing with a server I must make sure that the "proto" is released after a loseConnection invocation. So I must cleanup all cross and self! references in loseConnection and use weakrefs where needed. Now those structures that I'm leaking (like the protocol object) are so tiny that there's no chance that I could ever notice the memleak in real life, so I had to add debugging code to trap memleaks. You can imagine my server code like this: class cpushare_protocol(Int32StringReceiver): def connectionMade(self): [..] self.hard_handlers = { PROTO_SECCOMP : self.seccomp_handler, PROTO_LOG : self.log_handler, } [..] def log_handler(self, string): [..] def seccomp_handler(self, string): [..] def __del__(self): print 'protocol deleted' def connectionLost(self, reason): [..] # memleaks del self.hard_handlers print 'protocol refcount:', sys.getrefcount(self) #assert sys.getrefcount(self) == 4 For things like hard_handlers (that are self-referencing callbacks) I can't even use the weakref.WeakValueDictionary, because it wouldn't hold itself, the object gets released immediately. So the only chance I have to release the memory of the protocol object when the connection is dropped, is to do an explicit del self.hard_handlers in loseConnection. I wonder what other twisted developers do to avoid those troubles. Perhaps I shouldn't use self referencing callbacks to hold the state machine, and do like the smpt protocol that does this: def lookupMethod(self, command): return getattr(self, 'do_' + command.upper(), None) basically working with strings instead of pointers. Or I can simply make sure to cleanup all structures when I stop using them (like with the del self.hard_handlers above), but then I'll lose part of the automatic garbage collection features of python. I really want garbage collection or I could have written this in C++ if I'm forced to cleanup by hand. Help is appreciated, thanks! PS. Merry Christmas and Happy New 2006!

8 17

[Twisted-Python] Re: r15451 - Fix test failures under windows by changing the eventual-send operation to
by Brian Warner 02 Jan '06

02 Jan '06

>> Yes it does: order of method call. Ah. That's useful to know: I would prefer to use a non-threading-related call to achieve this eventual-send functionality. I'll add a TODO test to check that this promise is being met. > It should be fixable by storing the last time of an element added to the > queue, and adding epsilon if current time is the same. Wouldn't you have to guarantee that epsilon is smaller than the resolution of your time.time() return value? What if you sorted on a tuple of (time, counter) instead? (the down side is that you'd have to search for all existing timers with the same time value to figure out what counter value you ought to use. ick.). We should clarify what guarantees are made by callLater. I think there may be several separate ones here: 1: callLater(0, A); callLater(0, B) will result in A being invoked before B will result in both A and B being invoked before any other DelayedCalls 2: callLater(N, A); callLater(N, B) will result in A being invoked before B 3: callLater(N, A); callLater(N+M, C); callLater(N, B) will result in A being invoked before B (think of this as a unit test for the adding-epsilon concern above) The second and third ones are not so important to me, just in terms of what I need to use it as a plan-coordination tool. I only intend to use this with N=0. To that end, using a separate queue for timers that are ready to go "now" (i.e. ones that will be fired before calling select() or the like) might be useful, basically making N=0 a special case. This would avoid the overhead of inserting the DelayedCall into an arbitrary place and maintaining the ordering guarantees of #2 and #3, and would avoid an extra select() spin between the time an N=0 timer was inserted and the time it was fired. The existing threadCallQueue happens to behave exactly this way, although I'd want to write some additional tests to make sure it gets serviced as many times as it's supposed to be (specifically, when threads are unavailable and therefore wakeUp() is not used, does a call inserted from within an N=0 callback get serviced before the reactor sleeps again?). The problem is both the word "thread" in the name, and the fact that we might not be making the same guarantees about the behavior of callFromThread as we are about that of callLater. Hmm. Most reactors split off a list of timers that are ready to go "now" on each spin, right? And/or there's that _pendingTimedCalls list I see in t.i.base .. maybe we could take advantage of one of those, just appending the call to those lists and making sure they'll be serviced again, rather than adding the overhead of maintaining ordering guarantees #2 and #3. hmm-ingly, -Brian

4 4

[Twisted-Python] Depreciation of trial.unittest functions?
by Paul Swartz 02 Jan '06

02 Jan '06

I'm not sure if Conch is the only project that did this, but it used the trial.unittest.* functions instead of methods on TestCases. I used them to avoid having to pass TestCase instances around all over the places, or using a global variable. To avoid the depreciation warnings, I've starting using the unittest._inst global variable, but I doubt that this is the appropriate way to change my code. Is this just not a supported way of using Trial, or is there something else I should be doing? -p -- Paul Swartz paulswartz at gmail dot com http://z3p.livejournal.com/ AIM: z3penguin GPG: 5CF0B1C9

3 3

[Twisted-Python] Twisted windows hackers - help the tests to pass!
by glyph＠divmod.com 01 Jan '06

01 Jan '06

Twisted fails scads of tests on Windows. You can see that here: http://twistedmatrix.com/buildbot/ I know that there are at least a few people on this list that use Twisted on Windows. Doesn't it bother you that it's broken? If you're a windows developer, help out and fix some tests! A few Twisted hackers, Justin J. in particular, are hard at work to make it scale and perform well on Windows, but in addition to the rocket-science of IOCP there is quite a bit of grunt work to do, especially in the area of understanding why certain tests fail when they pass on other (more UNIX-y) platforms. So please, try debugging some Twisted unit tests and either submitting bug reports explaining what's going on so an existing Twisted dev can fix them, or submitting patches that we can apply. The real bonus for you if you're using Twisted on Windows in some commercial capacity is that once the buildbot turns green, test fascists like JP (or, on off days, myself) can revert changes which break it, and enforce a higher standard of quality for portability of Twisted code. Right now it's hard to convince developers to not make it fail _more_ tests, since hey, it's broken already, right?

6 20

[Twisted-Python] Re: plus mode was Re: how winnt fileops work and what to do about it
by Brian Warner 01 Jan '06

01 Jan '06

glyph(a)divmod.com writes: > It seems like we can work around this more easily than that, considering > that flush and seek are available from Twisted; the file object causing > problems in the tests is being returned from the open() method of a > FilePath object, if I understand it correctly. FilePath could include the > workaround far in advance of Python deciding to. I'm pretty sure that the real problem we're trying to solve here is caused by a stuck process keeping a .pyd file open. Indeed, if you look at the buildslave's logs, you'll see the exception is as follows: exceptions.OSError: [Errno 13] Permission denied: 'c:\\buildslave\\win32-win32er\\W32-full2.4-win32er\\Twisted\\twisted\\protocols\\_c_urlarg.pyd' So changing the way Twisted or its unit tests open a file is just not going to help. What matters is the way python (or.. pyrex?) opens a file. (for context: the buildbot is currently configured to do SVN checkout/updates into one directory, then copy the tree into a second directory, then run tests on that second directory. This mode='copy' approach uses 'svn update' to minimizes network bandwidth, but at the expense of doubling the disk usage with the extra copy. At the beginning of each build, the buildslave deletes the second directory with a function named rmdirRecursive() that bear provided, which does a chmod() of any mis-permissioned files before deleting them. It was an os.remove() inside this rmdirRecursive which raised the exception). I've run into a similar problem in the past, under Solaris, using NFS, where a test case spawned off a daemon process which then didn't die when it was supposed to, somehow held on to a file (I think solaris won't let you delete a file that is being used as the backing store for an executable), and that prevented the unlink() from succeeding. In that environment, I just renamed the top-level directory to something unique, spawned off an 'rm -rf' into the background to delete the old directory if it was possible, then continued on with the next build. If the code had to try too hard to come up with a unique name, it would flag a warning that there might be a stuck process somewhere. Perhaps we could use something similar here? Of course, the real fix would be to find a way to let the testing code kill off any stuck processes, but that'll probably be very windows-specific. cheers, -Brian

3 4