Mailman 3 December 2005 - Twisted

[Twisted-Python] Twisted Python 0.7.0 Prerelease Testing
by Glyph Lefkowitz 05 Aug '21

05 Aug '21

hi there, folks: I'd really like to release 0.7.0 but I would like it to be at least a little bit tested before I do so. Could those of you with CVS trees check everything out and see if it performs as advertised? Deeper bugs than that will have to wait for the next release, but I'd at least like to know if it works for someone other than me. Thanks. ______ __ __ _____ _ _ | ____ | \_/ |_____] |_____| |_____| |_____ | | | | @ t w i s t e d m a t r i x . c o m http://www.twistedmatrix.com/~glyph/

4 5

[Twisted-Python] Get IP of connecting client in XMLRPC server
by Remy C. Cool 28 Apr '13

28 Apr '13

Hello, On 18 May 2004, the following message was posted to this mailinglist: Jp Calderone exarkun at divmod.com wrote: >Daniel Newton wrote: > > I have a simple XML-PRC server similar to the example below: > > from twisted.web import xmlrpc, server > > class Example(xmlrpc.XMLRPC): > """An example object to be published.""" > > def xmlrpc_add(self, a, b): > """Return sum of arguments.""" > return a + b def xmlrpc_whatIsMyAddress(self): return self.transport.getPeer().host > > if __name__ == '__main__': > from twisted.internet import reactor > r = Example() > reactor.listenTCP(7080, server.Site(r)) > reactor.run() > > I want to be able to get the address of the client that calls the xmlrpc > method can anyone help me with this? This solution didn't work because 'transport' isn't a property of the Example class. I'm currently in the process of changing from a customized SimpleXMLRPCServer to a twisted XMLRPC server solution and I need to insert the client IP into the attributes passed to the called xmlrpc method. Anyone who knows the answer and is willing to share the info? Regards, Remy

3 3

[Twisted-Python] Problem: spawnProcess() never ending when installSignalHandlers=False
by Matt Feifarek 03 Feb '07

03 Feb '07

Hello. I'm using Twisted 1.3.0 on python2.4 on an Ubuntu-Hoary x86 Linux box. I'm using twisted and spawnProcess() with a Queue structure to chain processes that need to be called (specifically, imagemagick commands). I use the "processEnded" event/method of t.i.p.ProcessProtocol to go from one task to the next. Twisted shares a python process with a multi-threaded web-app-kit (Webware) so it has to be run in a thread. This worked well on other older versions of twisted and older versions of python; also on linux. I've tracked down the problem to be twisted itself (or my incorrect use of it) rather than my code. This example illustrates it: ------------------------------------------------- from twisted.internet import reactor from twisted.internet.protocol import ProcessProtocol class MyProt( ProcessProtocol ): def processEnded(self, reason): print "ended.", reason executable = '/bin/touch' args = ( '/tmp/foo.txt', ) args = (executable,) + tuple(args) reactor.spawnProcess( MyProt(), executable, args ) reactor.callLater( 5, reactor.stop ) #reactor.run() reactor.run(installSignalHandlers=False) ------------------------------------------------- As is, this code never runs the "processEnded()" method of MyProt(). If I change the reactor.run() parameter to leave the signal handlers on, it works. Note that I need to leave the signal handlers turned off because I can't run the reactor in the main thread; Webware needs that. I also know that the subprocess is giving up its file descriptors; I can see that if I put more print statements into MyProt(). And I know that the subprocess is working; /tmp/foo.txt is there. It seems that somehow the reactor is using signals to notice when a process has ended; this makes a certain ammount of sense intuitively, though I don't understand the nitty-gritty of processes and signals at all. And more importantly, this has worked before on other combinations of twisted/python. Is this a bug? If so, has it been fixed? If not, what should I do to detect when a spawned process is finished? Thanks very much. Twisted rocks.

6 14

[Twisted-Python] Status of wxPython support in Twisted 2.1: use wxreactor
by Itamar Shtull-Trauring 02 Jul '06

02 Jul '06

wxreactor in Twisted 2.1 uses threadedselectreactor. This means that wx apps ought to use wxreactor, not TSR directly; there are some issues with vanilla TSR wx integration that are also present in wxreactor, but will be fixed in future versions of wxreactor. What's broken (there may be more): 1. reactor shutdown events aren't handled correctly (e.g. Deferred returned from during-shutdown event). 2. Signal handling for e.g. Ctrl-C is caught by wx, not twisted. In order to get these fixed I need your help. I have a version of wxreactor that *probably* fixes these in Subversion trunk. You can try it out by running doc/core/examples/wxdemo.py. Hitting Ctrl-C, choosing Exit from the menu or closing the menu all should have the same effect - the application logs "shutting down in 0.3 seconds...", "2", "1", "0" then exits. In addition, "Hello world" should continue to be printed while the dialog box is open or you are holding down a menu. Please try this out, and report success or failure at <http://twistedmatrix.com/bugs/issue1235>. I'm especially interested to hear if it works on Windows or Mac OS X. Please include your OS and version of wxPython/wxWidgets when submitting reports.

3 7

[Twisted-Python] Securing a custom protocol
by Brian Granger 07 Jan '06

07 Jan '06

Hi, I have written a custom line based protocol using twisted. I need to add authentication and encryption to my protocol and I am wondering what the best route is. Here are some options that I see: 1. Use SSL for encryption and cred for user authentication 2. Use SSH through conch As I understand it, both of these approaches can do authentication and encryption of any protocol. What are the advantages and disadvantages or each? Which is more robust and supported in twisted? Are there other approaches in twisted that I am missing? Some design constraints I have: 1. I need the protocol to be as fast as possible - low latency and high throughput. 2. I need to be able to run the protocol without encryption when speed is more important than security. Thanks! Brian

2 2

[Twisted-Python] help with refcounts and memleaks
by Andrea Arcangeli 06 Jan '06

06 Jan '06

Hello, I was just shoked today when I noticed this: ------------------- import sys class A(object): y = None def x(self): pass def __del__(self): print 'deleted' a = A() print sys.getrefcount(a) if 1: a.y = a.x print sys.getrefcount(a) del a ------------------- I understood the cross references memleaks well, like "x.y = y; y.x= x; del x,y", but I didn't imagine that "a.y = a.x" would be enough to generate a memleak. "a.y = a.x" isn't referencing another structure, it's referencing itself only. Infact if I do this the memleak goes away!! ------------------- import sys class A(object): def x(self): pass y = x def __del__(self): print 'deleted' a = A() print sys.getrefcount(a) a.x() a.y() print a.x, a.y del a ------------------- Now the fact a static field doesn't generate a reference but a dynamic one does is quite confusing to me and it also opened a can of worms in my code. I can handle that now that I know about it, but I wonder what people recommends to solve memleaks of this kind. I'd also like to know how other languages like ruby and java behave in terms of self-references of objects. Can't the language understand it's a self reference, and in turn it's the same as an integer or a string, like it already does when the member is initialized statically? Infact can't the language be smart enough to even understand when two cross referenced objects lost visibility from all points of view, and drop both objects even if they hold a reference on each other? I understand this is a lot more complicated but wouldn't it be possible in theory? What does the garbage collection of other languages like ruby and java, the same as python or more advanced? So far my python programs never really cared to released memory (so my not full understanding of python refcounts wasn't a problem), but now since I'm dealing with a server I must make sure that the "proto" is released after a loseConnection invocation. So I must cleanup all cross and self! references in loseConnection and use weakrefs where needed. Now those structures that I'm leaking (like the protocol object) are so tiny that there's no chance that I could ever notice the memleak in real life, so I had to add debugging code to trap memleaks. You can imagine my server code like this: class cpushare_protocol(Int32StringReceiver): def connectionMade(self): [..] self.hard_handlers = { PROTO_SECCOMP : self.seccomp_handler, PROTO_LOG : self.log_handler, } [..] def log_handler(self, string): [..] def seccomp_handler(self, string): [..] def __del__(self): print 'protocol deleted' def connectionLost(self, reason): [..] # memleaks del self.hard_handlers print 'protocol refcount:', sys.getrefcount(self) #assert sys.getrefcount(self) == 4 For things like hard_handlers (that are self-referencing callbacks) I can't even use the weakref.WeakValueDictionary, because it wouldn't hold itself, the object gets released immediately. So the only chance I have to release the memory of the protocol object when the connection is dropped, is to do an explicit del self.hard_handlers in loseConnection. I wonder what other twisted developers do to avoid those troubles. Perhaps I shouldn't use self referencing callbacks to hold the state machine, and do like the smpt protocol that does this: def lookupMethod(self, command): return getattr(self, 'do_' + command.upper(), None) basically working with strings instead of pointers. Or I can simply make sure to cleanup all structures when I stop using them (like with the del self.hard_handlers above), but then I'll lose part of the automatic garbage collection features of python. I really want garbage collection or I could have written this in C++ if I'm forced to cleanup by hand. Help is appreciated, thanks! PS. Merry Christmas and Happy New 2006!

8 17

[Twisted-Python] Re: r15451 - Fix test failures under windows by changing the eventual-send operation to
by Brian Warner 02 Jan '06

02 Jan '06

>> Yes it does: order of method call. Ah. That's useful to know: I would prefer to use a non-threading-related call to achieve this eventual-send functionality. I'll add a TODO test to check that this promise is being met. > It should be fixable by storing the last time of an element added to the > queue, and adding epsilon if current time is the same. Wouldn't you have to guarantee that epsilon is smaller than the resolution of your time.time() return value? What if you sorted on a tuple of (time, counter) instead? (the down side is that you'd have to search for all existing timers with the same time value to figure out what counter value you ought to use. ick.). We should clarify what guarantees are made by callLater. I think there may be several separate ones here: 1: callLater(0, A); callLater(0, B) will result in A being invoked before B will result in both A and B being invoked before any other DelayedCalls 2: callLater(N, A); callLater(N, B) will result in A being invoked before B 3: callLater(N, A); callLater(N+M, C); callLater(N, B) will result in A being invoked before B (think of this as a unit test for the adding-epsilon concern above) The second and third ones are not so important to me, just in terms of what I need to use it as a plan-coordination tool. I only intend to use this with N=0. To that end, using a separate queue for timers that are ready to go "now" (i.e. ones that will be fired before calling select() or the like) might be useful, basically making N=0 a special case. This would avoid the overhead of inserting the DelayedCall into an arbitrary place and maintaining the ordering guarantees of #2 and #3, and would avoid an extra select() spin between the time an N=0 timer was inserted and the time it was fired. The existing threadCallQueue happens to behave exactly this way, although I'd want to write some additional tests to make sure it gets serviced as many times as it's supposed to be (specifically, when threads are unavailable and therefore wakeUp() is not used, does a call inserted from within an N=0 callback get serviced before the reactor sleeps again?). The problem is both the word "thread" in the name, and the fact that we might not be making the same guarantees about the behavior of callFromThread as we are about that of callLater. Hmm. Most reactors split off a list of timers that are ready to go "now" on each spin, right? And/or there's that _pendingTimedCalls list I see in t.i.base .. maybe we could take advantage of one of those, just appending the call to those lists and making sure they'll be serviced again, rather than adding the overhead of maintaining ordering guarantees #2 and #3. hmm-ingly, -Brian

4 4

[Twisted-Python] Depreciation of trial.unittest functions?
by Paul Swartz 02 Jan '06

02 Jan '06

I'm not sure if Conch is the only project that did this, but it used the trial.unittest.* functions instead of methods on TestCases. I used them to avoid having to pass TestCase instances around all over the places, or using a global variable. To avoid the depreciation warnings, I've starting using the unittest._inst global variable, but I doubt that this is the appropriate way to change my code. Is this just not a supported way of using Trial, or is there something else I should be doing? -p -- Paul Swartz paulswartz at gmail dot com http://z3p.livejournal.com/ AIM: z3penguin GPG: 5CF0B1C9

3 3

[Twisted-Python] Twisted windows hackers - help the tests to pass!
by glyph＠divmod.com 01 Jan '06

01 Jan '06

Twisted fails scads of tests on Windows. You can see that here: http://twistedmatrix.com/buildbot/ I know that there are at least a few people on this list that use Twisted on Windows. Doesn't it bother you that it's broken? If you're a windows developer, help out and fix some tests! A few Twisted hackers, Justin J. in particular, are hard at work to make it scale and perform well on Windows, but in addition to the rocket-science of IOCP there is quite a bit of grunt work to do, especially in the area of understanding why certain tests fail when they pass on other (more UNIX-y) platforms. So please, try debugging some Twisted unit tests and either submitting bug reports explaining what's going on so an existing Twisted dev can fix them, or submitting patches that we can apply. The real bonus for you if you're using Twisted on Windows in some commercial capacity is that once the buildbot turns green, test fascists like JP (or, on off days, myself) can revert changes which break it, and enforce a higher standard of quality for portability of Twisted code. Right now it's hard to convince developers to not make it fail _more_ tests, since hey, it's broken already, right?

6 20

[Twisted-Python] Re: plus mode was Re: how winnt fileops work and what to do about it
by Brian Warner 31 Dec '05

31 Dec '05

glyph(a)divmod.com writes: > It seems like we can work around this more easily than that, considering > that flush and seek are available from Twisted; the file object causing > problems in the tests is being returned from the open() method of a > FilePath object, if I understand it correctly. FilePath could include the > workaround far in advance of Python deciding to. I'm pretty sure that the real problem we're trying to solve here is caused by a stuck process keeping a .pyd file open. Indeed, if you look at the buildslave's logs, you'll see the exception is as follows: exceptions.OSError: [Errno 13] Permission denied: 'c:\\buildslave\\win32-win32er\\W32-full2.4-win32er\\Twisted\\twisted\\protocols\\_c_urlarg.pyd' So changing the way Twisted or its unit tests open a file is just not going to help. What matters is the way python (or.. pyrex?) opens a file. (for context: the buildbot is currently configured to do SVN checkout/updates into one directory, then copy the tree into a second directory, then run tests on that second directory. This mode='copy' approach uses 'svn update' to minimizes network bandwidth, but at the expense of doubling the disk usage with the extra copy. At the beginning of each build, the buildslave deletes the second directory with a function named rmdirRecursive() that bear provided, which does a chmod() of any mis-permissioned files before deleting them. It was an os.remove() inside this rmdirRecursive which raised the exception). I've run into a similar problem in the past, under Solaris, using NFS, where a test case spawned off a daemon process which then didn't die when it was supposed to, somehow held on to a file (I think solaris won't let you delete a file that is being used as the backing store for an executable), and that prevented the unlink() from succeeding. In that environment, I just renamed the top-level directory to something unique, spawned off an 'rm -rf' into the background to delete the old directory if it was possible, then continued on with the next build. If the code had to try too hard to come up with a unique name, it would flag a warning that there might be a stuck process somewhere. Perhaps we could use something similar here? Of course, the real fix would be to find a way to let the testing code kill off any stuck processes, but that'll probably be very windows-specific. cheers, -Brian

3 4

Re: [Twisted-Python] Re: plus mode was Re: how winnt fileops work and what to do about it
by Brian Warner 31 Dec '05

31 Dec '05

Ah, sorry, there are two separate bugs, and this thread is dealing with the other one. Nevermind. -Brian (plus, this thread makes me realize that renaming the directory won't work anyway)

1 0

Re: [Twisted-Python] Re: r15451 - Fix test failures under windows by changing the eventual-send operation to
by Brian Warner 31 Dec '05

31 Dec '05

> I think that your eventual-send queue should probably be its own > event-queuing API. If you have: > > callLater(0, foo); callLater(0, bar) > > and you expect ordering between them, something like this is better: > > callLater(0, lambda : foo(); bar()) > or perhaps > myRunQueue.put(foo); myRunQueue.put(bar); callLater(0, myRunQueue.activate) > > You get the idea. The issue here is with multiple eventual-sends being called from other code. Specifically, the context was an ITransport-providing Loopback class, used in newpb unit tests, which needs to decouple transport.write() from the resulting proto.dataReceived(). The write() calls are being made from all over the place, with no good location to coordinate or batch them together. Technically I could rig it so that transport.write() queues the data and then oblige the writing Protocol to invoke transport.flush() (newpb has a place for this to go, to support more datagram-oriented transports someday), but then who's to say that whatever code invokes *that* won't want to run multiple times in a brief period of time? I heartily agree that callLater() is not the appropriate tool for this purpose. A week ago I thought it might be, but at that point I didn't realize that I need the ordering guarantee described above. The implementation of callFromThread() happens to match both my needs and my expectations about efficiency, but of course it isn't documented to provide either, plus this eventual-send functionality should be completely unrelated to threads. -Brian

1 0

[Twisted-Python] Re: Twisted-Python Digest, Vol 21, Issue 39
by Michael Schneider 31 Dec '05

31 Dec '05

On Windows, you might use time.clock() it is more precise (keep with time.time on UNIX Hope this helps, Mike > > >Message: 2 >Date: Sat, 31 Dec 2005 12:42:07 -0500 >From: James Y Knight <foom(a)fuhm.net> >Subject: Re: [Twisted-Python] Re: [Twisted-commits] r15451 - Fix test > failures under windows by changing the eventual-send operation to >To: Twisted general discussion <twisted-python(a)twistedmatrix.com> >Message-ID: <5F60A50B-53F9-4AC2-AECA-681882929FB3(a)fuhm.net> >Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed > > >On Dec 31, 2005, at 11:59 AM, James Y Knight wrote: > > > >>On Dec 31, 2005, at 1:39 AM, Brian Warner wrote: >> >> >> >>>callLater(0) does not guarantee >>>relative ordering of sequentially-scheduled calls, >>> >>> >>> >>Yes it does: order of method call. >> >> >> >> >>>and the windows >>>reactors in fact execute them in random order. >>> >>> >>> >>That is broken. >> >> > >I'm afraid this is my fault. Heapq does not guarantee stable ordering >amongst elements with the same key, and on windows, two successive >calls to time.time() are very likely to return the same number, >because of its poor resolution. The same error could probably occur >on unix/linux systems as well, although currently unlikely to the >point of impossibility. It should be fixable by storing the last time >of an element added to the queue, and adding epsilon if current time >is the same. > >Just as a point of interest, on my system, this loop never finishes: >import time >def f(): > while 1: > if time.time()==time.time(): > break > >But this one does: >from time import time >def f(): > while 1: > if time()==time(): > break > > >James > > > >------------------------------ > >_______________________________________________ >Twisted-Python mailing list >Twisted-Python(a)twistedmatrix.com >http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > >End of Twisted-Python Digest, Vol 21, Issue 39 >********************************************** > > > -- The greatest performance improvement occurs on the transition of from the non-working state to the working state.

3 2

[Twisted-Python] Re: [Twisted-commits] r15451 - Fix test failures under windows by changing the eventual-send operation to
by James Y Knight 31 Dec '05

31 Dec '05

On Dec 31, 2005, at 1:39 AM, Brian Warner wrote: > callLater(0) does not guarantee > relative ordering of sequentially-scheduled calls, Yes it does: order of method call. > and the windows > reactors in fact execute them in random order. That is broken. James

1 1

[Twisted-Python] Qtreactor problem on twisted 2.1
by Umit Oztosun 31 Dec '05

31 Dec '05

Hello, Our application uses twisted as the core network framework. It basically consists of a server and multiple clients. All communication is carried on a twisted.spread.pb channel. We have been using twisted 1.3 since the beginning, we decided to switch to twisted 2.1, the switch was easy but an interesting problem showed up. Our client is based on PyQt, so uses qtreactor. If something goes wrong on the server side, the error is transferred to the client (automagically, thanks to twisted & deferreds), and handled there (with an errback). We warn our users about the failure with a dialog. The interesting part is, with twisted 2.1, at the dialog's exec_loop() call, occasionally the connection between the server and client goes away with the following exception: ------------------------------------------- 2005/12/30 15:51 GTB Standard Time [-] Error in <<class 'twisted.internet.tcp.TLSConnection'> to (u'localhost', 4243) at 1312e50>.doRead() 2005/12/30 15:51 GTB Standard Time [-] Traceback (most recent call last): File "C:\Python24\Lib\site-packages\twisted\internet\defer.py", line 182, in addCallbacks self._runCallbacks() File "C:\Python24\Lib\site-packages\twisted\internet\defer.py", line 307, in _runCallbacks self.result = callback(self.result, *args, **kw) File ".\lq\lqtrxutils.py", line 126, in eb_localFailed File ".\lq\lqdialog.py", line 201, in showDialog --- <exception caught here> --- File "C:\Python24\Lib\site-packages\twisted\internet\qtreactor.py", line 62, in read why = w.doRead() File "C:\Python24\Lib\site-packages\twisted\internet\tcp.py", line 111, in doRead return Connection.doRead(self) File "C:\Python24\Lib\site-packages\twisted\internet\tcp.py", line 347, in doRead return self.protocol.dataReceived(data) File "C:\Python24\Lib\site-packages\twisted\spread\banana.py", line 173, in dataReceived gotItem(self.incomingVocabulary[num]) File "C:\Python24\Lib\site-packages\twisted\spread\banana.py", line 114, in gotItem self.callExpressionReceived(item) File "C:\Python24\Lib\site-packages\twisted\spread\banana.py", line 81, in callExpressionReceived self.expressionReceived(obj) File "C:\Python24\Lib\site-packages\twisted\spread\pb.py", line 567, in expressionReceived raise ProtocolError("Non-list expression received.") twisted.spread.pb.ProtocolError: Non-list expression received. 2005/12/30 15:51 GTB Standard Time [-] <twisted.internet.ssl.Connector instance at 0x0134BE90> will retry in 2 seconds 2005/12/30 15:51 GTB Standard Time [-] Stopping factory <lqclientfactory.lqClientFactory instance at 0x0134BDF0> ------------------------------------------- Line 201 at lqdialog.py reads as result = self.exec_loop() We are using a reconnecting client factory, the application reconnects in a matter of seconds but this behaviour is annoying. As I wrote above, the we had no problem when using 1.3. Any ideas? Regards, and happy new year! Umit Oztosun P.S.: Tested on both Windows XP and Ubuntu Linux (5.10 and 5.04). Disconnection occurs regardless of using or not using SSL.

2 2

[Twisted-Python] SSL client authentication
by dkarbott＠comcast.net 29 Dec '05

29 Dec '05

Is it possible to do client authentication with reactor.listenSSL() and reactor.connectSSL()? If so, are there any examples? Any info would be much appreciated. Regards, -- Darrell

2 1

[Twisted-Python] Weekly Bug Summary
by James Y Knight 28 Dec '05

28 Dec '05

Bug summary ______________________ Since 2005-12-21 07:00:01.630000: Bugs opened: 10 Bugs closed: 5 Total open bugs: 506 (+5) Urgent: 1 (+0) Bug: 241 (+1) Feature: 180 (+3) Wish: 84 (+1) New / Reopened Bugs ______________________ ==== Bug ==== twisted.web.client.HTTPClientFactory sends Host header (and others) more than once (yesterday) http://twistedmatrix.com/bugs/issue1382 opened by trickos [TESTS, TRIAL] t.trial.test.test_output tests fail on win32 select reactor (yesterday) http://twistedmatrix.com/bugs/issue1380 opened by jml [TRIAL] Implementing setUpClass should not require ClassSuite (2 days ago) http://twistedmatrix.com/bugs/issue1379 opened by jml [TRIAL] Fail test suite on import error (4 days ago) CLOSED http://twistedmatrix.com/bugs/issue1377 opened by jml iocpreactor WindowsError: (0, 'De bewerking is voltooid') (3 days ago) http://twistedmatrix.com/bugs/issue1378 opened by ghazel [TRIAL] Organise Trial's unit tests (5 days ago) CLOSED http://twistedmatrix.com/bugs/issue1374 opened by jml ==== Feature ==== [CONCH] hashed host entries in known_hosts for conch (SSH) (4 days ago) http://twistedmatrix.com/bugs/issue1376 opened by count0 hashed host entries in known_hosts for conch (SSH) (4 days ago) http://twistedmatrix.com/bugs/issue1375 opened by count0 [CORE] Support twisted.internet.stdio on Windows (6 days ago) http://twistedmatrix.com/bugs/issue1373 opened by itamarst ==== Wish ==== [CORE] Remove string import (yesterday) http://twistedmatrix.com/bugs/issue1381 opened by therve Bugs Closed ______________________ ==== Bug ==== [TESTS, WEB] twisted.web2.test.test_http calls reactor.iterate() (3 months ago) http://twistedmatrix.com/bugs/issue1186 opened by exarkun, closed by jknight [TRIAL] Fail test suite on import error (4 days ago) http://twistedmatrix.com/bugs/issue1377 opened by jml, closed by jml [TRIAL] Organise Trial's unit tests (5 days ago) http://twistedmatrix.com/bugs/issue1374 opened by jml, closed by jml [TRIAL] Clean up Trial's Runner and Loader concepts (2 weeks ago) http://twistedmatrix.com/bugs/issue1355 opened by jml, closed by jml [CORE, TRIAL] Raising SkipTest in setUp is treated as an error (7 days ago) http://twistedmatrix.com/bugs/issue1372 opened by exarkun, closed by jml

1 0

Re: [Twisted-Python] Allowing a type to be transmitted by jelly?
by Brian Warner 27 Dec '05

27 Dec '05

>>I am pretty sure newpb, which Brian Warner was threatening to merge any >>day now, will support this much better than PB does. Yup. In oldpb, object graphs are serialized through recursive calls to Jelly, which has a big switch statement (in twisted.spread.jelly._Jellier.jelly) that uses a different one-object serializer for each type. To teach oldpb how to handle a new type, you have to modify this function, and the corresponding unjellier, and then arrange for your Broker to use the new _Jellier classes instead of the normal ones. Subclassing pb.Copyable does away with this, but as you've noticed it is both slightly inefficent (particularly if you're using __slots__ anyway) and requires extensive surgery or pre-marshalling to use with third-party code that doesn't happen to inherit from pb.Copyable. In newpb, object graphs are serialized through iterative calls to a collection of Slicers. Each time a new object needs to be serialized, the enclosing Slicers help decide (based upon both the type of the object and any adapters that have been registered for it) which new Slicer to use. To teach newpb how to serialize a new type, you register an adapter for it (e.g. registerAdapter(BooleanSlicer, bool, ISlicer)). To teach newpb how to unserialize the resulting stream, you define an Unslicer class (there is some metaclass magic which auto-registers the Unslicer for you.. I'm still going back and forth on the best way to do this). The advantage of the adapter/registration scheme is that you don't have to modify the inheritance set of the classes you want to serialize, so third-party code is easy to accomodate without surgery or pre-marshalling. Pre-marshalling is a serious nuisance, because to do it correctly means looking inside every possible container for further instances of the type that needs to be converted into something jelly can handle, which is just as much work as the final serialization itself. Take a look at branches/newpb-again, in the doc/pb directory, for some examples and descriptions of how to use newpb. The code itself is in the twisted/pb directory (unless we come up with a different name for it). It's still in development: it was in the trunk for a few hours last week, but hopefully I'll get the remaining test failures resolved and it will go back into the trunk some time this week. cheers, -Brian

1 0

Re: [Twisted-Python] Receiving a root from a pb client
by Brian Warner 27 Dec '05

27 Dec '05

william.lewis(a)uk.bnpparibas.com writes: > I am using pb to communicate in a client-server configuration. My client > inherits from PBClientFactory. My server has a remote_take_root() method > by which the client can connect to the server and pass in its root, thus > allowing two-way communication. Is there anyway for the server to > re-establish the connection (should it go down) in this setup, or is > this the sole responsibility of the client? The other posters in this thread have it exactly right. The RemoteReference that is received by your remote_take_root() method does not carry enough information to allow anyone else (including your server) to initiate a new connection to the Broker which hosts it. In this configuration, the "client" side (specifically the side which initiates the PB connection with a PBClientFactory) must be responsible for any sort of reconnection. Incidentally, in newpb this won't necessarily be the case. Both ends can listen on a TCP socket, and the RemoteReference that you receive may contain enough information to connect to its source. It is still a good idea to make exactly one side be responsible for things like reconnection, though, because otherwise you get a bunch of ugly race conditions as both sides attempt to connect at the same time. (newpb handles this for you, so you'll only get a single connection between the two Tubs, but it's still ugly). Also, the application level should be aware of the reconnection events, otherwise it becomes more difficult to be sure about which methods have been successfully invoked and which have not. cheers, -Brian

1 0

[Twisted-Python] Python 2.2 Support
by Jonathan Lange 27 Dec '05

27 Dec '05

Should we drop Python 2.2 support in Subversion head for Twisted 2.2? - Debian stable now comes with Python 2.3 - Twisted 2.1 has been released with Python 2.2 support Dropping Python 2.2 support has some good consequences: - We get to delete code - An entire class of bugs gets eliminated - Disc space can be freed on the muon buildslave Some bad consequences: - People wanting to run Twisted 2.2 (or svn) on ancient versions of Python won't be able to. cheers, jml

13 30

[Twisted-Python] Receiving a root from a pb client
by william.lewis＠uk.bnpparibas.com 26 Dec '05

26 Dec '05

Hi, I am using pb to communicate in a client-server configuration. My client inherits from PBClientFactory. My server has a remote_take_root() method by which the client can connect to the server and pass in its root, thus allowing two-way communication. Is there anyway for the server to re-establish the connection (should it go down) in this setup, or is this the sole responsibility of the client? Thanks William Lewis This message and any attachments (the "message") is intended solely for the addressees and is confidential. If you receive this message in error, please delete it and immediately notify the sender. Any use not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. The internet can not guarantee the integrity of this message. BNP PARIBAS (and its subsidiaries) shall (will) not therefore be liable for the message if modified. ********************************************************************************************** BNP Paribas Private Bank London Branch is authorised by CECEI & AMF and is regulated by the Financial Services Authority for the conduct of its investment business in the United Kingdom. BNP Paribas Securities Services London Branch is authorised by CECEI & AMF and is regulated by the Financial Services Authority for the conduct of its investment business in the United Kingdom. BNP Paribas Fund Services UK Limited is authorised and regulated by the Financial Services Authority

3 2

[Twisted-Python] Custom transports
by Adam Atlas 24 Dec '05

24 Dec '05

Hello, What would I do to implement a custom transport within the Twisted framework? That is, I'd like to be able to add custom listen___ and connect___ methods to the reactor, so that existing protocols can run over a new backend transport protocol. What API would I use for this? -- Adam

3 2

[Twisted-Python] Integrate other API that uses select
by Wolfgang Eichler 23 Dec '05

23 Dec '05

Hi, newbie question: I would like to integrate another c-api, that uses select ( for some tcp - connection ); If I could extend this c-api, to question for the bit-arry it uses for the select, could I enter this in the twisted framework to get the control, if some event for these sockets are coming in and then notifiy the c-api to handle the events on his sockets. The aim is to keep the single waiting point in twisted and not being forced to poll on the c-api. Thanks for any help ... Wolfgang

2 1

Re: [Twisted-Python] Python 2.2 Support
by Ed Suominen 23 Dec '05

23 Dec '05

As a humble user, that makes perfect sense to me. I wonder how many people really are stuck with Python 2.2 at this point, through ignorance (not necessarily their own, but perhaps of people higher up) or otherwise. Best regards, Ed Suominen On Fri, 2005-12-23 at 07:23 -0600, twisted-python-request(a)twistedmatrix.com wrote: > I'm going to vote for dropping Python 2.2 support in Twisted trunk > now. Twisted 2.1 is a fine release to use if you want to use an > ancient version of python. Keeping compat is a pain in the ass. And > lest exarkun complain about lack of enthusiasm: I'll declare myself > quite enthusiastic about the concept. :) > > James

1 0

[Twisted-Python] Trial and ImportErrors
by Jonathan Lange 23 Dec '05

23 Dec '05

Hello, For a while I've been thinking of changing Trial so that import errors flag the test suite as failed. This would mean the buildbot would go red if things fail to import. What do you all think? jml

3 3

[Twisted-Python] Re: [Twisted-commits] r15363 - Merge the DAV branch to trunk.
by Jonathan Lange 23 Dec '05

23 Dec '05

On 12/22/05, Wilfredo Sanchez <wsanchez(a)wolfwood.twistedmatrix.com> wrote: > Author: wsanchez > Date: Wed Dec 21 17:28:58 2005 > New Revision: 15363 > > Added: > trunk/twisted/python/pymodules/LICENSE-pypy.txt > - copied unchanged from r15360, branches/wsanchez/dav/twisted/python/pymodules/LICENSE-pypy.txt > trunk/twisted/python/pymodules/datetime.py > - copied unchanged from r15360, branches/wsanchez/dav/twisted/python/pymodules/datetime.py > trunk/twisted/web2/dav/ > - copied from r15360, branches/wsanchez/dav/twisted/web2/dav/ <snip> twisted.web2.dav.test.util uses tempfile.mkdtemp, which is unavailable in Python 2.2. We still support Python 2.2 (ungh!).

2 1

[Twisted-Python] python crash when running sample code from Twisted Book....
by John Case 22 Dec '05

22 Dec '05

Hello! I am working through the Twisted Network Programming Essentials book by Abe Fettig, using python24 and twisted 2.1 on a Win2k sp4 Server platform. Python crashes with a memory reference error on 'reactor.run()' when I run this code (connectiontest.py from Chap 2): Any suggestions greatly appreciated!!! **** from twisted.internet import reactor, defer, protocol class CallbackAndDisconnectProtocol(protocol.Protocol): def connectionMade(self): self.factory.deferred.callback("Connected!") self.transport.loseConnection() class ConnectionTestFactory(protocol.ClientFactory): protocol = CallbackAndDisconnectProtocol def __init__(self): self.deferred = defer.Deferred() def clientConnectionFailed(self, connector, reason): self.deferred.errback(reason) def testConnect(host, port): testFactory = ConnectionTestFactory() reactor.connectTCP(host, port, testFactory) return testFactory.deferred def handleSuccess(result, port): print "Connected to port %i" % port reactor.stop() def handleFailure(failure, port): print "Error connecting to port %i: %s" % ( port, failure.getErrorMessage()) reactor.stop() if __name__ == "__main__": import sys #if not len(sys.argv) == 3: # print "Usage: connectiontest.py host port" # sys.exit(1) host = 'localhost' port = 82 connecting = testConnect(host, port) connecting.addCallback(handleSuccess, port) connecting.addErrback(handleFailure, port) reactor.run()

1 0

[Twisted-Python] Analog to asyncore.dispatcher.handle_write()?
by dkarbott＠comcast.net 22 Dec '05

22 Dec '05

Hi, I'm new to twisted so this may be a simple question. I want to write a twisted.internet.protocol.Protocol implementation which doesn't write too far ahead of the socket it's writing into. How do I get a notification when the underlying socket is ready to write more data? Any info would be much appreciated. -- Darrell

2 1

[Twisted-Python] Allowing a type to be transmitted by jelly?
by Mike C. Fletcher 22 Dec '05

22 Dec '05

This is one of those problems where I've got a solution, but it *seems* like it should be something that can be done that's a little less drastic. Here's the problem: Want to expose a simple data-type (in this case a sub-class of tuple, the PySNMP OID object) to allow for transmission across pb queries and returns. Now, I've tried quite a number of iterations on this, from: jelly.globalSecurity.allowInstancesOf( oid.OID, ) through using classes like this: class PBTypeAllowingServerFactory( pb.PBServerFactory ): """Allow various types to be transported""" def buildProtocol(self, addr): """Return a Broker attached to me (as the service provider). """ broker = pb.PBServerFactory.buildProtocol( self, addr ) broker.security.allowInstancesOf( oid.OID, ) return broker class PBTypeAllowingClientFactory( pb.PBClientFactory ): def clientConnectionMade(self, broker): """Expand the types allowed to be communicated""" broker.security.allowInstancesOf( oid.OID, ) return pb.PBClientFactory.clientConnectionMade( self, broker ) but whenever I try using jelly for the encoding it complains that: Traceback (most recent call last): File "/home/mcfletch/pylive/Twisted/twisted/internet/defer.py", line 190, in addCallbacks self._runCallbacks() File "/home/mcfletch/pylive/Twisted/twisted/internet/defer.py", line 358, in _runCallbacks self.result = callback(self.result, *args, **kw) File "/home/mcfletch/pylive/cinemon/cinespread/metaperspective.py", line 216, in callWhenFree return perspective.callRemote( *args, **named ).addErrback( File "/home/mcfletch/pylive/Twisted/twisted/spread/pb.py", line 382, in callRemote _name, args, kw) --- <exception caught here> --- File "/home/mcfletch/pylive/Twisted/twisted/spread/pb.py", line 857, in _sendMessage netArgs = self.serialize(args, perspective=perspective, method=message) File "/home/mcfletch/pylive/Twisted/twisted/spread/pb.py", line 811, in serialize return jelly(object, self.security, None, self) File "/home/mcfletch/pylive/Twisted/twisted/spread/jelly.py", line 896, in jelly return _Jellier(taster, persistentStore, invoker).jelly(object) File "/home/mcfletch/pylive/Twisted/twisted/spread/jelly.py", line 480, in jelly sxp.append(self.jelly(item)) File "/home/mcfletch/pylive/Twisted/twisted/spread/jelly.py", line 476, in jelly sxp.append(self.jelly(item)) File "/home/mcfletch/pylive/Twisted/twisted/spread/jelly.py", line 511, in jelly raise InsecureJelly("Type not allowed for object: %s %s" % twisted.spread.jelly.InsecureJelly: Type not allowed for object: <class 'cinemon.snmp.oidobject.OID'> .1.3.6.1.2.1.1.4.0 to hack around it I've built my own generic (security restricted) pickle-like system, and freeze the parameters and return values before passing them across pb... which just seems *so* wrong. My system is almost as large as jelly, and is just entirely redundant code other than its allowing me to add new classes to the set allowed. Anyway, I'm sure I'm once again missing something blindingly obvious, but given that I missed the logical equivalent of turning on the power switch earlier I thought I'd ask if someone had a recipe for this type of operation. BTW, this seems like something that should be covered in the tutorial for pb, as it would seem like a common need to add a low-level object-type or two to the set allowed. Freezer-isn't-so-wonderful-I-want-to-use-it-pointlessly y'rs, Mike -- ________________________________________________ Mike C. Fletcher Designer, VR Plumber, Coder http://www.vrplumber.com http://blog.vrplumber.com

5 7

[Twisted-Python] Twisted conch bad key signatures
by Phil Mayers 21 Dec '05

21 Dec '05

All, I've got an SSH server (Extremeware 7.3 router) which OpenSSH can SFTP into without problem but Twisted Conch reports: 2005/12/20 17:50 GMT [-] Log opened. 2005/12/20 17:50 GMT [-] Enabling Multithreading. 2005/12/20 17:50 GMT [Transport,client] kex alg, key alg: diffie-hellman-group1-sha1 ssh-dss 2005/12/20 17:50 GMT [Transport,client] client->server: 3des-cbc hmac-md5 none 2005/12/20 17:50 GMT [Transport,client] server->client: 3des-cbc hmac-md5 none 2005/12/20 17:50 GMT [Transport,client] host key fingerprint: 51:11:e1:76:89:f8:cd:af:8c:09:42:9e:37:a8:0a:36 2005/12/20 17:50 GMT [Transport,client] Disconnecting with error, code 3 reason: bad signature 2005/12/20 17:50 GMT [Transport,client] connection lost 2005/12/20 17:50 GMT [Transport,client] Stopping factory <twisted.internet.protocol._InstanceFactory instance at 0xb71b6b0c> 2005/12/20 17:50 GMT [-] Received SIGINT, shutting down. 2005/12/20 17:50 GMT [-] Main loop terminated. (cftp does the same). What debugging do I need to do to identify the cause?

2 2

[Twisted-Python] Re: [Twisted-commits] r15355 - Add twisted.python.pymodules.datetime and use it if "import datetime" fails.
by Itamar Shtull-Trauring 21 Dec '05

21 Dec '05

On Tue, 2005-12-20 at 18:26 -0700, Wilfredo Sanchez wrote: > +try: > + import datetime > +except ImportError: > + import twisted.python.pymodules.datetime as datetime twisted.python.compat is an even better place, so other modules can benefit (check out how other compat stuff is handled).

1 0

[Twisted-Python] Weekly Bug Summary
by James Y Knight 21 Dec '05

21 Dec '05

Bug summary ______________________ Since 2005-12-14 07:00:01.820000: Bugs opened: 10 Bugs closed: 9 Total open bugs: 501 (+1) Urgent: 1 (+0) Bug: 240 (+2) Feature: 177 (-1) Wish: 83 (+0) New / Reopened Bugs ______________________ ==== Bug ==== [CORE, TRIAL] Raising SkipTest in setUp is treated as an error (12 hours ago) http://twistedmatrix.com/bugs/issue1372 opened by exarkun [CORE, TESTS] Possible race condition in test_tcp (yesterday) CLOSED http://twistedmatrix.com/bugs/issue1370 opened by jml [CORE, TESTS] Possible race condition in test_tcp (yesterday) http://twistedmatrix.com/bugs/issue1371 opened by jml iocpreactor (udp) socket.error: (10049, "Can't assign requested address") (yesterday) http://twistedmatrix.com/bugs/issue1369 opened by ghazel [CORE, FTP] FTP server tests are a mess (2 days ago) http://twistedmatrix.com/bugs/issue1368 opened by exarkun [CORE, TRIAL] If setUp returns an unfired Deferred, test methods cannot use wait() (2 days ago) http://twistedmatrix.com/bugs/issue1367 opened by exarkun [CORE, DOCUMENTATION] TwistedQuotes documentation references external source files (2 days ago) http://twistedmatrix.com/bugs/issue1366 opened by exarkun select.error: (10022, 'Windows Error 0x2726') (3 days ago) http://twistedmatrix.com/bugs/issue1365 opened by ghazel ==== Feature ==== [CORE] [PATCH] re-installation of the same reactor (6 days ago) http://twistedmatrix.com/bugs/issue1364 opened by antony [CORE] [PATCH] broadcasting over udp causes warning (6 days ago) http://twistedmatrix.com/bugs/issue1363 opened by antony Bugs Closed ______________________ ==== Bug ==== Separate reporter result handling from output (2 weeks ago) http://twistedmatrix.com/bugs/issue1349 opened by jml, closed by jml [CORE, TESTS] Possible race condition in test_tcp (yesterday) http://twistedmatrix.com/bugs/issue1370 opened by jml, closed by jml [CORE, TRIAL] Test case run() methods to use Deferreds internally (3 weeks ago) http://twistedmatrix.com/bugs/issue1341 opened by jml, closed by jml [TRIAL] Trial doesn't cut tracebacks properly (13 months ago) http://twistedmatrix.com/bugs/issue773 opened by jknight, closed by jml [TRIAL] [PATCH] Traceback running a test with broken setUp method (1 week ago) http://twistedmatrix.com/bugs/issue1359 opened by headly, closed by jml [BACKPORT, WEB2] twisted.web2.stream and wsgi (7 days ago) http://twistedmatrix.com/bugs/issue1362 opened by mkerrin, closed by jknight ==== Feature ==== Trial spews color codes by default on win32 (4 months ago) http://twistedmatrix.com/bugs/issue1141 opened by TimothyFitz, closed by jml [DOCUMENTATION, TRIAL] Draft release notes for the recent and upcoming Trial changes (3 months ago) http://twistedmatrix.com/bugs/issue1172 opened by jml, closed by jml [TRIAL] TASK: Deprecate setUpClass/tearDownClass. Provide replacement (2 months ago) http://twistedmatrix.com/bugs/issue1206 opened by jml, closed by jml

1 0

[Twisted-Python] Problem w/ TNPE Example on Windows
by Viktor Ransmayr 21 Dec '05

21 Dec '05

Hello, I'm new to Twisted and just started reading "Twisted Network Programming Essentials". The example "dataforward.py" in chapter 2 creates a traceback and an import error, if executed on my laptop running WinXP/Pro-SP2, using Python-2.4.2 and Twisted-2.1.0 The import error complains about a missing module "fcntl" required by ...\twisted\internet\fdesc.py Reading the python documentation states that the "fcntl" module is available on Unix only. - Does that mean, that this example is working on Unix only or what am I missing? Kind regards, Viktor Ransmayr

6 8

[Twisted-Python] I'm missing something on SSL-encrypted PB connections...
by Mike C. Fletcher 21 Dec '05

21 Dec '05

Hi all, I'm working on refining a load-sharing mechanism to run across a number of pb connections (sharing processor time and/or access to a network). One of the requirements I have is that I must be able to run the pb connections over encrypted channels. However, when I try to set up an SSL-encrypted server (using the same mechanisms I've used for setting up SSL-encrypted Nevow sites and the same ones in the echo examples) I always get a connection refused error on the client when I try to connect. I'm using self-generated keys from: openssl genrsa > privkey.pem openssl req -new -x509 -key privkey.pem -out cacert.pem -days 1000 I'm guessing this has to be some incredibly stupid error, but I'm not seeing it in a few hours of poking now, so this is basically a request for more eyes. I've reduced the failing code down to a modification of the pbecho* example code (attached). With keys generated as above in the current directory the server starts fine, but the client always receives: Failure: twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused. when it tries to connect. The only other code I've found trying to use SSL + PB is connecting it up to a multi-service of some sort, which I don't *think* should be necessary. Thoughts appreciated, Mike -- ________________________________________________ Mike C. Fletcher Designer, VR Plumber, Coder http://www.vrplumber.com http://blog.vrplumber.com

2 2

[Twisted-Python] Twisted SCP
by Phil Mayers 20 Dec '05

20 Dec '05

All, This is a very very horrible bit of code (stupid undocumented RCP junk), but a basic single-file remote -> local SCP can be done using the attached code. It's clearly as dumb as a box of rocks, but seems to work against OpenSSH and Cisco IOS, so is fine for what we need. If anyone knows why the sftp stuff in it doesn't work that would be useful. Anyway, hope it's helpful to someone. If I get time over or after xmas I'll clean it up - it seems pretty clear to me how to make the code pretty modular without it descending into the mess that OpenSSH and Putty's SCP/SFTP code has become (with the greatest of respect to the developers involved of course) #!/usr/bin/env python from twisted.conch.ssh import transport, userauth, connection, channel from twisted.conch.ssh.common import NS from twisted.internet import defer, protocol, reactor from twisted.python import log import sys, os, getpass USER, PASS, HOST, CMD, SRC, DST = None, None, None, None, None, None # The "Transport" is the crypto layer on top of port 22 class Transport(transport.SSHClientTransport): def verifyHostKey(self, hostKey, fingerprint): # For info only log.msg('host key fingerprint: %s' % fingerprint) # FIXME: this is insecure return defer.succeed(1) def connectionSecure(self): # Once we're secure (server key valid) we ask for userauth service # on a new "connection" object self.requestService(UserAuth(USER, Connection())) class UserAuth(userauth.SSHUserAuthClient): def getPassword(self): # Ack! Globals! return defer.succeed(PASS) def getPublicKey(self): # returning None means always use username/password auth return class Connection(connection.SSHConnection): def serviceStarted(self): # Once userauth has succeeded we ask for a channel on this # connection self.openChannel(ScpChannel(2**16, 2**15, self)) class XferChannelBase(channel.SSHChannel): name = 'session' state = None todo = 0 buf = '' def openFailed(self, reason): log.err(reason) def channelOpen(self, data): # Might display/process welcome screen self.welcome = data # We might be an SCP or SFTP requests if 'scp' in CMD or CMD.startswith('/'): kind = 'exec' else: kind = 'subsystem' # Call our handler d = self.conn.sendRequest(self, kind, NS(CMD), wantReply=1) d.addCallbacks(self.channelOpened, log.err) def closed(self): self.loseConnection() reactor.stop() class SftpChannel(XferChannelBase): def channelOpened(self, data): log.msg("channelOpened: %r" % (data,)) self.client = filetransfer.FileTransferClient() self.client.makeConnection(self) self.dataReceived = self.client.dataReceived d = self.client.openFile(SRC, filetransfer.FXF_READ, {}) d.addCallbacks(self.fileOpened, log.err) def fileOpened(self, rfile): rfile.getAttrs().addCallbacks(self.fileStatted, log.err, (rfile,)) def fileStatted(self, attrs, rfile): rfile.readChunk(0, 4096).addCallbacks(self.did_read, log.err, (rfile, 0, attrs['size'])).addCallback(self.done) def did_read(self, data, f, pos, todo): if len(data)>todo: log.msg("got %i bytes more than expected, trimming" % (len(data)-todo,)) data = data[:todo] DST.write(data) todo -= len(data) pos += len(data) if todo<=0: return pos return f.readChunk(pos, 4096).addCallbacks(self.did_read, log.err, (f, pos, todo)) def done(self, l): log.msg("done %i bytes" % (l,)) self.loseConnection() class ScpChannel(XferChannelBase): def channelOpened(self, data): # once the scp is exec'ed, start the SCP transfer self.write('\0') # we're a state machine self.state = 'waiting' def dataReceived(self, data): #log.msg('dataReceived: %s %r' % (self.state, data)) # What we do with the data depends on where we are if self.state=='waiting': # we've started the transfer, and are expecting a C # Coctalperms size filename\n # might not get it all at once, buffer self.buf += data if not self.buf.endswith('\n'): return b = self.buf self.buf = '' # Must be a C if not b.startswith('C'): log.msg("expecting C command: %r" % (self.buf,)) self.loseConnection() return # Get the file info p, l, n = b[1:-1].split(' ') perms = int(p, 8) self.todo = int(l) log.msg("getting file %s mode %s len %i" % (n, oct(perms), self.todo)) # Tell the far end to start sending the content self.state = 'receiving' self.write('\0') elif self.state=='receiving': # we've started the file body #log.msg('got %i bytes' % (len(data),)) if len(data)>self.todo: extra = data[self.todo:] data = data[:self.todo] if extra!='\0': log.msg("got %i more bytes than we expected, ignoring: %r" % (len(extra), extra)) DST.write(data) self.todo -= len(data) if self.todo<=0: log.msg('done') self.loseConnection() else: log.err("dataReceived in unknown state: %r" % (self.state,)) def usage(ex): print >>sys.stderr, """%s: [user[:pass]@]hostname:sourcefile destfile""" % (sys.argv[0],) if ex: sys.exit(ex) if __name__=='__main__': args = sys.argv[1:] if len(args)<2: usage(1) SRC = args[0] DST = args[1] if '@' in SRC: USER, SRC = SRC.split('@', 1) else: USER = os.environ['USERNAME'] if not ':' in SRC: usage(1) HOST, SRC = SRC.split(':', 1) if ':' in USER: USER, PASS = USER.split(':', 1) else: PASS = getpass.getpass('password for %s@%s: ' % (USER, HOST)) DST = open(DST, 'wb') if 'sftp' in sys.argv[0]: CMD = 'sftp' else: CMD = 'scp -f %s' % (SRC,) protocol.ClientCreator(reactor, Transport).connectTCP(HOST, 22) log.startLogging(sys.stderr) reactor.run()

1 0

[Twisted-Python] xpost: multiple job openings
by Paul G 19 Dec '05

19 Dec '05

folks, i'm crossposting this to twisted-python-l and twisted-web-l for the benefit of those who don't read both. apologies to those who will consequently have to read this twice. --- We've got multiple job openings for great engineers familiar with python and twisted, senior and junior positions available. We're building a small team of folks who like working hard, solving interesting problems, writing clean code and having fun doing it. If you got into coding because that's where the money was in the '90s, please skip this one and see my next job posting where I'm recruiting bodyguards for work in the Persian Gulf. The dev team is run by an engineer ('run' in the sense that I shield you from the crazy business folk), so it is a strict meritocracy. You will be working on rebuilding/refactoring an existing PHP (ugh) web application into python services built on top of twisted as well as implementing new features. This is a high volume system, with millions of transactions a day - if you dig concurrency, you won't be bored. Must be able to handle the excitement ;) Requirements: * We're looking for great engineers - while your ability to discuss Wittgenstein over our 5th beer is welcome, a piece of paper^W^W^Wdiploma is not required. * You should be old enough not to get us in trouble with Amnesty International for child exploitation. * Experience working on hobby, open source and commercial projects counts with me. * Python/Twisted and SQL a must. * Reading/grokking bad PHP code a big plus. Writing PHP code useful, since we'll be hacking in the twisted/python bits over time. * Javascript/DHTML/AJAX(ugh, buzzowrd) familiarity needed in at least one hire, so a plus. * Experience with C and ASM good in that it is a good predictor of your understanding of memory allocation and performance issues. * Don't mention Java, unless it has caused a medical (eg mental health) problem we should know about. <grin> * Understanding of performance and scalability principles and issues. Work environment: * Onsite in NYC/NY Metro area strongly preferred. * Don't let the above deter you - we can make telecommuting work under the right conditions. * Competitive compensation - I know what good people are worth and have managed to beat that into the management with a clue-by-four. * Flex time - relaxed cluture - getting your job done is all that matters. --- Shoot me a resume, CV, writeup of what you've done, absolutely anything. Put [twisted-resume] in the subject. Include your phone number, time zone and a good time to call. I'll respond to everyone. cheers, -p

1 0

[Twisted-Python] Is it necessary to create a new reactor to daemonize a File Alteration Monitoring system
by Kaladan 19 Dec '05

19 Dec '05

Hello, I manage to write a file system monitoring daemon based on gamin ( http://gnome.org/~veillard/gamin/) thanks to twisted API. Therefore I have created a new reactor, which inherit from the ReactorBase. In order to use the twistd application runner I have a GaminService which define a simple StartService. Am I in the good way ? The code is in attachment -- Olivier Guilloux

2 1

[Twisted-Python] Some beginner questions about "twisted.names.client" and ".tac" environment
by Jesus Cea 18 Dec '05

18 Dec '05

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Twisted 2.1, twisted.named 0.2, here. I'm taking my first steps with Twisted (documentation -inexistence- nightmare :-), and my first project will be a bulk mailer as the backend of my mailing list system. The application would take the message and the subscriber list and a) resolve the MX for the domains and b) connect to the MX's and send the message, trying to minimice traffic sending a single envelope for several recipients sharing the domain or the MX's. I'm doing currently the DNS stuff. The result are promising, resolving about 200 domains per second in a 1.4GHz P4, so my biggest mailing list (about 31500 unique domains, múltiple subscribers per domain) is "resolved" in less than three minutes. Nice so far. The demo code (2Kbytes) is the following (if I'm violating the rules posting this code, please tell me): ===== # File "dns.tac" from twisted.application import service application = service.Application("DNS test") import time t=time.time() class resolucion(object) : def __init__(self,dominio) : from twisted.names import client d = client.lookupMailExchange(dominio,timeout=(60,)) d.addCallbacks(self._cbMailExchange, self._ebMailExchange) self.dominio=dominio def _cbMailExchange(self,results): # Callback for MX query global aun_pendientes aun_pendientes-=1 if not aun_pendientes : print "OK",time.time()-t return from twisted.internet import reactor reactor.stop() return if not len(pendientes) : return resolucion(pendientes.pop()) from twisted.names.dns import QUERY_TYPES for i in results[0] : n=i.payload.name tipo=QUERY_TYPES[i.payload.TYPE] if tipo=="MX" : return p=i.payload.preference print n,p, for j in results[2] : if n==j.name : print j.payload.dottedQuad(),"(%d)" %j.ttl break else : print "???" elif tipo=="CNAME" : redirigidos.append((self.dominio,i.payload.name)) def _ebMailExchange(self,failure): # Error callback for MX query global aun_pendientes aun_pendientes-=1 if not aun_pendientes : print "ERROR",time.time()-t return from twisted.internet import reactor reactor.stop() return if not len(pendientes) : return resolucion(pendientes.pop()) print "XXX",self.dominio print 'Lookup failed:' failure.printTraceback() pendientes=[] redirigidos=[] f=open("domain_list") for i in f : pendientes.append(i) aun_pendientes=len(pendientes) concurrencia=1000 for i in pendientes[:concurrencia] : resolucion(i) from twisted.names import client client.theResolver.resolvers[-1].dynServers=[('127.0.0.1', 53)] # client.theResolver.resolvers=[client.theResolver.resolvers[-1]] pendientes=pendientes[concurrencia:] ===== I launch the code as "twistd -ny dns.tac". The demo does 1000 resolutions in parallel. If you experiment with the code, reduce the value. Questions: 1. I get a warning: "[Uninitialized] /usr/local/lib/python2.4/site-packages/twisted/names/dns.py:1227: exceptions.DeprecationWarning: Deferred.setTimeout is deprecated. Look for timeout support specific to the API you are using instead." I'm using, the native "twisted.names" timeout API, as far as I know... 2. By default "twisted.names.client" uses the "/etc/resolv.conf" file to know which nameservers to use. I, nevertheless, want to use a particular nameserver, so: 2.1. I couldn't to find an appropiate API. I had to do a "hack", reading the "twisted.names" core to know implementation details: "client.theResolver.resolvers[-1].dynServers=[('127.0.0.1', 53)]" 2.2. The previous "hack" is only effective for future "twisted.names.client" instances. The previous ones use the "/etc/resolv.conf" entries. Putting the "hack" code before any instance creation doesn't work. 2.3. While reading the framework code, I saw that "client" uses a resolver chain: host, cache, network. But the cache is initially clear (of course) and NEVER ever gets populated, so we are not using it but checking missing entries eats CPU: 155 seconds for the unchanged code, 125 seconds if I drop the host and cache resolvers. A caching client would be very nice, if the client is long running (my original idea). 2.4. The resolution failure code is only called if the resolution timeouts. But if the domain doesn't exists, the code called is the "success" one, with a "nil" answer. So we can't diferenciate between inexistant domains and inexistant RRs. 3. How can I stop this ".tac"?. If I do "reactor.stop()", I get an infinite error, repeated forever: ===== [twisted.names.dns.DNSDatagramProtocol (UDP)] Traceback (most recent call last): File "/usr/local/lib/python2.4/site-packages/twisted/python/log.py", line 43, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/local/lib/python2.4/site-packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/local/lib/python2.4/site-packages/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) File "/usr/local/lib/python2.4/site-packages/twisted/internet/selectreactor.py", line 139, in _doReadOrWrite why = getattr(selectable, method)() --- <exception caught here> --- File "/usr/local/lib/python2.4/site-packages/twisted/internet/udp.py", line 113, in doRead data, addr = self.socket.recvfrom(self.maxPacketSize) exceptions.AttributeError: 'Port' object has no attribute 'socket' ===== I must kill -9 the "twistd" process. Thank you for your time and attention. Help greatly appreciated :-) - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea(a)argo.es http://www.argo.es/~jcea/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/_/_/_/ PGP Key Available at KeyServ _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBQ6SNsplgi5GaxT1NAQLMQwP/czYFLQ6+olTCvM0jdmMlaBgwxHsHdvxT /2mhWqtyhIf1Kdh6FioFQq13xqCfZxFIkwuUwTlG+ZmkSYK1iWZEmaS0CGa5YmuA d7miIFfL9Tfa3OLyV1nvqdCR3YtzH/ws9UuJ2DGnACRI++Of6gBVwGlhFDa7S57o wZcsYWAS6Sk= =/Pks -----END PGP SIGNATURE-----

2 5

[Twisted-Python] WSGI Thread-management strategy
by Jim Fulton 16 Dec '05

16 Dec '05

The strategy used by twisted WSGI, as I understand it, doesn't meet our needs. Currently, a thread is created for each request. The total number of threads is throttled, I gather using a general Twisted thread limit. WSGI applications are called as soon as input headers have been received completely. An application may be called before all body input is received. We need application calls to be delayed until all request input has been received, In an application like Zope, application threads are expensive. There are application resources, such as database connections and caches needed by the application threads. To limit resource consumption, we have to limit the number of application threads. Because the number of threads is limited, application threads are scarce and thus valuable resources that need to be used very efficiently. In particular, we don't want to tie up an application thread waiting for request body input. I propose that the default thread-management strategy should be to delay calling an application until all request input has been received. If this isn't the default, then there should at least be an option to get this behavior. (Of course, the buffering strategy needs to be clever enough to switch to a file when the input gets over some size.) Thoughts? Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

2 3

[Twisted-Python] how to run a twisted daemon without tcpserver
by Kaladan 15 Dec '05

15 Dec '05

Can I run a twisted daemon (twistd) without TCPServer, basically I want to run a reactor within an application, something like : application = service.Application('myApp', uid=505, gid=506) reactor.run(application) Thank's -- Olivier Guilloux

1 0

[Twisted-Python] How can a tcp client connect with multi servers?
by Khoa Nguyen 15 Dec '05

15 Dec '05

Hi Xu, How did you solve this problem? I face a similar issue and would appreciate if you post a working sample code Thanks, Khoa

2 1

[Twisted-Python] threading issues with DB connection pooling and ReconnectingClientFactory
by Andreas Poisel 14 Dec '05

14 Dec '05

Hi Twisted Experts, in my application I use ReconnectingClientFactory to handle a TCP connection which is supposed to be permanent. The server side (which is not a twisted application) is very fragile and closes the connection frequently. The client has to do database queries on a regular basis and talk to the server depending on the query results. The client runs into a problem with the following application flow: - Client and server are up and running, the database is queried frequently, server talks to client and vice versa, everything is fine. - The server closes the connection and the client stops it's database queries. - The server is reachable again, the client reconnects. - A database query on the client side is performed and we find something we want to transmit to the server. But we fail because the transport object we use after our database query is not connected (transport.connected == 0 and transport.disconnected == 1). These are my conclusions (maybe that's garbage): The reconnection of the client works perfectly for the main reactor thread. But when we try a database query, we get a cached thread which doesn't have a working transport object anymore. Some experiments with protocol and transport object id()s seem to second this assumption. I've some ideas how to solve this problem: - Empty the thread pool after reconnecting and use "fresh threads" with valid transport objects. I can't find a hint how to do this. - Pass a valid transport object to the thread which just did the database query. I'm not sure how to do this properly. - Pass the query result back to the main thread. Hmmm... I would very much appreciate some hints on how to handle this problem properly. Thank you very much! -- Regards, Andi

2 2

[Twisted-Python] Rails? Twisted versus J2EE? Metaprogramming as self-modifying code?
by John Benson 14 Dec '05

14 Dec '05

Hi, I did some Python and Twisted two years ago got distracted, picked up some IBM WebSphere certifications, got turned off by the verbosity of J2EE and suspected that there was a cleaner way to do it. I just got through Bruce Tate's O'Reilly book /Beyond Java/ and am hearing the siren call of Ruby on Rails. Tate is unwilling to champion Ruby over Python or vice versa, but waxes poetic about Rails. I know there is a Rail-like Python project (Subway?) but I'd like to know if there is a Rail-alike that works with Twisted. Also, I'd like to hear about the advantages of J2EE over Twisted (if any), especially in the scalability and MQ Series connectivity areas. The little project I did with Python and Twisted was a joy and would easily have taken 4X the time and effort in a compiled environment. Now that I've seen what J2EE tries to do, I'm curious to see how Twisted stacks up against the J2EE specialties. Comparisons between Python and Ruby will also interest, especially regarding ease of metaprogramming since this seems to be the arena in which the Next Big Thing will hit software practice with aspect-oriented programming, dependency injection and mixins. Kind of like when COBOL programmers had to learn pointers to deal with C, or when C programmers had to learn about objects to deal with C++ and/or Java. By the way, I remember the campaign to stamp out self-modifying code back during the structured programming wars of the 1970's. Although the benefits of metaprogramming are attractive, isn't metaprogramming just self-modifying code at the object level? If so, should we just say no? If not, what has changed to make it reasonable now?

2 1

[Twisted-Python] Weekly Bug Summary
by James Y Knight 14 Dec '05

14 Dec '05

Bug summary ______________________ Since 2005-12-07 07:00:01.870000: Bugs opened: 9 Bugs closed: 2 Total open bugs: 500 (+7) Urgent: 1 (-1) Bug: 238 (+6) Feature: 178 (+2) Wish: 83 (+0) New / Reopened Bugs ______________________ ==== Bug ==== [CGI, WEB2, WSGI] createCGIEnvironment fails to include ;parameters in PATH_INFO (yesterday) http://twistedmatrix.com/bugs/issue1361 opened by cce [BACKPORT, WEB2] twisted.web2.stream and wsgi (yesterday) http://twistedmatrix.com/bugs/issue1362 opened by mkerrin [TRIAL] [PATCH] Traceback running a test with broken setUp method (2 days ago) http://twistedmatrix.com/bugs/issue1359 opened by headly [TRIAL] Clean up Trial's Runner and Loader concepts (5 days ago) http://twistedmatrix.com/bugs/issue1355 opened by jml [WEB] [PATCH] microdom is not case sensitive when doing laterClosers (2 days ago) http://twistedmatrix.com/bugs/issue1358 opened by Tv [CORE] [PATCH] t.p.usage error messages mangle option case (6 days ago) http://twistedmatrix.com/bugs/issue1354 opened by Tv ==== Feature ==== [MAIL] IMessageDelivery.validateTo API simplification (yesterday) http://twistedmatrix.com/bugs/issue1360 opened by Tv sendFile handy API for FTPClient (3 days ago) http://twistedmatrix.com/bugs/issue1357 opened by dialtone ==== Wish ==== [MAIL] Twisted Mail/OS X performance problem (3 days ago) CLOSED http://twistedmatrix.com/bugs/issue1356 opened by exarkun Bugs Closed ______________________ ==== Urgent ==== [TRIAL] trial loads test methods incorrectly (7 days ago) http://twistedmatrix.com/bugs/issue1353 opened by hagna, closed by jml ==== Wish ==== [MAIL] Twisted Mail/OS X performance problem (3 days ago) http://twistedmatrix.com/bugs/issue1356 opened by exarkun, closed by exarkun

1 0

[Twisted-Python] How can I keep data integrity in multiple threads environment
by Kaladan 12 Dec '05

12 Dec '05

Hello, I am new to twisted and I have a few questions: I have implemented a TCPServer and the associated Factory, which manages data and dispatches treatments (According to the tutorial it's not the best solution but actually this is not the point ;) ) All of the treatments run in threads and thus use shared data. The allData dictionnary is critical since it can be used by multiple threads. How can I keep the allData integrity ? Does Twisted core provide such facilites or do I have to use semaphore or other stuff ? class GestionnaireXXX(protocol.Protocol) def dataReceived(self, data): self.factory.dispatch(data) class GestionnaireXXXFactory(protocol.ServerFactory): def __init__(self): self.allData = {} self.initData() self._dispatcher = { 1: self.alert, 4: self.create, } def initData(self): if os.path.isfile(TEMP) self.allData = aot.unjellyFromSource(file(TEMP, 'r')) def dispatch(self, data): splitedData = data.split('\n') method = self._dispatcher.get(len(splitedData), self.doNothing) reactor.runInThread(method, splitedData) def create(self, *data): # Some treatments on allData ... aot.jellyToSource(file(TEMP, 'w')) def alert(self, *unused): # XXX sem = DeferredSemaphore() sem.acquire() allDataCopy = self.allData.copy() self.allData.clear() sem.release() # End XXX # some long treatments on allDataCopy ... Thank's, -- Olivier Guilloux

3 2

[Twisted-Python] Re: pb references
by william.lewis＠uk.bnpparibas.com 12 Dec '05

12 Dec '05

> In short: you seem to be sending an reference you got through pb to > another pb peer, which is not allowed. > > I hope that helps. Yes, that was indeed the problem. My server was taking a root object from a client and attempting to pass it on to another server as an argument to a remote method. Thanks. I now have a further question: When my server takes the root object from a client it keeps and uses that object, later on, to call remote methods on the client (as described in the docs). What I am not sure about is what happens to the root object if the TCP/IP connection goes down between the server and client. Presumably, nothing will happen until I try to call a remote method on the client? In which case, how can the server attempt to reconnect to the client if it only has the original root object given to it by the client and the connection has temporarily gone down? Or is the client responsible for keeping the connection open? Thanks William Lewis This message and any attachments (the "message") is intended solely for the addressees and is confidential. If you receive this message in error, please delete it and immediately notify the sender. Any use not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. The internet can not guarantee the integrity of this message. BNP PARIBAS (and its subsidiaries) shall (will) not therefore be liable for the message if modified. ********************************************************************************************** BNP Paribas Private Bank London Branch is authorised by CECEI & AMF and is regulated by the Financial Services Authority for the conduct of its investment business in the United Kingdom. BNP Paribas Securities Services London Branch is authorised by CECEI & AMF and is regulated by the Financial Services Authority for the conduct of its investment business in the United Kingdom. BNP Paribas Fund Services UK Limited is authorised and regulated by the Financial Services Authority

1 0

Re: [Twisted-Python] pb references
by Brian Warner 12 Dec '05

12 Dec '05

Micky Latowicki <ml.flex(a)gmail.com> writes: > Now, B is apparently trying to send this RemoteReference to some third > party, C. This is not allowed by twisted, possibly because that would > often require forwarding messages from C to A through B, which can be > inefficient and unreliable. In PB we call this "serializing third-party references". and it's illustrated concisely by the "Granovetter Diagram"[1] as described by papers on distributed computing[2]. I also sometimes call it the "Gift" pattern. Micky has described the situtation accurately, but I wanted to add some detail on the whole third-party reference thing, because it's changing in newpb. Some of this is pretty verbose, probably more information than anybody but a newpb implementor is likely to care about, but I figured I'll throw it out there so people can know what's going on behind the scenes. oldpb refuses to serialize third-party references because there's no reference-identification infrastructure in place to let the recipient establish a direct connection to the originator. Each RemoteReference is really just a connection-local ID number (the CLID), like "4", wrapped by a bunch of code that implements callRemote(). The CLID is only meaningful when it gets looked up by the Broker on the other end of the wire that it's scoped to. So if B wants to send his RemoteReference(A) to C, the only way to send something that C will actually be able to use would be to create a proxy object that provides a whole bunch of methods like the following: def remote_foo(self, *args, **kwargs): return self.refA.callRemote("foo", *args, **kwargs) (with suitable cleverness, you could make this into a generic Proxy object, such that you didn't have to manually duplicate this for every single method that A can respond to. Note that even in newpb this pattern is useful, so we'll figure out some way to make it convenient: Revocable Forwarders, Logging Proxies, and restricted Facets are all design patterns implemented with proxy objects like these). The issue of course is speed and resource consumption. With this proxy in place, B has to be involved in every message between A and C, even if he doesn't want to, adding at least an extra round trip for every method call. Worse yet, B has to carefully watch the arguments and return values of all the methods flying between A and C to see if either end has included a new RemoteReference to some local object. If so, B has to create a new proxy for that object too. (this is one of the obligations of a "Membrane", and is a nuisance that would be nice to avoid). Finally, one of the design goals of oldpb has been reasonable security: make it slightly harder to accidentally leak information or authority. To this end, objects must be declared pb.Referenceable to be remotely callable, instances must inherit from pb.Copyable to be transferred across the wire, and RemoteReferences cannot be sent to third parties. oldpb forces you to make certain designations explicit. For both these reasons (and because it's just too much of a hassle to implement), oldpb takes the easy way out and punts, by disallowing third-party references and forcing the user to implement a proxy if that's what they really want. NEWPB In newpb, things are much better. Assuming you make your Tub accessible to the outside world (by telling it to listen on a port, and by telling it what hostname+portno it's listening on), each Referenceable you publish gets a PB-URL, which is then accessible from any other Tub, anywhere. (the return value from tub.registerReference() is this PB-URL). To "gift" a RemoteReference to a third party, you simply include the RemoteReference as an argument in a callRemote (or return it from a remotely-invoked method), and PB will handle the rest for you. This works by passing a special sequence that includes the PB-URL of the target object, which the receiving side then submits to tub.getReference() to obtain their own RemoteReference before invoking the target method. If you don't want to pass live references around, you can turn any RemoteReference into a "SturdyRef" (which is like an object form of a PB-URL) with rref.getSturdyRef(), and then pass *that*, since SturdyRefs are pb.Copyable . You can also take a SturdyRef and pass it "live" to a remote Tub, by doing something like: rref.callRemote("introduce", sturdy.asLiveRef()) whereupon the recipient's remote_introduce() method will be invoked with a live RemoteReference to the target of the sturdyref. The API is still up in the air, but my plan is for every RemoteReference you pass over a wire to be given an unguessable PB-URL so that it is eligible for being sent as a gift to a third-party Tub. (one possibility is that you have to explicitly publish the ones that you want to be giftable.. there might be a switch to turn this sort of thing on or off, it's a tradeoff of memory consumption versus convenience). Another design question has to do with object lifetime: in the current implementation, when B sends rref(A) to C, B makes sure to keep it's handle on A alive until C confirms it has acquired its own. This improves the chances that C will be able to acquire a live reference, but it also allows a malicious C to force B to keep that rref alive forever, wasting memory. The alternative is to just tell C to take their chances, and maybe they'll wind up with a working RemoteReference, and maybe they'll be unlucky and A will have garbage-collected that object by the time they finish trying to acquire their own. Distributed garbage collection is very tricky. The object-lifetime design issues show up elsewhere too. Should an object, once it gets sent over the wire (any wire), stay alive forever, just in case somebody wrote down its URL and might some day come calling for it? Or should it be allowed to vanish as soon as the last live reference is released? In the current implementation, anything you submit to tub.registerReference() will stay alive forever (where "forever" is equal to the lifetime of the Tub), whereas objects that cross the wire in method invocations get reference-counted and released when there are no more live references to them. Correspondingly, objects submitted to tub.registerReference() get globally-reachable names (PB-URLs), and are therefore eligible for gifting, whereas objects merely crossing the wire do not (and cannot be gifted). The latter needs to change, since *all* objects should be giftable, but it's quite possible that the URL->object table will use weakrefs so that the giftable/non-giftable distinction can be orthogonal to the long-lived/ephemeral distinction. Tyler's trying to convince me to let objects stay alive "forever", get rid of garbage collection and distributed reference counting, and just use the Tub lifetime to reclaim memory or block access to old objects. I'm not sure yet, though, it may become a flag you set on the Tub. One of the issues with allowing Gifts is that it opens up the possibility that methods will be invoked out-of-order. In the present implementation, if you do: a.callRemote("introduce", gift_rref_to_B) a.callRemote("second", 1, 2, 3) then remote_second() will probably be invoked *first*, because the potential call to remote_introduce() is held up waiting for gift_rref_to_B to be turned into a real RemoteReference (which must wait for connection negotiation, etc). I'm thinking that this will be changed (by queueing all method invocations and stalling remote_second until remote_introduce has been invoked), but I might add a flag that lets you choose between the two behaviors. Setting the flag one way lets methods be invoked as quickly as possible, setting it the other way forces them to be called in-order, even if that adds arbitrarily long delays to deal with Gifts. I might add a flag which would disable Gifts altogether, since they're a moderately advanced feature and it might be confusing to have them work so transparently. Finally, there's the complex issue of what kind of ordering guarantees to make regarding methods invoked on gifted references. The E documents[3] describe a situation where you'd like to make certain promises about the relative ordering of methods invoked (by you) on some reference, versus methods invoked (on that same reference) by someone else you've just given that reference to. E (or VatTP, to be precise) suggests a funny kind of proxy behavior called the WormholeOp[4] to provide these promises. At the moment, newpb just punts on the issue, and only makes claims about the relative ordering of messages sent on a *single connection*. If you want to make sure that messages sent to or from different parties happen in some particular order, you must wait for the first to complete before allowing the second to occur. This might be improved in the future (once I understand the issue better, for starters), but for now newpb's ordering guarantees may not be as thorough as certain environments might prefer. (I *think* the only practical consequence is that certain kinds of promise-pipelining optimizations cannot be made, occasionally increasing the number of round trips, but really I don't understand enough yet, and neither of the two people on the planet who *do* understand enough have been able to explain it to me in a way that I can get my head around). Also, the proxy object described above (as the only way to accomplish third-party references in oldpb) is actually quite useful, so newpb will eventually make it easy and cheap to build them. One pattern is the Revocable Forwarder, where you want to extend your authority to somebody else, but you want to be able to cut them off if you change your mind. This is as simple as an object that does the same sort of "def remote_foo(): return self.refA.callRemote('foo')" thing as above, but adds an extra method (exposed in a separate capability) that does 'del self.refA' to turn off all forwarding at once. Another pattern is the restrictive Facet, which forwards access to some (but not all) methods. Either of things might add logging, where the caller doesn't notice anything special, but somebody else gets a record of each method invoked. Anyway, I just wanted to do a bit of a braindump on where third-party references are going in newpb. The summary is that using them is as easy as you think it ought to be. "make simple things simple", and all that :). cheers, -Brian [1]: "Ode to the Granovetter Diagram" http://www.erights.org/elib/capability/ode/index.html [2]: pretty much everything at http://www.erights.org/ [3]: http://www.erights.org/elib/concurrency/partial-order.html [4]: http://www.erights.org/elib/distrib/captp/WormholeOp.html

1 0

[Twisted-Python] twisted performance
by Alec Matusis 11 Dec '05

11 Dec '05

I am running Twisted 1.3 server with a fairly large number of clients. The hardware is two 64 bit 3.0 GHz Xeons with HT, 4GB RAM, and it's on 2.6.11 kernel. I am using poll reactor. Currently, when the number of clients approaches 5000, the "top" shows 99% CPU load for twistd process, the event loop slows down and weird racing conditions show up. My clients basically do nothing: they connect and just sit there, sending application-level keep-alives (null character string) to the server every 90 sec, which are echoed back. The connections/ disconnections for clients are rare- about 4-5 per sec. So I am mostly dealing with 5000 clients in the ESTABLISHED state. Very rarely (5x per second on average) one of the clients gets a message from the server, but I am sure that this is not what keeps the load so high (I slowed down message rates, and the CPU load did not change). What are your thoughts on this performance? Is this load mainly due to poll() overhead, polling a large number of file descriptors? I would love to try an epoll reactor, but as I understand, it's not yet available? Additionally, the process leaks about 100MB memory per day. I made the garbage collector print out the number of objects traced by it every 30 min, and this number stays more or less constant. When the memory held by the process becomes larger, the event loop further slows down, and the performance with the same number of clients degrades, as the memory footprint increases. Thanks for any advice. Alec.

3 2

[Twisted-Python] Non-terminating Process using gtkreactor2
by Doug Winter 11 Dec '05

11 Dec '05

I have a GTK program using twisted 2.1.0 and python 2.4. On both Linux and Windows the program reliably fails to terminate after calling gtk.main_quit() and reactor.stop(). It just sits there. All the windows close, but the process remains running. It will terminate happily if the program has merely been run, but if "things" have happened, the program refuses to die. I am guessing that this is because there are deferreds loitering around unfired, or some GTK resources remain that refuse to die. To kill the process after this, it must be sent a KILL signal, TERM is not sufficient. On windows it can be killed in the Task Manager processes tab. I admit I am at a loss as to how to debug this - can someone give me some pointers to diagnostics I can try to find out what the problem is? Cheers, Doug. -- doug(a)isotoma.com / Isotoma, Open Source Software Consulting Tel: 020 7620 1446 / Mobile: 07879 423002 / Fax: 020 79006980 Skype: dougwinter / http://www.isotoma.com Lincoln House, 75 Westminster Bridge Road, London, SE1 7HS

3 3

[Twisted-Python] pb references
by william.lewis＠uk.bnpparibas.com 11 Dec '05

11 Dec '05

Hi I am trying to call a remote method on a client from a server using a reference that has been given to the server by the client. I am trying to do this in the way described in your "References to Client-side objects" documentation. However, I keep getting the following error: "Can't send references to brokers other than their own." and I can't see why. Any help with this error would be greatly appreciated. This message and any attachments (the "message") is intended solely for the addressees and is confidential. If you receive this message in error, please delete it and immediately notify the sender. Any use not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. The internet can not guarantee the integrity of this message. BNP PARIBAS (and its subsidiaries) shall (will) not therefore be liable for the message if modified. ********************************************************************************************** BNP Paribas Private Bank London Branch is authorised by CECEI & AMF and is regulated by the Financial Services Authority for the conduct of its investment business in the United Kingdom. BNP Paribas Securities Services London Branch is authorised by CECEI & AMF and is regulated by the Financial Services Authority for the conduct of its investment business in the United Kingdom. BNP Paribas Fund Services UK Limited is authorised and regulated by the Financial Services Authority

2 1