Mailman 3 December 2005 - Twisted

[Twisted-Python] How can a tcp client connect with multi servers?
by Khoa Nguyen 14 Dec '05

14 Dec '05

Hi Xu, How did you solve this problem? I face a similar issue and would appreciate if you post a working sample code Thanks, Khoa

2 1

[Twisted-Python] threading issues with DB connection pooling and ReconnectingClientFactory
by Andreas Poisel 14 Dec '05

14 Dec '05

Hi Twisted Experts, in my application I use ReconnectingClientFactory to handle a TCP connection which is supposed to be permanent. The server side (which is not a twisted application) is very fragile and closes the connection frequently. The client has to do database queries on a regular basis and talk to the server depending on the query results. The client runs into a problem with the following application flow: - Client and server are up and running, the database is queried frequently, server talks to client and vice versa, everything is fine. - The server closes the connection and the client stops it's database queries. - The server is reachable again, the client reconnects. - A database query on the client side is performed and we find something we want to transmit to the server. But we fail because the transport object we use after our database query is not connected (transport.connected == 0 and transport.disconnected == 1). These are my conclusions (maybe that's garbage): The reconnection of the client works perfectly for the main reactor thread. But when we try a database query, we get a cached thread which doesn't have a working transport object anymore. Some experiments with protocol and transport object id()s seem to second this assumption. I've some ideas how to solve this problem: - Empty the thread pool after reconnecting and use "fresh threads" with valid transport objects. I can't find a hint how to do this. - Pass a valid transport object to the thread which just did the database query. I'm not sure how to do this properly. - Pass the query result back to the main thread. Hmmm... I would very much appreciate some hints on how to handle this problem properly. Thank you very much! -- Regards, Andi

2 2

[Twisted-Python] Rails? Twisted versus J2EE? Metaprogramming as self-modifying code?
by John Benson 14 Dec '05

14 Dec '05

Hi, I did some Python and Twisted two years ago got distracted, picked up some IBM WebSphere certifications, got turned off by the verbosity of J2EE and suspected that there was a cleaner way to do it. I just got through Bruce Tate's O'Reilly book /Beyond Java/ and am hearing the siren call of Ruby on Rails. Tate is unwilling to champion Ruby over Python or vice versa, but waxes poetic about Rails. I know there is a Rail-like Python project (Subway?) but I'd like to know if there is a Rail-alike that works with Twisted. Also, I'd like to hear about the advantages of J2EE over Twisted (if any), especially in the scalability and MQ Series connectivity areas. The little project I did with Python and Twisted was a joy and would easily have taken 4X the time and effort in a compiled environment. Now that I've seen what J2EE tries to do, I'm curious to see how Twisted stacks up against the J2EE specialties. Comparisons between Python and Ruby will also interest, especially regarding ease of metaprogramming since this seems to be the arena in which the Next Big Thing will hit software practice with aspect-oriented programming, dependency injection and mixins. Kind of like when COBOL programmers had to learn pointers to deal with C, or when C programmers had to learn about objects to deal with C++ and/or Java. By the way, I remember the campaign to stamp out self-modifying code back during the structured programming wars of the 1970's. Although the benefits of metaprogramming are attractive, isn't metaprogramming just self-modifying code at the object level? If so, should we just say no? If not, what has changed to make it reasonable now?

2 1

[Twisted-Python] Weekly Bug Summary
by James Y Knight 14 Dec '05

14 Dec '05

Bug summary ______________________ Since 2005-12-07 07:00:01.870000: Bugs opened: 9 Bugs closed: 2 Total open bugs: 500 (+7) Urgent: 1 (-1) Bug: 238 (+6) Feature: 178 (+2) Wish: 83 (+0) New / Reopened Bugs ______________________ ==== Bug ==== [CGI, WEB2, WSGI] createCGIEnvironment fails to include ;parameters in PATH_INFO (yesterday) http://twistedmatrix.com/bugs/issue1361 opened by cce [BACKPORT, WEB2] twisted.web2.stream and wsgi (yesterday) http://twistedmatrix.com/bugs/issue1362 opened by mkerrin [TRIAL] [PATCH] Traceback running a test with broken setUp method (2 days ago) http://twistedmatrix.com/bugs/issue1359 opened by headly [TRIAL] Clean up Trial's Runner and Loader concepts (5 days ago) http://twistedmatrix.com/bugs/issue1355 opened by jml [WEB] [PATCH] microdom is not case sensitive when doing laterClosers (2 days ago) http://twistedmatrix.com/bugs/issue1358 opened by Tv [CORE] [PATCH] t.p.usage error messages mangle option case (6 days ago) http://twistedmatrix.com/bugs/issue1354 opened by Tv ==== Feature ==== [MAIL] IMessageDelivery.validateTo API simplification (yesterday) http://twistedmatrix.com/bugs/issue1360 opened by Tv sendFile handy API for FTPClient (3 days ago) http://twistedmatrix.com/bugs/issue1357 opened by dialtone ==== Wish ==== [MAIL] Twisted Mail/OS X performance problem (3 days ago) CLOSED http://twistedmatrix.com/bugs/issue1356 opened by exarkun Bugs Closed ______________________ ==== Urgent ==== [TRIAL] trial loads test methods incorrectly (7 days ago) http://twistedmatrix.com/bugs/issue1353 opened by hagna, closed by jml ==== Wish ==== [MAIL] Twisted Mail/OS X performance problem (3 days ago) http://twistedmatrix.com/bugs/issue1356 opened by exarkun, closed by exarkun

1 0

[Twisted-Python] How can I keep data integrity in multiple threads environment
by Kaladan 12 Dec '05

12 Dec '05

Hello, I am new to twisted and I have a few questions: I have implemented a TCPServer and the associated Factory, which manages data and dispatches treatments (According to the tutorial it's not the best solution but actually this is not the point ;) ) All of the treatments run in threads and thus use shared data. The allData dictionnary is critical since it can be used by multiple threads. How can I keep the allData integrity ? Does Twisted core provide such facilites or do I have to use semaphore or other stuff ? class GestionnaireXXX(protocol.Protocol) def dataReceived(self, data): self.factory.dispatch(data) class GestionnaireXXXFactory(protocol.ServerFactory): def __init__(self): self.allData = {} self.initData() self._dispatcher = { 1: self.alert, 4: self.create, } def initData(self): if os.path.isfile(TEMP) self.allData = aot.unjellyFromSource(file(TEMP, 'r')) def dispatch(self, data): splitedData = data.split('\n') method = self._dispatcher.get(len(splitedData), self.doNothing) reactor.runInThread(method, splitedData) def create(self, *data): # Some treatments on allData ... aot.jellyToSource(file(TEMP, 'w')) def alert(self, *unused): # XXX sem = DeferredSemaphore() sem.acquire() allDataCopy = self.allData.copy() self.allData.clear() sem.release() # End XXX # some long treatments on allDataCopy ... Thank's, -- Olivier Guilloux

3 2

[Twisted-Python] Re: pb references
by william.lewis＠uk.bnpparibas.com 12 Dec '05

12 Dec '05

> In short: you seem to be sending an reference you got through pb to > another pb peer, which is not allowed. > > I hope that helps. Yes, that was indeed the problem. My server was taking a root object from a client and attempting to pass it on to another server as an argument to a remote method. Thanks. I now have a further question: When my server takes the root object from a client it keeps and uses that object, later on, to call remote methods on the client (as described in the docs). What I am not sure about is what happens to the root object if the TCP/IP connection goes down between the server and client. Presumably, nothing will happen until I try to call a remote method on the client? In which case, how can the server attempt to reconnect to the client if it only has the original root object given to it by the client and the connection has temporarily gone down? Or is the client responsible for keeping the connection open? Thanks William Lewis This message and any attachments (the "message") is intended solely for the addressees and is confidential. If you receive this message in error, please delete it and immediately notify the sender. Any use not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. The internet can not guarantee the integrity of this message. BNP PARIBAS (and its subsidiaries) shall (will) not therefore be liable for the message if modified. ********************************************************************************************** BNP Paribas Private Bank London Branch is authorised by CECEI & AMF and is regulated by the Financial Services Authority for the conduct of its investment business in the United Kingdom. BNP Paribas Securities Services London Branch is authorised by CECEI & AMF and is regulated by the Financial Services Authority for the conduct of its investment business in the United Kingdom. BNP Paribas Fund Services UK Limited is authorised and regulated by the Financial Services Authority

1 0

Re: [Twisted-Python] pb references
by Brian Warner 12 Dec '05

12 Dec '05

Micky Latowicki <ml.flex(a)gmail.com> writes: > Now, B is apparently trying to send this RemoteReference to some third > party, C. This is not allowed by twisted, possibly because that would > often require forwarding messages from C to A through B, which can be > inefficient and unreliable. In PB we call this "serializing third-party references". and it's illustrated concisely by the "Granovetter Diagram"[1] as described by papers on distributed computing[2]. I also sometimes call it the "Gift" pattern. Micky has described the situtation accurately, but I wanted to add some detail on the whole third-party reference thing, because it's changing in newpb. Some of this is pretty verbose, probably more information than anybody but a newpb implementor is likely to care about, but I figured I'll throw it out there so people can know what's going on behind the scenes. oldpb refuses to serialize third-party references because there's no reference-identification infrastructure in place to let the recipient establish a direct connection to the originator. Each RemoteReference is really just a connection-local ID number (the CLID), like "4", wrapped by a bunch of code that implements callRemote(). The CLID is only meaningful when it gets looked up by the Broker on the other end of the wire that it's scoped to. So if B wants to send his RemoteReference(A) to C, the only way to send something that C will actually be able to use would be to create a proxy object that provides a whole bunch of methods like the following: def remote_foo(self, *args, **kwargs): return self.refA.callRemote("foo", *args, **kwargs) (with suitable cleverness, you could make this into a generic Proxy object, such that you didn't have to manually duplicate this for every single method that A can respond to. Note that even in newpb this pattern is useful, so we'll figure out some way to make it convenient: Revocable Forwarders, Logging Proxies, and restricted Facets are all design patterns implemented with proxy objects like these). The issue of course is speed and resource consumption. With this proxy in place, B has to be involved in every message between A and C, even if he doesn't want to, adding at least an extra round trip for every method call. Worse yet, B has to carefully watch the arguments and return values of all the methods flying between A and C to see if either end has included a new RemoteReference to some local object. If so, B has to create a new proxy for that object too. (this is one of the obligations of a "Membrane", and is a nuisance that would be nice to avoid). Finally, one of the design goals of oldpb has been reasonable security: make it slightly harder to accidentally leak information or authority. To this end, objects must be declared pb.Referenceable to be remotely callable, instances must inherit from pb.Copyable to be transferred across the wire, and RemoteReferences cannot be sent to third parties. oldpb forces you to make certain designations explicit. For both these reasons (and because it's just too much of a hassle to implement), oldpb takes the easy way out and punts, by disallowing third-party references and forcing the user to implement a proxy if that's what they really want. NEWPB In newpb, things are much better. Assuming you make your Tub accessible to the outside world (by telling it to listen on a port, and by telling it what hostname+portno it's listening on), each Referenceable you publish gets a PB-URL, which is then accessible from any other Tub, anywhere. (the return value from tub.registerReference() is this PB-URL). To "gift" a RemoteReference to a third party, you simply include the RemoteReference as an argument in a callRemote (or return it from a remotely-invoked method), and PB will handle the rest for you. This works by passing a special sequence that includes the PB-URL of the target object, which the receiving side then submits to tub.getReference() to obtain their own RemoteReference before invoking the target method. If you don't want to pass live references around, you can turn any RemoteReference into a "SturdyRef" (which is like an object form of a PB-URL) with rref.getSturdyRef(), and then pass *that*, since SturdyRefs are pb.Copyable . You can also take a SturdyRef and pass it "live" to a remote Tub, by doing something like: rref.callRemote("introduce", sturdy.asLiveRef()) whereupon the recipient's remote_introduce() method will be invoked with a live RemoteReference to the target of the sturdyref. The API is still up in the air, but my plan is for every RemoteReference you pass over a wire to be given an unguessable PB-URL so that it is eligible for being sent as a gift to a third-party Tub. (one possibility is that you have to explicitly publish the ones that you want to be giftable.. there might be a switch to turn this sort of thing on or off, it's a tradeoff of memory consumption versus convenience). Another design question has to do with object lifetime: in the current implementation, when B sends rref(A) to C, B makes sure to keep it's handle on A alive until C confirms it has acquired its own. This improves the chances that C will be able to acquire a live reference, but it also allows a malicious C to force B to keep that rref alive forever, wasting memory. The alternative is to just tell C to take their chances, and maybe they'll wind up with a working RemoteReference, and maybe they'll be unlucky and A will have garbage-collected that object by the time they finish trying to acquire their own. Distributed garbage collection is very tricky. The object-lifetime design issues show up elsewhere too. Should an object, once it gets sent over the wire (any wire), stay alive forever, just in case somebody wrote down its URL and might some day come calling for it? Or should it be allowed to vanish as soon as the last live reference is released? In the current implementation, anything you submit to tub.registerReference() will stay alive forever (where "forever" is equal to the lifetime of the Tub), whereas objects that cross the wire in method invocations get reference-counted and released when there are no more live references to them. Correspondingly, objects submitted to tub.registerReference() get globally-reachable names (PB-URLs), and are therefore eligible for gifting, whereas objects merely crossing the wire do not (and cannot be gifted). The latter needs to change, since *all* objects should be giftable, but it's quite possible that the URL->object table will use weakrefs so that the giftable/non-giftable distinction can be orthogonal to the long-lived/ephemeral distinction. Tyler's trying to convince me to let objects stay alive "forever", get rid of garbage collection and distributed reference counting, and just use the Tub lifetime to reclaim memory or block access to old objects. I'm not sure yet, though, it may become a flag you set on the Tub. One of the issues with allowing Gifts is that it opens up the possibility that methods will be invoked out-of-order. In the present implementation, if you do: a.callRemote("introduce", gift_rref_to_B) a.callRemote("second", 1, 2, 3) then remote_second() will probably be invoked *first*, because the potential call to remote_introduce() is held up waiting for gift_rref_to_B to be turned into a real RemoteReference (which must wait for connection negotiation, etc). I'm thinking that this will be changed (by queueing all method invocations and stalling remote_second until remote_introduce has been invoked), but I might add a flag that lets you choose between the two behaviors. Setting the flag one way lets methods be invoked as quickly as possible, setting it the other way forces them to be called in-order, even if that adds arbitrarily long delays to deal with Gifts. I might add a flag which would disable Gifts altogether, since they're a moderately advanced feature and it might be confusing to have them work so transparently. Finally, there's the complex issue of what kind of ordering guarantees to make regarding methods invoked on gifted references. The E documents[3] describe a situation where you'd like to make certain promises about the relative ordering of methods invoked (by you) on some reference, versus methods invoked (on that same reference) by someone else you've just given that reference to. E (or VatTP, to be precise) suggests a funny kind of proxy behavior called the WormholeOp[4] to provide these promises. At the moment, newpb just punts on the issue, and only makes claims about the relative ordering of messages sent on a *single connection*. If you want to make sure that messages sent to or from different parties happen in some particular order, you must wait for the first to complete before allowing the second to occur. This might be improved in the future (once I understand the issue better, for starters), but for now newpb's ordering guarantees may not be as thorough as certain environments might prefer. (I *think* the only practical consequence is that certain kinds of promise-pipelining optimizations cannot be made, occasionally increasing the number of round trips, but really I don't understand enough yet, and neither of the two people on the planet who *do* understand enough have been able to explain it to me in a way that I can get my head around). Also, the proxy object described above (as the only way to accomplish third-party references in oldpb) is actually quite useful, so newpb will eventually make it easy and cheap to build them. One pattern is the Revocable Forwarder, where you want to extend your authority to somebody else, but you want to be able to cut them off if you change your mind. This is as simple as an object that does the same sort of "def remote_foo(): return self.refA.callRemote('foo')" thing as above, but adds an extra method (exposed in a separate capability) that does 'del self.refA' to turn off all forwarding at once. Another pattern is the restrictive Facet, which forwards access to some (but not all) methods. Either of things might add logging, where the caller doesn't notice anything special, but somebody else gets a record of each method invoked. Anyway, I just wanted to do a bit of a braindump on where third-party references are going in newpb. The summary is that using them is as easy as you think it ought to be. "make simple things simple", and all that :). cheers, -Brian [1]: "Ode to the Granovetter Diagram" http://www.erights.org/elib/capability/ode/index.html [2]: pretty much everything at http://www.erights.org/ [3]: http://www.erights.org/elib/concurrency/partial-order.html [4]: http://www.erights.org/elib/distrib/captp/WormholeOp.html

1 0

[Twisted-Python] twisted performance
by Alec Matusis 11 Dec '05

11 Dec '05

I am running Twisted 1.3 server with a fairly large number of clients. The hardware is two 64 bit 3.0 GHz Xeons with HT, 4GB RAM, and it's on 2.6.11 kernel. I am using poll reactor. Currently, when the number of clients approaches 5000, the "top" shows 99% CPU load for twistd process, the event loop slows down and weird racing conditions show up. My clients basically do nothing: they connect and just sit there, sending application-level keep-alives (null character string) to the server every 90 sec, which are echoed back. The connections/ disconnections for clients are rare- about 4-5 per sec. So I am mostly dealing with 5000 clients in the ESTABLISHED state. Very rarely (5x per second on average) one of the clients gets a message from the server, but I am sure that this is not what keeps the load so high (I slowed down message rates, and the CPU load did not change). What are your thoughts on this performance? Is this load mainly due to poll() overhead, polling a large number of file descriptors? I would love to try an epoll reactor, but as I understand, it's not yet available? Additionally, the process leaks about 100MB memory per day. I made the garbage collector print out the number of objects traced by it every 30 min, and this number stays more or less constant. When the memory held by the process becomes larger, the event loop further slows down, and the performance with the same number of clients degrades, as the memory footprint increases. Thanks for any advice. Alec.

3 2

[Twisted-Python] Non-terminating Process using gtkreactor2
by Doug Winter 11 Dec '05

11 Dec '05

I have a GTK program using twisted 2.1.0 and python 2.4. On both Linux and Windows the program reliably fails to terminate after calling gtk.main_quit() and reactor.stop(). It just sits there. All the windows close, but the process remains running. It will terminate happily if the program has merely been run, but if "things" have happened, the program refuses to die. I am guessing that this is because there are deferreds loitering around unfired, or some GTK resources remain that refuse to die. To kill the process after this, it must be sent a KILL signal, TERM is not sufficient. On windows it can be killed in the Task Manager processes tab. I admit I am at a loss as to how to debug this - can someone give me some pointers to diagnostics I can try to find out what the problem is? Cheers, Doug. -- doug(a)isotoma.com / Isotoma, Open Source Software Consulting Tel: 020 7620 1446 / Mobile: 07879 423002 / Fax: 020 79006980 Skype: dougwinter / http://www.isotoma.com Lincoln House, 75 Westminster Bridge Road, London, SE1 7HS

3 3

[Twisted-Python] pb references
by william.lewis＠uk.bnpparibas.com 11 Dec '05

11 Dec '05

Hi I am trying to call a remote method on a client from a server using a reference that has been given to the server by the client. I am trying to do this in the way described in your "References to Client-side objects" documentation. However, I keep getting the following error: "Can't send references to brokers other than their own." and I can't see why. Any help with this error would be greatly appreciated. This message and any attachments (the "message") is intended solely for the addressees and is confidential. If you receive this message in error, please delete it and immediately notify the sender. Any use not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. The internet can not guarantee the integrity of this message. BNP PARIBAS (and its subsidiaries) shall (will) not therefore be liable for the message if modified. ********************************************************************************************** BNP Paribas Private Bank London Branch is authorised by CECEI & AMF and is regulated by the Financial Services Authority for the conduct of its investment business in the United Kingdom. BNP Paribas Securities Services London Branch is authorised by CECEI & AMF and is regulated by the Financial Services Authority for the conduct of its investment business in the United Kingdom. BNP Paribas Fund Services UK Limited is authorised and regulated by the Financial Services Authority

2 1