Mailman 3 November 2004 - Twisted

[Twisted-Python] Re: Hello
by Iap Nov. 6, 2004

Nov. 6, 2004

1 0

[Twisted-Python] Re: Hello
by Iap Nov. 6, 2004

Nov. 6, 2004

1 0

[Twisted-Python] Weekly Bug Summary
by James Y Knight Nov. 6, 2004

Nov. 6, 2004

Bug summary ______________________ Since 2004-10-30 03:12:02.480000: Bugs opened: 5 Bugs closed: 5 Total open bugs: 327 (+0) New / Reopened Bugs ______________________ [TRIAL] trial/pdb integration is broken wrt readline (3 hours ago) http://www.twistedmatrix.com/bugs/issue775 opened by exarkun Move SRVConnector into twisted.names (3 days ago) http://www.twistedmatrix.com/bugs/issue772 opened by itamarst Switch getHandle() to IPythonSocket (7 hours ago) http://www.twistedmatrix.com/bugs/issue774 opened by itamarst [TRIAL] Trial doesn't cut tracebacks properly (yesterday) http://www.twistedmatrix.com/bugs/issue773 opened by jknight [CORE] Support unicode and non-directory sys.path entries in t.p.plugin (6 days ago) http://www.twistedmatrix.com/bugs/issue771 opened by exarkun Bugs Closed ______________________ Something broke twisted.web or twisted.internet.tcp in the latests commits (7 days ago) http://www.twistedmatrix.com/bugs/issue767 opened by dialtone [TRIAL] Traceback from trial in trivial usage (7 days ago) http://www.twistedmatrix.com/bugs/issue768 opened by exarkun [CONCH] conch client can't read password when running in a conch server shell (2 weeks ago) http://www.twistedmatrix.com/bugs/issue750 opened by z3p [CORE] ReactorTime has serious performance issues (1 month ago) http://www.twistedmatrix.com/bugs/issue707 opened by stefan [CORE] The spewer (i.e. twistd --spew) only spews for the main thread (1 week ago) http://www.twistedmatrix.com/bugs/issue762 opened by spiv

1 0

[Twisted-Python] Re: Hi
by Iap Nov. 6, 2004

Nov. 6, 2004

1 0

[Twisted-Python] Re: Server with several outgoing connections
by Paul Campbell Nov. 5, 2004

Nov. 5, 2004

On Thu, Nov 04, 2004 at 02:37:57AM -0700, twisted-python-request(a)twistedmatrix.com wrote: > I'm new to Twisted and I have tried to read and understand the > intricate sides of it. > And have found as so many other that the learning curve pretty steep, > especially I guess if one comes from being a C programmer :-) I've been there myself recently. It also doesn't help that the standard idioms (design patterns?) in Python are significantly different in some ways from C. > Writing the 'client' side is not so easy and here I guess my C legacy > is hampering me. > > So I'd like someone to give me, or point me, to an example on how to > dynamically create new connections to receivers. I first though these > connections should be shortlived that is 'open the connection, send the > message, receive the ACK and close the connection' but now I think it > might be wiser to keep the connections as long as possible. That is, > until that receiver disappears. Not really. You just have to do a lot of tracking by creating the protocols in the protocol factory. But there's an easier way for your particular case since essentially you are doing RPC's. > Another side of this is that the influx of messages might be higher > than the possible output, that is each node has to keep queues for > messages not yet sent and messages sent but not ack'ed. > It might also happen that recivers of messages are not accessible in > which case the message has to be queue until the receiver pops up > again. The system is not allowed to drop messages. > Note, that each incomming message might have several receivers, so one > incomming message might end up in the sendqueue of several receivers. > > And here in lies one of the problems, I guess I have to keep a > 'centrally' managed queue, that should then be notified on the change > of status of a message on route to a recevier. How should I implement > this ? All of this is a lot easier to do at a higher level. Implement your code in the perspective broker. Then all the ACK's and such are automatically handled. In PB, the code is "almost symmetric" so there's no real difference between implementing a "server" and a "client". The server/client code ends up looking almost identical. Plus, PB semi-automatically handles packet formatting so that you can just pass around high level messages, and it does active cache updating for you semi-automatically. You can also implement a login protocol fairly easily if you have a need for that type of function. Otherwise, RPC's are already implemented in the XML code. Simply steal the XML/HTTP stuff and use it for "non-web" related functions. This is a common technique and issues about client/server management go away (handled automatically in the HTTP code). In your case, there's no "return" since you are simply passing messages. A "return" is just an ACK for you. Otherwise, if you want something more primitive but you can forgo using TCP, it's a lot easier to implement in UDP. For your protocol, what you described sounds very un-TCP-like. It doesn't really fit the mold of a bidirectional stream of bytes in a client/server framework. In the datagram protocol, keep your messages within the limits of a packet or else you have to deal with packet fragmenting and reassembly issues. Here's how I got around it: class queueNode: "Placeholder for holding data" pass class MyMesgProtocol(DatagramProtocol): sendTimeout = 30 # Seconds to retry def __init__(self): sendQueue = dict() def sendMessage(self, data, (host,port)): "Sends outgoing messages" self.transport.write("W"+data, (host, port)) q=queueNode q.message = data q.retries = 3 q.address = (host,port) m = mangle(host, port) q.call = reactor.callLater(sendTimeout, retryMesg, m) sendQueue[m] = q return def datagramReceived(self, data, (host,port)): "Handles incoming packets" if data[0] == "W": self.transport.write("A", (host,port)) self.handleMessage(data[1:], (host,port)) elif data[0] == "A": m = mangle(host,port) if m in sendQueue: q = sendQueue[m] q.call.cancel() del sendQueue[m] else: pass # Hmmm...got an acknowledgement but we didn't have a message queued else: pass # Hmmm...not a valid packet type return def handleMessage(self, data, (host,port)): "User-level handler. Subclass your message system here." pass def retryMesg(self, index): "Handles retry timeouts" q = sendQueue[m] q.retries -=1 if not retries: self.nodeFailure(q.address) del sendQueue[m] return self.transport.write("W"+data,q.address) q.call = reactor.callLater(sendTimeout, retryMesg, m) return def nodeFailure(self, (address, port)): "User-level node failure handler. Subclass your code here." pass To use the code, simply subclass it and define the two functions handleMessage and nodeFailure. Send messages via sendMessage. Note that right now, the code will NOT properly handle multiple messages queued to the same destination address for brevity. To fix this, you'll need to insert nonces on the messages so that the ACK's can be paired up with the corresponding entries in the message queue. I made several references to a "mangle" function. This function takes an (address, port) call and returns a string or a long integer or whatever you feel appropriate to use for indexing the dictionary structure. It's a simple function to write so I left it for the reader. Plus, it is a simple matter to extend the same code to add a few bytes for a nonce to the message "header" (currently a single ASCII letter). Then add the nonce to your address mangling code for the queue, and then the same queue is now extended to handle multiple messages in flight to the same destination as well. In a similar fashion, the ACK's are about 15 bytes right now (just a single byte attached to the UDP header). There's nothing to say that you can't extend the code to a full blown RPC code. Simply edit the code that calls the message handler to accept a return string and then send that string on the write message. The send message code has to be changed to return a deferred (q.deferred=defer.Deferred()). Then later on, the ACK code uses callBack to return data to the original message calling function. I noticed that you had a lot of things referencing buffer sizes and such. This version doesn't pay any attention to buffers at all. If you want to control buffer sizes, then not only will you need the above extra code to insert and delete nonces, but you'll also need to use the HTB code in Twisted or roll your own similar function. Just subclass/wrap the above code as appropriate in the HTB libraries.

1 0

[Twisted-Python] Re: Thanks :)
by Stephen Nov. 3, 2004

Nov. 3, 2004

1 0

[Twisted-Python] Re: Twisted problem
by exarkun＠divmod.com Nov. 2, 2004

Nov. 2, 2004

(Cross posting to twisted-python(a)twistedmatrix.com) On 2 Nov 2004 08:43:26 -0800, elbertlev(a)hotmail.com (Elbert Lev) wrote: >Hi! > > > I'm having a problem with twisted on windows. Installed it and started > finger tutorial. All was working fine untill stem 11 - running twistw > (twistd in documentation). It dies with now error indication. Tried > debugin it, but also with no succes: process terminates before getting > to the run() method > > def run(): > app.run(runApp, ServerOptions) > > > Please help. I can't quite tell what you're doing, or what error you're actually getting. Can you include the full command line you are running, the full source file you are working with (or, preferably, a shorter one that demonstrates the same problem), and the full text of the error you receive, including the traceback if there is one? Jp

1 0

[Twisted-Python] callLater(0) is A BIG LIE
by Jonathan Simms Nov. 2, 2004

Nov. 2, 2004

I'd like to offer some reflections on the reactor and facilitate some conversation on a subject that's been causing me much frustration: the twisted idiom, "callLater(0)" While it seems that this is a DelayedCall, it really is a lie. It has nothing to do with time at all, it is a very *convenient* lie that we all have accepted, but is a lie nonetheless. We all know this means "call function f in the next reactor iteration", but the semantics of callLater have to do with time, not reactor iterations. In fact, in the API documentation, there is no reference to reactor iterations, at all. I think the implicit nature of this idiom is bad. For example in tcp.py:377 reactor.callLater(0, self.failIfNotConnected, error) what this really means is: "run this method at the beginning of the next reactor iteration, if my socket isn't connected, that method will fail" Now, if the reactor were a "pure" event loop (in the way libevent is, with no scheduling), callLater(0) would mean, "Call function f *ASAP* after this function exits". There would be no guarantees as to what would get run next. The workaround (in terms of my implementation) is to emulate this scheduling behavior the select reactor provides, but this kills some of the performance gains one gets from using libevent. When I asked the libevent guys about this, they said it seemed like a brokenness of our library. I guess the question is, is there a better way for twisted to handle this type of thing? Would it be more to our advantage to make scheduling a different abstraction with explicit well-defined methods? -Jonathan

9 15

[Twisted-Python] Talk on Twisted in Boston on Nov. 11th
by Itamar Shtull-Trauring Nov. 1, 2004

Nov. 1, 2004

2 1