Mailman 3 November 2004 - Twisted

[Twisted-Python] multiple connectTCP bindaddress problem
by Chris Laws Nov. 8, 2004

Nov. 8, 2004

Looks like I'm moving to "OS specified addresses" anyway, as I can't get my current impementation working properly. Using the implementation where I increment the local port for each bainaddress arg to the connectTCP call, I was receiving "connectbinderror" on the client whenever I closed and restarted a Server. Interstingly I would also get this error if I started the client only (Ie. no servers running which it could connect to). In my logs I would see the initial connection attempt, a … [View More]

1 0

[Twisted-Python] multiple connectTCP bindaddress problem
by Chris Laws Nov. 8, 2004

Nov. 8, 2004

I would like to specify my client ports for the previously mentioned reasons. We also use integration tools such as LAN message crackers to help us verify message contents. These use information such as "port" to determine the appropriate cracking method. If I let the OS determine the client's ports I will have to change the port information in our LAN message cracker everytime I restart the client application. Admittedly this will only be necessary during the integration phase, however I'm … [View More]

2 1

[Twisted-Python] multiple connectTCP bindaddress problem
by Chris Laws Nov. 8, 2004

Nov. 8, 2004

I probably have not explained myself very well. I'm only using a single protocol. The server side of my application is all fine, no problems with conflicts. The server side uses a single port. There are approx 30 machines each of which runs a separate instance of the server side application. They all bind the same port number on their local machine as a listen socket. My client side application will attempt to maintain a connection with all server side applications. The design allows for … [View More]

2 1

[Twisted-Python] Re: Thank you!
by Stephen Nov. 8, 2004

Nov. 8, 2004

1 0

[Twisted-Python] multiple connectTCP bindaddress problem
by Chris Laws Nov. 8, 2004

Nov. 8, 2004

I only have about 30 servers which I connect the client(s) with. There are between 1 and 10 client applications which connect to the 30 servers, though it is very unlikely that more than two client side applications would be running at any one time. I have control over the ports I am using (within reason) so the client(s) can go on any port I choose. I simply wanted the clients to use the same local port when establishing their connections with the servers to reduce the likelyhood for … [View More]

2 1

[Twisted-Python] multiple connectTCP bindaddress problem
by Chris Laws Nov. 7, 2004

Nov. 7, 2004

I'm in the process of building a twisted application which will be used to automate control of processes running on remote machines. The basic architecture is a client to many server configuration. 1. many (~30), twisted based, network server applications running on different machines, 2. a client side, twisted based, application which will be used to control and monitor the servers. This is a small component of a large project and as part of the project we have defined port ranges which … [View More]

4 5

[Twisted-Python] Deprecated function removal policy
by James Y Knight Nov. 7, 2004

Nov. 7, 2004

So, I recently stumbled across t.i.base.ReactorBase.clientTCP/SSL/UNIX. They have been deprecated since Aug 2002. (version 0.99.0rc5 or so) Is there some policy for when deprecated functionality can be removed, or is it just going to stay forever? In addition, it would be really nice if every time something got deprecated, it got marked with the last version in which it was the recommended interface, and the date too, so less investigation needs to be done. James

2 1

[Twisted-Python] Re: Thank you!
by Stephen Nov. 7, 2004

Nov. 7, 2004

1 0

[Twisted-Python] etag and last-modified
by Jacob Friis Nov. 6, 2004

Nov. 6, 2004

I have a script that downloads multiple rss/atom feeds via Feedparser. The script uses twisted.internet but the developer tells me there is no way to use etag and last-modified with twisted. Instead I'll let Feedparser do the download and use twisted for threads. What is the maximum pool size I can use? Thanks, Jacob

4 7

[Twisted-Python] Server with several outgoing connections
by Roland Hedberg Nov. 6, 2004

Nov. 6, 2004

Hi! I'm new to Twisted and I have tried to read and understand the intricate sides of it. And have found as so many other that the learning curve pretty steep, especially I guess if one comes from being a C programmer :-) Anyway, this is my problem: I'm building a system that consists of a number of nodes, each node acting as a receiver and transmitter of messages. The protocol used between the nodes might be a message passing system like xmpp or something else. The number of nodes are … [View More]

10 15

[Twisted-Python] Re: Hello
by Iap Nov. 6, 2004

Nov. 6, 2004

1 0

[Twisted-Python] Re: Hello
by Iap Nov. 6, 2004

Nov. 6, 2004

1 0

[Twisted-Python] Weekly Bug Summary
by James Y Knight Nov. 6, 2004

Nov. 6, 2004

Bug summary ______________________ Since 2004-10-30 03:12:02.480000: Bugs opened: 5 Bugs closed: 5 Total open bugs: 327 (+0) New / Reopened Bugs ______________________ [TRIAL] trial/pdb integration is broken wrt readline (3 hours ago) http://www.twistedmatrix.com/bugs/issue775 opened by exarkun Move SRVConnector into twisted.names (3 days ago) http://www.twistedmatrix.com/bugs/issue772 opened by itamarst Switch getHandle() to IPythonSocket (7 hours ago) http:… [View More]

1 0

[Twisted-Python] Re: Hi
by Iap Nov. 5, 2004

Nov. 5, 2004

1 0

[Twisted-Python] Re: Server with several outgoing connections
by Paul Campbell Nov. 5, 2004

Nov. 5, 2004

On Thu, Nov 04, 2004 at 02:37:57AM -0700, twisted-python-request(a)twistedmatrix.com wrote: > I'm new to Twisted and I have tried to read and understand the > intricate sides of it. > And have found as so many other that the learning curve pretty steep, > especially I guess if one comes from being a C programmer :-) I've been there myself recently. It also doesn't help that the standard idioms (design patterns?) in Python are significantly different in some ways from C. > … [View More]Writing the 'client' side is not so easy and here I guess my C legacy > is hampering me. > > So I'd like someone to give me, or point me, to an example on how to > dynamically create new connections to receivers. I first though these > connections should be shortlived that is 'open the connection, send the > message, receive the ACK and close the connection' but now I think it > might be wiser to keep the connections as long as possible. That is, > until that receiver disappears. Not really. You just have to do a lot of tracking by creating the protocols in the protocol factory. But there's an easier way for your particular case since essentially you are doing RPC's. > Another side of this is that the influx of messages might be higher > than the possible output, that is each node has to keep queues for > messages not yet sent and messages sent but not ack'ed. > It might also happen that recivers of messages are not accessible in > which case the message has to be queue until the receiver pops up > again. The system is not allowed to drop messages. > Note, that each incomming message might have several receivers, so one > incomming message might end up in the sendqueue of several receivers. > > And here in lies one of the problems, I guess I have to keep a > 'centrally' managed queue, that should then be notified on the change > of status of a message on route to a recevier. How should I implement > this ? All of this is a lot easier to do at a higher level. Implement your code in the perspective broker. Then all the ACK's and such are automatically handled. In PB, the code is "almost symmetric" so there's no real difference between implementing a "server" and a "client". The server/client code ends up looking almost identical. Plus, PB semi-automatically handles packet formatting so that you can just pass around high level messages, and it does active cache updating for you semi-automatically. You can also implement a login protocol fairly easily if you have a need for that type of function. Otherwise, RPC's are already implemented in the XML code. Simply steal the XML/HTTP stuff and use it for "non-web" related functions. This is a common technique and issues about client/server management go away (handled automatically in the HTTP code). In your case, there's no "return" since you are simply passing messages. A "return" is just an ACK for you. Otherwise, if you want something more primitive but you can forgo using TCP, it's a lot easier to implement in UDP. For your protocol, what you described sounds very un-TCP-like. It doesn't really fit the mold of a bidirectional stream of bytes in a client/server framework. In the datagram protocol, keep your messages within the limits of a packet or else you have to deal with packet fragmenting and reassembly issues. Here's how I got around it: class queueNode: "Placeholder for holding data" pass class MyMesgProtocol(DatagramProtocol): sendTimeout = 30 # Seconds to retry def __init__(self): sendQueue = dict() def sendMessage(self, data, (host,port)): "Sends outgoing messages" self.transport.write("W"+data, (host, port)) q=queueNode q.message = data q.retries = 3 q.address = (host,port) m = mangle(host, port) q.call = reactor.callLater(sendTimeout, retryMesg, m) sendQueue[m] = q return def datagramReceived(self, data, (host,port)): "Handles incoming packets" if data[0] == "W": self.transport.write("A", (host,port)) self.handleMessage(data[1:], (host,port)) elif data[0] == "A": m = mangle(host,port) if m in sendQueue: q = sendQueue[m] q.call.cancel() del sendQueue[m] else: pass # Hmmm...got an acknowledgement but we didn't have a message queued else: pass # Hmmm...not a valid packet type return def handleMessage(self, data, (host,port)): "User-level handler. Subclass your message system here." pass def retryMesg(self, index): "Handles retry timeouts" q = sendQueue[m] q.retries -=1 if not retries: self.nodeFailure(q.address) del sendQueue[m] return self.transport.write("W"+data,q.address) q.call = reactor.callLater(sendTimeout, retryMesg, m) return def nodeFailure(self, (address, port)): "User-level node failure handler. Subclass your code here." pass To use the code, simply subclass it and define the two functions handleMessage and nodeFailure. Send messages via sendMessage. Note that right now, the code will NOT properly handle multiple messages queued to the same destination address for brevity. To fix this, you'll need to insert nonces on the messages so that the ACK's can be paired up with the corresponding entries in the message queue. I made several references to a "mangle" function. This function takes an (address, port) call and returns a string or a long integer or whatever you feel appropriate to use for indexing the dictionary structure. It's a simple function to write so I left it for the reader. Plus, it is a simple matter to extend the same code to add a few bytes for a nonce to the message "header" (currently a single ASCII letter). Then add the nonce to your address mangling code for the queue, and then the same queue is now extended to handle multiple messages in flight to the same destination as well. In a similar fashion, the ACK's are about 15 bytes right now (just a single byte attached to the UDP header). There's nothing to say that you can't extend the code to a full blown RPC code. Simply edit the code that calls the message handler to accept a return string and then send that string on the write message. The send message code has to be changed to return a deferred (q.deferred=defer.Deferred()). Then later on, the ACK code uses callBack to return data to the original message calling function. I noticed that you had a lot of things referencing buffer sizes and such. This version doesn't pay any attention to buffers at all. If you want to control buffer sizes, then not only will you need the above extra code to insert and delete nonces, but you'll also need to use the HTB code in Twisted or roll your own similar function. Just subclass/wrap the above code as appropriate in the HTB libraries. [View Less]

1 0

[Twisted-Python] Re: Thanks :)
by Stephen Nov. 3, 2004

Nov. 3, 2004

1 0

[Twisted-Python] Re: Twisted problem
by exarkun＠divmod.com Nov. 2, 2004

Nov. 2, 2004

(Cross posting to twisted-python(a)twistedmatrix.com) On 2 Nov 2004 08:43:26 -0800, elbertlev(a)hotmail.com (Elbert Lev) wrote: >Hi! > > > I'm having a problem with twisted on windows. Installed it and started > finger tutorial. All was working fine untill stem 11 - running twistw > (twistd in documentation). It dies with now error indication. Tried > debugin it, but also with no succes: process terminates before getting > to the run() method > > def run(): … [View More]

1 0

[Twisted-Python] callLater(0) is A BIG LIE
by Jonathan Simms Nov. 2, 2004

Nov. 2, 2004

I'd like to offer some reflections on the reactor and facilitate some conversation on a subject that's been causing me much frustration: the twisted idiom, "callLater(0)" While it seems that this is a DelayedCall, it really is a lie. It has nothing to do with time at all, it is a very *convenient* lie that we all have accepted, but is a lie nonetheless. We all know this means "call function f in the next reactor iteration", but the semantics of callLater have to do with time, not … [View More]

9 15

[Twisted-Python] Talk on Twisted in Boston on Nov. 11th
by Itamar Shtull-Trauring Nov. 1, 2004

Nov. 1, 2004

2 1