Stephen Thorne wrote:
Have a look at htb.py, you should be able to rate limit the udp throughput rather easily.
For me doing web stuff it was (in its entirety) like this:
<bandwidth throttler clipped>
Hope that helps.
If for no other reason than that it convinced me there wasn't a simple and elegant solution readily available to be dropped in for a UDP system :) . In the final analysis, what was happening is that individual queries were timing out *before the queries were even sent* because it was taking so incredibly long to format and send all of the queries. The manually written system was "pull" based. It marked the time-of-sending for timeout purposes deep in the select loop. The Twisted version is setting the timeout via a callLater when the request is *submitted*. With 2 or 3 thousand queries, the 20 seconds of timeout was finishing before any messages were processed from the incoming port, then the timeouts would all get called, which kept eating up processing time, preventing any messages from getting read, so the port's internal buffer would fill up and more timeouts occurred, so basically nothing would get through. Queueing up the outputs before sending didn't help, btw. Nor would simple bandwidth throttling without modifying the timeout characteristics. My solution, which is rather low tech, is to do an effective "yield" after each message is sent, to allow for processing incoming responses. This is done with this little method: def smallBatch( self, oids, tables, index=0, iterDelay=.01 ): if index < len(self.proxies): proxy = self.proxies[index] self.singleProxy( proxy,oids,tables ) reactor.callLater( iterDelay, self.smallBatch, oids, tables, index+1 ) else: dl = defer.DeferredList( self.partialDefers ) dl.addCallback( self.returnFinal ) (which also has a side-effect of introducing a bandwidth throttling), but most importantly, it delays setting the timeout on any given query until when the query is actually sent (or fairly close to then). Put another way, it gives the system a chance to send the message as soon after submitting it as possible. Have fun all, and thanks, Stephen, Mike
Stephen Thorne
On Mon, 2004-02-09 at 20:51, Mike C. Fletcher wrote:
...
only a few dozen dropped messages. However, with the Twisted equivalent (UDPTransport with my simple protocol object), I was seeing huge drop rates, so, gathering that Twisted isn't queueing up the UDP requests, I wrote a (byzantine) query-throttling mechanism with Twisted defers.
...
_______________________________________ Mike C. Fletcher Designer, VR Plumber, Coder http://members.rogers.com/mcfletch/