Mailman 3 October 2004 - Twisted

[Twisted-Python] Using wxreactor on Windows
by Joshua Ritter Oct. 6, 2004

Oct. 6, 2004

On Windows, I was having a problem with my wxreactor application intermittently freezing on initialization. It was also taking a long time to exit. wxdemo.py doesn't exhibit the problem at startup, most likely due to the simplicity of the GUI. The demo does however exhibit the exit problem. What I did to alleviate both problems was to move the timer creation and starting code: class MoMWorldServer(wx.App): def OnInit(self): #SNIP wx.EVT_IDLE(self,self.OnFirstIdle) def OnFirstIdle(self,evt): #We catch the first idle message and use it to kick start #the twisted reactor, this avoids a possible deadlock, at least on Windows, gulp print "Starting Network Service" wx.EVT_IDLE(self,None) #remove handler #create the timer and start it reactor.timer = wxreactor.ReactorTimer(reactor) reactor.timer.Start(1) For the exiting problem, I stop the timer on the close event to the main frame of the application: class MainFrame(wx.Frame): def __init__(self, parent, id, title): #SNIP wx.EVT_CLOSE(self,self.OnClose) def OnClose(self,evt): evt.Skip() #stop the timer here reactor.timer.Stop() -Josh Ritter Technical Director http://www.prairiegames.com

1 0

[Twisted-Python] Sending other things than strings in UDP packets
by Ibrahim Mubarak Oct. 6, 2004

Oct. 6, 2004

Hi, I am working on a UDP server. I can send and recieve packets with their data in strings with no problems. However, I was wondering if there is a way to populate the data portion of the packet with data types other than strings. And to be able to stick a number of values, might be of different types, together in the same packet. I looked at PB. It is not what I need. I need a low level control on what I add to the packets I send and how I can read packets I recieve. Thanks, ib __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

2 1

[Twisted-Python] assignKeyAttr in row object
by Tsai Li Ming Oct. 5, 2004

Oct. 5, 2004

Dear all, What is the assignKeyAttr method used for in a row object? Is it similar to a primary key that can be used to identity the instance? Liming

3 4

[Twisted-Python] About adodb
by Simon Oct. 5, 2004

Oct. 5, 2004

Dave, thanks for your reply. >I'm not too familiar with ADOdb, what are the benefits? Quoting the author: "You might ask why Python needs a database abstraction library when Python provides the official DB API. Unfortunately the DB API does not encapsulate differences in the database implementations. For example, to select a limited number of rows, say 10 rows, you would have to use very different SQL for different databases. [...] These differences are handled by ADOdb (using SelectLimit), but not by the Python DB API. Other important database differences transparently handled by ADOdb include date-formating, associative arrays (records as dictionaries) and LOB-handling." >By rewrite, do you mean that the new version only uses >ADOdb, or have you added ADOdb as a supported back-end? ADOdb adds a level of abstraction on top of db-api. I wrote a version of ConnectionPool that uses adodb connections instead of the regular dbapi connections and cursors. If this is something python developers find useful, i don't know. I guess it isn't as necessary as in php (which i'm switching from) where no common db-api exists, but I still think it's quite a nice thing. >sqlreflector and row could definitely use a reworking. >I don't think they are used much. I think people mainly >use adbapi and write their own object mapping system. >What sort of things did you have in mind? Well mainly validating the data assigned to a rowobject so one can be sure it makes it into the db (checking data length, proper dates etc). But if row and sqlreflector isn't used I'll probably build something new instead of going through that code. I'm thinking of using the db_row module (http://opensource.theopalgroup.com) which seems to offer efficient row data access. /Simon (STemplar)

2 3

[Twisted-Python] Twisted scalability with hundreds of outbound (client) connections
by Phil Mayers Oct. 4, 2004

Oct. 4, 2004

I have an SNMP poller written in Python that I'd like to turn into a long-running server process. The obvious thing (it is obvious - I really need the support infrastructure) is to convert it to twisted, since it provides easy support to turn the process into a server as well as a client (e.g. XMLRPC query mapping onto an SNMP fetch; using DBAPI to update node status, all hooked in via deferred, etc.) However, I've done a couple of proof-of-concept ports and am having scalability problems related to the way reactors work. I'm not entirely certain I've got the problem nailed, since the reactor sources are... complex... but I think what happens is this: The framework is heavily oriented towards getting IO out as soon as possible, and since there's no "scheduling" of receive events, protocol instances talking to "fast" clients eat all the CPU, and the socket buffer overflows since each datagramReceived triggers another write(). Possibly an example would help: class agent: def start(self): d = self.get_table('ifTable') d.addCallbacks(self.step2, self.die) def step2(self, ifTable): # Do some processing, then more data fetch d = self.get_table('ipAddrTable') d.addCallbacks(self.step3, self.die) def step3(self, ipAddrTable): # etc. etc. def do_timeouts(reactor): # get the list of sent PDUs now = time.time() for pdu in pdus: if pdu.due < now: pdu.deferred.errback(Timeout()) reactor.callLater(1, do_timeouts, reactor) import sys from twisted.internet import reactor for host in sys.argv[1:]: a = agent(host) reactor.callLater(0, a.start) reactor.callLater(1, do_timeouts, reactor) reactor.run() Now, I know there are scalability problems with callLater, but that's only being used for the startup because I can't think of a cleaner way of doing it - future activity is all triggered by deferreds. I'm handling the timeouts by running a once-per-second expiry which is sufficient for my needs. The *problem* is that the SNMP agents I'm talking to have an extremely wide range of performances, response times and quantity of data to retrieve, and the faster endpoints are consuming a disproportionate quantity of the CPU time; so much so in fact, that the UDP socket receive queue limit is being exceeded (I'm using a single UDP socket for all the clients, and breaking the responses back out based on SNMP message ID - this is to overcome FD limits). We're talking 750-1000 clients here BTW. What I really need is some way of making the reactor do a fair-queuing style of transmit/receive; I have some not-very-clear notion in my head of transport.write not actually doing the write, but just putting the output data onto a per-agent queue, and the reactor emptying those queues in a strictly round-robin fashion, *but* stopping immediately to service input data if it arrives. Hmm. I suppose I could implement that by subclassing the UDP transport: class queuedtransport: def write(self, data, host=None): if not host in self.host2queue: queue = max(self.host2queue.values()) + 1 self.host2queue[host] = queue else: queue = self.host2queue[host] self.queues.setdefault(queue, []).append(data) def queue(self): if not self.queues: return reactor.callLater(1, self.queue) while True: self.pos = (self.pos + 1) % len(self.queues) if not self.queues[self.pos]: continue self._write(self.queues[self.pos].pop(0)) break Or something - I'm fairly sure that code wouldn't quite work, but something similar. Does anyone have any suggestions how to handle this? My existing (home-grown, nasty) async framework can handle a lot of queries a second; upwards of several hundred, and it does a very good job of sharing out the load evenly so that fast systems complete quickly but the slower ones still get enough time to a) not suffer PDU timeouts and b) not exhaust inbound socket buffers. The way my current code works is: def receive(): r,w,x = select.select(fds,[],[]) for rfd in r: while True: try: pdu = r.read() if not pdu: break mark_answered(pdu) except: pass return len(r) else: return None while self.jobs: while True: if not receive(): break job = job.nextptr if job.waiting: if job.waiting.answered: job.receive() elif job.waiting.due < now: try: job.waiting.retry() except: # Out of retries job.cancel() You can probably see the way this works - on every loop, receive IO takes absolute precedence but goes into an input queue, and the input queue only has one "receive" event per IO-empty - *and* there's a pointer into the queue of jobs to ensure that they're serviced in a round-robin fashion if ready. Sadly, the pseudo-code above makes it look much cleaner than the real code, and I want the other protocol / service support along with other things like the deferred pattern and such. I would welcome any suggestions.

4 11

[Twisted-Python] problems with trial and adbapi
by donfu Oct. 4, 2004

Oct. 4, 2004

Dear Twisters I'm diving into the twisted matrix. Mostly really enjoying it, but sometimes it's a tough ground :) Now i'm trying to set up unittests with trial for a small twisted datastore. the store itself works outside of trial, but strange things happen when running the tests. The first thing I had to learn, is to explicitly start the dbpool with dbpool.start(). Now the tests work sometimes, but sometimes not. I guess the problem has something to do with adbapi's usage of threads and my using of 'deferredResult()' in the tests. can someone help me here? thanks, donfu.

6 8

[Twisted-Python] Proposed change to buffering
by Itamar Shtull-Trauring Oct. 4, 2004

Oct. 4, 2004

Now that we don't do write-at-once, lists are easier to use. Benefits: Allows protocols with many small writes to not have to do their own buffering (e.g. http, imap4 as currently coded, PB, IRC, telnet, etc. etc.) Problems: Bit more overhead for large writes. There's also a bit of code change in tcp.py for the SSL code. Index: abstract.py =================================================================== --- abstract.py (revision 11908) +++ abstract.py (working copy) @@ -43,7 +43,9 @@ if not reactor: from twisted.internet import reactor self.reactor = reactor - + self._tempDataBuffer = [] # will be added to dataBuffer in doWrite + self._tempDataLen = 0 + def connectionLost(self, reason): """The connection was lost. @@ -81,6 +83,9 @@ there; a result of 0 implies no write was done, and a result of None indicates that a write was done. """ + self.dataBuffer += "".join(self._tempDataBuffer) + self._tempDataBuffer = [] + self._tempDataLen = 0 # Send as much data as you can. if self.offset: l = self.writeSomeData(buffer(self.dataBuffer, self.offset)) @@ -131,15 +136,16 @@ if not self.connected: return if data: - self.dataBuffer = self.dataBuffer + data + self._tempDataBuffer.append(data) + self._tempDataLen += len(data) if self.producer is not None: - if len(self.dataBuffer) > self.bufferSize: + if len(self.dataBuffer) + self._tempDataLen > self.bufferSize: self.producerPaused = 1 self.producer.pauseProducing() self.startWriting() def writeSequence(self, iovec): - self.write("".join(iovec)) + self._tempDataBuffer.extend(iovec) def loseConnection(self): """Close the connection at the next available opportunity.

1 0

[Twisted-Python] Weekly Bug Summary
by James Y Knight Oct. 3, 2004

Oct. 3, 2004

Bug summary ______________________ Since 2004-09-25 17:50:20.840000: Bugs opened: 3 Bugs closed: 12 Total open bugs: 302 (-9) New / Reopened Bugs ______________________ twisted/xish/domish.py bug breaks jabber protocol implementation (12 hours ago) http://www.twistedmatrix.com/bugs/issue734 opened by doko twisted's SIGCHLD handler breaks popen. (3 days ago) http://www.twistedmatrix.com/bugs/issue733 opened by jknight test_journal.Service defines addtime twice in the same scope and uses undefined name "journal" (7 days ago) CLOSED http://www.twistedmatrix.com/bugs/issue732 opened by exarkun Bugs Closed ______________________ t.p.c.MetaInterface.__getattr__ causes infinite recursion (1 week ago) http://www.twistedmatrix.com/bugs/issue730 opened by slyphon test_journal.Service defines addtime twice in the same scope and uses undefined name "journal" (7 days ago) http://www.twistedmatrix.com/bugs/issue732 opened by exarkun udp.Port and udp.ConnectedPort use isinstance (2 months ago) http://www.twistedmatrix.com/bugs/issue651 opened by burris [PATCH] twisted.protocols.portforward has a couple of problems. (7 months ago) http://www.twistedmatrix.com/bugs/issue527 opened by Omnifarious [MAIL] SMTPSenderFactory has unintended side effect (7 months ago) http://www.twistedmatrix.com/bugs/issue540 opened by mjs twistd -y fails with dos-style newlines (5 months ago) http://www.twistedmatrix.com/bugs/issue595 opened by phed twisted.internet.{tcp,udp}.Port lie when port == 0 (1 week ago) http://www.twistedmatrix.com/bugs/issue725 opened by anthony twisted.names.dns.Record_NULL is stupid (1 week ago) http://www.twistedmatrix.com/bugs/issue729 opened by anthony ReconnectingClientFactory can fail to reconnect on connection timeouts (2 weeks ago) http://www.twistedmatrix.com/bugs/issue722 opened by exarkun [CORE] Add process support to default reactor on windows (5 months ago) http://www.twistedmatrix.com/bugs/issue598 opened by itamarst Persistant Adapters at adapter registration time (6 months ago) http://www.twistedmatrix.com/bugs/issue556 opened by ironfroggy tempfile.mkstemp in twisted.python.compat (2 weeks ago) http://www.twistedmatrix.com/bugs/issue713 opened by yacc

1 0

[Twisted-Python] PATCH: twisted.web.client Basic/Digest Authentication
by Mathieu Fenniak Oct. 1, 2004

Oct. 1, 2004

The attached patch adds support for HTTP authentication to twisted.web.client. It supports both basic and digest authentication. This patch adds a new module, twisted.web.auth, which I hope to expand in the future to support digest server-side authentication. A new class is also added to twisted.web.error, which passes on the WWW-Authenticate headers such that authentication could be handled externally. I have never contributed before to Twisted, so I'd appreciate feedback on this patch. It works, and is pretty clean code, but I don't know if there are some practices I might be breaking without being aware. This patch is based on repository v.11878. -- Mathieu Fenniak <mfenniak(a)pobox.com> http://stompstompstomp.com/

2 1

[Twisted-Python] ANN: Fusion 0.1, C++ / Twisted integration
by Itamar Shtull-Trauring Oct. 1, 2004

Oct. 1, 2004

This allows you to write protocols for UDP, multicast and TCP in C++. In theory this can make the protocol much faster since it's easier to prevent data copying etc.. The TCP writing support is not efficient yet, though, as it still uses Twisted's current buffering system. Tested with 1.3.0 and 2.0 from trunk. No API stability guaranteed. http://itamarst.org/software/fusion-0.1.tar.gz or sandbox/itamar/cppreactor in Twisted SVN. Disclaimer: if you want to make your Twisted code go faster this is NOT the first thing you should try. Nor the second, third or fourth. -- Itamar Shtull-Trauring http://itamarst.org

1 1