Mailman 3 December 2005 - Twisted

[Twisted-Python] Re: [Twisted-commits] r15316 - Reorganize davxml.py into a few files, still exposed as davxml.py.
by Jean-Paul Calderone 10 Dec '05

10 Dec '05

On Fri, 09 Dec 2005 19:18:17 -0700, Wilfredo Sanchez <wsanchez(a)wolfwood.twistedmatrix.com> wrote: >Author: wsanchez >Date: Fri Dec 9 19:18:17 2005 >New Revision: 15316 > >Added: > branches/wsanchez/dav/twisted/web2/dav/elements/ > branches/wsanchez/dav/twisted/web2/dav/elements/__init__.py > branches/wsanchez/dav/twisted/web2/dav/elements/base.py > - copied, changed from r15315, branches/wsanchez/dav/twisted/web2/dav/davxml.py > branches/wsanchez/dav/twisted/web2/dav/elements/rfc2518.py > - copied, changed from r15315, branches/wsanchez/dav/twisted/web2/dav/davxml.py > branches/wsanchez/dav/twisted/web2/dav/elements/rfc3253.py > - copied, changed from r15315, branches/wsanchez/dav/twisted/web2/dav/davxml.py > branches/wsanchez/dav/twisted/web2/dav/elements/rfc3744.py > - copied, changed from r15315, branches/wsanchez/dav/twisted/web2/dav/davxml.py >Modified: > branches/wsanchez/dav/twisted/web2/dav/davxml.py >Log: >Reorganize davxml.py into a few files, still exposed as davxml.py. >davxml.py was getting way too big. Now elements are in files split up by RFC. >We no longer have to manage __all__. These elements don't constitute part of the application-level API, right? Jean-Paul

1 0

RE: [Twisted-Python] RE: waiting on transport return
by jmbenski＠micron.com 09 Dec '05

09 Dec '05

Maybe I misspoke. The way I understand it, is that deferreds do not get CPU time until the reactor resumes control. Once the reactor gets control it allows events to happen, some of which may be deferreds. I'm not sure what does the scheduling or what order deferred objects get control of the thread. I had assumed that the reactor or whatever scheduler kicked off deferreds in the order that they were created. Once that deferred chain got control of the main loop it would run to completion and the next event whether a deferred or something else would run. In my case I need the deferred to wait until my commandComplete function runs. Is the aDeferred.callback(result) the trigger that starts the deferred? If that is the case, then I understand how my current code is working and I understand that there IS no scheduler for the deferreds. Deferreds fire when a callback or errback function is called on them. End of story and no magic done behind the scenes. Jon -----Original Message----- From: twisted-python-bounces(a)twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of Itamar Shtull-Trauring Sent: Friday, December 09, 2005 11:29 AM To: Twisted general discussion Subject: RE: [Twisted-Python] RE: waiting on transport return On Fri, 2005-12-09 at 11:02 -0700, jmbenski(a)micron.com wrote: > After looking at the code a bit more and your change below, I'm > surprised that my code works. Why isn't the reactor kicking off my > deferreds that are stored in the queue? Deferreds have nothing to do with the reactor. So, doing aDeferred.callback(result) or .errback(error) are the only way it gets a value. _______________________________________________ Twisted-Python mailing list Twisted-Python(a)twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

2 1

RE: [Twisted-Python] RE: waiting on transport return
by jmbenski＠micron.com 09 Dec '05

09 Dec '05

I guess I'm still trying to wrap my head around deferreds and how best to use them as they were intended. The way that code works, I have to pass my deferred chain to sendCommand. The protocol needs to store that deferred on its deferred stack, so that I can pass the results of the command back to the deferred function. After looking at the code a bit more and your change below, I'm surprised that my code works. Why isn't the reactor kicking off my deferreds that are stored in the queue? My current logic is dependent on the protocol running the commandComplete function before the deferreds fire in the protocol's deferred queue. #Do some thing #Send the command self.processProtocol.sendCommand #Do some more #return control to the reactor ??Here the reactor should kick off my command and run any deferreds. Am I just lucky that my server returns the response(commandComplete) before the deferreds in the queue fire? The return of the data and the chain of deferreds are not linked in any way, which I guess is what my original question is. How can my commandComplete return a deferred and kick off a logic chain? How is it working now? -----Original Message----- From: twisted-python-bounces(a)twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of Jean-Paul Calderone Sent: Friday, December 09, 2005 10:01 AM To: Twisted general discussion Subject: RE: [Twisted-Python] RE: waiting on transport return On Fri, 9 Dec 2005 09:03:52 -0700, jmbenski(a)micron.com wrote: >I'm not trying to bump my question, but if someone could tell me if I'm just being stupid and missing something with this code or that it isn't possible to do what I need with a spawned process? If I know it isn't possible, I can just continue down the path I am. It's not too much of a pain, but it just feels...wrong. > The code mostly seems good to me. The only thing I might change is completely trivial: instead of passing a Deferred in to sendCommand, I'd have sendCommand be responsible for creating the Deferred and then return it, so your application code would look like this - d = self.processprotocol.sendCommand("syncCmd", cmd) d.addCallback(self.handleTheCommand) d.addCallback(Self.theRestOfTheCommands) I'm not sure from where your unhappyness with the version you posted stems, so I'm not sure what other comments to make. Jean-Paul _______________________________________________ Twisted-Python mailing list Twisted-Python(a)twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

2 1

RE: [Twisted-Python] RE: waiting on transport return
by jmbenski＠micron.com 09 Dec '05

09 Dec '05

I'm not trying to bump my question, but if someone could tell me if I'm just being stupid and missing something with this code or that it isn't possible to do what I need with a spawned process? If I know it isn't possible, I can just continue down the path I am. It's not too much of a pain, but it just feels...wrong. Thanks, Jon -----Original Message----- From: twisted-python-bounces(a)twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of jmbenski Sent: Tuesday, December 06, 2005 7:58 PM To: twisted-python(a)twistedmatrix.com Subject: [Twisted-Python] RE: waiting on transport return Ok, I need some help to write some code using deferred's the way they were meant, not the way I've bastardized them. My framework... I have a client that will connect to a server and MAINTAIN that connection. I can't create a protocol factory. The server and the client communicate commands through stdin and stdout. I need to write my commands to the server and wait for the response in some cases. I've setup a system that passes a deferred chain to my process protocol and once my process knows that the data it needs is received it will kick off the deferred chain that happens after that response. This seems backwards. From my reading and understanding of deferreds, they should bubble down and not have to be pushed up. How do I handle the case of some protocol command kicking off the rest of a deferred chain or of passing back a deferred, so that I can add to it the things that need to run? class SyncManagerProtocol(protocol.ProcessProtocol): def __init__(self ): self.dataStore = "" ##queue that holds the commands self.cmdQueue = list() ##queue that holds the callback function to call once the call returns self.deferredQueue = list() def outReceived(self, data): if len(self.cmdQueue) > 0: ##take the data and shove it into our internal store self.dataStore += data ##determine what is good data cmd = self.cmdQueue[0] ##look for the command the is next in the queue. grab all the return data ##between the command block. ## syncCmd { ## return data goes here ## } syncCmd cmdBlock = r"%s {(?P<cmd>.*)} %s(?P<rest>.*)" % (cmd, cmd) errBlock = r"invalid {(?P<errMsg>.*)}" cmdMatch = re.compile( cmdBlock, re.S ).search( self.dataStore ) if cmdMatch: ##set the rest of the data to the dataStore. This clears out the current contents ##but keeps any extra data that may be coming in from the next command self.dataStore = cmdMatch.group( "rest" ) ##clean up this command self.commandComplete( cmdMatch.group("cmd").lstrip() ) ##This is called if all the data we need has been returned. def commandComplete(self, cmdData): ##pop this command since we are done with it self.cmdQueue.pop(0) ##pop the deferred that matches this command deferred = self.deferredQueue.pop(0) ##pass the results to the deferred deferred.callback( cmdData ) ##interface to the process that sends a command to execute def sendCommand( self, cmd, args, deferred ): ##store the command in the queue, so we can check it when we get data self.cmdQueue.append( cmd ) ##store the deferred in the queue that we need to run when this command has been completed self.deferredQueue.append( deferred ) ##write out command to the process cmd = cmd + " " + args + "\n" self.transport.write( cmd ) ## This code is my client interface class to the server that holds the processProtocol instance. ## #setup a command that will handle errors or good data d = defer.Deferred().addCallback( self.handleThisCommand ) #run the calling function handler that will handle an error or returned data. This will allow something like a GUI to process data that will now be available d.addCallback( self.theRestOfTheCommands ) cmd = "some command goes here" self.processProtocol.sendCommand( "syncCmd", cmd, d)

2 1

[Twisted-Python] (no subject)
by kwhitesell＠adelphia.net 09 Dec '05

09 Dec '05

Hi Frank, Best that I can tell, it's not your various classes that need to be converted to deferreds, just your program that calls class1, or even class1 itself. The bottom line is that _something_ higher up the calling stack needs to be aware that this is an asynchronous process - and that a deferred gets generated along the way. I believe it would be helpful if the originating function is coded for this. I'm not a Twisted guru, so there may be an easier way to do this - but as I see it... I would be writing the classes to look more like the following: class1(): def check1(data): perform test if test failed: return defer.succeed(False) return class2.check2(data): class2(): def check2(data): perform test if test failed: return defer.succeed(False) return class3.check3(data): class3(): def check3(data): perform test if test failed: return defer.succeed(False) aDeferred = avatar.callRemote(check4,data) return aDeferred Then, my originating class (and possibly becomming a wrapper class if this is used in many places). class0(): def check0(data): class1.check1(data).addCallback(afterCheck) def afterCheck(result): return result This way, all the functions return a deferred back up to the calling program - but the only one that causes a delay is the call to check4. Ken

3 3

[Twisted-Python] ftpClient.py, retrieveFile, and line ending issues
by Keith.Gunderson＠act.org 07 Dec '05

07 Dec '05

Hello, If any of you have a spare minute, I can't figure out else where I might solve my present dilemma. Its certainly likely that my only issues are ignorance about Python's file operations or the features that are a part of other FTP clients that I use. I'm using the ftpclient.py code to write a script which downloads file via FTP. My example code shows the basics. The file is transferred, but the lines seem to have different line endings when I view them 'hexlified' in xemacs. My twisted-created files has hex 0D0D 0D0D 0D0D at the end of lines while a file downloaded using SmartFTP has hex 0D0A 0D0A. Is it clear, based on that information, if there is an issue in my code? Or an easy way to fix it? class FileWriterProtocol(Protocol): def __init__(self, filename): self.file = open(filename, 'w') def dataReceived(self, data): self.file.write(data) def connectionMade(ftpClient): proto2 = FileWriterProtocol( 'c:/ftpfile1.txt' ) # Get the file, and quit when done d = ftpClient.retrieveFile( 's1/s2/T2005C.processed' , proto2) d.addCallback(lambda result: reactor.stop()) Thanks very much, Keith Gunderson

3 3

[Twisted-Python] Weekly Bug Summary
by James Y Knight 07 Dec '05

07 Dec '05

KIND == 'create' Bug summary ______________________ Since 2005-11-30 07:00:02.250000: Bugs opened: 11 Bugs closed: 2 Total open bugs: 493 (+9) Urgent: 2 (+1) Bug: 232 (+5) Feature: 176 (+3) Wish: 83 (+0) New / Reopened Bugs ______________________ ==== Urgent ==== [TRIAL] trial passes deferredGenerator test cases without running the test case (8 hours ago) http://twistedmatrix.com/bugs/issue1353 opened by hagna [CORE] FilePath sucks (5 days ago) CLOSED http://twistedmatrix.com/bugs/issue1348 opened by glyph ==== Bug ==== [CORE] threadedselectreactor queue size (5 days ago) http://twistedmatrix.com/bugs/issue1347 opened by Toby Dickenson Separate reporter result handling from output (5 days ago) http://twistedmatrix.com/bugs/issue1349 opened by jml [CORE, WIN32] iocpreactor [Errno 10014] The system detected an invalid pointer address in attempting to use a pointer argument in a call (6 days ago) http://twistedmatrix.com/bugs/issue1346 opened by ghazel [CORE, WIN32] socket.error: (19, 'No such device') (6 days ago) http://twistedmatrix.com/bugs/issue1345 opened by ghazel [CORE] posixbase win32 waker problem (6 days ago) http://twistedmatrix.com/bugs/issue1344 opened by antony [CORE] readers and writers not removed properly from threadedselectreactor (6 days ago) http://twistedmatrix.com/bugs/issue1343 opened by antony ==== Feature ==== [TRIAL] [PATCH] HTML reporter for trial (4 days ago) http://twistedmatrix.com/bugs/issue1350 opened by dialtone log observer bridging to python logging facility (4 days ago) http://twistedmatrix.com/bugs/issue1351 opened by ctheune ==== Wish ==== twisted + dbus 0.60 howto (yesterday) http://twistedmatrix.com/bugs/issue1352 opened by senko Bugs Closed ______________________ ==== Urgent ==== [CORE] FilePath sucks (5 days ago) http://twistedmatrix.com/bugs/issue1348 opened by glyph, closed by <unknown> ==== Bug ==== twisted.web.html.output bug (IRC) (1 week ago) http://twistedmatrix.com/bugs/issue1337 opened by kyle, closed by exarkun

1 0

[Twisted-Python] Question on deferreds
by Ken Whitesell 07 Dec '05

07 Dec '05

Frank, Ok, your clarifications make things a little more difficult, but not impossible. Standard disclaimer: I'm no Twisted-guru. There may be an easier way to do this. I can only take this as far as my knowledge extends. So having said that... There's no way to get around the fact that the calling program needs to be 'broken' down into two parts, 'separated' by the deferred. In other words, to greatly simplify the situation, your flow should be something like: function 1 do stuff... do more stuff... do more stuff... callRemote... addCallback(function2) return function 2 do more stuff... do more stuff... do more stuff... end of tests. The bottom line is that your entire program flow needs to be reworked around the Twisted methodology. (Or, in other words, you need to Twist your program. <g>) I don't remember where I saw this first, but the best way I know to describe this situation is to stop thinking of your program as a program and Twisted as a library. Twisted is the program, and your code is the library. Your functions get called by the reactor because of some event. You handle this event then return control to the reactor. The reactor then calls other functions as necessary. So you perform some tests, then call remote to invoke a process on the server, register your callback and yield control back to the reactor. When the reply is received, a different function gets control and you continue processing. One way this can look, is to take your original program (section 1) and modify it to look more like the pseudo code below. *** Original code, with modified class 3 *** class1(): def check1(data): perform test if test failed: return False if not class2.check2(data): return False do_additional_stuff_1 return True class2(): def check2(data): perform test if test failed: return False if not class3.check3(data): return False do_additional_stuff_2 return True class3(): def check3(data): perform test if test failed: return False *** This is where the flow breaks *** avatar.callRemote(check4,data).addCallback(afterCheck) if not class4.check4(data): return False do_additional_stuff_3 return True def afterCheck(result): return result The simplest rework I can come up with for this is: class0(): def check0(data): part1 = class1.check1(data) # Assuming part1 is either false or a deferred if part1: part1.addCallback(afterCheck) else callLater(0, afterCheck, False) def afterCheck(result): # part 2 # Do the stuff after check4 # This would include the stuff that needs to be done # after the tests. So code from # do_additional_stuff_x # can be called from here. finalResults = class1.do_additional_stuff_1(data) return finalResult class1(): def check1(data): perform test if test failed: return False if not class2.check2(data): return False # Note, we can't get here so this code can go # return True def do_additional_stuff_1(data): class2.do_additional_stuff_2(data) more suff goes here. class2(): def check2(data): perform test if test failed: return False if not class3.check3(data): return False # Note, we can't get here so this code can go # return True def do_additional_stuff_2(data): class3.do_additional_stuff_3(data) more suff goes here. class3(): def check3(data): perform test if test failed: return False *** This is where the flow breaks *** aDeferred = avatar.callRemote(check4,data).addCallback(afterCheck) return aDeferred def do_additional_stuff_3(data): suff goes here. ************************************** As you can see, it does chop things up a bit - and I'm still not sure that I've really captured what you're trying to do. I've just tried to match your existing class structure - which I believe to be sub-optimal in this context, given that I'm working from some very generic descriptions. Specifically, I'm not sure if these "class1, class2 and class3" definitions are truly classes, or if they're simply "organizational usage groups". (I'm not judging one way or the other, I'm only working from what you've written and what assumptions I can make - this isn't meant to disparage your code, just that I don't feel like I have enough solid information to make a more accurate judgement.) Hope this helps spark some ideas on your end. Ken

1 0

[Twisted-Python] RE: waiting on transport return
by jmbenski＠micron.com 07 Dec '05

07 Dec '05

Ok, I need some help to write some code using deferred's the way they were meant, not the way I've bastardized them. My framework... I have a client that will connect to a server and MAINTAIN that connection. I can't create a protocol factory. The server and the client communicate commands through stdin and stdout. I need to write my commands to the server and wait for the response in some cases. I've setup a system that passes a deferred chain to my process protocol and once my process knows that the data it needs is received it will kick off the deferred chain that happens after that response. This seems backwards. From my reading and understanding of deferreds, they should bubble down and not have to be pushed up. How do I handle the case of some protocol command kicking off the rest of a deferred chain or of passing back a deferred, so that I can add to it the things that need to run? class SyncManagerProtocol(protocol.ProcessProtocol): def __init__(self ): self.dataStore = "" ##queue that holds the commands self.cmdQueue = list() ##queue that holds the callback function to call once the call returns self.deferredQueue = list() def outReceived(self, data): if len(self.cmdQueue) > 0: ##take the data and shove it into our internal store self.dataStore += data ##determine what is good data cmd = self.cmdQueue[0] ##look for the command the is next in the queue. grab all the return data ##between the command block. ## syncCmd { ## return data goes here ## } syncCmd cmdBlock = r"%s {(?P<cmd>.*)} %s(?P<rest>.*)" % (cmd, cmd) errBlock = r"invalid {(?P<errMsg>.*)}" cmdMatch = re.compile( cmdBlock, re.S ).search( self.dataStore ) if cmdMatch: ##set the rest of the data to the dataStore. This clears out the current contents ##but keeps any extra data that may be coming in from the next command self.dataStore = cmdMatch.group( "rest" ) ##clean up this command self.commandComplete( cmdMatch.group("cmd").lstrip() ) ##This is called if all the data we need has been returned. def commandComplete(self, cmdData): ##pop this command since we are done with it self.cmdQueue.pop(0) ##pop the deferred that matches this command deferred = self.deferredQueue.pop(0) ##pass the results to the deferred deferred.callback( cmdData ) ##interface to the process that sends a command to execute def sendCommand( self, cmd, args, deferred ): ##store the command in the queue, so we can check it when we get data self.cmdQueue.append( cmd ) ##store the deferred in the queue that we need to run when this command has been completed self.deferredQueue.append( deferred ) ##write out command to the process cmd = cmd + " " + args + "\n" self.transport.write( cmd ) ## This code is my client interface class to the server that holds the processProtocol instance. ## #setup a command that will handle errors or good data d = defer.Deferred().addCallback( self.handleThisCommand ) #run the calling function handler that will handle an error or returned data. This will allow something like a GUI to process data that will now be available d.addCallback( self.theRestOfTheCommands ) cmd = "some command goes here" self.processProtocol.sendCommand( "syncCmd", cmd, d)

1 0

[Twisted-Python] XML consumers and protocols
by Keith.Gunderson＠act.org 06 Dec '05

06 Dec '05

Another question I'm trying to hunt down regarding my plans to develop an ftpClient that will download XML files... How can I teach myself what is important to know about the relationship between Protocols and Consumers? How much of these concepts are twisted specific... or Components/ Interfaces ... or just native Python stuff. I have a notion of abstractly passing data from an ftpClient to an XML parser, but I'm obviously not there yet. Thanks! Keith Gunderson

1 0

[Twisted-Python] [patch] issue 630: timezone in twisted log
by Alexey Shamrin 05 Dec '05

05 Dec '05

Hello! Nearly half a year ago I took issue 630 from twisted bug database [1] and tried to fix it. This bug is good to fix, because it makes the experience of looking at the console after issuing "twistd -noy some.tac" much better. But after adding several comments on the issue and attaching patches, I recieved ZERO response... :-( [1] http://twistedmatrix.com/bugs/issue630 -- Alexey

1 0

[Twisted-Python] Question on deferreds
by Frank Millman 04 Dec '05

04 Dec '05

Hi all, I am trying to convert my app to use Twisted. The original version handles a gui, some business logic, and a database connection, all in one program. I have split it into a 'client' program, that handles the gui; and a server program, that handles the business logic and the database connection, and communicates with the client using pb. The original program has some logic to validate user input. The data to be validated is passed up through 3 or 4 classes. Here is some very simplified pseudo code - class1(): def check1(data): perform test if test failed: return False if not class2.check2(data): return False return True class2(): def check2(data): perform test if test failed: return False if not class3.check3(data): return False return True class3(): def check3(data): perform test if test failed: return False if not class4.check4(data): return False return True class4(): def check4(data): perform test if test failed: return False return True In my new program, classes 1-3 run on the client, class4 runs on the server. Therefore I tried to change class3 as follows - class3(): def check3(data): avatar.callRemote(check4,data).addCallback(afterCheck) def afterCheck(result): return result This does not work, as check1 and check2 never receive the result. I would like to avoid going through my entire program and changing each occurrence of check1 and check2 into a deferred and a callback. I do not particularly mind if the client program blocks, as I do not want the user to proceed until the result is known. Is there any way to accomplish this? Thanks for any advice Frank Millman

1 0

[Twisted-Python] Request for ideas
by Ken Whitesell 04 Dec '05

04 Dec '05

Warning - fairly long post follows for what I hope is a simple question that only needs (possibly) a short answer. (I'm not looking for any code, just one of a couple possible ideas.) Environment: A LAN with a server and multiple PCs. Full control is available over all the PCs. I have the ability to run whatever I want on each - in this case it means Python 2.4 / Twisted 2. Background / Use Case: Imagine a simultaneous silent auction. (More like an RFP-type situation.) Any of the PCs can put an item up for bid. Other PCs can respond (or ignore!) that item. There are two timeout requirements - an "acknowledgement" timeout, where the PC has a certain period of time to respond "I want to make a bid", and a response timeout that restricts the clients to how much time they have to make the bid. These timeouts can change depending on the item going up for bid. (How and 'what' goes up for bid is irrelevant to my question.) What I've done so far: I've created a class called "CommTrans" to manage the auction. Each time an item goes up for bid, an instance of CommTrans is created. CommTrans creates two objects, one each to manage the acknowlegements and the other to manage the replies (AckMgr, RepMgr). The RepMgr is a Viewable that can be called by the client to submit its bid. (RepMgr is passed to the client in the callRemote call.) CommTrans issues the callRemote, passing RepMgr to the client. Then it sets a function in the AckMgr as the callback. The client receives the request, and decides whether or not to respond to the request. If yes, it issues a callLater on the function to determine the bid and submit it to the RepMgr. Either way, the original function returns either "Yes" or "No", triggering the callback on the CommTrans. Both the AckMgr and RepMgr issue callLater to schedule the timeout. If all possible machines reply before the timeout occurs, the timeout is cancelled. If the timeout occurs first, a flag is set to have the objects reject all further information from the clients. (It's simply ignored, nothing special occurs.) QUESTION #1: Did I miss something in the Twisted API that would handle this type of situation? I don't think a Deferred list would work in this situation, because it's not going to fire until all deferrs complete - something which might not occur. But this isn't the problem - the code that I have written for these functions all work. The specific problem occurs when objects are reused. I've had situations where a client fails to reply for significantly longer than the timeout period. In that interval, the original CommTrans, AckMgr and RepMgr have been deleted and replaced by new instances. The client finally replies to the original request, and the reply is caught by those new objects! (There is what I think is a very small, but non-zero probability that I'm misinterpreting what I'm seeing.) From what I can see in the source code, the internal ID is what's set by the builtin 'id' function. I'm printing these at key times and I can see where they're being duplicated. The only time I see this problem occur is when a new object has been created with the same id as a previous object. So this leads me to QUESTION #2: Anyone have ideas on how to resolve this? (Note: I can't channel everything through a single CommTrans object, because multiple bids might be happening concurrently with overlapping timeouts.) About the only thought I've had so far is to generate a GUID-like object for each CommTrans, and include that in every communication to and from the clients. The server would check each ack and each reply to ensure that the ack and reply are for the proper auction. Sending another message to the client isn't going to be a good solution, because the client might be tied up doing other things - starving the reactor - so by the time the message is processed, it may be too late. This also brings up QUESTION #3, although it hasn't happened yet: Server does a callRemote to the client from a commTrans object. Before the remote function returns (client is _extremely_ busy) the commTrans object goes away. What will happen when the function exits? (I know, I ought to generate a test-case for this - I'm more wondering if anyone knows right off-hand.) All thoughts, ideas, comments, recommendations are encouraged. Thanks to any/everyone who managed to get this far through my writing. Finally, I'm most pleased to announce that my proposal to provide a "Getting Started with Twisted" tutorial at PyCon has been accepted! I know I can't turn people into Twisted gurus in 3 hours, but I hope I can get a few over that initial hurdle. Thanks, Ken

2 1

[Twisted-Python] Twisted Words: throttling server output
by Seventh Holy Scripture 03 Dec '05

03 Dec '05

I've been writing an IRC bot using Twisted Words, and am having difficulty keeping it from flooding itself off the server. Currently my output subroutines look like this: def queueMsg(self, channel, msg): self.chanQueue.append(channel) self.msgQueue.append(msg) def emptyMsgQueue(self): for msg in self.msgQueue: channel = self.chanQueue.pop(0) self.msg(channel, msg) if channel[0] == "#": self.logger.log("<%s> %s" % (self.nickname, msg)) time.sleep(0.5) self.msgQueue = [] This ought to work as far as I know... but the bot waits until after all the sleep statements have executed to spew the output to the IRC server, which (if there's a lot of it) results in a forced disconnect due to "Excess Flood". What am I doing wrong here? My guess is that twisted has its own message queue, and waits until after all host program subroutines have ended to process it... I'm not good at python, though, so it could be a stupid mistake on my part. In any case, is there another way to introduce a delay between the messages sent to server that will work?

3 2

[Twisted-Python] Non-list expression received, 'Server' object has no attribute 'socket'
by Alex Couper 02 Dec '05

02 Dec '05

I have an application that requires each node using it to update each other, and to fetch updates automatically. Each node, therefore, has a server and a client on it. I am receiving different errors each time I run the program, and they are coming from within the twisted packages, (shown below) Error 1: File "C:\Python24\lib\site-packages\twisted\internet\tcp.py", line 89, in _closeSocket skt = self.socket exceptions.AttributeError: 'Server' object has no attribute 'socket' Error 2: File "C:\Python24\lib\site-packages\twisted\spread\pb.py", line 572, in expressionReceived raise ProtocolError("Non-list expression received.") twisted.spread.pb.ProtocolError: Non-list expression received. I suspect that they are caused by the way that I am running my server. On occasion, reactor.run() finishes, and thus there is no longer a server running in effect. I therefore use a while loop to always be running the server. The server is launched as a thread. def run(self): while 1: application = service.Application("copy_receiver") internet.TCPServer(8800, pb.PBServerFactory (self)).setServiceParent( service.IServiceCollection(application)) self.listener = reactor.listenTCP(8800, pb.PBServerFactory (self)) #installSignalHandlers=0 allows the reactor to run in a non-main thread reactor.run(installSignalHandlers=0) Is this known to cause problems? If not, any ideas of why i might be getting the error messages? or any ideas how I can keep the server running so that it doesn't exit? Alex Couper

2 1

[Twisted-Python] How can a tcp client connect with multi servers?
by Xu Ryan 02 Dec '05

02 Dec '05

Here is the situation: a tcp control client need connect to multi-agent. Is there a light way to connect with server , instead make a factory to do each connect?

2 18

[Twisted-Python] Help needed with SmartHostSMTPRelayingManager
by James Tanis 02 Dec '05

02 Dec '05

Right now I've got this SMTP server that I've slowly been piecing together using Twisted 2.1. I was able to get everything working as far as local delivery, but after adding a relaymanager.Queue the server was able to receive and queue messages from a mail client as well as deliver to local accounts. The problem comes in with relaymanager.SmartHostSMTPRelayingManager. Here is the log with traceback: 2005/12/01 21:24 EST [-] Log opened. 2005/12/01 21:24 EST [-] twistd 2.1.0 (/usr/local/bin/python 2.4.2) starting up 2005/12/01 21:24 EST [-] reactor class: twisted.internet.selectreactor.SelectReactor 2005/12/01 21:24 EST [-] Loading ./h2smtp.py... 2005/12/01 21:24 EST [-] Set 71757_1133487291.33_0_138982188 waiting 2005/12/01 21:24 EST [-] Set 71757_1133487291.33_0_138982188 waiting 2005/12/01 21:24 EST [-] /etc/resolv.conf changed, reparsing 2005/12/01 21:24 EST [-] Resolver added ('24.197.160.17', 53) to server list 2005/12/01 21:24 EST [-] Resolver added ('24.197.160.18', 53) to server list 2005/12/01 21:24 EST [-] twisted.names.dns.DNSDatagramProtocol starting on 62457 2005/12/01 21:24 EST [-] Loaded. 2005/12/01 21:24 EST [-] twisted.mail.protocols.ESMTPFactory starting on 2500 2005/12/01 21:24 EST [-] Starting factory <twisted.mail.protocols.ESMTPFactory instance at 0x85b04ec> 2005/12/01 21:24 EST [-] twisted.mail.protocols.POP3Factory starting on 1100 2005/12/01 21:24 EST [-] Starting factory <twisted.mail.protocols.POP3Factory instance at 0x85b052c> 2005/12/01 21:24 EST [twisted.names.dns.DNSDatagramProtocol (UDP)] Starting factory <twisted.mail.relaymanager.SMTPManagedRelayerFactory instance at 0x85fff0c> 2005/12/01 21:24 EST [Uninitialized] Traceback (most recent call last): File "/usr/local/lib/python2.4/site-packages/twisted/python/log.py", line 58, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/usr/local/lib/python2.4/site-packages/twisted/python/log.py", line 43, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/local/lib/python2.4/site-packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/local/lib/python2.4/site-packages/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) --- <exception caught here> --- File "/usr/local/lib/python2.4/site-packages/twisted/internet/selectreactor.py", line 139, in _doReadOrWrite why = getattr(selectable, method)() File "/usr/local/lib/python2.4/site-packages/twisted/internet/tcp.py", line 542, in doConnect self._connectDone() File "/usr/local/lib/python2.4/site-packages/twisted/internet/tcp.py", line 545, in _connectDone self.protocol = self.connector.buildProtocol(self.getPeer()) File "/usr/local/lib/python2.4/site-packages/twisted/internet/base.py", line 669, in buildProtocol return self.factory.buildProtocol(addr) File "/usr/local/lib/python2.4/site-packages/twisted/mail/relaymanager.py", line 103, in buildProtocol protocol = self.protocol(self.messages, self.manager, *self.pArgs, File "/usr/local/lib/python2.4/site-packages/twisted/mail/relaymanager.py", line 79, in __init__ relay.SMTPRelayer.__init__(self, messages, *args, **kw) File "/usr/local/lib/python2.4/site-packages/twisted/mail/relay.py", line 108, in __init__ smtp.SMTPClient.__init__(self, *args, **kw) exceptions.TypeError: __init__() takes at least 2 arguments (1 given) 2005/12/01 21:24 EST [Uninitialized] Backing off on delivery of ['71757_1133487291.33_0_138982188'] 2005/12/01 21:24 EST [Uninitialized] Stopping factory <twisted.mail.relaymanager.SMTPManagedRelayerFactory instance at 0x85fff0c> >From what I can tell smtp.SMTPClient isn't getting passed identity. Since this error is generated with a call to the relaying manager's .checkState(self) function which only takes self as a argument.. well lets just say I can't figure out what I'm missing from my own tracing through the code in the relay/relaymanager/smtp modules. You should find my code attached, the error should be easily reproducible if you want to run it. -- James Tanis jtanis(a)pycoder.org http://pycoder.org

1 1

[Twisted-Python] Need a SmartHost(E)SMTPRelayingManager guru
by James Tanis 02 Dec '05

02 Dec '05

Previously my only experience with Twisted was with 1.3 when I was developing a protocol from the ground up. Right now I've got this SMTP server that I've slowly been piecing together using Twisted 2.1. I was able to get everything working but after experiencing the learning curve firsthand I decided to buy the shiny new Twisted: Network Programming Essentials -- which is a great book I might add. Although the book did open my eyes to things I had missed as well as help me to improve my own code for local delivery it does not seem to cover (E)SMTP relaying which is a valuable component to me. Anyway, after adding a relaymanager.Queue the server was able to receive and queue messages from a mail client as well as deliver to local accounts. The problem comes in with relaymanager.SMTPRelayingManager (relaymanager.ESMTPRelayingManager gave me errors even earlier so I dropped it for the time being). Here is the log with traceback: 2005/12/01 21:24 EST [-] Log opened. 2005/12/01 21:24 EST [-] twistd 2.1.0 (/usr/local/bin/python 2.4.2) starting up 2005/12/01 21:24 EST [-] reactor class: twisted.internet.selectreactor.SelectReactor 2005/12/01 21:24 EST [-] Loading ./h2smtp.py... 2005/12/01 21:24 EST [-] Set 71757_1133487291.33_0_138982188 waiting 2005/12/01 21:24 EST [-] Set 71757_1133487291.33_0_138982188 waiting 2005/12/01 21:24 EST [-] /etc/resolv.conf changed, reparsing 2005/12/01 21:24 EST [-] Resolver added ('24.197.160.17', 53) to server list 2005/12/01 21:24 EST [-] Resolver added ('24.197.160.18', 53) to server list 2005/12/01 21:24 EST [-] twisted.names.dns.DNSDatagramProtocol starting on 62457 2005/12/01 21:24 EST [-] Loaded. 2005/12/01 21:24 EST [-] twisted.mail.protocols.ESMTPFactory starting on 2500 2005/12/01 21:24 EST [-] Starting factory <twisted.mail.protocols.ESMTPFactory instance at 0x85b04ec> 2005/12/01 21:24 EST [-] twisted.mail.protocols.POP3Factory starting on 1100 2005/12/01 21:24 EST [-] Starting factory <twisted.mail.protocols.POP3Factory instance at 0x85b052c> 2005/12/01 21:24 EST [twisted.names.dns.DNSDatagramProtocol (UDP)] Starting factory <twisted.mail.relaymanager.SMTPManagedRelayerFactory instance at 0x85fff0c> 2005/12/01 21:24 EST [Uninitialized] Traceback (most recent call last): File "/usr/local/lib/python2.4/site-packages/twisted/python/log.py", line 58, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/usr/local/lib/python2.4/site-packages/twisted/python/log.py", line 43, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/local/lib/python2.4/site-packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/local/lib/python2.4/site-packages/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) --- <exception caught here> --- File "/usr/local/lib/python2.4/site-packages/twisted/internet/selectreactor.py", line 139, in _doReadOrWrite why = getattr(selectable, method)() File "/usr/local/lib/python2.4/site-packages/twisted/internet/tcp.py", line 542, in doConnect self._connectDone() File "/usr/local/lib/python2.4/site-packages/twisted/internet/tcp.py", line 545, in _connectDone self.protocol = self.connector.buildProtocol(self.getPeer()) File "/usr/local/lib/python2.4/site-packages/twisted/internet/base.py", line 669, in buildProtocol return self.factory.buildProtocol(addr) File "/usr/local/lib/python2.4/site-packages/twisted/mail/relaymanager.py", line 103, in buildProtocol protocol = self.protocol(self.messages, self.manager, *self.pArgs, File "/usr/local/lib/python2.4/site-packages/twisted/mail/relaymanager.py", line 79, in __init__ relay.SMTPRelayer.__init__(self, messages, *args, **kw) File "/usr/local/lib/python2.4/site-packages/twisted/mail/relay.py", line 108, in __init__ smtp.SMTPClient.__init__(self, *args, **kw) exceptions.TypeError: __init__() takes at least 2 arguments (1 given) 2005/12/01 21:24 EST [Uninitialized] Backing off on delivery of ['71757_1133487291.33_0_138982188'] 2005/12/01 21:24 EST [Uninitialized] Stopping factory <twisted.mail.relaymanager.SMTPManagedRelayerFactory instance at 0x85fff0c> >From what I can tell smtp.SMTPClient isn't getting passed identity. Since this error is generated with a call to the relaying manager's .checkState(self) function which only takes self as a argument.. well lets just say I can't figure out what I'm missing from my own tracing through the code in the relay/relaymanager/smtp modules. You should find my code attached, the error should be easily reproducible if you want to run it. -- James Tanis jtanis(a)pycoder.org http://pycoder.org

1 0

[Twisted-Python] Re: [Twisted-commits] r15274 - Significantly improve twisted.python.filepath, adding several new features and tests.
by Jonathan Lange 02 Dec '05

02 Dec '05

On 02/12/05, Glyph Lefkowitz <glyph(a)wolfwood.twistedmatrix.com> wrote: > Author: glyph > Date: Thu Dec 1 14:47:17 2005 > New Revision: 15274 > > Modified: > trunk/twisted/python/filepath.py > trunk/twisted/test/test_paths.py > Log: > Significantly improve twisted.python.filepath, adding several new features and tests. <snip> > Modified: trunk/twisted/python/filepath.py > ============================================================================== > --- trunk/twisted/python/filepath.py (original) > +++ trunk/twisted/python/filepath.py Thu Dec 1 14:47:17 2005 > @@ -5,6 +5,10 @@ > from __future__ import generators > > import os > +import errno > +import base64 > +import sha > + > from os.path import isabs, exists, normpath, abspath, splitext > from os.path import basename, dirname > from os.path import join as joinpath > @@ -25,8 +29,13 @@ > class InsecurePath(Exception): > pass > > +def _secureEnoughString(): > + """ > + Create a pseudorandom, 16-character string for use in secure filenames. > + """ > + return base64.urlsafe_b64encode(sha.new(os.urandom(64)).digest())[:16] > + <snip> This breaks Twisted. base64.urlsafe_b64encode is not available in Python before version 2.4. cheers, jml

1 0

[Twisted-Python] Re: user interaction & the reactor
by Lenny G Arbage 01 Dec '05

01 Dec '05

Okay, I think I've got it now. This: http://twistedmatrix.com/projects/core/documentation/howto/threading.html combined with this: http://mail.python.org/pipermail/python-list/2004-February/209632.html contained some details I was woefully ignorant of. --- Lenny G Arbage <alengarbage(a)yahoo.com> wrote: > Where can I find examples or instructions on > combining > the reactor with [command-line] user interaction? > I've been searching every resource I can, and know > it > must be possible based on threads like this one: > http://twistedmatrix.com/pipermail/twisted-python/2005-June/010625.html > . I /can/ block on a call made from within the > reactor while waiting for user's raw_input, but that > doesn't allow me to give live feedback to the user > from incoming status messages, and might cause > problems if a large number of status messages arrive > while the user is 'away'. All the example clients I > can find inside of twisted are not [user-] > interactive > (ftpclient, echoclient, simpleclient, etc). Am I > looking in the wrong place? Have I missed a howto? > > Specifically I have a server that takes commands via > a > protocol I define and a client that sends commands > to > the server. The commands on the server take some > time > (on the order of 20 seconds to many minutes) to > complete or error out. Once they do, status is sent > back to the client, i.e., the user can issue many > commands that are concurrently 'pending', completing > when the server sends back status. > > I've started by running the client reactor in a > thread, and then using the main thread to get input > from the user. Am I making this more difficult than > it needs to be? Is there some way to integrate user > action (keyboard input) as events into the reactor's > event model? Or is there a better approach? __________________________________________ Yahoo! DSL Something to write home about. Just $16.99/mo. or less. dsl.yahoo.com

1 0

[Twisted-Python] user interaction & the reactor
by Lenny G Arbage 01 Dec '05

01 Dec '05

Where can I find examples or instructions on combining the reactor with [command-line] user interaction? I've been searching every resource I can, and know it must be possible based on threads like this one: http://twistedmatrix.com/pipermail/twisted-python/2005-June/010625.html . I /can/ block on a call made from within the reactor while waiting for user's raw_input, but that doesn't allow me to give live feedback to the user from incoming status messages, and might cause problems if a large number of status messages arrive while the user is 'away'. All the example clients I can find inside of twisted are not [user-] interactive (ftpclient, echoclient, simpleclient, etc). Am I looking in the wrong place? Have I missed a howto? Specifically I have a server that takes commands via a protocol I define and a client that sends commands to the server. The commands on the server take some time (on the order of 20 seconds to many minutes) to complete or error out. Once they do, status is sent back to the client, i.e., the user can issue many commands that are concurrently 'pending', completing when the server sends back status. I've started by running the client reactor in a thread, and then using the main thread to get input from the user. Am I making this more difficult than it needs to be? Is there some way to integrate user action (keyboard input) as events into the reactor's event model? Or is there a better approach? Thanks, Lenny __________________________________ Yahoo! Music Unlimited Access over 1 million songs. Try it free. http://music.yahoo.com/unlimited/

3 2

[Twisted-Python] New to twisted
by Fred Hauck 01 Dec '05

01 Dec '05

I am not only new to twisted, but to python as well. I need what twisted does, so went out and got the "twisted" book by abe fettig. I am coding each example in the book as i get there. the connectiontest.py example on page 16 gives me a call back on success, but not on error. from twisted.internet import reactor,defer, protocol class CallbackAndDisconnectProtocol(protocol.Protocol): def connectionMade(self): self.factory.deferred.callback("Connected!") self.transport.loseConnection() class ConnectionTestFactory(protocol.ClientFactory): protocol = CallbackAndDisconnectProtocol def __init__(self): self.deferred = defer.Deferred() def clientConnnectionFailed(self, connector, reason): self.deferred.errback(reason) def testConnect(host, port): testFactory = ConnectionTestFactory() reactor.connectTCP(host, port, testFactory) return testFactory.deferred def handleSuccess( result,port): print "Connected to port %i" % port reactor.stop() def handleFailure(failure, port): print "Error connecting to port %i: %s" % (port,failure.getErrorMessage()) reactor.stop() if __name__ == "__main__": import sys if not len( sys.argv) == 3: print "Usage: connectiontest.py host port" sys.exit(1) host = sys.argv[1] port = int(sys.argv[2]) connecting = testConnect(host, port) # connecting.addCallback( handleSuccess, port) # connecting.addErrback( handleFailure,port) connecting.addBoth( handleFailure,port) reactor.run() The environment is: Windows XP sp2 python 2.4.1 from activestate twisted_nodocs-2.1.0.win32-py24.exe (installed) When chasing this down, go into the main loop and there are comments about "signals" not working in windows, etc. etc. CallLater works fine, successful callback works fine, the errback does not thanks in advance fred

2 1

[Twisted-Python] FTPS server
by Vijay Rao 01 Dec '05

01 Dec '05

Hi , I am trying to use a 'virtual' twisted FTP server ( using 2.1.0 ) in an application which can use a database instead of the file system. It seems to work fine except I could not connect over SSL/TLS since the FTP security extensions enhancement http://www.ietf.org/rfc/rfc2228.txt is not yet implemented. Is there any plan to add this in the near future / any simple way to add it cleanly ? Thanks Vijay -- No virus found in this outgoing message. Checked by AVG Anti-Virus. Version: 7.1.362 / Virus Database: 267.13.10/188 - Release Date: 11/29/2005

2 1

[Twisted-Python] My review of Abe's book "Twisted Network Programming Essentials"
by Tommi Virtanen 01 Dec '05

01 Dec '05

Hi. I just finished a review of the new O'Reilly book on Twisted, by Abe Fettig (hi Abe!), and thought you all might be interested in it: http://tv.debian.net/articles/review-snakeball/

4 4

[Twisted-Python] Re: [Twisted-commits] r15230 - Merge of jml/trial-deferred-logic -r15183:15229. Resolves #1341.
by Jean-Paul Calderone 01 Dec '05

01 Dec '05

On Sun, 27 Nov 2005 23:35:45 -0700, Jonathan Lange <jml(a)wolfwood.twistedmatrix.com> wrote: >Author: jml >Date: Sun Nov 27 23:35:45 2005 >New Revision: 15230 > >Modified: > trunk/twisted/mail/test/test_mail.py > trunk/twisted/test/test_ftp.py > trunk/twisted/test/test_ssl.py > trunk/twisted/test/test_tcp.py > trunk/twisted/trial/reporter.py > trunk/twisted/trial/runner.py > trunk/twisted/trial/test/detests.py > trunk/twisted/trial/test/test_deferred.py > trunk/twisted/trial/test/test_failure_formatting.py > trunk/twisted/trial/test/test_keyboard.py > trunk/twisted/trial/test/timeouts.py > trunk/twisted/trial/unittest.py > trunk/twisted/trial/util.py > trunk/twisted/web/test/test_webclient.py >Log: >Merge of jml/trial-deferred-logic -r15183:15229. Resolves #1341. > The Gtk reactor can no longer complete the test suite: http://twistedmatrix.com/buildbot/reactors/builds/884/step-gtk/0 On my machine, the suite hangs in twisted.web.test.test_webclient.CookieTestCase.testNoCookies, but it seems to die in twisted.test.test_tcp.HalfClose2TestCase.testNoNotification on buildbot every time. This may just be a race (or several races) going different ways on a somewhat reliable fashion on the different hardware. It seems like the remaining reactor.iterate() calls are the likely cause of this? Jean-Paul

3 3

[Twisted-Python] implementing scp using Conch
by erick_bodine＠comcast.net 01 Dec '05

01 Dec '05

Using a suggestion from David Mertz's old article I am implementing scp using Conch. I can scp from the server to the client, it does give an exception listed below. Question #1 is why is the exception happening? Question #2 is how to scp from the client to the server? Can I use sendRequest() to open a file and then Channel.write('some data') to send the client file contents? Traceback (most recent call last): File "client_mertz.py", line 80, in ? reactor.run() File "/usr/lib/python2.3/site-packages/twisted/internet/posixbase.py", line 206, in run self.mainLoop() File "/usr/lib/python2.3/site-packages/twisted/internet/posixbase.py", line 214, in mainLoop self.runUntilCurrent() --- <exception caught here> --- File "/usr/lib/python2.3/site-packages/twisted/internet/base.py", line 541, in runUntilCurrent call.func(*call.args, **call.kw) File "/usr/lib/python2.3/site-packages/twisted/internet/base.py", line 397, in _continueSystemEvent for callList in sysEvtTriggers[1], sysEvtTriggers[2]: exceptions.TypeError: unsubscriptable object -- --ERick

2 1