Mailman 3 December 2005 - Twisted

[Twisted-Python] Re: [Twisted-commits] r15316 - Reorganize davxml.py into a few files, still exposed as davxml.py.
by Jean-Paul Calderone Dec. 10, 2005

Dec. 10, 2005

On Fri, 09 Dec 2005 19:18:17 -0700, Wilfredo Sanchez <wsanchez(a)wolfwood.twistedmatrix.com> wrote: >Author: wsanchez >Date: Fri Dec 9 19:18:17 2005 >New Revision: 15316 > >Added: > branches/wsanchez/dav/twisted/web2/dav/elements/ > branches/wsanchez/dav/twisted/web2/dav/elements/__init__.py > branches/wsanchez/dav/twisted/web2/dav/elements/base.py > - copied, changed from r15315, branches/wsanchez/dav/twisted/web2/dav/davxml.py > branches/… [View More]

1 0

RE: [Twisted-Python] RE: waiting on transport return
by jmbenski＠micron.com Dec. 9, 2005

Dec. 9, 2005

Maybe I misspoke. The way I understand it, is that deferreds do not get CPU time until the reactor resumes control. Once the reactor gets control it allows events to happen, some of which may be deferreds. I'm not sure what does the scheduling or what order deferred objects get control of the thread. I had assumed that the reactor or whatever scheduler kicked off deferreds in the order that they were created. Once that deferred chain got control of the main loop it would run to completion … [View More]

2 1

RE: [Twisted-Python] RE: waiting on transport return
by jmbenski＠micron.com Dec. 9, 2005

Dec. 9, 2005

I guess I'm still trying to wrap my head around deferreds and how best to use them as they were intended. The way that code works, I have to pass my deferred chain to sendCommand. The protocol needs to store that deferred on its deferred stack, so that I can pass the results of the command back to the deferred function. After looking at the code a bit more and your change below, I'm surprised that my code works. Why isn't the reactor kicking off my deferreds that are stored in the queue? … [View More]My current logic is dependent on the protocol running the commandComplete function before the deferreds fire in the protocol's deferred queue. #Do some thing #Send the command self.processProtocol.sendCommand #Do some more #return control to the reactor ??Here the reactor should kick off my command and run any deferreds. Am I just lucky that my server returns the response(commandComplete) before the deferreds in the queue fire? The return of the data and the chain of deferreds are not linked in any way, which I guess is what my original question is. How can my commandComplete return a deferred and kick off a logic chain? How is it working now? -----Original Message----- From: twisted-python-bounces(a)twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of Jean-Paul Calderone Sent: Friday, December 09, 2005 10:01 AM To: Twisted general discussion Subject: RE: [Twisted-Python] RE: waiting on transport return On Fri, 9 Dec 2005 09:03:52 -0700, jmbenski(a)micron.com wrote: >I'm not trying to bump my question, but if someone could tell me if I'm just being stupid and missing something with this code or that it isn't possible to do what I need with a spawned process? If I know it isn't possible, I can just continue down the path I am. It's not too much of a pain, but it just feels...wrong. > The code mostly seems good to me. The only thing I might change is completely trivial: instead of passing a Deferred in to sendCommand, I'd have sendCommand be responsible for creating the Deferred and then return it, so your application code would look like this - d = self.processprotocol.sendCommand("syncCmd", cmd) d.addCallback(self.handleTheCommand) d.addCallback(Self.theRestOfTheCommands) I'm not sure from where your unhappyness with the version you posted stems, so I'm not sure what other comments to make. Jean-Paul _______________________________________________ Twisted-Python mailing list Twisted-Python(a)twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python [View Less]

2 1

RE: [Twisted-Python] RE: waiting on transport return
by jmbenski＠micron.com Dec. 9, 2005

Dec. 9, 2005

I'm not trying to bump my question, but if someone could tell me if I'm just being stupid and missing something with this code or that it isn't possible to do what I need with a spawned process? If I know it isn't possible, I can just continue down the path I am. It's not too much of a pain, but it just feels...wrong. Thanks, Jon -----Original Message----- From: twisted-python-bounces(a)twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of jmbenski Sent: … [View More]Tuesday, December 06, 2005 7:58 PM To: twisted-python(a)twistedmatrix.com Subject: [Twisted-Python] RE: waiting on transport return Ok, I need some help to write some code using deferred's the way they were meant, not the way I've bastardized them. My framework... I have a client that will connect to a server and MAINTAIN that connection. I can't create a protocol factory. The server and the client communicate commands through stdin and stdout. I need to write my commands to the server and wait for the response in some cases. I've setup a system that passes a deferred chain to my process protocol and once my process knows that the data it needs is received it will kick off the deferred chain that happens after that response. This seems backwards. From my reading and understanding of deferreds, they should bubble down and not have to be pushed up. How do I handle the case of some protocol command kicking off the rest of a deferred chain or of passing back a deferred, so that I can add to it the things that need to run? class SyncManagerProtocol(protocol.ProcessProtocol): def __init__(self ): self.dataStore = "" ##queue that holds the commands self.cmdQueue = list() ##queue that holds the callback function to call once the call returns self.deferredQueue = list() def outReceived(self, data): if len(self.cmdQueue) > 0: ##take the data and shove it into our internal store self.dataStore += data ##determine what is good data cmd = self.cmdQueue[0] ##look for the command the is next in the queue. grab all the return data ##between the command block. ## syncCmd { ## return data goes here ## } syncCmd cmdBlock = r"%s {(?P<cmd>.*)} %s(?P<rest>.*)" % (cmd, cmd) errBlock = r"invalid {(?P<errMsg>.*)}" cmdMatch = re.compile( cmdBlock, re.S ).search( self.dataStore ) if cmdMatch: ##set the rest of the data to the dataStore. This clears out the current contents ##but keeps any extra data that may be coming in from the next command self.dataStore = cmdMatch.group( "rest" ) ##clean up this command self.commandComplete( cmdMatch.group("cmd").lstrip() ) ##This is called if all the data we need has been returned. def commandComplete(self, cmdData): ##pop this command since we are done with it self.cmdQueue.pop(0) ##pop the deferred that matches this command deferred = self.deferredQueue.pop(0) ##pass the results to the deferred deferred.callback( cmdData ) ##interface to the process that sends a command to execute def sendCommand( self, cmd, args, deferred ): ##store the command in the queue, so we can check it when we get data self.cmdQueue.append( cmd ) ##store the deferred in the queue that we need to run when this command has been completed self.deferredQueue.append( deferred ) ##write out command to the process cmd = cmd + " " + args + "\n" self.transport.write( cmd ) ## This code is my client interface class to the server that holds the processProtocol instance. ## #setup a command that will handle errors or good data d = defer.Deferred().addCallback( self.handleThisCommand ) #run the calling function handler that will handle an error or returned data. This will allow something like a GUI to process data that will now be available d.addCallback( self.theRestOfTheCommands ) cmd = "some command goes here" self.processProtocol.sendCommand( "syncCmd", cmd, d) [View Less]

2 1

[Twisted-Python] (no subject)
by kwhitesell＠adelphia.net Dec. 9, 2005

Dec. 9, 2005

Hi Frank, Best that I can tell, it's not your various classes that need to be converted to deferreds, just your program that calls class1, or even class1 itself. The bottom line is that _something_ higher up the calling stack needs to be aware that this is an asynchronous process - and that a deferred gets generated along the way. I believe it would be helpful if the originating function is coded for this. I'm not a Twisted guru, so there may be an easier way to do this - but as I see it... … [View More]

3 3

[Twisted-Python] ftpClient.py, retrieveFile, and line ending issues
by Keith.Gunderson＠act.org Dec. 7, 2005

Dec. 7, 2005

Hello, If any of you have a spare minute, I can't figure out else where I might solve my present dilemma. Its certainly likely that my only issues are ignorance about Python's file operations or the features that are a part of other FTP clients that I use. I'm using the ftpclient.py code to write a script which downloads file via FTP. My example code shows the basics. The file is transferred, but the lines seem to have different line endings when I view them 'hexlified' in xemacs. My … [View More]

3 3

[Twisted-Python] Weekly Bug Summary
by James Y Knight Dec. 7, 2005

Dec. 7, 2005

KIND == 'create' Bug summary ______________________ Since 2005-11-30 07:00:02.250000: Bugs opened: 11 Bugs closed: 2 Total open bugs: 493 (+9) Urgent: 2 (+1) Bug: 232 (+5) Feature: 176 (+3) Wish: 83 (+0) New / Reopened Bugs ______________________ ==== Urgent ==== [TRIAL] trial passes deferredGenerator test cases without running the test case (8 hours ago) http://twistedmatrix.com/bugs/issue1353 opened by hagna [CORE] FilePath sucks (5 days ago) CLOSED … [View More]

1 0

[Twisted-Python] Question on deferreds
by Ken Whitesell Dec. 7, 2005

Dec. 7, 2005

Frank, Ok, your clarifications make things a little more difficult, but not impossible. Standard disclaimer: I'm no Twisted-guru. There may be an easier way to do this. I can only take this as far as my knowledge extends. So having said that... There's no way to get around the fact that the calling program needs to be 'broken' down into two parts, 'separated' by the deferred. In other words, to greatly simplify the situation, your flow should be something like: function 1 do stuff..… [View More]. do more stuff... do more stuff... callRemote... addCallback(function2) return function 2 do more stuff... do more stuff... do more stuff... end of tests. The bottom line is that your entire program flow needs to be reworked around the Twisted methodology. (Or, in other words, you need to Twist your program. <g>) I don't remember where I saw this first, but the best way I know to describe this situation is to stop thinking of your program as a program and Twisted as a library. Twisted is the program, and your code is the library. Your functions get called by the reactor because of some event. You handle this event then return control to the reactor. The reactor then calls other functions as necessary. So you perform some tests, then call remote to invoke a process on the server, register your callback and yield control back to the reactor. When the reply is received, a different function gets control and you continue processing. One way this can look, is to take your original program (section 1) and modify it to look more like the pseudo code below. *** Original code, with modified class 3 *** class1(): def check1(data): perform test if test failed: return False if not class2.check2(data): return False do_additional_stuff_1 return True class2(): def check2(data): perform test if test failed: return False if not class3.check3(data): return False do_additional_stuff_2 return True class3(): def check3(data): perform test if test failed: return False *** This is where the flow breaks *** avatar.callRemote(check4,data).addCallback(afterCheck) if not class4.check4(data): return False do_additional_stuff_3 return True def afterCheck(result): return result The simplest rework I can come up with for this is: class0(): def check0(data): part1 = class1.check1(data) # Assuming part1 is either false or a deferred if part1: part1.addCallback(afterCheck) else callLater(0, afterCheck, False) def afterCheck(result): # part 2 # Do the stuff after check4 # This would include the stuff that needs to be done # after the tests. So code from # do_additional_stuff_x # can be called from here. finalResults = class1.do_additional_stuff_1(data) return finalResult class1(): def check1(data): perform test if test failed: return False if not class2.check2(data): return False # Note, we can't get here so this code can go # return True def do_additional_stuff_1(data): class2.do_additional_stuff_2(data) more suff goes here. class2(): def check2(data): perform test if test failed: return False if not class3.check3(data): return False # Note, we can't get here so this code can go # return True def do_additional_stuff_2(data): class3.do_additional_stuff_3(data) more suff goes here. class3(): def check3(data): perform test if test failed: return False *** This is where the flow breaks *** aDeferred = avatar.callRemote(check4,data).addCallback(afterCheck) return aDeferred def do_additional_stuff_3(data): suff goes here. ************************************** As you can see, it does chop things up a bit - and I'm still not sure that I've really captured what you're trying to do. I've just tried to match your existing class structure - which I believe to be sub-optimal in this context, given that I'm working from some very generic descriptions. Specifically, I'm not sure if these "class1, class2 and class3" definitions are truly classes, or if they're simply "organizational usage groups". (I'm not judging one way or the other, I'm only working from what you've written and what assumptions I can make - this isn't meant to disparage your code, just that I don't feel like I have enough solid information to make a more accurate judgement.) Hope this helps spark some ideas on your end. Ken [View Less]

1 0

[Twisted-Python] RE: waiting on transport return
by jmbenski＠micron.com Dec. 6, 2005

Dec. 6, 2005

Ok, I need some help to write some code using deferred's the way they were meant, not the way I've bastardized them. My framework... I have a client that will connect to a server and MAINTAIN that connection. I can't create a protocol factory. The server and the client communicate commands through stdin and stdout. I need to write my commands to the server and wait for the response in some cases. I've setup a system that passes a deferred chain to my process protocol and once my … [View More]process knows that the data it needs is received it will kick off the deferred chain that happens after that response. This seems backwards. From my reading and understanding of deferreds, they should bubble down and not have to be pushed up. How do I handle the case of some protocol command kicking off the rest of a deferred chain or of passing back a deferred, so that I can add to it the things that need to run? class SyncManagerProtocol(protocol.ProcessProtocol): def __init__(self ): self.dataStore = "" ##queue that holds the commands self.cmdQueue = list() ##queue that holds the callback function to call once the call returns self.deferredQueue = list() def outReceived(self, data): if len(self.cmdQueue) > 0: ##take the data and shove it into our internal store self.dataStore += data ##determine what is good data cmd = self.cmdQueue[0] ##look for the command the is next in the queue. grab all the return data ##between the command block. ## syncCmd { ## return data goes here ## } syncCmd cmdBlock = r"%s {(?P<cmd>.*)} %s(?P<rest>.*)" % (cmd, cmd) errBlock = r"invalid {(?P<errMsg>.*)}" cmdMatch = re.compile( cmdBlock, re.S ).search( self.dataStore ) if cmdMatch: ##set the rest of the data to the dataStore. This clears out the current contents ##but keeps any extra data that may be coming in from the next command self.dataStore = cmdMatch.group( "rest" ) ##clean up this command self.commandComplete( cmdMatch.group("cmd").lstrip() ) ##This is called if all the data we need has been returned. def commandComplete(self, cmdData): ##pop this command since we are done with it self.cmdQueue.pop(0) ##pop the deferred that matches this command deferred = self.deferredQueue.pop(0) ##pass the results to the deferred deferred.callback( cmdData ) ##interface to the process that sends a command to execute def sendCommand( self, cmd, args, deferred ): ##store the command in the queue, so we can check it when we get data self.cmdQueue.append( cmd ) ##store the deferred in the queue that we need to run when this command has been completed self.deferredQueue.append( deferred ) ##write out command to the process cmd = cmd + " " + args + "\n" self.transport.write( cmd ) ## This code is my client interface class to the server that holds the processProtocol instance. ## #setup a command that will handle errors or good data d = defer.Deferred().addCallback( self.handleThisCommand ) #run the calling function handler that will handle an error or returned data. This will allow something like a GUI to process data that will now be available d.addCallback( self.theRestOfTheCommands ) cmd = "some command goes here" self.processProtocol.sendCommand( "syncCmd", cmd, d) [View Less]

1 0

[Twisted-Python] XML consumers and protocols
by Keith.Gunderson＠act.org Dec. 6, 2005

Dec. 6, 2005

Another question I'm trying to hunt down regarding my plans to develop an ftpClient that will download XML files... How can I teach myself what is important to know about the relationship between Protocols and Consumers? How much of these concepts are twisted specific... or Components/ Interfaces ... or just native Python stuff. I have a notion of abstractly passing data from an ftpClient to an XML parser, but I'm obviously not there yet. Thanks! Keith Gunderson

1 0