Mailman 3 December 2005 - Twisted

[Twisted-Python] Re: [Twisted-commits] r15316 - Reorganize davxml.py into a few files, still exposed as davxml.py.
by Jean-Paul Calderone 10 Dec '05

10 Dec '05

On Fri, 09 Dec 2005 19:18:17 -0700, Wilfredo Sanchez <wsanchez(a)wolfwood.twistedmatrix.com> wrote: >Author: wsanchez >Date: Fri Dec 9 19:18:17 2005 >New Revision: 15316 > >Added: > branches/wsanchez/dav/twisted/web2/dav/elements/ > branches/wsanchez/dav/twisted/web2/dav/elements/__init__.py > branches/wsanchez/dav/twisted/web2/dav/elements/base.py > - copied, changed from r15315, branches/wsanchez/dav/twisted/web2/dav/davxml.py > branches/wsanchez/dav/twisted/web2/dav/elements/rfc2518.py > - copied, changed from r15315, branches/wsanchez/dav/twisted/web2/dav/davxml.py > branches/wsanchez/dav/twisted/web2/dav/elements/rfc3253.py > - copied, changed from r15315, branches/wsanchez/dav/twisted/web2/dav/davxml.py > branches/wsanchez/dav/twisted/web2/dav/elements/rfc3744.py > - copied, changed from r15315, branches/wsanchez/dav/twisted/web2/dav/davxml.py >Modified: > branches/wsanchez/dav/twisted/web2/dav/davxml.py >Log: >Reorganize davxml.py into a few files, still exposed as davxml.py. >davxml.py was getting way too big. Now elements are in files split up by RFC. >We no longer have to manage __all__. These elements don't constitute part of the application-level API, right? Jean-Paul

1 0

RE: [Twisted-Python] RE: waiting on transport return
by jmbenski＠micron.com 09 Dec '05

09 Dec '05

Maybe I misspoke. The way I understand it, is that deferreds do not get CPU time until the reactor resumes control. Once the reactor gets control it allows events to happen, some of which may be deferreds. I'm not sure what does the scheduling or what order deferred objects get control of the thread. I had assumed that the reactor or whatever scheduler kicked off deferreds in the order that they were created. Once that deferred chain got control of the main loop it would run to completion and the next event whether a deferred or something else would run. In my case I need the deferred to wait until my commandComplete function runs. Is the aDeferred.callback(result) the trigger that starts the deferred? If that is the case, then I understand how my current code is working and I understand that there IS no scheduler for the deferreds. Deferreds fire when a callback or errback function is called on them. End of story and no magic done behind the scenes. Jon -----Original Message----- From: twisted-python-bounces(a)twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of Itamar Shtull-Trauring Sent: Friday, December 09, 2005 11:29 AM To: Twisted general discussion Subject: RE: [Twisted-Python] RE: waiting on transport return On Fri, 2005-12-09 at 11:02 -0700, jmbenski(a)micron.com wrote: > After looking at the code a bit more and your change below, I'm > surprised that my code works. Why isn't the reactor kicking off my > deferreds that are stored in the queue? Deferreds have nothing to do with the reactor. So, doing aDeferred.callback(result) or .errback(error) are the only way it gets a value. _______________________________________________ Twisted-Python mailing list Twisted-Python(a)twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

2 1

RE: [Twisted-Python] RE: waiting on transport return
by jmbenski＠micron.com 09 Dec '05

09 Dec '05

I guess I'm still trying to wrap my head around deferreds and how best to use them as they were intended. The way that code works, I have to pass my deferred chain to sendCommand. The protocol needs to store that deferred on its deferred stack, so that I can pass the results of the command back to the deferred function. After looking at the code a bit more and your change below, I'm surprised that my code works. Why isn't the reactor kicking off my deferreds that are stored in the queue? My current logic is dependent on the protocol running the commandComplete function before the deferreds fire in the protocol's deferred queue. #Do some thing #Send the command self.processProtocol.sendCommand #Do some more #return control to the reactor ??Here the reactor should kick off my command and run any deferreds. Am I just lucky that my server returns the response(commandComplete) before the deferreds in the queue fire? The return of the data and the chain of deferreds are not linked in any way, which I guess is what my original question is. How can my commandComplete return a deferred and kick off a logic chain? How is it working now? -----Original Message----- From: twisted-python-bounces(a)twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of Jean-Paul Calderone Sent: Friday, December 09, 2005 10:01 AM To: Twisted general discussion Subject: RE: [Twisted-Python] RE: waiting on transport return On Fri, 9 Dec 2005 09:03:52 -0700, jmbenski(a)micron.com wrote: >I'm not trying to bump my question, but if someone could tell me if I'm just being stupid and missing something with this code or that it isn't possible to do what I need with a spawned process? If I know it isn't possible, I can just continue down the path I am. It's not too much of a pain, but it just feels...wrong. > The code mostly seems good to me. The only thing I might change is completely trivial: instead of passing a Deferred in to sendCommand, I'd have sendCommand be responsible for creating the Deferred and then return it, so your application code would look like this - d = self.processprotocol.sendCommand("syncCmd", cmd) d.addCallback(self.handleTheCommand) d.addCallback(Self.theRestOfTheCommands) I'm not sure from where your unhappyness with the version you posted stems, so I'm not sure what other comments to make. Jean-Paul _______________________________________________ Twisted-Python mailing list Twisted-Python(a)twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

2 1

RE: [Twisted-Python] RE: waiting on transport return
by jmbenski＠micron.com 09 Dec '05

09 Dec '05

I'm not trying to bump my question, but if someone could tell me if I'm just being stupid and missing something with this code or that it isn't possible to do what I need with a spawned process? If I know it isn't possible, I can just continue down the path I am. It's not too much of a pain, but it just feels...wrong. Thanks, Jon -----Original Message----- From: twisted-python-bounces(a)twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of jmbenski Sent: Tuesday, December 06, 2005 7:58 PM To: twisted-python(a)twistedmatrix.com Subject: [Twisted-Python] RE: waiting on transport return Ok, I need some help to write some code using deferred's the way they were meant, not the way I've bastardized them. My framework... I have a client that will connect to a server and MAINTAIN that connection. I can't create a protocol factory. The server and the client communicate commands through stdin and stdout. I need to write my commands to the server and wait for the response in some cases. I've setup a system that passes a deferred chain to my process protocol and once my process knows that the data it needs is received it will kick off the deferred chain that happens after that response. This seems backwards. From my reading and understanding of deferreds, they should bubble down and not have to be pushed up. How do I handle the case of some protocol command kicking off the rest of a deferred chain or of passing back a deferred, so that I can add to it the things that need to run? class SyncManagerProtocol(protocol.ProcessProtocol): def __init__(self ): self.dataStore = "" ##queue that holds the commands self.cmdQueue = list() ##queue that holds the callback function to call once the call returns self.deferredQueue = list() def outReceived(self, data): if len(self.cmdQueue) > 0: ##take the data and shove it into our internal store self.dataStore += data ##determine what is good data cmd = self.cmdQueue[0] ##look for the command the is next in the queue. grab all the return data ##between the command block. ## syncCmd { ## return data goes here ## } syncCmd cmdBlock = r"%s {(?P<cmd>.*)} %s(?P<rest>.*)" % (cmd, cmd) errBlock = r"invalid {(?P<errMsg>.*)}" cmdMatch = re.compile( cmdBlock, re.S ).search( self.dataStore ) if cmdMatch: ##set the rest of the data to the dataStore. This clears out the current contents ##but keeps any extra data that may be coming in from the next command self.dataStore = cmdMatch.group( "rest" ) ##clean up this command self.commandComplete( cmdMatch.group("cmd").lstrip() ) ##This is called if all the data we need has been returned. def commandComplete(self, cmdData): ##pop this command since we are done with it self.cmdQueue.pop(0) ##pop the deferred that matches this command deferred = self.deferredQueue.pop(0) ##pass the results to the deferred deferred.callback( cmdData ) ##interface to the process that sends a command to execute def sendCommand( self, cmd, args, deferred ): ##store the command in the queue, so we can check it when we get data self.cmdQueue.append( cmd ) ##store the deferred in the queue that we need to run when this command has been completed self.deferredQueue.append( deferred ) ##write out command to the process cmd = cmd + " " + args + "\n" self.transport.write( cmd ) ## This code is my client interface class to the server that holds the processProtocol instance. ## #setup a command that will handle errors or good data d = defer.Deferred().addCallback( self.handleThisCommand ) #run the calling function handler that will handle an error or returned data. This will allow something like a GUI to process data that will now be available d.addCallback( self.theRestOfTheCommands ) cmd = "some command goes here" self.processProtocol.sendCommand( "syncCmd", cmd, d)

2 1

[Twisted-Python] (no subject)
by kwhitesell＠adelphia.net 09 Dec '05

09 Dec '05

Hi Frank, Best that I can tell, it's not your various classes that need to be converted to deferreds, just your program that calls class1, or even class1 itself. The bottom line is that _something_ higher up the calling stack needs to be aware that this is an asynchronous process - and that a deferred gets generated along the way. I believe it would be helpful if the originating function is coded for this. I'm not a Twisted guru, so there may be an easier way to do this - but as I see it... I would be writing the classes to look more like the following: class1(): def check1(data): perform test if test failed: return defer.succeed(False) return class2.check2(data): class2(): def check2(data): perform test if test failed: return defer.succeed(False) return class3.check3(data): class3(): def check3(data): perform test if test failed: return defer.succeed(False) aDeferred = avatar.callRemote(check4,data) return aDeferred Then, my originating class (and possibly becomming a wrapper class if this is used in many places). class0(): def check0(data): class1.check1(data).addCallback(afterCheck) def afterCheck(result): return result This way, all the functions return a deferred back up to the calling program - but the only one that causes a delay is the call to check4. Ken

3 3

[Twisted-Python] ftpClient.py, retrieveFile, and line ending issues
by Keith.Gunderson＠act.org 07 Dec '05

07 Dec '05

Hello, If any of you have a spare minute, I can't figure out else where I might solve my present dilemma. Its certainly likely that my only issues are ignorance about Python's file operations or the features that are a part of other FTP clients that I use. I'm using the ftpclient.py code to write a script which downloads file via FTP. My example code shows the basics. The file is transferred, but the lines seem to have different line endings when I view them 'hexlified' in xemacs. My twisted-created files has hex 0D0D 0D0D 0D0D at the end of lines while a file downloaded using SmartFTP has hex 0D0A 0D0A. Is it clear, based on that information, if there is an issue in my code? Or an easy way to fix it? class FileWriterProtocol(Protocol): def __init__(self, filename): self.file = open(filename, 'w') def dataReceived(self, data): self.file.write(data) def connectionMade(ftpClient): proto2 = FileWriterProtocol( 'c:/ftpfile1.txt' ) # Get the file, and quit when done d = ftpClient.retrieveFile( 's1/s2/T2005C.processed' , proto2) d.addCallback(lambda result: reactor.stop()) Thanks very much, Keith Gunderson

3 3

[Twisted-Python] Weekly Bug Summary
by James Y Knight 07 Dec '05

07 Dec '05

KIND == 'create' Bug summary ______________________ Since 2005-11-30 07:00:02.250000: Bugs opened: 11 Bugs closed: 2 Total open bugs: 493 (+9) Urgent: 2 (+1) Bug: 232 (+5) Feature: 176 (+3) Wish: 83 (+0) New / Reopened Bugs ______________________ ==== Urgent ==== [TRIAL] trial passes deferredGenerator test cases without running the test case (8 hours ago) http://twistedmatrix.com/bugs/issue1353 opened by hagna [CORE] FilePath sucks (5 days ago) CLOSED http://twistedmatrix.com/bugs/issue1348 opened by glyph ==== Bug ==== [CORE] threadedselectreactor queue size (5 days ago) http://twistedmatrix.com/bugs/issue1347 opened by Toby Dickenson Separate reporter result handling from output (5 days ago) http://twistedmatrix.com/bugs/issue1349 opened by jml [CORE, WIN32] iocpreactor [Errno 10014] The system detected an invalid pointer address in attempting to use a pointer argument in a call (6 days ago) http://twistedmatrix.com/bugs/issue1346 opened by ghazel [CORE, WIN32] socket.error: (19, 'No such device') (6 days ago) http://twistedmatrix.com/bugs/issue1345 opened by ghazel [CORE] posixbase win32 waker problem (6 days ago) http://twistedmatrix.com/bugs/issue1344 opened by antony [CORE] readers and writers not removed properly from threadedselectreactor (6 days ago) http://twistedmatrix.com/bugs/issue1343 opened by antony ==== Feature ==== [TRIAL] [PATCH] HTML reporter for trial (4 days ago) http://twistedmatrix.com/bugs/issue1350 opened by dialtone log observer bridging to python logging facility (4 days ago) http://twistedmatrix.com/bugs/issue1351 opened by ctheune ==== Wish ==== twisted + dbus 0.60 howto (yesterday) http://twistedmatrix.com/bugs/issue1352 opened by senko Bugs Closed ______________________ ==== Urgent ==== [CORE] FilePath sucks (5 days ago) http://twistedmatrix.com/bugs/issue1348 opened by glyph, closed by <unknown> ==== Bug ==== twisted.web.html.output bug (IRC) (1 week ago) http://twistedmatrix.com/bugs/issue1337 opened by kyle, closed by exarkun

1 0

[Twisted-Python] Question on deferreds
by Ken Whitesell 07 Dec '05

07 Dec '05

Frank, Ok, your clarifications make things a little more difficult, but not impossible. Standard disclaimer: I'm no Twisted-guru. There may be an easier way to do this. I can only take this as far as my knowledge extends. So having said that... There's no way to get around the fact that the calling program needs to be 'broken' down into two parts, 'separated' by the deferred. In other words, to greatly simplify the situation, your flow should be something like: function 1 do stuff... do more stuff... do more stuff... callRemote... addCallback(function2) return function 2 do more stuff... do more stuff... do more stuff... end of tests. The bottom line is that your entire program flow needs to be reworked around the Twisted methodology. (Or, in other words, you need to Twist your program. <g>) I don't remember where I saw this first, but the best way I know to describe this situation is to stop thinking of your program as a program and Twisted as a library. Twisted is the program, and your code is the library. Your functions get called by the reactor because of some event. You handle this event then return control to the reactor. The reactor then calls other functions as necessary. So you perform some tests, then call remote to invoke a process on the server, register your callback and yield control back to the reactor. When the reply is received, a different function gets control and you continue processing. One way this can look, is to take your original program (section 1) and modify it to look more like the pseudo code below. *** Original code, with modified class 3 *** class1(): def check1(data): perform test if test failed: return False if not class2.check2(data): return False do_additional_stuff_1 return True class2(): def check2(data): perform test if test failed: return False if not class3.check3(data): return False do_additional_stuff_2 return True class3(): def check3(data): perform test if test failed: return False *** This is where the flow breaks *** avatar.callRemote(check4,data).addCallback(afterCheck) if not class4.check4(data): return False do_additional_stuff_3 return True def afterCheck(result): return result The simplest rework I can come up with for this is: class0(): def check0(data): part1 = class1.check1(data) # Assuming part1 is either false or a deferred if part1: part1.addCallback(afterCheck) else callLater(0, afterCheck, False) def afterCheck(result): # part 2 # Do the stuff after check4 # This would include the stuff that needs to be done # after the tests. So code from # do_additional_stuff_x # can be called from here. finalResults = class1.do_additional_stuff_1(data) return finalResult class1(): def check1(data): perform test if test failed: return False if not class2.check2(data): return False # Note, we can't get here so this code can go # return True def do_additional_stuff_1(data): class2.do_additional_stuff_2(data) more suff goes here. class2(): def check2(data): perform test if test failed: return False if not class3.check3(data): return False # Note, we can't get here so this code can go # return True def do_additional_stuff_2(data): class3.do_additional_stuff_3(data) more suff goes here. class3(): def check3(data): perform test if test failed: return False *** This is where the flow breaks *** aDeferred = avatar.callRemote(check4,data).addCallback(afterCheck) return aDeferred def do_additional_stuff_3(data): suff goes here. ************************************** As you can see, it does chop things up a bit - and I'm still not sure that I've really captured what you're trying to do. I've just tried to match your existing class structure - which I believe to be sub-optimal in this context, given that I'm working from some very generic descriptions. Specifically, I'm not sure if these "class1, class2 and class3" definitions are truly classes, or if they're simply "organizational usage groups". (I'm not judging one way or the other, I'm only working from what you've written and what assumptions I can make - this isn't meant to disparage your code, just that I don't feel like I have enough solid information to make a more accurate judgement.) Hope this helps spark some ideas on your end. Ken

1 0

[Twisted-Python] RE: waiting on transport return
by jmbenski＠micron.com 07 Dec '05

07 Dec '05

Ok, I need some help to write some code using deferred's the way they were meant, not the way I've bastardized them. My framework... I have a client that will connect to a server and MAINTAIN that connection. I can't create a protocol factory. The server and the client communicate commands through stdin and stdout. I need to write my commands to the server and wait for the response in some cases. I've setup a system that passes a deferred chain to my process protocol and once my process knows that the data it needs is received it will kick off the deferred chain that happens after that response. This seems backwards. From my reading and understanding of deferreds, they should bubble down and not have to be pushed up. How do I handle the case of some protocol command kicking off the rest of a deferred chain or of passing back a deferred, so that I can add to it the things that need to run? class SyncManagerProtocol(protocol.ProcessProtocol): def __init__(self ): self.dataStore = "" ##queue that holds the commands self.cmdQueue = list() ##queue that holds the callback function to call once the call returns self.deferredQueue = list() def outReceived(self, data): if len(self.cmdQueue) > 0: ##take the data and shove it into our internal store self.dataStore += data ##determine what is good data cmd = self.cmdQueue[0] ##look for the command the is next in the queue. grab all the return data ##between the command block. ## syncCmd { ## return data goes here ## } syncCmd cmdBlock = r"%s {(?P<cmd>.*)} %s(?P<rest>.*)" % (cmd, cmd) errBlock = r"invalid {(?P<errMsg>.*)}" cmdMatch = re.compile( cmdBlock, re.S ).search( self.dataStore ) if cmdMatch: ##set the rest of the data to the dataStore. This clears out the current contents ##but keeps any extra data that may be coming in from the next command self.dataStore = cmdMatch.group( "rest" ) ##clean up this command self.commandComplete( cmdMatch.group("cmd").lstrip() ) ##This is called if all the data we need has been returned. def commandComplete(self, cmdData): ##pop this command since we are done with it self.cmdQueue.pop(0) ##pop the deferred that matches this command deferred = self.deferredQueue.pop(0) ##pass the results to the deferred deferred.callback( cmdData ) ##interface to the process that sends a command to execute def sendCommand( self, cmd, args, deferred ): ##store the command in the queue, so we can check it when we get data self.cmdQueue.append( cmd ) ##store the deferred in the queue that we need to run when this command has been completed self.deferredQueue.append( deferred ) ##write out command to the process cmd = cmd + " " + args + "\n" self.transport.write( cmd ) ## This code is my client interface class to the server that holds the processProtocol instance. ## #setup a command that will handle errors or good data d = defer.Deferred().addCallback( self.handleThisCommand ) #run the calling function handler that will handle an error or returned data. This will allow something like a GUI to process data that will now be available d.addCallback( self.theRestOfTheCommands ) cmd = "some command goes here" self.processProtocol.sendCommand( "syncCmd", cmd, d)

1 0

[Twisted-Python] XML consumers and protocols
by Keith.Gunderson＠act.org 06 Dec '05

06 Dec '05

Another question I'm trying to hunt down regarding my plans to develop an ftpClient that will download XML files... How can I teach myself what is important to know about the relationship between Protocols and Consumers? How much of these concepts are twisted specific... or Components/ Interfaces ... or just native Python stuff. I have a notion of abstractly passing data from an ftpClient to an XML parser, but I'm obviously not there yet. Thanks! Keith Gunderson

1 0