[Twisted-Python] Need replacement for subprocess.call within a pyAMF method - any suggestions?
I am trying to use the twisted reactor as a basis for a Python server for a Flex app, using pyAMF. My problem is that one of the server methods requires me to run a series of command-line tools to complete parsing data from a file into an XML object, with lots of Python processing on the results of one command-line tool to provide the input to the next one. I have been using subprocess.call to invoke the command-line tools, but these intermittently - and frequently - fail with an error "IOError: [Errno 4] Interrupted system call". I do know that this was supposed to be fixed in twisted for Python 2.6, but it doesn't seem to be. I have python 2.6.3 under Mac OSX 10.5.8, and twisted 8.2.0. Is there a work-around for this problem, or a better way to do this with twisted? I do know that I can make a command-line call with reactor.callLater, and that I can return a deferred from my server method, but I don't see how to easily accomplish the chain of calls that I need, short of breaking up my serverMethod1 into the several sequential parts, and calling each from the result handler of the previous one. I'd rather not go that route, as the logic of which command-line call gets made when is actually quite complicated. What I have now looks like (enormously simplified): class MyPythonServer: def serverMethod1(self, dummyPost, inputFilePath): # Open inputFilePath, do stuff, save results to file2 subprocess.call(["command1", "file2", "file3"]) # Open file3, do stuff to the data, save results to file4 subprocess.call(["command2", "file4", "file5"]) # Open file5, convert data to a large string containing XML. return xmlString - Read Roberts
On 3 Nov, 11:55 pm, rroberts@adobe.com wrote:
I am trying to use the twisted reactor as a basis for a Python server for a Flex app, using pyAMF. My problem is that one of the server methods requires me to run a series of command-line tools to complete parsing data from a file into an XML object, with lots of Python processing on the results of one command-line tool to provide the input to the next one. I have been using subprocess.call to invoke the command-line tools, but these intermittently - and frequently - fail with an error "IOError: [Errno 4] Interrupted system call".
I do know that this was supposed to be fixed in twisted for Python 2.6, but it doesn't seem to be.
I'm not sure why you know this. I sure would be great if that were the case, but it's not. More changes in Twisted are necessary to let things like subprocess.call work properly alongside Twisted's process API.
I have python 2.6.3 under Mac OSX 10.5.8, and twisted 8.2.0. Is there a work-around for this problem, or a better way to do this with twisted? I do know that I can make a command-line call with reactor.callLater, and that I can return a deferred from my server method, but I don't see how to easily accomplish the chain of calls that I need, short of breaking up my serverMethod1 into the several sequential parts, and calling each from the result handler of the previous one. I'd rather not go that route, as the logic of which command-line call gets made when is actually quite complicated.
Are you aware that this means your server will only be able to handle one request at a time? If so and that's what you're shooting for, then you can fix this easily, by passing False to reactor.run. This disable's Twisted's own process support and lets things like subprocess.call work (blockingly, of course).
What I have now looks like (enormously simplified): class MyPythonServer:
def serverMethod1(self, dummyPost, inputFilePath): # Open inputFilePath, do stuff, save results to file2 subprocess.call(["command1", "file2", "file3"]) # Open file3, do stuff to the data, save results to file4 subprocess.call(["command2", "file4", "file5"]) # Open file5, convert data to a large string containing XML. return xmlString
Of course, you can still have methods that look like this if you use things like inlineCallbacks (which you can, since you said you're using Python 2.6). That way you preserve the style you seem to prefer and avoid blocking your server any time you need to launch a child process. Jean-Paul
Hello Jean-Paul ;
Thank you very much! For my simply workflow, blocking during a reactor call is acceptable, and simply calling reactor.run with installSignalHandlers=False does allow subprocess.Popen to work. My application is working, and I can continue development - I was totally blocked until I had this problem solved.
I have not yet learned how to use inlineCallbacks, and will study up in this, as in the long run, I would like to server side to be able to handle multiple calls.
Much appreciated,
Read Roberts
On 11/3/09 4:08 PM, "exarkun@twistedmatrix.com"
I am trying to use the twisted reactor as a basis for a Python server for a Flex app, using pyAMF. My problem is that one of the server methods requires me to run a series of command-line tools to complete parsing data from a file into an XML object, with lots of Python processing on the results of one command-line tool to provide the input to the next one. I have been using subprocess.call to invoke the command-line tools, but these intermittently - and frequently - fail with an error "IOError: [Errno 4] Interrupted system call".
I do know that this was supposed to be fixed in twisted for Python 2.6, but it doesn't seem to be.
I'm not sure why you know this. I sure would be great if that were the case, but it's not. More changes in Twisted are necessary to let things like subprocess.call work properly alongside Twisted's process API.
I have python 2.6.3 under Mac OSX 10.5.8, and twisted 8.2.0. Is there a work-around for this problem, or a better way to do this with twisted? I do know that I can make a command-line call with reactor.callLater, and that I can return a deferred from my server method, but I don't see how to easily accomplish the chain of calls that I need, short of breaking up my serverMethod1 into the several sequential parts, and calling each from the result handler of the previous one. I'd rather not go that route, as the logic of which command-line call gets made when is actually quite complicated.
Are you aware that this means your server will only be able to handle one request at a time? If so and that's what you're shooting for, then you can fix this easily, by passing False to reactor.run. This disable's Twisted's own process support and lets things like subprocess.call work (blockingly, of course).
What I have now looks like (enormously simplified): class MyPythonServer:
def serverMethod1(self, dummyPost, inputFilePath): # Open inputFilePath, do stuff, save results to file2 subprocess.call(["command1", "file2", "file3"]) # Open file3, do stuff to the data, save results to file4 subprocess.call(["command2", "file4", "file5"]) # Open file5, convert data to a large string containing XML. return xmlString
Of course, you can still have methods that look like this if you use things like inlineCallbacks (which you can, since you said you're using Python 2.6). That way you preserve the style you seem to prefer and avoid blocking your server any time you need to launch a child process. Jean-Paul _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (2)
-
exarkun＠twistedmatrix.com -
Read Roberts