[Twisted-Python] ProcessProtocol randomly not returning all of stdout
I recently started running into a problem where my process protocol isn't extracting stdout accurately in all cases. I have a method on a service that loops through a list and runs a command, passing in each item on the list one at a time. Out of ~50 items in the list, and thus ~50 invocations of the command, one or two will fail to extract all of stdout from the command. Instead the first character of stdout is extracted and that is it. The process protocol I'm using is below. Does anyone see anything wrong with the code? I am using the win32event reactor and Twisted 1.3.0. from twisted.internet import reactor from twisted.internet import protocol, error, defer from twisted.python import log from ratcontrol import errors, config import os, sys class BasicProcessProtocol(protocol.ProcessProtocol): def __init__(self): self.deferred = defer.Deferred() self.stdout = "" self.stderr = "" self.status = 0 def connectionMade(self): self.transport.closeStdin() def outReceived(self, data): self.stdout += data def errReceived(self, data): self.stderr += data def processEnded(self, reason): if reason.check(error.ProcessDone): self.status = 0 # pass the items we want down the callback chain self.deferred.callback((self.status, self.stdout, self.stderr)) else: self.status = 1 theError = errors.CmdError(self.cmd, self.status, self.stdout, self.stderr) log.msg(theError) self.deferred.errback(theError) def openProcess(cmd_and_args): cmd = cmd_and_args[0] args = cmd_and_args[1:] # Spawn the process using the reactor so we don't block proto = BasicProcessProtocol() proto.cmd = cmd_and_args reactor.spawnProcess(proto, cmd, cmd_and_args, env=os.environ) return proto.deferred
On Thu, 2005-01-13 at 13:47 -0600, Justin Johnson wrote:
The process protocol I'm using is below. Does anyone see anything wrong with the code? I am using the win32event reactor and Twisted 1.3.0.
The win32event process support is known to be buggy. On Windows, popen with a thread may be a better choice.
Crud! It's been working for me for most everything else. Did you add support to the default reactor to spawn threads on Windows? I thought I saw a closed bug related to this, but was waiting for the next release to try it out. On Thu, 13 Jan 2005 15:07:37 -0500, Itamar Shtull-Trauring <itamar@itamarst.org> wrote:
On Thu, 2005-01-13 at 13:47 -0600, Justin Johnson wrote:
The process protocol I'm using is below. Does anyone see anything wrong with the code? I am using the win32event reactor and Twisted 1.3.0.
The win32event process support is known to be buggy. On Windows, popen with a thread may be a better choice.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Thu, 2005-01-13 at 14:11 -0600, Justin Johnson wrote:
Crud! It's been working for me for most everything else.
Did you add support to the default reactor to spawn threads on Windows? I thought I saw a closed bug related to this, but was waiting for the next release to try it out.
Thread support has always been fine on Windows.
On Thu, 2005-01-13 at 15:07 -0500, Itamar Shtull-Trauring wrote:
The win32event process support is known to be buggy. On Windows, popen with a thread may be a better choice.
Is this still true in trunk? I keep thinking that we've fixed this issue and it keeps not being fixed. What issue number is it? If popen with a thread is a better choice, can we provide an implementation of that that comes with the reactor so that at least people don't have to rewrite their code?
On Thu, 2005-01-13 at 16:23 -0500, Glyph Lefkowitz wrote:
The win32event process support is known to be buggy. On Windows, popen with a thread may be a better choice.
Is this still true in trunk? I keep thinking that we've fixed this issue and it keeps not being fixed. What issue number is it?
Cory checked in fixed code and then backed it out.
If popen with a thread is a better choice, can we provide an implementation of that that comes with the reactor so that at least people don't have to rewrite their code?
Cory's code may have been similar to doing this.
I was referring to Issue591, which based on the comments is done. Can anyone verify that? Maybe I'll have to do a windows build instead of wait for the next release. On Thu, 13 Jan 2005 16:34:22 -0500, Itamar Shtull-Trauring <itamar@itamarst.org> wrote:
On Thu, 2005-01-13 at 16:23 -0500, Glyph Lefkowitz wrote:
The win32event process support is known to be buggy. On Windows, popen with a thread may be a better choice.
Is this still true in trunk? I keep thinking that we've fixed this issue and it keeps not being fixed. What issue number is it?
Cory checked in fixed code and then backed it out.
If popen with a thread is a better choice, can we provide an implementation of that that comes with the reactor so that at least people don't have to rewrite their code?
Cory's code may have been similar to doing this.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Thu, 2005-01-13 at 15:49 -0600, Justin Johnson wrote:
I was referring to Issue591, which based on the comments is done. Can anyone verify that? Maybe I'll have to do a windows build instead of wait for the next release.
I think Cory then backed the code out, so the bug ought to be reopened.
participants (3)
-
Glyph Lefkowitz -
Itamar Shtull-Trauring -
Justin Johnson