spawning a xvfb and firefox with a timeout
I'm attempting to spawn xvfb with firefox using twisted's spawnProcess. I'm having two issues: 1) when I call a normal process (not xvfb) it seems to work, but when I call xvfb the process goes defunct 2) perhaps because it goes defunct but the timeout I set in place in the processprotocol does not trigger. ============== here is the code for spawning a process ============== 108 command = ["/usr/bin/xvfb-run", "--auto-servernum", "/usr/bin/firefox", "-P", profile_id] 109 subprocess = reactor.spawnProcess(TimedProcessProtocol(20), 110 command[0], command) 111 logging.debug(type(subprocess)) 112 logging.debug("spawned a process: pid: %d", subprocess.pid) 113 114 # this file will be created when firefox starts 115 while (not os.path.exists(self.profile_dir + "/importantfile")): 116 logging.debug("in while loop waiting for firesharkReady file to exist") 117 pass 118 119 logging.debug("file exist") ============== here is the process class ============== 27 class TimedProcessProtocol(protocol.ProcessProtocol): 28 29 def __init__(self, timeout): 30 self.timeout = timeout 31 32 def connectionMade(self): 33 logging.debug("connection made timeout = %d", self.timeout) 34 @defer.inlineCallbacks 35 def killIfAlive(): 36 logging.debug("timeout reached - killing process") 37 try: 38 yield self.transport.signalProcess('KILL') 39 except error.ProcessExitedAlready: 40 logging.debug("process already exited") 41 pass 42 43 d = reactor.callLater(self.timeout, killIfAlive) 44 45 def outReceived(self, data): 46 logging.debug("output: %s", data) 47 48 def errReceived(self, data): 49 logging.debug("errReceived %s", data)
On Aug 19, 2011, at 12:42 AM, Stephan wrote:
I'm attempting to spawn xvfb with firefox using twisted's spawnProcess.
I'm having two issues:
1) when I call a normal process (not xvfb) it seems to work, but when I call xvfb the process goes defunct
Defunct processes have exited, but not been reaped. In older versions of Twisted, this might happen if some other library were competing to register the SIGCHLD signal. What version of Twisted are you using?
2) perhaps because it goes defunct but the timeout I set in place in the processprotocol does not trigger.
It should never go defunct, whether or not your timeout triggers.
============== here is the code for spawning a process ============== 108 command = ["/usr/bin/xvfb-run", "--auto-servernum", "/usr/bin/firefox", "-P", profile_id] 109 subprocess = reactor.spawnProcess(TimedProcessProtocol(20), 110 command[0], command) 111 logging.debug(type(subprocess)) 112 logging.debug("spawned a process: pid: %d", subprocess.pid) 113 114 # this file will be created when firefox starts 115 while (not os.path.exists(self.profile_dir + "/importantfile")): 116 logging.debug("in while loop waiting for firesharkReady file to exist") 117 pass
You should really not busy-loop like this waiting for the subprocess. Have a repeating callLater (or LoopingCall) that does this check cooperatively with the reactor, so if something goes wrong (for example: firefox fails to launch), you'll be able to react to it effectively rather than just hanging your Twisted process forever. If firefox isn't actually starting correctly, or that file doesn't actually get created for some reason, this could cause the defunct process.
118 119 logging.debug("file exist")
============== here is the process class ==============
27 class TimedProcessProtocol(protocol.ProcessProtocol): 28 29 def __init__(self, timeout): 30 self.timeout = timeout 31 32 def connectionMade(self): 33 logging.debug("connection made timeout = %d", self.timeout) 34 @defer.inlineCallbacks 35 def killIfAlive(): 36 logging.debug("timeout reached - killing process") 37 try: 38 yield self.transport.signalProcess('KILL') 39 except error.ProcessExitedAlready: 40 logging.debug("process already exited") 41 pass 42 43 d = reactor.callLater(self.timeout, killIfAlive) 44 45 def outReceived(self, data): 46 logging.debug("output: %s", data) 47 48 def errReceived(self, data): 49 logging.debug("errReceived %s", data)
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Thanks for the prompt reply response inline. I'm still having issues even though I've changed some code based on your advice. On Thu, Aug 18, 2011 at 9:51 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
On Aug 19, 2011, at 12:42 AM, Stephan wrote:
I'm attempting to spawn xvfb with firefox using twisted's spawnProcess.
I'm having two issues:
1) when I call a normal process (not xvfb) it seems to work, but when I call xvfb the process goes defunct
Defunct processes have exited, but not been reaped. In older versions of Twisted, this might happen if some other library were competing to register the SIGCHLD signal. What version of Twisted are you using?
I'm using the latest version (I installed it today). 11.0.0 on an ubuntu machine
2) perhaps because it goes defunct but the timeout I set in place in the processprotocol does not trigger.
It should never go defunct, whether or not your timeout triggers.
I agree, I can't figure out why it's going defunct.
============== here is the code for spawning a process ============== 108 command = ["/usr/bin/xvfb-run", "--auto-servernum", "/usr/bin/firefox", "-P", profile_id] 109 subprocess = reactor.spawnProcess(TimedProcessProtocol(20), 110 command[0], command) 111 logging.debug(type(subprocess)) 112 logging.debug("spawned a process: pid: %d", subprocess.pid) 113 114 # this file will be created when firefox starts 115 while (not os.path.exists(self.profile_dir + "/importantfile")): 116 logging.debug("in while loop waiting for firesharkReady file to exist") 117 pass
You should really not busy-loop like this waiting for the subprocess. Have a repeating callLater (or LoopingCall) that does this check cooperatively with the reactor, so if something goes wrong (for example: firefox fails to launch), you'll be able to react to it effectively rather than just hanging your Twisted process forever. If firefox isn't actually starting correctly, or that file doesn't actually get created for some reason, this could cause the defunct process.
I agree with your point, I've changed it a bit (see the code at the bottom) I'm not sure if this is correct or not.
118 119 logging.debug("file exist")
============== here is the process class ==============
27 class TimedProcessProtocol(protocol.ProcessProtocol): 28 29 def __init__(self, timeout): 30 self.timeout = timeout 31 32 def connectionMade(self): 33 logging.debug("connection made timeout = %d", self.timeout) 34 @defer.inlineCallbacks 35 def killIfAlive(): 36 logging.debug("timeout reached - killing process") 37 try: 38 yield self.transport.signalProcess('KILL') 39 except error.ProcessExitedAlready: 40 logging.debug("process already exited") 41 pass 42 43 d = reactor.callLater(self.timeout, killIfAlive) 44 45 def outReceived(self, data): 46 logging.debug("output: %s", data) 47 48 def errReceived(self, data): 49 logging.debug("errReceived %s", data)
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
============ new code (still does not work) ============= 108 profile_id = self.profile['name'] 109 command = ["/usr/bin/xvfb-run", "--auto-servernum", "/usr/bin/firefox", "-P", profile_id] 110 subprocess = reactor.spawnProcess(TimedProcessProtocol(20), 111 command[0], command) 112 logging.debug(type(subprocess)) 113 logging.debug("spawned a process: pid: %d", subprocess.pid) 114 115 lc = LoopingCall(self.check_if_file) 116 self.check = lc 117 lc.start(0.5) 118 119 def check_if_file(self): 120 if os.path.exists(self.fireshark_profile_dir + "/firesharkReady"): 121 logging.debug("firesharkReady file to exist") 122 self.check.stop() ============ new code (log) ============= 478C0-CA07-11E0-B595-E88E810DE385 2011-08-18 22:06:58,038 DEBUG:FirefoxProcess run called 2011-08-18 22:06:58,038 DEBUG:intialize firefox files 2011-08-18 22:06:58,040 DEBUG:connection made timeout = 20 2011-08-18 22:06:58,041 DEBUG:<class 'twisted.internet.process.Process'> 2011-08-18 22:06:58,041 DEBUG:spawned a process: pid: 30562 2011-08-18 22:07:01,374 DEBUG:inConnectionLost 2011-08-18 22:07:01,374 DEBUG:errConnectionLost 2011-08-18 22:07:01,375 DEBUG:process exited, status 1 ======================= this time the xvfb process did not go defunct but it did exit, which shouldn't really happen, firefox should just stay open really.
On Aug 19, 2011, at 1:14 AM, Stephan wrote:
Thanks for the prompt reply response inline. I'm still having issues even though I've changed some code based on your advice.
Perhaps you could reduce this to a complete, runnable example so that someone could reproduce the behavior you're seeing. At this point further advice would be guessing on my part.
Hi Glyph, I've created a specially crafted runnable example you can find it here: http://pastebin.com/dyhTFPjM as you'll see in the debug log that is created: 2011-08-20 10:36:10,484 DEBUG:errReceived xvfb-run: error: Xvfb failed to start 2011-08-20 10:36:10,486 DEBUG:inConnectionLost 2011-08-20 10:36:10,486 DEBUG:errConnectionLost 2011-08-20 10:36:10,487 DEBUG:process exited, status 1 Any help is appreciated. to install xvfb on linux if not already installed sudo apt-get install xvfb On Fri, Aug 19, 2011 at 1:22 AM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
On Aug 19, 2011, at 1:14 AM, Stephan wrote:
Thanks for the prompt reply response inline. I'm still having issues even though I've changed some code based on your advice.
Perhaps you could reduce this to a complete, runnable example so that someone could reproduce the behavior you're seeing. At this point further advice would be guessing on my part. _______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
I've attempted to switch out spawnProcess for subprocess.Popen e.g. command = "xvfb-run --auto-servernum firefox" args = shlex.split(command) #p = subprocess.Popen(args) #logging.debug("spawned a process: pid: %d", p.pid) subprocess = reactor.spawnProcess(TimedProcessProtocol(20), args[0], args) logging.debug(type(subprocess)) logging.debug("spawned a process: pid: %d", subprocess.pid) when using subprocess.Popen, xvfb does start up correctly, but then I have a python issue which is that xfb will start firefox-bin and make it the parent and exit, thus, making it hard to kill the new process of which I don't know the pid of. in any case, I'd like to use spawnProcess, so any help as to why this isn't working would be helpful. Stephan On Sat, Aug 20, 2011 at 10:52 AM, Stephan <schenette@gmail.com> wrote:
Hi Glyph,
I've created a specially crafted runnable example you can find it here: http://pastebin.com/dyhTFPjM
as you'll see in the debug log that is created:
2011-08-20 10:36:10,484 DEBUG:errReceived xvfb-run: error: Xvfb failed to start 2011-08-20 10:36:10,486 DEBUG:inConnectionLost 2011-08-20 10:36:10,486 DEBUG:errConnectionLost 2011-08-20 10:36:10,487 DEBUG:process exited, status 1
Any help is appreciated. to install xvfb on linux if not already installed sudo apt-get install xvfb
On Fri, Aug 19, 2011 at 1:22 AM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
On Aug 19, 2011, at 1:14 AM, Stephan wrote:
Thanks for the prompt reply response inline. I'm still having issues even though I've changed some code based on your advice.
Perhaps you could reduce this to a complete, runnable example so that someone could reproduce the behavior you're seeing. At this point further advice would be guessing on my part. _______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
On 19 Aug, 04:42 am, schenette@gmail.com wrote:
I'm attempting to spawn xvfb with firefox using twisted's spawnProcess.
I'm having two issues:
1) when I call a normal process (not xvfb) it seems to work, but when I call xvfb the process goes defunct
2) perhaps because it goes defunct but the timeout I set in place in the processprotocol does not trigger.
============== here is the code for spawning a process ============== 108 command = ["/usr/bin/xvfb-run", "--auto-servernum", "/usr/bin/firefox", "-P", profile_id] 109 subprocess = reactor.spawnProcess(TimedProcessProtocol(20), 110 command[0], command)
Does it make any difference if you pass on the parent's environment to the xvfb-run process? Try something like spawnProcess(..., env=os.environ) Jean-Paul
Jean-Paul, adding env=os.environ to spanProcess fixed the issue, thank you. There is still one problem I brought up that I'm attempting to figure out with twisted that is related to the initial question... current status is that now with the addition of setting the env params xvfb starts up correctly and starts up firefox. the ProcessProtocol object that I created, has a calllater function that kills the process after 20seconds. the issue is that xvfb has one process id and any process it's asked to create will cause another process to be spawned, thus another process id, where the parent process is that of xvfb. when I issue the kill command, xvfb does indeed die, but firefox-bin stays around and it's parent is set to 1. here is the ProcessProtocol code, should I be killing the parent process differently so that the children die as well? 33 class TimedProcessProtocol(protocol.ProcessProtocol): 34 35 def __init__(self, timeout): 36 self.timeout = timeout 37 38 def connectionMade(self): 39 logging.debug("connection made timeout = %d", self.timeout) 40 @defer.inlineCallbacks 41 def killIfAlive(): 42 logging.debug("timeout reached - killing process") 43 try: 44 yield self.transport.signalProcess('TERM') 45 except error.ProcessExitedAlready: 46 logging.debug("process already exited") 47 pass 48 49 d = reactor.callLater(self.timeout, killIfAlive) 50 51 def outReceived(self, data): 52 logging.debug("output: %s", data) Stephan ps both TERM and KILL act the same way On Sat, Aug 20, 2011 at 2:01 PM, <exarkun@twistedmatrix.com> wrote:
On 19 Aug, 04:42 am, schenette@gmail.com wrote:
I'm attempting to spawn xvfb with firefox using twisted's spawnProcess.
I'm having two issues:
1) when I call a normal process (not xvfb) it seems to work, but when I call xvfb the process goes defunct
2) perhaps because it goes defunct but the timeout I set in place in the processprotocol does not trigger.
============== here is the code for spawning a process ============== 108 command = ["/usr/bin/xvfb-run", "--auto-servernum", "/usr/bin/firefox", "-P", profile_id] 109 subprocess = reactor.spawnProcess(TimedProcessProtocol(20), 110 command[0], command)
Does it make any difference if you pass on the parent's environment to the xvfb-run process? Try something like spawnProcess(..., env=os.environ)
Jean-Paul
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
I think I might have figured out that you have to add usePTY=1 to associate the process group, so that the children will be killed when the parent process is killed. e.g. subprocess = reactor.spawnProcess(self.pp, args[0], args, env = os.e nviron, usePTY=1) but I tried to separate out the killing in another process and now the new method isnt' called. are ProcessProtocol classes allowed to have other methods added to it for calling? 45 def killProcessIfAlive(self): 46 logging.debug("killProcessIfAlive called") 47 try: 48 yield self.transport.signalProcess('KILL') 49 except error.ProcessExitedAlready: 50 logging.debug("process already exited") 51 pass 52 53 def connectionMade(self): 54 logging.debug("connection made timeout = %d", self.timeout) 55 @defer.inlineCallbacks 56 def onTimer(): 57 logging.debug("timeout triggered") 58 self.killProcessIfAlive() 59 d = reactor.callLater(self.timeout, onTimer) On Mon, Aug 22, 2011 at 2:16 AM, Stephan <schenette@gmail.com> wrote:
Jean-Paul,
adding
env=os.environ to spanProcess fixed the issue, thank you.
There is still one problem I brought up that I'm attempting to figure out with twisted that is related to the initial question...
current status is that now with the addition of setting the env params xvfb starts up correctly and starts up firefox.
the ProcessProtocol object that I created, has a calllater function that kills the process after 20seconds.
the issue is that xvfb has one process id and any process it's asked to create will cause another process to be spawned, thus another process id, where the parent process is that of xvfb.
when I issue the kill command, xvfb does indeed die, but firefox-bin stays around and it's parent is set to 1.
here is the ProcessProtocol code, should I be killing the parent process differently so that the children die as well?
33 class TimedProcessProtocol(protocol.ProcessProtocol): 34 35 def __init__(self, timeout): 36 self.timeout = timeout 37 38 def connectionMade(self): 39 logging.debug("connection made timeout = %d", self.timeout) 40 @defer.inlineCallbacks 41 def killIfAlive(): 42 logging.debug("timeout reached - killing process") 43 try: 44 yield self.transport.signalProcess('TERM') 45 except error.ProcessExitedAlready: 46 logging.debug("process already exited") 47 pass 48 49 d = reactor.callLater(self.timeout, killIfAlive) 50 51 def outReceived(self, data): 52 logging.debug("output: %s", data)
Stephan
ps both TERM and KILL act the same way
On Sat, Aug 20, 2011 at 2:01 PM, <exarkun@twistedmatrix.com> wrote:
On 19 Aug, 04:42 am, schenette@gmail.com wrote:
I'm attempting to spawn xvfb with firefox using twisted's spawnProcess.
I'm having two issues:
1) when I call a normal process (not xvfb) it seems to work, but when I call xvfb the process goes defunct
2) perhaps because it goes defunct but the timeout I set in place in the processprotocol does not trigger.
============== here is the code for spawning a process ============== 108 command = ["/usr/bin/xvfb-run", "--auto-servernum", "/usr/bin/firefox", "-P", profile_id] 109 subprocess = reactor.spawnProcess(TimedProcessProtocol(20), 110 command[0], command)
Does it make any difference if you pass on the parent's environment to the xvfb-run process? Try something like spawnProcess(..., env=os.environ)
Jean-Paul
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
On 04:32 pm, schenette@gmail.com wrote:
I think I might have figured out that you have to add usePTY=1 to associate the process group, so that the children will be killed when the parent process is killed. e.g. subprocess = reactor.spawnProcess(self.pp, args[0], args, env = os.e nviron, usePTY=1)
This is a not bad approach. Another possibility might be to leave usePTY out but change how you kill the process. There is the notion of "process groups", which contain one or more processes, and all of the processes in a group can be sent a signal by passing a negative number to os.kill. For your case, the process group would probably be the same as the pid of the xvfb-run process, so you could do something like: os.kill(-process.transport.pid, SIGTERM) Whether this is better or worse than using a PTY depends on the particulars of how xvfb-run manages its child process, whether it is okay to actually have a pty allocated, and probably some other POSIX arcana.
but I tried to separate out the killing in another process and now the new method isnt' called. are ProcessProtocol classes allowed to have other methods added to it for calling?
45 def killProcessIfAlive(self): 46 logging.debug("killProcessIfAlive called") 47 try: 48 yield self.transport.signalProcess('KILL') 49 except error.ProcessExitedAlready: 50 logging.debug("process already exited") 51 pass
It looks like you wanted killProcessIfAlive to be decorated with inlineCallbacks, but I don't see that here. So that would prevent the signal from ever actually being sent. signalProcess doesn't return a Deferred though, so you can skip inlineCallbacks and the yield and that should also fix the problem.
52 53 def connectionMade(self): 54 logging.debug("connection made timeout = %d", self.timeout) 55 @defer.inlineCallbacks 56 def onTimer(): 57 logging.debug("timeout triggered") 58 self.killProcessIfAlive() 59 d = reactor.callLater(self.timeout, onTimer)
Ah, there's the inlineCallbacks. It won't work there, though, it needs to go onto the generator function (the function that has yield in it). Jean-Paul
Hi Jean-Paul, decorated with inlineCallbacks? why would I need that? I'm trying to figure that out... but would it look like this? def killProcessIfAlive(self): logging.debug("killProcessIfAlive called") @defer.inlineCallbacks try: yield self.transport.signalProcess('KILL') except error.ProcessExitedAlready: logging.debug("process already exited") pass def connectionMade(self): logging.debug("connection made timeout = %d", self.timeout) #@defer.inlineCallbacks def onTimer(): logging.debug("timeout triggered") self.killProcessIfAlive() d = reactor.callLater(self.timeout, onTimer) ---- also I tried before you had sent out the email os.kill(self.pid, SIGTERM) and os.kill(self.pid, SIGTERM) but not os.kill(-process.transport.pid, SIGTERM) (with the dash, does that make a difference?) On Mon, Aug 22, 2011 at 12:53 PM, <exarkun@twistedmatrix.com> wrote:
On 04:32 pm, schenette@gmail.com wrote:
I think I might have figured out that you have to add usePTY=1 to associate the process group, so that the children will be killed when the parent process is killed. e.g. subprocess = reactor.spawnProcess(self.pp, args[0], args, env = os.e nviron, usePTY=1)
This is a not bad approach. Another possibility might be to leave usePTY out but change how you kill the process. There is the notion of "process groups", which contain one or more processes, and all of the processes in a group can be sent a signal by passing a negative number to os.kill. For your case, the process group would probably be the same as the pid of the xvfb-run process, so you could do something like:
os.kill(-process.transport.pid, SIGTERM)
Whether this is better or worse than using a PTY depends on the particulars of how xvfb-run manages its child process, whether it is okay to actually have a pty allocated, and probably some other POSIX arcana.
but I tried to separate out the killing in another process and now the new method isnt' called. are ProcessProtocol classes allowed to have other methods added to it for calling?
45 def killProcessIfAlive(self): 46 logging.debug("killProcessIfAlive called") 47 try: 48 yield self.transport.signalProcess('KILL') 49 except error.ProcessExitedAlready: 50 logging.debug("process already exited") 51 pass
It looks like you wanted killProcessIfAlive to be decorated with inlineCallbacks, but I don't see that here. So that would prevent the signal from ever actually being sent. signalProcess doesn't return a Deferred though, so you can skip inlineCallbacks and the yield and that should also fix the problem.
52 53 def connectionMade(self): 54 logging.debug("connection made timeout = %d", self.timeout) 55 @defer.inlineCallbacks 56 def onTimer(): 57 logging.debug("timeout triggered") 58 self.killProcessIfAlive() 59 d = reactor.callLater(self.timeout, onTimer)
Ah, there's the inlineCallbacks. It won't work there, though, it needs to go onto the generator function (the function that has yield in it).
Jean-Paul
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
participants (3)
-
exarkun＠twistedmatrix.com -
Glyph Lefkowitz -
Stephan