[Twisted-Python] twisted.internet.process for Win32

I've attached my current code, which seems to be working. This will hopefully be checked in today, but some testing by others would be useful. """ Win32 process support. """ # Twisted imports from twisted.internet import task, abstract # System imports import win32api import win32pipe import win32file import win32process import win32security import win32con import win32event import pywintypes import msvcrt import os import sys import threading import Queue import string class Process(abstract.FileDescriptor): """A process that integrates with the Twisted event loop.""" buffer = '' def __init__(self, command, args, environment, path): # security attributes for pipes sAttrs = win32security.SECURITY_ATTRIBUTES() sAttrs.bInheritHandle = 1 # create pipes hStdin_r, self.hStdin_w = win32pipe.CreatePipe(sAttrs, 0) self.hStdout_r, hStdout_w = win32pipe.CreatePipe(sAttrs, 0) self.hStderr_r, hStderr_w = win32pipe.CreatePipe(sAttrs, 0) # set the info structure for the new process. StartupInfo = win32process.STARTUPINFO() StartupInfo.hStdInput = hStdin_r StartupInfo.hStdOutput = hStdout_w StartupInfo.hStdError = hStderr_w StartupInfo.dwFlags = win32process.STARTF_USESTDHANDLES # Mark doesn't support wShowWindow yet. # StartupInfo.dwFlags = StartupInfo.dwFlags | win32process.STARTF_USESHOWWINDOW # StartupInfo.wShowWindow = win32con.SW_HIDE # Create new output read handles and the input write handle. Set # the inheritance properties to FALSE. Otherwise, the child inherits # the these handles; resulting in non-closeable handles to the pipes # being created. pid = win32api.GetCurrentProcess() tmp = win32api.DuplicateHandle( pid, self.hStdin_w, pid, 0, 0, # non-inheritable!! win32con.DUPLICATE_SAME_ACCESS) # Close the inhertible version of the handle win32file.CloseHandle(self.hStdin_w) self.hStdin_w = tmp tmp = win32api.DuplicateHandle( pid, self.hStdout_r, pid, 0, 0, # non-inheritable! win32con.DUPLICATE_SAME_ACCESS) # Close the inhertible version of the handle win32file.CloseHandle(self.hStdout_r) self.hStdout_r = tmp tmp = win32api.DuplicateHandle( pid, self.hStderr_r, pid, 0, 0, # non-inheritable! win32con.DUPLICATE_SAME_ACCESS) # Close the inhertible version of the handle win32file.CloseHandle(self.hStderr_r) self.hStderr_r = tmp # start the process. print "creating process" cmdline = "%s %s" % (command, string.join(args, ' ')) hProcess, hThread, dwPid, dwTid = win32process.CreateProcess( None, # program cmdline,# command line None, # process security attributes None, # thread attributes 1, # inherit handles, or USESTDHANDLES won't work. # creation flags. Don't access the console. 0, # Don't need anything here. # If you're in a GUI app, you should use # CREATE_NEW_CONSOLE here, or any subprocesses # might fall victim to the problem described in: # KB article: Q156755, cmd.exe requires # an NT console in order to perform redirection.. environment, # new environment path, # new directory StartupInfo) # normally, we would save the pid etc. here... print "process created" # Child is launched. Close the parents copy of those pipe handles # that only the child should have open. # You need to make sure that no handles to the write end of the # output pipe are maintained in this process or else the pipe will # not close when the child process exits and the ReadFile will hang. win32file.CloseHandle(hStderr_w) win32file.CloseHandle(hStdout_w) win32file.CloseHandle(hStdin_r) self.outQueue = Queue.Queue() self.closed = 0 threading.Thread(target=self.doWrite).start() threading.Thread(target=self.doReadOut).start() threading.Thread(target=self.doReadErr).start() def write(self, data): """Write data to the process' stdin.""" self.outQueue.put(data) def closeStdin(self): """Close the process' stdin.""" self.outQueue.put(None) def connectionLost(self): """Will be called twice, by the stdout and stderr threads.""" if not self.closed: abstract.FileDescriptor.connectionLost(self) print "connection lost" self.closed = 1 self.closeStdin() win32file.CloseHandle(self.hStdout_r) win32file.CloseHandle(self.hStderr_r) def doWrite(self): """Runs in thread.""" while 1: data = self.outQueue.get() if data == None: break try: win32file.WriteFile(self.hStdin_w, data, None) except win32api.error: break win32file.CloseHandle(self.hStdin_w) def doReadOut(self): """Runs in thread.""" while 1: try: hr, data = win32file.ReadFile(self.hStdout_r, 8192, None) except win32api.error: task.schedule(self.connectionLost) return task.schedule(self.handleChunk, data) def doReadErr(self): """Runs in thread.""" while 1: try: hr, data = win32file.ReadFile(self.hStderr_r, 8192, None) except win32api.error: task.schedule(self.connectionLost) return task.schedule(self.handleError, data) if __name__ == '__main__': from twisted.internet import main def printer(x): print "Got", repr(x) exe = win32api.GetModuleFileName(0) print exe p = Process(exe, ['-u', 'processtest.py'], None, None) print "ok, made process object" p.handleChunk = printer p.handleError = printer p.write("hello, world") p.closeStdin() main.run()

On Fri, Mar 08, 2002 at 09:50:48AM -0500, Itamar Shtull-Trauring wrote:
I've attached my current code, which seems to be working. This will hopefully be checked in today, but some testing by others would be useful.
Looks good; I'm not sure that you need to spawn three threads per process, though. I'll muck around a bit and see if I can do it better :) (It *should* be easy to get it down to one thread by using WaitForMultipleObjects, but we'll see...) -Andrew.

On Fri, Mar 08, 2002 at 09:50:48AM -0500, Itamar Shtull-Trauring wrote:
I've attached my current code, which seems to be working. This will hopefully be checked in today, but some testing by others would be useful.
I'm not sure if it works on Win98 -- is this error harmless? --- D:\PYTHON22\PYTHON.EXE creating process process created ok, made process object connection lost Exception in thread Thread-3: Traceback (most recent call last): File "D:\PYTHON22\lib\threading.py", line 408, in __bootstrap self.run() File "D:\PYTHON22\lib\threading.py", line 396, in run apply(self.__target, self.__args, self.__kwargs) File "process_win32.py", line 175, in doReadErr task.schedule(self.handleError, data) AttributeError: Process instance has no attribute 'handleError' --- And now on to me... :) On Sat, Mar 09, 2002 at 10:16:55PM +1100, Andrew Bennetts wrote:
(It *should* be easy to get it down to one thread by using WaitForMultipleObjects, but we'll see...)
I'm attaching a version which should work with only one thread per process, not three (and with some work, could be limited to one thread overall). However, I'm getting the following error: --- Exception in thread Thread-1: Traceback (most recent call last): File "D:\PYTHON22\lib\threading.py", line 408, in __bootstrap self.run() File "D:\PYTHON22\lib\threading.py", line 396, in run apply(self.__target, self.__args, self.__kwargs) File "process_win32.py", line 158, in waitOnPipe val = win32event.WaitForMultipleObjects(handles, 0, api_error: (6, 'WaitForMultipleObjects', 'The handle is invalid.') --- Anyone have any idea as to what the problem might be? -Andrew.

A bit further reading showed a bit of a race condition in the writing part, and the fact that writing pipe might never get closed in certain situations. All solveable of course - but it might be easier to do a 2 thread solution, one for writing and one with WaitForMultipleObjects for reading. Or... there could be a single thread for *all* Processes stdout and stderr reading, and then another thread per Process for writing.

On Fri, Mar 08, 2002 at 09:50:48AM -0500, Itamar Shtull-Trauring wrote:
I've attached my current code, which seems to be working. This will hopefully be checked in today, but some testing by others would be useful.
Looks good; I'm not sure that you need to spawn three threads per process, though. I'll muck around a bit and see if I can do it better :) (It *should* be easy to get it down to one thread by using WaitForMultipleObjects, but we'll see...) -Andrew.

On Fri, Mar 08, 2002 at 09:50:48AM -0500, Itamar Shtull-Trauring wrote:
I've attached my current code, which seems to be working. This will hopefully be checked in today, but some testing by others would be useful.
I'm not sure if it works on Win98 -- is this error harmless? --- D:\PYTHON22\PYTHON.EXE creating process process created ok, made process object connection lost Exception in thread Thread-3: Traceback (most recent call last): File "D:\PYTHON22\lib\threading.py", line 408, in __bootstrap self.run() File "D:\PYTHON22\lib\threading.py", line 396, in run apply(self.__target, self.__args, self.__kwargs) File "process_win32.py", line 175, in doReadErr task.schedule(self.handleError, data) AttributeError: Process instance has no attribute 'handleError' --- And now on to me... :) On Sat, Mar 09, 2002 at 10:16:55PM +1100, Andrew Bennetts wrote:
(It *should* be easy to get it down to one thread by using WaitForMultipleObjects, but we'll see...)
I'm attaching a version which should work with only one thread per process, not three (and with some work, could be limited to one thread overall). However, I'm getting the following error: --- Exception in thread Thread-1: Traceback (most recent call last): File "D:\PYTHON22\lib\threading.py", line 408, in __bootstrap self.run() File "D:\PYTHON22\lib\threading.py", line 396, in run apply(self.__target, self.__args, self.__kwargs) File "process_win32.py", line 158, in waitOnPipe val = win32event.WaitForMultipleObjects(handles, 0, api_error: (6, 'WaitForMultipleObjects', 'The handle is invalid.') --- Anyone have any idea as to what the problem might be? -Andrew.

A bit further reading showed a bit of a race condition in the writing part, and the fact that writing pipe might never get closed in certain situations. All solveable of course - but it might be easier to do a 2 thread solution, one for writing and one with WaitForMultipleObjects for reading. Or... there could be a single thread for *all* Processes stdout and stderr reading, and then another thread per Process for writing.
participants (2)
-
Andrew Bennetts
-
Itamar Shtull-Trauring