[Twisted-Python] Running twistd and tac files as Windows service
I have been looking at running some twisted programs as a windows service. So in the end I took a copy of twistd.py and adapted it to load my tac file and run it as a windows service. Of course the windows service will run it in a separate thread. I now have it running on XP, 2003 Server Twisted 10 and 12 and it all seemed fine. However when I install it on Windows 7 I get an error 2012-10-23 07:54:49+0100 [-] Unhandled Error Traceback (most recent call last): File "C:\Python27\lib\site-packages\twisted\internet\base.py", line 413, in fireEvent DeferredList(beforeResults).addCallback(self._continueFiring) File "C:\Python27\lib\site-packages\twisted\internet\defer.py", line 297, in addCallback callbackKeywords=kw) File "C:\Python27\lib\site-packages\twisted\internet\defer.py", line 286, in addCallbacks self._runCallbacks() File "C:\Python27\lib\site-packages\twisted\internet\defer.py", line 542, in _runCallbacks current.result = callback(current.result, *args, **kw) --- <exception caught here> --- File "C:\Python27\lib\site-packages\twisted\internet\base.py", line 426, in _continueFiring callable(*args, **kwargs) File "C:\Python27\lib\site-packages\twisted\internet\base.py", line 1153, in _reallyStartRunning self._handleSignals() File "C:\Python27\lib\site-packages\twisted\internet\posixbase.py", line 277, in _handleSignals _SignalReactorMixin._handleSignals(self) File "C:\Python27\lib\site-packages\twisted\internet\base.py", line 1118, in _handleSignals signal.signal(signal.SIGINT, self.sigInt) exceptions.ValueError: signal only works in main thread Looking around it would seem I need to tell the reactor 'installsignalhandlers=0'. I am not sure if I should do this with windows or not. Has anyone got twistd.py to run as a service on windows because I would be glad to know what other people have done to solve the problem. Thanks for any info. *John Aherne* * * * * * * *www.rocs.co.uk * 020 7223 7567
On 08:23 am, johnaherne@rocs.co.uk wrote:
To start off with, I'll point out that this isn't a supported mode of running Twisted (which I'm sure you noticed already :). This feature has been requested several times and worked on by a couple people, but not quite to the point of completion. Here's a related ticket: http://twistedmatrix.com/trac/ticket/4073 There's also some more code in the sandbox: http://twistedmatrix.com/trac/browser/sandbox/moonfallen BuildBot also has some code, though I'm not exactly sure where it is. It would be really great for someone to actually complete and generalize one or more of these initial attempts so that Windows service integration could be a real, working feature. :)
Yes, you'll need to do that. It probably won't even break child process support (although other aspects of Windows services probably will). Notice it's failing on installing a SIGINT handler - ie, a Control-C handler. Since you want the thing to run as a service, you probably aren't very interested in having Control-C do anything. Jean-Paul
Some of the code seems pretty ancient and for setting up the service I think the win32com stuff from Mark Hammond seems to have all covered. I have started to look around twisted to see where I can stop the signal handlers being installed.. I have landed in postApplication where startReactor is called, but I have yet to work out how I can set installSignalHandlers=False. Am I right in assuming this is where I should be looking. I don't have much idea about signalhandlers but I assume the one I am interested in here is to stop ControlC. The others might need to stay on. I'm not too sure about that without knowing more about windows services and how they react to these signals. Thanks for the pointers John Aherne
-- *John Aherne* * * * * * * *www.rocs.co.uk * 020 7223 7567
On 10/24/2012 08:43 AM, John Aherne wrote:
No, you need more than that removed. Otherwise you get a "signal" (can't remember which - SIGBRK?) when the logged-in workstation logs off, which terminates your twisted services. Very confusing. [FWIW we don't use twistd/.tac on windows]
Am 24.10.2012 11:01, schrieb Phil Mayers:
There are no signals on windows. But what you are talking about is the SetConsoleCtrlHandler function. http://docs.activestate.com/activepython/2.4/pywin32/win32api__SetConsoleCtr... http://msdn.microsoft.com/en-us/library/windows/desktop/ms686016%28v=vs.85%2... Thats relevant if your process has a console, e.g. is not a GUI app. (pythonw.exe vs. python.exe). If you run as a GUI app you might get a WM_QUERYENDSESSION or similar message. You get a CTRL_LOGOFF_EVENT when a user on the machine logs off and you run as a subprocess of a service and need to ignore it. The service itself should deal with this already. Michael -- Michael Schlenker Software Architect CONTACT Software GmbH Tel.: +49 (421) 20153-80 Wiener Straße 1-3 Fax: +49 (421) 20153-41 28359 Bremen http://www.contact.de/ E-Mail: msc@contact.de Sitz der Gesellschaft: Bremen Geschäftsführer: Karl Heinz Zachries, Ralf Holtgrefe Eingetragen im Handelsregister des Amtsgerichts Bremen unter HRB 13215
n 24/10/12 12:33, Michael Schlenker wrote:
But Python for windows certain calls "signal()" handlers in response to certain events, in certain configurations. Whether they're actually "signals" is kind of moot, though interesting.
Interesting. I've just looked over the code we've got. It actually does something rather more involved than I'd first thought. Specifcally, the "ntsvc.py" code is pure-python, no Twisted. It actually runs the Twisted daemon as a sub-process, like this: import os class MyService(win32serviceutil.ServiceFramework): """NT Service.""" _svc_name_ = "opimport-wrap" _svc_display_name_ = "Marval opimport wrapper" def SvcDoRun(self): self.childServer = Popen([ "c:/python25/pythonw.exe",SCRIPT,"svc" ]) self.childServer.wait() ...and the Twisted script does this: def main(): args = sys.argv[1:] if args and args[0]=='svc': as_service = True else: as_service = False ... if as_service: log.startLogging(some_log_observer) reactor.run(installSignalHandlers=0) else: log.startLogging(sys.stderr) reactor.run() My memory is hazy on this, but if I recall correctly the main reason it works this way is that I can execute the twisted script directly from the console (for debugging) or via the service wrapper. Are you suggesting that if I change that subprocess to use "python.exe" versus "pythonw.exe" I would not need to disable the signal handlers? Cheers, Phil
Am 24.10.2012 14:14, schrieb Phil Mayers: this looks dubious, are you sure you can still react to ServiceManager actions like 'Stop' or 'Query' this way, isn't this wait() blocking?
Won't probably help, not sure though. If you ever import the 'time' module, as Python registers a control handler there, to allow time.sleep() to wake up when ctrl-c is pressed or the system shuts down, so i assume it has a console anyway. from timemodule.c #ifdef MS_WINDOWS /* Helper to allow interrupts for Windows. If Ctrl+C event delivered while not sleeping it will be ignored. */ main_thread = PyThread_get_thread_ident(); hInterruptEvent = CreateEvent(NULL, TRUE, FALSE, NULL); SetConsoleCtrlHandler( PyCtrlHandler, TRUE); #endif /* MS_WINDOWS */ The usual workaround to keep a service subprocess with a console alive is to simply register your own ConsoleHandler to override the default (which is ExitProcess()) and return TRUE, so the default handler doesn't run. If you spawn a lot of subprocess (around 100 at a time) you will additionally have fun with random crashes once you hit the DesktopHeap Memory limit. Michael -- Michael Schlenker Software Architect CONTACT Software GmbH Tel.: +49 (421) 20153-80 Wiener Straße 1-3 Fax: +49 (421) 20153-41 28359 Bremen http://www.contact.de/ E-Mail: msc@contact.de Sitz der Gesellschaft: Bremen Geschäftsführer: Karl Heinz Zachries, Ralf Holtgrefe Eingetragen im Handelsregister des Amtsgerichts Bremen unter HRB 13215
On 24/10/12 17:14, Michael Schlenker wrote:
this looks dubious, are you sure you can still react to ServiceManager
Yes.
actions like 'Stop' or 'Query' this way, isn't this wait() blocking?
AIUI, the "svc" methods are called from various threads? I didn't create this code; I cribbed it from somewhere, but this was a long time ago (since I avoid Windows for anything other than playing games).
This entire Twisted process exists to spawn processes that can only be run under Windows. Fortunately, the app in particular is so crappy that if I ever get *close* to 100 processes, I'm sure the app will die well before pywin does ;o)
As JP said, buildbot has some code for this at https://github.com/buildbot/buildbot/blob/master/master/contrib/windows/buil...
On Tue, Oct 23, 2012 at 6:09 PM, Tom Prince <tom.prince@ualberta.net> wrote:
As JP said, buildbot has some code for this at
https://github.com/buildbot/buildbot/blob/master/master/contrib/windows/buil...
-- *John Aherne* * * * * * * *www.rocs.co.uk * 020 7223 7567
On 23 October 2012 20:09, Tom Prince <tom.prince@ualberta.net> wrote:
As JP said, buildbot has some code for this at https://github.com/buildbot/buildbot/blob/master/master/contrib/windows/buil...
Just in case it might be of any help. I am using something based on this code: http://code.activestate.com/recipes/551780/ The code with Twisted reactor is here: https://github.com/chevah/commons/blob/master/commons/compat/nt_service.py It does not use .tac files. 'initialize' is used for creating the services, 'start' for start listening and 'stop' for stop listening. Cheers -- Adi Roiban
On 08:23 am, johnaherne@rocs.co.uk wrote:
To start off with, I'll point out that this isn't a supported mode of running Twisted (which I'm sure you noticed already :). This feature has been requested several times and worked on by a couple people, but not quite to the point of completion. Here's a related ticket: http://twistedmatrix.com/trac/ticket/4073 There's also some more code in the sandbox: http://twistedmatrix.com/trac/browser/sandbox/moonfallen BuildBot also has some code, though I'm not exactly sure where it is. It would be really great for someone to actually complete and generalize one or more of these initial attempts so that Windows service integration could be a real, working feature. :)
Yes, you'll need to do that. It probably won't even break child process support (although other aspects of Windows services probably will). Notice it's failing on installing a SIGINT handler - ie, a Control-C handler. Since you want the thing to run as a service, you probably aren't very interested in having Control-C do anything. Jean-Paul
Some of the code seems pretty ancient and for setting up the service I think the win32com stuff from Mark Hammond seems to have all covered. I have started to look around twisted to see where I can stop the signal handlers being installed.. I have landed in postApplication where startReactor is called, but I have yet to work out how I can set installSignalHandlers=False. Am I right in assuming this is where I should be looking. I don't have much idea about signalhandlers but I assume the one I am interested in here is to stop ControlC. The others might need to stay on. I'm not too sure about that without knowing more about windows services and how they react to these signals. Thanks for the pointers John Aherne
-- *John Aherne* * * * * * * *www.rocs.co.uk * 020 7223 7567
On 10/24/2012 08:43 AM, John Aherne wrote:
No, you need more than that removed. Otherwise you get a "signal" (can't remember which - SIGBRK?) when the logged-in workstation logs off, which terminates your twisted services. Very confusing. [FWIW we don't use twistd/.tac on windows]
Am 24.10.2012 11:01, schrieb Phil Mayers:
There are no signals on windows. But what you are talking about is the SetConsoleCtrlHandler function. http://docs.activestate.com/activepython/2.4/pywin32/win32api__SetConsoleCtr... http://msdn.microsoft.com/en-us/library/windows/desktop/ms686016%28v=vs.85%2... Thats relevant if your process has a console, e.g. is not a GUI app. (pythonw.exe vs. python.exe). If you run as a GUI app you might get a WM_QUERYENDSESSION or similar message. You get a CTRL_LOGOFF_EVENT when a user on the machine logs off and you run as a subprocess of a service and need to ignore it. The service itself should deal with this already. Michael -- Michael Schlenker Software Architect CONTACT Software GmbH Tel.: +49 (421) 20153-80 Wiener Straße 1-3 Fax: +49 (421) 20153-41 28359 Bremen http://www.contact.de/ E-Mail: msc@contact.de Sitz der Gesellschaft: Bremen Geschäftsführer: Karl Heinz Zachries, Ralf Holtgrefe Eingetragen im Handelsregister des Amtsgerichts Bremen unter HRB 13215
n 24/10/12 12:33, Michael Schlenker wrote:
But Python for windows certain calls "signal()" handlers in response to certain events, in certain configurations. Whether they're actually "signals" is kind of moot, though interesting.
Interesting. I've just looked over the code we've got. It actually does something rather more involved than I'd first thought. Specifcally, the "ntsvc.py" code is pure-python, no Twisted. It actually runs the Twisted daemon as a sub-process, like this: import os class MyService(win32serviceutil.ServiceFramework): """NT Service.""" _svc_name_ = "opimport-wrap" _svc_display_name_ = "Marval opimport wrapper" def SvcDoRun(self): self.childServer = Popen([ "c:/python25/pythonw.exe",SCRIPT,"svc" ]) self.childServer.wait() ...and the Twisted script does this: def main(): args = sys.argv[1:] if args and args[0]=='svc': as_service = True else: as_service = False ... if as_service: log.startLogging(some_log_observer) reactor.run(installSignalHandlers=0) else: log.startLogging(sys.stderr) reactor.run() My memory is hazy on this, but if I recall correctly the main reason it works this way is that I can execute the twisted script directly from the console (for debugging) or via the service wrapper. Are you suggesting that if I change that subprocess to use "python.exe" versus "pythonw.exe" I would not need to disable the signal handlers? Cheers, Phil
Am 24.10.2012 14:14, schrieb Phil Mayers: this looks dubious, are you sure you can still react to ServiceManager actions like 'Stop' or 'Query' this way, isn't this wait() blocking?
Won't probably help, not sure though. If you ever import the 'time' module, as Python registers a control handler there, to allow time.sleep() to wake up when ctrl-c is pressed or the system shuts down, so i assume it has a console anyway. from timemodule.c #ifdef MS_WINDOWS /* Helper to allow interrupts for Windows. If Ctrl+C event delivered while not sleeping it will be ignored. */ main_thread = PyThread_get_thread_ident(); hInterruptEvent = CreateEvent(NULL, TRUE, FALSE, NULL); SetConsoleCtrlHandler( PyCtrlHandler, TRUE); #endif /* MS_WINDOWS */ The usual workaround to keep a service subprocess with a console alive is to simply register your own ConsoleHandler to override the default (which is ExitProcess()) and return TRUE, so the default handler doesn't run. If you spawn a lot of subprocess (around 100 at a time) you will additionally have fun with random crashes once you hit the DesktopHeap Memory limit. Michael -- Michael Schlenker Software Architect CONTACT Software GmbH Tel.: +49 (421) 20153-80 Wiener Straße 1-3 Fax: +49 (421) 20153-41 28359 Bremen http://www.contact.de/ E-Mail: msc@contact.de Sitz der Gesellschaft: Bremen Geschäftsführer: Karl Heinz Zachries, Ralf Holtgrefe Eingetragen im Handelsregister des Amtsgerichts Bremen unter HRB 13215
On 24/10/12 17:14, Michael Schlenker wrote:
this looks dubious, are you sure you can still react to ServiceManager
Yes.
actions like 'Stop' or 'Query' this way, isn't this wait() blocking?
AIUI, the "svc" methods are called from various threads? I didn't create this code; I cribbed it from somewhere, but this was a long time ago (since I avoid Windows for anything other than playing games).
This entire Twisted process exists to spawn processes that can only be run under Windows. Fortunately, the app in particular is so crappy that if I ever get *close* to 100 processes, I'm sure the app will die well before pywin does ;o)
As JP said, buildbot has some code for this at https://github.com/buildbot/buildbot/blob/master/master/contrib/windows/buil...
On Tue, Oct 23, 2012 at 6:09 PM, Tom Prince <tom.prince@ualberta.net> wrote:
As JP said, buildbot has some code for this at
https://github.com/buildbot/buildbot/blob/master/master/contrib/windows/buil...
-- *John Aherne* * * * * * * *www.rocs.co.uk * 020 7223 7567
On 23 October 2012 20:09, Tom Prince <tom.prince@ualberta.net> wrote:
As JP said, buildbot has some code for this at https://github.com/buildbot/buildbot/blob/master/master/contrib/windows/buil...
Just in case it might be of any help. I am using something based on this code: http://code.activestate.com/recipes/551780/ The code with Twisted reactor is here: https://github.com/chevah/commons/blob/master/commons/compat/nt_service.py It does not use .tac files. 'initialize' is used for creating the services, 'start' for start listening and 'stop' for stop listening. Cheers -- Adi Roiban
participants (6)
-
Adi Roiban -
exarkun＠twistedmatrix.com -
John Aherne -
Michael Schlenker -
Phil Mayers -
Tom Prince