Fantastic, thanks for sharing this code. I will definitely have a look. If it turns out to work for us, would you mind releasing this under a BSD license so we can include it in IPython (BSD open source project)? In either case, the ideas will be very helpful to us. Cheers, Brian It is possible to daemonize a process on Windows. I experimented with

adding that support to the twistd script, but got swamped with other work and couldn't finish it. Below is the code that I have so far. You can save it in a module and call the daemonize() function from your script.

The process of daemonization is similar to the one on UNIX -- you have to spawn a child process twice, the first child is responsible for breaking away from any job objects (somewhat similar to becoming a session leader on UNIX), becoming a new process group leader and closing all handles (file descriptors) that might have been inherited.

The second child has to open dummy std* files and a new (hidden) console, otherwise the signals stop working. There is a slight complication with window stations and desktops. Each console creates at least one window and some other user objects, so we have to create a separate desktop, or other processes would be able to manipulate them and send us arbitrary (Windows) messages.

Regards, Ziga

import os import sys import msvcrt

import win32con import win32process import win32security import win32service

from twisted.python import win32

def getPythonArgs(): """ Return the list of command line args that were used to start the current Python interpreter and were not stored in C{sys.argv}.

These are the options that control the Python interpreter itself, like the Python executable, optimization level, warning filters, division behaviour and literal string handling. """ args = [sys.executable] for warnoption in sys.warnoptions: args.append("-W") args.append(warnoption) if type(1 / 2) is not int: args.append("-Qnew") if type("") is not str: args.append("-U") if not __debug__: if getPythonArgs.__doc__ is None: args.append("-OO") else: args.append("-O") return args

def daemonize(): args = [os.path.abspath(__file__)] + sys.argv executable = sys.executable cmdline = win32.quoteArguments(getPythonArgs() + args) inherit = False

flags = (win32process.CREATE_BREAKAWAY_FROM_JOB | # session leader win32process.CREATE_NEW_PROCESS_GROUP | # group leader win32process.DETACHED_PROCESS) # no controlling terminal

info = win32process.STARTUPINFO() win32process.CreateProcess(executable, cmdline, None, None, inherit, flags, None, None, info) # Do what exec* functions do, let the OS do the cleanup. os._exit(0)

def daemonize2(): args = [sys.argv[1], "--nodaemon"] + sys.argv[2:] executable = sys.executable cmdline = win32.quoteArguments(getPythonArgs() + args) inherit = True # create an invisible console flags = (win32process.CREATE_NO_WINDOW attributes = win32security.SECURITY_ATTRIBUTES() attributes.bInheritHandle = True station = win32service.CreateWindowStation(None, 0, win32con.GENERIC_READ | win32con.GENERIC_WRITE, attributes) station.SetProcessWindowStation() sname = win32service.GetUserObjectInformation(station, win32service.UOI_NAME) dname = str(os.getpid()) desktop = win32service.CreateDesktop(dname, 0, win32con.GENERIC_READ | win32con.GENERIC_WRITE, attributes) desktop.SetThreadDesktop() null = os.open("NUL", os.O_RDWR) handle = msvcrt.get_osfhandle(null) info = win32process.STARTUPINFO() info.lpDesktop = "%s\\%s" % (sname, dname) info.dwFlags = win32process.STARTF_USESTDHANDLES info.hStdInput = info.hStdOutput = info.hStdError = handle win32process.CreateProcess(executable, cmdline, None, None, inherit, flags, None, None, info) # Same as above, exit as fast as possible. os._exit(0)

if __name__ == "__main__": daemonize2()

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Sure, you can find a slightly updated version of the code in the attachment, licensed under the two clause BSD license and without the syntax error. Thank you very much for being willing to share this code under the BSD license. I will try this out and see if it does the job for us.

It would be even better if you could try to integrate the functionality into Twisted itself, if the core developers agree that it would be useful. I opened a ticket for this:

http://twistedmatrix.com/trac/ticket/4073

Yes, it would be nice if this was in twisted itself. I will probably include it in IPython for now and see how it works for us - use that as a medium term test of the approach. I will provide feeback and any changes to the twisted ticket as appropriate. Thanks again, Brian

Regards, Ziga

# Copyright (c) 2009 Ziga Seilnacht. All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: # # 1. Redistributions of source code must retain the above copyright notice, # this list of conditions and the following disclaimer. # 2. Redistributions in binary form must reproduce the above copyright notice, # this list of conditions and the following disclaimer in the documentation # and/or other materials provided with the distribution. # # THIS SOFTWARE IS PROVIDED BY THE FREEBSD PROJECT ''AS IS'' AND ANY EXPRESS OR # IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF # MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO # EVENT SHALL THE FREEBSD PROJECT OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, # INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, # BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF # LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE # OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF # ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import os import sys import msvcrt

try: import win32con import win32process import win32security import win32service except ImportError: win32process = None

from twisted.python import win32

def getPythonArgs(): """ Return the list of command line args that were used to start the current Python interpreter and were not stored in C{sys.argv}.

These are the options that control the Python interpreter itself, like the Python executable, optimization level, warning filters, division behaviour and literal string handling. """ args = [sys.executable] for warnoption in sys.warnoptions: args.append("-W") args.append(warnoption) if type(1 / 2) is not int: args.append("-Qnew") if type("") is not str: args.append("-U") if not __debug__: if getPythonArgs.__doc__ is None: args.append("-OO") else: args.append("-O") return args

def daemonize(): args = [os.path.abspath(__file__)] + sys.argv executable = sys.executable cmdline = win32.quoteArguments(getPythonArgs() + args) inherit = False priority = win32process.GetPriorityClass(win32process.GetCurrentProcess()) flags = (win32process.CREATE_BREAKAWAY_FROM_JOB | # session leader win32process.CREATE_NEW_PROCESS_GROUP | # group leader win32process.DETACHED_PROCESS | # no controlling terminal priority) info = win32process.STARTUPINFO() win32process.CreateProcess(executable, cmdline, None, None, inherit, flags, None, None, info) # Do what exec* functions do, let the OS do the cleanup. os._exit(0)

def daemonize2(): args = [sys.argv[1], "--nodaemon"] + sys.argv[2:] executable = sys.executable cmdline = win32.quoteArguments(getPythonArgs() + args) inherit = True priority = win32process.GetPriorityClass(win32process.GetCurrentProcess()) flags = (win32process.CREATE_NO_WINDOW | # create an invisible console win32process.CREATE_NEW_PROCESS_GROUP | # group leader priority) attributes = win32security.SECURITY_ATTRIBUTES() attributes.bInheritHandle = True station = win32service.CreateWindowStation(None, 0, win32con.GENERIC_READ | win32con.GENERIC_WRITE, attributes) station.SetProcessWindowStation() sname = win32service.GetUserObjectInformation(station, win32service.UOI_NAME) dname = str(os.getpid()) desktop = win32service.CreateDesktop(dname, 0, win32con.GENERIC_READ | win32con.GENERIC_WRITE, attributes) desktop.SetThreadDesktop() null = os.open("NUL", os.O_RDWR) handle = msvcrt.get_osfhandle(null) info = win32process.STARTUPINFO() info.lpDesktop = "%s\\%s" % (sname, dname) info.dwFlags = win32process.STARTF_USESTDHANDLES info.hStdInput = info.hStdOutput = info.hStdError = handle win32process.CreateProcess(executable, cmdline, None, None, inherit, flags, None, None, info) # Same as above, exit as fast as posible. os._exit(0)

if __name__ == "__main__": daemonize2()

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

David, While this process is certainly doable, I'll also point out that on

Windows, the more "natural" approach to this is to implement the process as a service. That also buys you some regular Windows approaches to management (net stop/start from command line, services UI from the control panel) and status inquiry (sc query from command line). So while it may be a little platform-specific code you're final result will integrate more naturally into that environment for any system administrators.

It is very possible I might have to go this route. We have some odd requrements, such as needing to run the server using the Windows Server 2008 HPC job scheduler. the job scheduler does some funky things and I am finding that some of the "usual" assumptions you make on Windows don't hold. So at this point, I need all the tricks in my bad that I can get. Thanks for the example!

While I haven't done this with a twistd based Twisted service, I've done it with a lot of Twisted code in general. It's easiest if you have pywin32 installed and can use it's utility service wrappers, for which the pywin32 package has examples.

OK, nice to know about these examples.

I've tended to offload the service definition to its own module, and have it just import and execute the main code (with the twisted reactor.run call) after the service gets started - that way you can also manually run the twisted code without involving any of the service support, or to support execution on non-Windows platforms.

Here's one example of a service object that invoices Twisted-based code. When stopping, you use callFromThread because the thread that handles the request from the Windows service manager is separate from the thread executing the main code (which is a thread created during the startup process).

That's also why you start the reactor without signal initialization, since it'll be in a secondary thread. In SvcDoRun is where you could import platform-generic modules to set up your twisted code, and then this code starts the reactor.

Bummer, then I can't use this approach. My "server" uses reactor.spawnProcess which needs the signal handlers to be installed (SIGCHLD specifically) to work properly... do you know if it can be done without the dual thread trick. Thanks, Brian Oh, and I left some code in that shows retrieving service-specific

parameters from the registry as is typical for Windows services (stored in HKLM/System/CurrentControlSet/Services/<svc_name>/Parameters if I recall corrrectly):

- - - - -

class DataImportService(win32serviceutil.ServiceFramework):

__version__ = '1.0.0'

_svc_name_ = 'fdi' _svc_display_name_ = 'SomeCompany Data Integration'

stopping = False debug = False

def __init__(self, *args, **kwargs): win32serviceutil.ServiceFramework.__init__(self, *args, **kwargs) self.initialize() self._stopped = threading.Event() self.log = None

def initialize(self): # This is separate from __init__ so during debugging the bootstrap # code can override __init__ but still finish initialization.

def _getOption(self, option, default): return win32serviceutil.GetServiceCustomOption(self._svc_name_, option, default)

def SvcStop(self): self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING) print '%s Service stopping' % self._svc_display_name_ reactor.callFromThread(reactor.stop) self._stopped.wait(5) print '%s Service stopped' % self._svc_display_name_ if self.log: self.log.close()

def SvcDoRun(self): """Main entry point for service execution"""

if hasattr(sys, 'frozen'): home_dir = os.path.dirname(sys.executable) else: home_dir = os.path.dirname(__file__)

# First, let's set up some logging if self.debug: dupstdout = True logprefix = None else: dupstdout = False logprefix = os.path.join(home_dir, 'logs', 'dataimport.log')

self.log = LogClass(logprefix, dupstdout)

# And then reroute stdout/stderr to that file object sys.stdout = self.log sys.stderr = self.log

print '%s Service %s starting' % (self._svc_display_name_, self.__version__)

try:

# Process config file config_file = self._getOption('config', os.path.join(home_dir, 'data', 'dataimport.ini'))

# ... other service-related initialization ...

# ... do any Twisted related initialization ...

# Start everything up reactor.callLater(0, self.log.write, '%s Service operational\n' % self._svc_display_name_) reactor.run(installSignalHandlers=False)

# We're shutting down.

# ... do any shutdown processing ...

# Flag that we're exiting so service thread can be more # accurate in terms of declaring shutdown. self._stopped.set()

except:

# For right now just log a traceback and abort log.err() # But try to gracefully close down log to let final traceback # information make it out to the log file. if self.log: self.log.close()

- - - - -

Your main startup code in the service wrapper module can use helper functions from win32serviceutil for startup (see the pywin32 examples) which provides some automatic support for installing/uninstalling the service, or you can implement your own startup if you want to support different options and what not. This all works under py2exe as well if you want to package things up and have a nice self-contained exe as a Windows service.

-- David

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Bummer, then I can't use this approach. My "server" uses reactor.spawnProcess

which needs the signal handlers to be installed (SIGCHLD specifically) to work properly... do you know if it can be done without the dual thread trick.

...On windows? I don't think Windows has SIGCHLD.

Yes, that may be. I will look into this...I may have some hope yet. Cheers, Brian

James

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python