[Twisted-Python] untwisting twistd
![](https://secure.gravatar.com/avatar/a0ae40818943eee65b2f25c020c98b17.jpg?s=120&d=mm&r=g)
I am currently working on writing an application using the twisted framework. The problem that I have is that I do not want to use twistd. In fact, I find twistd to be inappropriate as it defaults pidfile and logfile to something specific to twisted, not my application, and it seem completely silly to have to pass in -pidfile and -logfile params when stating it up. Those parameters should be specified as part of the Application object itself, and should allow for CLI override. Further, as I want my daemon that I'm writing to look and act like a daemon (UNIX), I do not want to be invoking some general "twistd" command and passing a ton of options. Ideally, I should be able to write a Pythons script that acts like twistd, but can bake in default options and hide/expose other ServerOptions as I so see fit. For example, if I run, my_daemon It should fire up the daemon my_daemon -help should dump the help my_daemon -pidfile=/var/run/my_daemon.pid my_daemon stop my_daemon restart should all function accordingly. I want to avoid all of this "twistd -y" business, as the daemon, isn't "twistd" it's my code. I should be able to use the ApplicationRunner, UnixApplicationRunner, etc "runners", however, as they are currently written, they are twistd specific. For example, in 8.2, the def checkPid function bakes in, " sys.exit("""\ Another twistd server is running, PID %s\n This could either be a previously started instance of your application or a different application entirely. To start a new one, either run it in some other directory, or use the --pidfile and --logfile parameters to avoid clashes. """ % pid)" Which is iditic as my application may not even expose a -pidfile or a -logfile and it's definatley not called twistd. The platform specific application runners need to be more generalized so users can write their own twistd equivalent to include only what they want. Perhaps I'm missing some key point of twisted that already does this. If so, I'd love to be guided in the right direction. If not, and I've got a valid point here, I'd love to know how I can help to remedy this problem. I think it's great that twistd provides application management classes, but they are unnecessarily intertwined with twistd which should be completely optional. -Jared
![](https://secure.gravatar.com/avatar/dd1243740a09f0676ef225404105cfc0.jpg?s=120&d=mm&r=g)
On Jun 22, 2009, at 4:30 PM, Jared Gisin wrote:
This topic comes up continuously, but like the other occurrences, it makes a gigantic mountain out of a small molehill. The critics are usually not thinking about the real requirements of deploying a configurable demonizing application in a production environment. Look at the init script used to start MySQL sometime, particularly when installed from a package on a finicky distro like debian. Or Apache. Or bind. Or any one of a number of other complex daemons -- you'll find that it's not usually advisable (or possible) to launch any of these daemons without an endless list of command-line options. The fact is, twistd does a *lot* of stuff for you, so you don't have to. This includes dealing with daemonizing in a UNIX-standard way (leaving pidfiles behind and logging to a file), and a myriad other uses that you may have no need for yet. Conversely, the things people write with Twisted (and then use twistd to execute) come in a million forms from the most complex to the very simple, so there's no way to address all the desires of all the functionality that someone would want in a tool like twistd. You said yourself that you should be able to write a python script that functions in a certain way, but what makes you think it should be integrated into twistd? You could write a wrapper that made use of the many other convenient scripting features Twisted provides (such as twisted.python.usage.Options) and which would then in turn launch the twistd process with the arguments you require, much like MySQL's mysqld_safe bootstrap utility. -phil
![](https://secure.gravatar.com/avatar/d6328babd9f9a98ecc905e1ccac2495e.jpg?s=120&d=mm&r=g)
On 09:06 pm, phil@bubblehouse.org wrote:
Sure, but we should be able to do better than that, right? :)
Actually, it's even simpler than that. The script is itself a function, which you can just call as a function without starting a separate process. Axiom had requirements like Mr. Gisin: we wanted to put the .pid file and log files inside the database directory, which is a location that obviously has to be computed. I wouldn't object to a more graceful and structured way to do this, but the approach "axiomatic" uses is pretty simple: provide a twistd plugin that does the meat of the work, then have a small wrapper script that builds a command line that includes the plugin and the appropriate options, such as logfile and pidfile. You can see that code here: http://divmod.org/trac/browser/trunk/Axiom/axiom/scripts/axiomatic.py http://divmod.org/trac/browser/trunk/Axiom/twisted/plugins/axiom_plugins.py
![](https://secure.gravatar.com/avatar/dd1243740a09f0676ef225404105cfc0.jpg?s=120&d=mm&r=g)
On Jun 22, 2009, at 5:38 PM, glyph@divmod.com wrote:
Ha, almost definitely ;-) But in my particular use cases, it doesn't often matter. For example, I usually like to deploy production servers under non-privileged accounts using authbind, which means that I'll usually need to make some kind of init script. Of course, these days I hardly ever write anything that *doesn't* launch at startup, and my command-line options are almost always 'server-specific' instead of 'job-specific'. Consequently most of my CLI options can just be set once for each machine and generally forgotten about, so I'm not really a good candidate to judge the importance of an enhanced twistd API... -phil
![](https://secure.gravatar.com/avatar/605b00ce4b27ebdae813603623c8202c.jpg?s=120&d=mm&r=g)
IMHO there is a misunderstanding here. The point, at least from my POV is to have the same functionality provided by twistd but with a different interface. Right now twistd is a starter script that is used like: twistd -noy -l my.log my_app.tac As far as I can understand the OP is missing (me as well) the possibility to have the same functionality with: python my_app.py -noy -l my.log or my_app -noy -l my.log without writing any kind of wrapping shell/python scripts. In other words, if we consider example from the book: file reverse_app.py: from twisted.application import service import reverse application = service.Application("Reverser") reverserService = reverse.ReverserService(() reverserServise.setServiceParent(application) and: twistd -y reverse_app.py I'd like to be able to have: file reverse_app2.py: from twisted.application import service import reverse application = service.Application("Reverser") reverserService = reverse.ReverserService(() reverserServise.setServiceParent(application) if __name__ == '__main__': from twisted.SOMETHING import run run(...) Regards, Mikhail
![](https://secure.gravatar.com/avatar/dd1243740a09f0676ef225404105cfc0.jpg?s=120&d=mm&r=g)
On Jul 2, 2009, at 12:08 AM, Mikhail wrote:
I think this point is well understood, but thank you for clarifying.
So, how is this not a wrapper script? The solution I provided in my last email is already workable, doesn't require shell scripting or launching a child process, and doesn't interfere with any of the functionality of the twistd feature. On top of that, it's only a few lines longer than the example you included here. Perhaps you could point out some things that you *couldn't* do with the example I sent, or other stumbling blocks in using it? At any rate, I don't think there's any disagreement about the fact that it would be nice for the twistd feature to be more flexible in the ways suggested, but it's a non-trivial undertaking. OTOH, I'm quite sure that a patch would be welcomed (hint, hint) ;-) -phil
![](https://secure.gravatar.com/avatar/605b00ce4b27ebdae813603623c8202c.jpg?s=120&d=mm&r=g)
Phil Christensen <phil <at> bubblehouse.org> writes:
So, how is this not a wrapper script? The solution I provided in my
Yes, in a sense everything is a wrapper for something. :)
In my example I should have written run(application) instead of run(...) Then the difference would be more apparent. Namely, in my 'wrapper' it is explicit what application will be run and where all the services came from, your 'wrapper' is just a customized version of twistd and what will be run depends on the command line and what will be found in the file system. In some cases I'd like to _explicitly_ code into main script what functionality my application provides and I do not want twistd search file system for plugins at all.
Perhaps you could point out some things that you *couldn't* do with the example I sent, or other stumbling blocks in using it?
I couldn't figure out how to use it without plugins. Obviously I'm missing something simple here. If you could extend your example to show how to use twistd functionality without writing plugins then it would be just great.
Yes, that is probably why it is called twisted :) Mikhail
![](https://secure.gravatar.com/avatar/0da8d6ba99ebd77ed8a31c889a28f542.jpg?s=120&d=mm&r=g)
On 2 Jul 2009, at 23:50, Mikhail wrote:
I'm not using plugins, but I had the same requirement as you to explicitly create the app and then run it - because I need the freeze scripts to make a single application. My solution looked something like: # myapp.py from twisted.scripts._twistd_unix import UnixApplicationRunner # Of course, that's just because the app is unix only class MyRunner(UnixApplicationRunner): def createOrGetApplication(self): # The stuff that's usually in a tac file application = # ... return application def main(): setup_logging() config = { # stuff pinched by dumping 'config' during a twistd run } # Also do the ServerOptions thing here if necessary MyRunner(config).run() if __name__ == '__main__' main()
![](https://secure.gravatar.com/avatar/a0ae40818943eee65b2f25c020c98b17.jpg?s=120&d=mm&r=g)
Mikhail is correct in what needs to be done here. Twistd should provide helpers to writing an application without being the application. Not every wants to make it look like they're using twisted, even if they are. Having a twistd process appear in my process list is unacceptable, and my applications takes a well know and singular config. Twistd should be nothing more than a general helper. Ultimately what Mikhail shows is very similar to what I have done by import directly the ApplicationRunner I want to use (my app is also Unix-only), and then subclassing ServerOptions. I also modified createorGetApplication to return an application object instead of loading it from some damned .tac file that is passed in on the CLI, which was exactly the thing that was trying to be avoided. My application is defined already in code, so I should be able to create my application object, and pass it right into a runner with proper application options and have it fire up avoiding the generalities of twistd that don't apply to me. Again, twistd is great at supporting one use case. It needs to support others, and it needs to be refactored just a bit more to do that. It wouldn't be as difficult to do this as you probably imagine, and proper documentation on more ways it could be used would save developers a ton of time. Now, if I only had the time to get this done.... I'll see if I can refactor and provide some sample code to at serves the use case we are discussing here. As I was told early on in this inquiry, this problem isn't an uncommon complaint, so solving it seems useful. -Jared From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of Paul Thomas Sent: Friday, July 03, 2009 2:31 AM To: Twisted general discussion Subject: Re: [Twisted-Python] untwisting twistd On 2 Jul 2009, at 23:50, Mikhail wrote: In my example I should have written run(application) instead of run(...) Then the difference would be more apparent. Namely, in my 'wrapper' it is explicit what application will be run and where all the services came from, your 'wrapper' is just a customized version of twistd and what will be run depends on the command line and what will be found in the file system. In some cases I'd like to _explicitly_ code into main script what functionality my application provides and I do not want twistd search file system for plugins at all. I'm not using plugins, but I had the same requirement as you to explicitly create the app and then run it - because I need the freeze scripts to make a single application. My solution looked something like: # myapp.py from twisted.scripts._twistd_unix import UnixApplicationRunner # Of course, that's just because the app is unix only class MyRunner(UnixApplicationRunner): def createOrGetApplication(self): # The stuff that's usually in a tac file application = # ... return application def main(): setup_logging() config = { # stuff pinched by dumping 'config' during a twistd run } # Also do the ServerOptions thing here if necessary MyRunner(config).run() if __name__ == '__main__' main()
![](https://secure.gravatar.com/avatar/d7d18c48316f83f1bf99618559ed2444.jpg?s=120&d=mm&r=g)
Hi Jared, First off, I'd like to say that I'm basically in agreement with exactly what you are saying, and that is: "I should be able to write a Python script that acts like twistd - not *be* twistd". Phil does have some really good points, such as: """ The critics are usually not thinking about the real requirements of deploying a configurable demonizing application in a production environment. """ And lastly, I'm glad to see that response from Glyph, as the methodology he describes is basically how I understand one would create a "personalized twistd". In fact, I just got an email from a co-worker that wanted to know how to either side-step "twistd" or write a custom version of it. Here is my response to him (I'd love some critic of my response, so I can provide a better answer): ============= Email to co-worker: =============== Yes, there is a better way of doing things. This better way of doing things is to write a "twisted plugin" and then, after that is working, using can wrap calls to "twistd" with python's "os.system" calls, or shell script calls, or whatever else you want. I must admit, this whole procedure is pretty confusing (it was for me at least) at first, but let me try to explain why you might want to write a "twisted plugin" and _then_ (possibly) wrap it with a shell script or a simple python script. The main reasons are the following: 1) You _do_ want to use "twistd", because it does lots of useful things like daemonizing, handling logfiles and pidfiles, etc. 2) You also want to pass config code to the app you are starting up (via "twistd") - optimally from the command line and a settings file. Now, once the above is in place, you then can write some wrapper code on top of the "twistd" calls for convenience for users. Here is the "plugin" documentation: http://twistedmatrix.com/projects/core/documentation/howto/plugin.html But I've found that it is easier to go from an example, so try typing this: twistd -n web --path="." Now open up firefox to http://localhost:8080 This starts up a twisted web server, serving files from the local directory. This is a twisted web plugin in action, and it is implemented with these 2 files: http://twistedmatrix.com/trac/browser/trunk/twisted/plugins/twisted_web.py http://twistedmatrix.com/trac/browser/trunk/twisted/web/tap.py ============ end email to co-worker ============== So basically I've found that the best thing to do is to extend "twistd" via writing a plugin, and then you can write all kinds of simple wrappers on top of it to provide a more user-friendly and a your-app-specific command line tool. Thoughts from Twisted gurus? thanks, Alex On Mon, Jun 22, 2009 at 1:30 PM, Jared Gisin<jared.gisin@isilon.com> wrote:
-- Alex Clemesha clemesha.org
![](https://secure.gravatar.com/avatar/a0ae40818943eee65b2f25c020c98b17.jpg?s=120&d=mm&r=g)
Thanks for the thoughts. I suppose I should have been a bit clearer on my specific use case that is not handled. Certainly, I do want to take advantage of twistd for the reasons listed below (daemonizing, handling log files, pidfile, etc); however, the problem that I have is that I have a very specific application that is going to be built into a product. There's no way that it could ever be installed or deployed elsewhere. Now, having said that I'd love to take advantage of the ApplicationRunner objects to fire up my application. The path to the application is baked in, the location for the pidfile and logfile should have sane defaults, and other various application parameters shouldn't change. In order to fire up the app with sane default, I should be able to issue one command with no arguments as the defaults for all of those should be baked into the system. Sure, twistd provides default application values, but there's no easy way to override those values. There's no ServerOptions constructor function where I can configure the default values, or have some options removed from the list that appears if the user issues --help. So, what I'm essentially arguing for is that although twistd is meant to solve a generic problem of generalizing server startups on various platforms for any number of applications, it's severely lacking in how I can take advantage of the ServerOptions to create one for my application. As it is now, I'm going to have to copy and paste the code and modify it as I want to create my own ServerOptions object that does have proper defaults. Right, now twisted.scripts._twistd_unis.ServerOptions inherits from app.ServerOptions, which is completely in appropriate as the UNIX ServerOptions have absolutely nothing to do with twistd itself. For example, I don't there to ever be a --file or --xml --source --report-profile --nothotshot --no_save or --savestats options. Really, this is a matter of presentation and locking down the application server options. I should be able to do something like this ServerOptions( hide_items=('xml', 'file', 'source', 'report-profile'), = defaults={'logfile':'/var/log/foo.log', 'pidfile':'/var/run/foo.pid'} ) To properly construct the subset of default twistd ServerOptions I need to have. I guess in the mean time, I'll just write my own custom ServerOptions, which oddly, will mirror a bunch of code that already exists. Thoughts? -Jared -----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of Alex Clemesha Sent: Monday, June 22, 2009 2:36 PM To: Twisted general discussion Subject: Re: [Twisted-Python] untwisting twistd Hi Jared, First off, I'd like to say that I'm basically in agreement with exactly what you are saying, and that is: "I should be able to write a Python script that acts like twistd - not *be* twistd". Phil does have some really good points, such as: """ The critics are usually not thinking about the real requirements of deploying a configurable demonizing application in a production environment. """ And lastly, I'm glad to see that response from Glyph, as the methodology he describes is basically how I understand one would create a "personalized twistd". In fact, I just got an email from a co-worker that wanted to know how to either side-step "twistd" or write a custom version of it. Here is my response to him (I'd love some critic of my response, so I can provide a better answer): ============= Email to co-worker: =============== Yes, there is a better way of doing things. This better way of doing things is to write a "twisted plugin" and then, after that is working, using can wrap calls to "twistd" with python's "os.system" calls, or shell script calls, or whatever else you want. I must admit, this whole procedure is pretty confusing (it was for me at least) at first, but let me try to explain why you might want to write a "twisted plugin" and _then_ (possibly) wrap it with a shell script or a simple python script. The main reasons are the following: 1) You _do_ want to use "twistd", because it does lots of useful things like daemonizing, handling logfiles and pidfiles, etc. 2) You also want to pass config code to the app you are starting up (via "twistd") - optimally from the command line and a settings file. Now, once the above is in place, you then can write some wrapper code on top of the "twistd" calls for convenience for users. Here is the "plugin" documentation: http://twistedmatrix.com/projects/core/documentation/howto/plugin.html But I've found that it is easier to go from an example, so try typing this: twistd -n web --path="." Now open up firefox to http://localhost:8080 This starts up a twisted web server, serving files from the local directory. This is a twisted web plugin in action, and it is implemented with these 2 files: http://twistedmatrix.com/trac/browser/trunk/twisted/plugins/twisted_web.py http://twistedmatrix.com/trac/browser/trunk/twisted/web/tap.py ============ end email to co-worker ============== So basically I've found that the best thing to do is to extend "twistd" via writing a plugin, and then you can write all kinds of simple wrappers on top of it to provide a more user-friendly and a your-app-specific command line tool. Thoughts from Twisted gurus? thanks, Alex On Mon, Jun 22, 2009 at 1:30 PM, Jared Gisin<jared.gisin@isilon.com> wrote:
-- Alex Clemesha clemesha.org _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
![](https://secure.gravatar.com/avatar/dd1243740a09f0676ef225404105cfc0.jpg?s=120&d=mm&r=g)
On Jun 23, 2009, at 11:27 AM, Jared Gisin wrote:
Based on your use case, you really shouldn't need to duplicate much functionality at all. Here's one way to make a custom runner that uses all the guts of the twistd stuff, but overrides various options. This example sets the pidfile to 'web.pid' and automatically runs the 'web' sub-command: from twisted.application import app from twisted.scripts.twistd import runApp from twisted.python.runtime import platformType if platformType == "win32": from twisted.scripts._twistw import ServerOptions else: from twisted.scripts._twistd_unix import ServerOptions class CustomServerOptions(ServerOptions): defaultSubCommand = 'web' optParameters = [ ['pidfile','','web.pid', "Name of the pidfile"], ] if __name__ == '__main__': app.run(runApp, CustomServerOptions) The biggest complication here is selecting the correct ServerOptions file to use as a base. As it is, this example will pick the correct ServerOptions base, but sets the pidfile default no matter what, which isn't applicable to the windows options, AFAIK. I'm not suggesting it couldn't be better, but I don't know what specific changes would need to be made to satisfy everyone (or mostly everyone). -phil
![](https://secure.gravatar.com/avatar/a142e8af65192f6acd029c0378f9d174.jpg?s=120&d=mm&r=g)
On Jun 22, 2009, at 10:30 PM, Jared Gisin wrote:
[...]
I must say I agree with everything you're saying. Another thing I'm missing is that there's no way to specify _when_ to daemonize the application. Plus I would love if startService could return a deferred, just like stopService does.
![](https://secure.gravatar.com/avatar/dd1243740a09f0676ef225404105cfc0.jpg?s=120&d=mm&r=g)
On Jun 22, 2009, at 4:30 PM, Jared Gisin wrote:
This topic comes up continuously, but like the other occurrences, it makes a gigantic mountain out of a small molehill. The critics are usually not thinking about the real requirements of deploying a configurable demonizing application in a production environment. Look at the init script used to start MySQL sometime, particularly when installed from a package on a finicky distro like debian. Or Apache. Or bind. Or any one of a number of other complex daemons -- you'll find that it's not usually advisable (or possible) to launch any of these daemons without an endless list of command-line options. The fact is, twistd does a *lot* of stuff for you, so you don't have to. This includes dealing with daemonizing in a UNIX-standard way (leaving pidfiles behind and logging to a file), and a myriad other uses that you may have no need for yet. Conversely, the things people write with Twisted (and then use twistd to execute) come in a million forms from the most complex to the very simple, so there's no way to address all the desires of all the functionality that someone would want in a tool like twistd. You said yourself that you should be able to write a python script that functions in a certain way, but what makes you think it should be integrated into twistd? You could write a wrapper that made use of the many other convenient scripting features Twisted provides (such as twisted.python.usage.Options) and which would then in turn launch the twistd process with the arguments you require, much like MySQL's mysqld_safe bootstrap utility. -phil
![](https://secure.gravatar.com/avatar/d6328babd9f9a98ecc905e1ccac2495e.jpg?s=120&d=mm&r=g)
On 09:06 pm, phil@bubblehouse.org wrote:
Sure, but we should be able to do better than that, right? :)
Actually, it's even simpler than that. The script is itself a function, which you can just call as a function without starting a separate process. Axiom had requirements like Mr. Gisin: we wanted to put the .pid file and log files inside the database directory, which is a location that obviously has to be computed. I wouldn't object to a more graceful and structured way to do this, but the approach "axiomatic" uses is pretty simple: provide a twistd plugin that does the meat of the work, then have a small wrapper script that builds a command line that includes the plugin and the appropriate options, such as logfile and pidfile. You can see that code here: http://divmod.org/trac/browser/trunk/Axiom/axiom/scripts/axiomatic.py http://divmod.org/trac/browser/trunk/Axiom/twisted/plugins/axiom_plugins.py
![](https://secure.gravatar.com/avatar/dd1243740a09f0676ef225404105cfc0.jpg?s=120&d=mm&r=g)
On Jun 22, 2009, at 5:38 PM, glyph@divmod.com wrote:
Ha, almost definitely ;-) But in my particular use cases, it doesn't often matter. For example, I usually like to deploy production servers under non-privileged accounts using authbind, which means that I'll usually need to make some kind of init script. Of course, these days I hardly ever write anything that *doesn't* launch at startup, and my command-line options are almost always 'server-specific' instead of 'job-specific'. Consequently most of my CLI options can just be set once for each machine and generally forgotten about, so I'm not really a good candidate to judge the importance of an enhanced twistd API... -phil
![](https://secure.gravatar.com/avatar/605b00ce4b27ebdae813603623c8202c.jpg?s=120&d=mm&r=g)
IMHO there is a misunderstanding here. The point, at least from my POV is to have the same functionality provided by twistd but with a different interface. Right now twistd is a starter script that is used like: twistd -noy -l my.log my_app.tac As far as I can understand the OP is missing (me as well) the possibility to have the same functionality with: python my_app.py -noy -l my.log or my_app -noy -l my.log without writing any kind of wrapping shell/python scripts. In other words, if we consider example from the book: file reverse_app.py: from twisted.application import service import reverse application = service.Application("Reverser") reverserService = reverse.ReverserService(() reverserServise.setServiceParent(application) and: twistd -y reverse_app.py I'd like to be able to have: file reverse_app2.py: from twisted.application import service import reverse application = service.Application("Reverser") reverserService = reverse.ReverserService(() reverserServise.setServiceParent(application) if __name__ == '__main__': from twisted.SOMETHING import run run(...) Regards, Mikhail
![](https://secure.gravatar.com/avatar/dd1243740a09f0676ef225404105cfc0.jpg?s=120&d=mm&r=g)
On Jul 2, 2009, at 12:08 AM, Mikhail wrote:
I think this point is well understood, but thank you for clarifying.
So, how is this not a wrapper script? The solution I provided in my last email is already workable, doesn't require shell scripting or launching a child process, and doesn't interfere with any of the functionality of the twistd feature. On top of that, it's only a few lines longer than the example you included here. Perhaps you could point out some things that you *couldn't* do with the example I sent, or other stumbling blocks in using it? At any rate, I don't think there's any disagreement about the fact that it would be nice for the twistd feature to be more flexible in the ways suggested, but it's a non-trivial undertaking. OTOH, I'm quite sure that a patch would be welcomed (hint, hint) ;-) -phil
![](https://secure.gravatar.com/avatar/605b00ce4b27ebdae813603623c8202c.jpg?s=120&d=mm&r=g)
Phil Christensen <phil <at> bubblehouse.org> writes:
So, how is this not a wrapper script? The solution I provided in my
Yes, in a sense everything is a wrapper for something. :)
In my example I should have written run(application) instead of run(...) Then the difference would be more apparent. Namely, in my 'wrapper' it is explicit what application will be run and where all the services came from, your 'wrapper' is just a customized version of twistd and what will be run depends on the command line and what will be found in the file system. In some cases I'd like to _explicitly_ code into main script what functionality my application provides and I do not want twistd search file system for plugins at all.
Perhaps you could point out some things that you *couldn't* do with the example I sent, or other stumbling blocks in using it?
I couldn't figure out how to use it without plugins. Obviously I'm missing something simple here. If you could extend your example to show how to use twistd functionality without writing plugins then it would be just great.
Yes, that is probably why it is called twisted :) Mikhail
![](https://secure.gravatar.com/avatar/0da8d6ba99ebd77ed8a31c889a28f542.jpg?s=120&d=mm&r=g)
On 2 Jul 2009, at 23:50, Mikhail wrote:
I'm not using plugins, but I had the same requirement as you to explicitly create the app and then run it - because I need the freeze scripts to make a single application. My solution looked something like: # myapp.py from twisted.scripts._twistd_unix import UnixApplicationRunner # Of course, that's just because the app is unix only class MyRunner(UnixApplicationRunner): def createOrGetApplication(self): # The stuff that's usually in a tac file application = # ... return application def main(): setup_logging() config = { # stuff pinched by dumping 'config' during a twistd run } # Also do the ServerOptions thing here if necessary MyRunner(config).run() if __name__ == '__main__' main()
![](https://secure.gravatar.com/avatar/a0ae40818943eee65b2f25c020c98b17.jpg?s=120&d=mm&r=g)
Mikhail is correct in what needs to be done here. Twistd should provide helpers to writing an application without being the application. Not every wants to make it look like they're using twisted, even if they are. Having a twistd process appear in my process list is unacceptable, and my applications takes a well know and singular config. Twistd should be nothing more than a general helper. Ultimately what Mikhail shows is very similar to what I have done by import directly the ApplicationRunner I want to use (my app is also Unix-only), and then subclassing ServerOptions. I also modified createorGetApplication to return an application object instead of loading it from some damned .tac file that is passed in on the CLI, which was exactly the thing that was trying to be avoided. My application is defined already in code, so I should be able to create my application object, and pass it right into a runner with proper application options and have it fire up avoiding the generalities of twistd that don't apply to me. Again, twistd is great at supporting one use case. It needs to support others, and it needs to be refactored just a bit more to do that. It wouldn't be as difficult to do this as you probably imagine, and proper documentation on more ways it could be used would save developers a ton of time. Now, if I only had the time to get this done.... I'll see if I can refactor and provide some sample code to at serves the use case we are discussing here. As I was told early on in this inquiry, this problem isn't an uncommon complaint, so solving it seems useful. -Jared From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of Paul Thomas Sent: Friday, July 03, 2009 2:31 AM To: Twisted general discussion Subject: Re: [Twisted-Python] untwisting twistd On 2 Jul 2009, at 23:50, Mikhail wrote: In my example I should have written run(application) instead of run(...) Then the difference would be more apparent. Namely, in my 'wrapper' it is explicit what application will be run and where all the services came from, your 'wrapper' is just a customized version of twistd and what will be run depends on the command line and what will be found in the file system. In some cases I'd like to _explicitly_ code into main script what functionality my application provides and I do not want twistd search file system for plugins at all. I'm not using plugins, but I had the same requirement as you to explicitly create the app and then run it - because I need the freeze scripts to make a single application. My solution looked something like: # myapp.py from twisted.scripts._twistd_unix import UnixApplicationRunner # Of course, that's just because the app is unix only class MyRunner(UnixApplicationRunner): def createOrGetApplication(self): # The stuff that's usually in a tac file application = # ... return application def main(): setup_logging() config = { # stuff pinched by dumping 'config' during a twistd run } # Also do the ServerOptions thing here if necessary MyRunner(config).run() if __name__ == '__main__' main()
![](https://secure.gravatar.com/avatar/d7d18c48316f83f1bf99618559ed2444.jpg?s=120&d=mm&r=g)
Hi Jared, First off, I'd like to say that I'm basically in agreement with exactly what you are saying, and that is: "I should be able to write a Python script that acts like twistd - not *be* twistd". Phil does have some really good points, such as: """ The critics are usually not thinking about the real requirements of deploying a configurable demonizing application in a production environment. """ And lastly, I'm glad to see that response from Glyph, as the methodology he describes is basically how I understand one would create a "personalized twistd". In fact, I just got an email from a co-worker that wanted to know how to either side-step "twistd" or write a custom version of it. Here is my response to him (I'd love some critic of my response, so I can provide a better answer): ============= Email to co-worker: =============== Yes, there is a better way of doing things. This better way of doing things is to write a "twisted plugin" and then, after that is working, using can wrap calls to "twistd" with python's "os.system" calls, or shell script calls, or whatever else you want. I must admit, this whole procedure is pretty confusing (it was for me at least) at first, but let me try to explain why you might want to write a "twisted plugin" and _then_ (possibly) wrap it with a shell script or a simple python script. The main reasons are the following: 1) You _do_ want to use "twistd", because it does lots of useful things like daemonizing, handling logfiles and pidfiles, etc. 2) You also want to pass config code to the app you are starting up (via "twistd") - optimally from the command line and a settings file. Now, once the above is in place, you then can write some wrapper code on top of the "twistd" calls for convenience for users. Here is the "plugin" documentation: http://twistedmatrix.com/projects/core/documentation/howto/plugin.html But I've found that it is easier to go from an example, so try typing this: twistd -n web --path="." Now open up firefox to http://localhost:8080 This starts up a twisted web server, serving files from the local directory. This is a twisted web plugin in action, and it is implemented with these 2 files: http://twistedmatrix.com/trac/browser/trunk/twisted/plugins/twisted_web.py http://twistedmatrix.com/trac/browser/trunk/twisted/web/tap.py ============ end email to co-worker ============== So basically I've found that the best thing to do is to extend "twistd" via writing a plugin, and then you can write all kinds of simple wrappers on top of it to provide a more user-friendly and a your-app-specific command line tool. Thoughts from Twisted gurus? thanks, Alex On Mon, Jun 22, 2009 at 1:30 PM, Jared Gisin<jared.gisin@isilon.com> wrote:
-- Alex Clemesha clemesha.org
![](https://secure.gravatar.com/avatar/a0ae40818943eee65b2f25c020c98b17.jpg?s=120&d=mm&r=g)
Thanks for the thoughts. I suppose I should have been a bit clearer on my specific use case that is not handled. Certainly, I do want to take advantage of twistd for the reasons listed below (daemonizing, handling log files, pidfile, etc); however, the problem that I have is that I have a very specific application that is going to be built into a product. There's no way that it could ever be installed or deployed elsewhere. Now, having said that I'd love to take advantage of the ApplicationRunner objects to fire up my application. The path to the application is baked in, the location for the pidfile and logfile should have sane defaults, and other various application parameters shouldn't change. In order to fire up the app with sane default, I should be able to issue one command with no arguments as the defaults for all of those should be baked into the system. Sure, twistd provides default application values, but there's no easy way to override those values. There's no ServerOptions constructor function where I can configure the default values, or have some options removed from the list that appears if the user issues --help. So, what I'm essentially arguing for is that although twistd is meant to solve a generic problem of generalizing server startups on various platforms for any number of applications, it's severely lacking in how I can take advantage of the ServerOptions to create one for my application. As it is now, I'm going to have to copy and paste the code and modify it as I want to create my own ServerOptions object that does have proper defaults. Right, now twisted.scripts._twistd_unis.ServerOptions inherits from app.ServerOptions, which is completely in appropriate as the UNIX ServerOptions have absolutely nothing to do with twistd itself. For example, I don't there to ever be a --file or --xml --source --report-profile --nothotshot --no_save or --savestats options. Really, this is a matter of presentation and locking down the application server options. I should be able to do something like this ServerOptions( hide_items=('xml', 'file', 'source', 'report-profile'), = defaults={'logfile':'/var/log/foo.log', 'pidfile':'/var/run/foo.pid'} ) To properly construct the subset of default twistd ServerOptions I need to have. I guess in the mean time, I'll just write my own custom ServerOptions, which oddly, will mirror a bunch of code that already exists. Thoughts? -Jared -----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of Alex Clemesha Sent: Monday, June 22, 2009 2:36 PM To: Twisted general discussion Subject: Re: [Twisted-Python] untwisting twistd Hi Jared, First off, I'd like to say that I'm basically in agreement with exactly what you are saying, and that is: "I should be able to write a Python script that acts like twistd - not *be* twistd". Phil does have some really good points, such as: """ The critics are usually not thinking about the real requirements of deploying a configurable demonizing application in a production environment. """ And lastly, I'm glad to see that response from Glyph, as the methodology he describes is basically how I understand one would create a "personalized twistd". In fact, I just got an email from a co-worker that wanted to know how to either side-step "twistd" or write a custom version of it. Here is my response to him (I'd love some critic of my response, so I can provide a better answer): ============= Email to co-worker: =============== Yes, there is a better way of doing things. This better way of doing things is to write a "twisted plugin" and then, after that is working, using can wrap calls to "twistd" with python's "os.system" calls, or shell script calls, or whatever else you want. I must admit, this whole procedure is pretty confusing (it was for me at least) at first, but let me try to explain why you might want to write a "twisted plugin" and _then_ (possibly) wrap it with a shell script or a simple python script. The main reasons are the following: 1) You _do_ want to use "twistd", because it does lots of useful things like daemonizing, handling logfiles and pidfiles, etc. 2) You also want to pass config code to the app you are starting up (via "twistd") - optimally from the command line and a settings file. Now, once the above is in place, you then can write some wrapper code on top of the "twistd" calls for convenience for users. Here is the "plugin" documentation: http://twistedmatrix.com/projects/core/documentation/howto/plugin.html But I've found that it is easier to go from an example, so try typing this: twistd -n web --path="." Now open up firefox to http://localhost:8080 This starts up a twisted web server, serving files from the local directory. This is a twisted web plugin in action, and it is implemented with these 2 files: http://twistedmatrix.com/trac/browser/trunk/twisted/plugins/twisted_web.py http://twistedmatrix.com/trac/browser/trunk/twisted/web/tap.py ============ end email to co-worker ============== So basically I've found that the best thing to do is to extend "twistd" via writing a plugin, and then you can write all kinds of simple wrappers on top of it to provide a more user-friendly and a your-app-specific command line tool. Thoughts from Twisted gurus? thanks, Alex On Mon, Jun 22, 2009 at 1:30 PM, Jared Gisin<jared.gisin@isilon.com> wrote:
-- Alex Clemesha clemesha.org _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
![](https://secure.gravatar.com/avatar/dd1243740a09f0676ef225404105cfc0.jpg?s=120&d=mm&r=g)
On Jun 23, 2009, at 11:27 AM, Jared Gisin wrote:
Based on your use case, you really shouldn't need to duplicate much functionality at all. Here's one way to make a custom runner that uses all the guts of the twistd stuff, but overrides various options. This example sets the pidfile to 'web.pid' and automatically runs the 'web' sub-command: from twisted.application import app from twisted.scripts.twistd import runApp from twisted.python.runtime import platformType if platformType == "win32": from twisted.scripts._twistw import ServerOptions else: from twisted.scripts._twistd_unix import ServerOptions class CustomServerOptions(ServerOptions): defaultSubCommand = 'web' optParameters = [ ['pidfile','','web.pid', "Name of the pidfile"], ] if __name__ == '__main__': app.run(runApp, CustomServerOptions) The biggest complication here is selecting the correct ServerOptions file to use as a base. As it is, this example will pick the correct ServerOptions base, but sets the pidfile default no matter what, which isn't applicable to the windows options, AFAIK. I'm not suggesting it couldn't be better, but I don't know what specific changes would need to be made to satisfy everyone (or mostly everyone). -phil
![](https://secure.gravatar.com/avatar/a142e8af65192f6acd029c0378f9d174.jpg?s=120&d=mm&r=g)
On Jun 22, 2009, at 10:30 PM, Jared Gisin wrote:
[...]
I must say I agree with everything you're saying. Another thing I'm missing is that there's no way to specify _when_ to daemonize the application. Plus I would love if startService could return a deferred, just like stopService does.
participants (7)
-
Alex Clemesha
-
glyph@divmod.com
-
Jared Gisin
-
Johan Rydberg
-
Mikhail
-
Paul Thomas
-
Phil Christensen