[Twisted-Python] process manager for twisted daemons?
We have a few twisted daemons running on a linux machine under Supervisord as the process manager. Unfortunately, Supervisord is missing a feature that is pretty critical to our needs, so must be replaced with something that supports it -- the feature is the ability to persist state across reboots. We've run into a situation more than once where we "stop" a Twisted service that caused issues, begun to triage it, and then the machine hits a reboot which causes the broken service to start up again and re-corrupts the data. The only way to maintain a "stop" across reboots in Supervisor is to "stop", then edit the config files so it doesn't start up again. If someone remembers to shut things off this way, they often forget to turn them back on again. We're at the point where using supervisor in our stack has caused more problems than conveniences. Is anyone deploying their twisted services with something other than Supervisor? If so, I'd love to know.
why not use systemd to manage the daemon? On Wed, May 03, 2017 at 07:21:24PM -0400, jonathan vanasco wrote:
We have a few twisted daemons running on a linux machine under Supervisord as the process manager. Unfortunately, Supervisord is missing a feature that is pretty critical to our needs, so must be replaced with something that supports it -- the feature is the ability to persist state across reboots.
We've run into a situation more than once where we "stop" a Twisted service that caused issues, begun to triage it, and then the machine hits a reboot which causes the broken service to start up again and re-corrupts the data. The only way to maintain a "stop" across reboots in Supervisor is to "stop", then edit the config files so it doesn't start up again. If someone remembers to shut things off this way, they often forget to turn them back on again. We're at the point where using supervisor in our stack has caused more problems than conveniences.
Is anyone deploying their twisted services with something other than Supervisor? If so, I'd love to know.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Thu, 4 May 2017 at 01:21 jonathan vanasco
Is anyone deploying their twisted services with something other than Supervisor? If so, I'd love to know.
It sounds like systemd has what you want here, and should work fine for your Twisted services, assuming you don't have other constraints that exclude its use. systemctl stop <service> will temporarily stop a service, but it will be restarted on boot if it was configured to do so. However, you can do systemctl disable --now <service> which will remove the configuration to start it on boot as well as stopping it. To reverse this, you would do systemctl enable --now <service> which would reenable it and start it. This is basically doing the same as you were doing with supervisord, but since you can combine the operations into a single command, you don't have to worry about forgetting to do half of the process.
On May 3, 2017, at 7:46 PM, Tristan Seligmann wrote:
It sounds like systemd has what you want here, and should work fine for your Twisted services, assuming you don't have other constraints that exclude its use.
we were avoiding it because of a few reasons, but will need to rethink that now. thanks!
Hello, On 05/04/17 02:21, jonathan vanasco wrote:
Is anyone deploying their twisted services with something other than Supervisor? If so, I'd love to know.
We're using djb's daemontools (with the -encore patchset) to manage our twisted and non-twisted (C++) daemons. Two advantages: 1. Biggest advantage for us: cross-platform. Every Linux / BSD distro I know of has daemontools supported. You won't have to depend on systemd/openrc etc. Slightly related: http://cr.yp.to/compatibility.html 2. svc, daemontools' service manager is pretty straightforward to use. See: https://cr.yp.to/daemontools/svc.html 3. Takes care of forking, logging, log rotation, etc. Can filter logs based on a given regexp. Logs only stdout but if you do "exec 2>&1" before starting your daemon, you'll get stderr too. I also find it dead simple to integrate your app with daemontools. Examples and other advantages: https://cr.yp.to/daemontools/faq/create.html The encore patchset (https://untroubled.org/daemontools-encore/) adds stuff like "email me when the process is restarted". It's not strictly needed as you can implement it in your run script but it's a nice-to-have. I hope this helps. Best, Burak
participants (4)
-
Burak Arslan -
dawuud -
jonathan vanasco -
Tristan Seligmann