[Twisted-Python] How to restart Twisted service in-process
On Feb 2, 2017, at 1:17 AM, Роман Мещеряков <romanmescheryakov@yandex.ru> wrote:
Hello everyone! I wrote Twisted-based TCP server which is capable of running in several relatively different modes. When mode switch is needed, I would like the server to restart itself by some means, for it to read new mode from configuration file and create corresponding implementation. Also I wanted the restart to occur without re-running twistd process, because it seems to me more appropriate for Linux daemon.
Hi Roman, Sorry it took a while to get to this one :). If your solution is working for you, then it's fine. You've correctly understood the API associated with IServiceCollection and your implementation (at least at first glance, I don't have time to test it exhaustively :)) is correct. However, there's a more philosophical question as well: if you really want to re-start from scratch, what exactly are you shutting down and starting up again? Wouldn't you want to be able to load new code? Simply stopping and starting objects won't re-initialize the process from the beginning, only from some indeterminate middle state where parts of the program are already set up; this doesn't have the usual desired effect of a "restart" where potentially unknown or buggy state is cleared away. You may want to consider simply calling execv https://docs.python.org/2.7/library/os.html#os.execv <https://docs.python.org/2.7/library/os.html#os.execv> with [sys.executable] + sys.argv rather than trying to do this in terms of Twisted's APIs. -glyph
On Sat, Feb 18, 2017 at 2:25 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
On Feb 2, 2017, at 1:17 AM, Роман Мещеряков <romanmescheryakov@yandex.ru> wrote:
*Hello everyone! I wrote Twisted-based TCP server which is capable of running in several relatively different modes. When mode switch is needed, I would like the server to restart itself by some means, for it to read new mode from configuration file and create corresponding implementation. Also I wanted the restart to occur without re-running twistd process, because it seems to me more appropriate for Linux daemon.*
Hi Roman,
Sorry it took a while to get to this one :).
If your solution is working for you, then it's fine. You've correctly understood the API associated with IServiceCollection and your implementation (at least at first glance, I don't have time to test it exhaustively :)) is correct.
However, there's a more philosophical question as well: if you really want to *re-start* from scratch, what exactly are you shutting down and starting up again? Wouldn't you want to be able to load new code? Simply stopping and starting objects won't re-initialize the process from the beginning, only from some indeterminate middle state where parts of the program are already set up; this doesn't have the usual desired effect of a "restart" where potentially unknown or buggy state is cleared away.
You may want to consider simply calling execv https://docs.python.org/ 2.7/library/os.html#os.execv with [sys.executable] + sys.argv rather than trying to do this in terms of Twisted's APIs.
Or better yet. Don't be afraid to just exit. Your process is being managed by a supervisor with a rich feature set - like easily controlled re-restart behavior, right (If not, what do you do when your daemon simply *crashes*)? Jean-Paul
On Sat, Feb 18, 2017 at 6:32 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
On Feb 18, 2017, at 3:19 PM, Jean-Paul Calderone < exarkun@twistedmatrix.com> wrote:
(If not, what do you do when your daemon simply *crashes*)?
Twisted programs can crash!??!
The Linux OOM killer is pretty crazy, man. Jean-Paul
-g
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Feb 20, 2017, at 00:58, Роман Мещеряков <romanmescheryakov@yandex.ru> wrote:
Hi everyone,
The answers are below
-- Kind regards, Roman Mescheryakov
19.02.2017, 02:21, "Jean-Paul Calderone" <exarkun@twistedmatrix.com <mailto:exarkun@twistedmatrix.com>>:
On Sat, Feb 18, 2017 at 2:25 PM, Glyph Lefkowitz <glyph@twistedmatrix.com <mailto:glyph@twistedmatrix.com>> wrote:
On Feb 2, 2017, at 1:17 AM, Роман Мещеряков <romanmescheryakov@yandex.ru <mailto:romanmescheryakov@yandex.ru>> wrote:
Hello everyone! I wrote Twisted-based TCP server which is capable of running in several relatively different modes. When mode switch is needed, I would like the server to restart itself by some means, for it to read new mode from configuration file and create corresponding implementation. Also I wanted the restart to occur without re-running twistd process, because it seems to me more appropriate for Linux daemon.
Hi Roman,
Sorry it took a while to get to this one :).
If your solution is working for you, then it's fine. You've correctly understood the API associated with IServiceCollection and your implementation (at least at first glance, I don't have time to test it exhaustively :)) is correct.
However, there's a more philosophical question as well: if you really want to re-start from scratch, what exactly are you shutting down and starting up again? Wouldn't you want to be able to load new code? Simply stopping and starting objects won't re-initialize the process from the beginning, only from some indeterminate middle state where parts of the program are already set up; this doesn't have the usual desired effect of a "restart" where potentially unknown or buggy state is cleared away.
You may want to consider simply calling execv https://docs.python.org/2.7/library/os.html#os.execv <https://docs.python.org/2.7/library/os.html#os.execv> with [sys.executable] + sys.argv rather than trying to do this in terms of Twisted's APIs.
You are right, Glyph, using os.execv would be more simple, reliable and straightforward way of achieving my goal. The only thing that bothers me due to lack of Linux experience: is it OK to os.execv inside daemon process? Because I run my process using twistd, not python directly.
If you're actually daemonizing ('twistd' without '-n') then there might be a small issue. However, a modern best practice is to never daemonize; this is why the new `twist` runner (which you should probably switch to) doesn't daemonize and expects your supervisor to handle backgrounding.
Or better yet. Don't be afraid to just exit. Your process is being managed by a supervisor with a rich feature set - like easily controlled re-restart behavior, right (If not, what do you do when your daemon simply crashes)?
Well, in fact I'm in the development stage now and did't have time to deal with this question yet :) As of now I understand very little about daemons and their supervisors (I belive it is systemd for Raspbian...).
Yes, systemd will do this.
Jean-Paul and Glyph, thank you for guiding me in the right direction! :)
Happy to help! -g
Jean-Paul
, _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com <mailto:Twisted-Python@twistedmatrix.com> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python <http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python>_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com <mailto:Twisted-Python@twistedmatrix.com> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python <http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python>
On Feb 28, 2017, at 8:11 AM, Роман Мещеряков <romanmescheryakov@yandex.ru> wrote:
Hello everyone, today I tried restarting twisted service using sys.execv, and it fails with the following message:
2017-02-28T18:39:47+0300 [ready_to_setup#debug] Before os.execv: sys.executable is /usr/bin/python and sys.argv are ['/usr/local/bin/twistd', '-ny', 'master_player.tac'] Another twistd server is running, PID 1939
This could either be a previously started instance of your application or a different application entirely. To start a new one, either run it in some other directory, or use the --pidfile and --logfile parameters to avoid clashes.
My service is started in the following way:
PYTHONPATH=. twistd -ny master_player.tac
(i.e. without daemonization) Code to restart is:
d = task.deferLater(self._reactor, seconds, shared_code.restart_script, self._log, self._reactor) # _restart_script_done should never be called d.addCallback(self._restart_script_done).addErrback(self._restart_script_failure)
where shared_code.restart_script is:
def restart_script(logger, reactor): logger.info("Restarting script...") reactor.stop() sys.stdout.flush() #TODO: force logging system to flush all log files logger.debug("Before os.execv: sys.executable is {sysexec} and sys.argv are {argv}", sysexec = sys.executable, argv = sys.argv) os.execv(sys.executable, [sys.executable] + sys.argv)
Could someone explain to me why does the "Another twistd server is running" error occur and how can I avoid it?
twistd writes a .pid file and then attempts to check whether another process on the system has the same PID contained in that file while starting up, to avoid conflicts. Yours is an odd case, because the process with that process-ID is the _current_ process, and perhaps the check ought to even be modified to account for this. Two other options you have are: switch to `twist`, which doesn't write a pidfile use the option "--pidfile ''" to disable writing a pidfile. I strongly suggest the first option :). -glyph
On Mar 1, 2017, at 9:46 AM, Роман Мещеряков <romanmescheryakov@yandex.ru> wrote:
Then the next question is: how to run my .tac file with twist? I don't see option like '-y' in it's --help output.
We haven't implemented .tac files for `twist` yet; you can either add that feature :) or you can write a twist plugin. (It's the same as a twistd plugin.) -glyph
participants (3)
-
Glyph Lefkowitz -
Jean-Paul Calderone -
Роман Мещеряков