[Twisted-Python] Graceful shutdown of twistd application
Hello list, I need to implement a graceful shutdown procedure for a twistd application. The application is made up of two services: an internet.TCPClient and an internet.TCPServer. They're glued together with a MultiService instance, which is in turn set to have 'application' as parent. The server and the client work together, making a proxy (SMTP server and AMQP client). My goal is the following: - intercept a SIGTERM signal - 'block' on the server side: since it's SMTP I get this by setting a variable that makes the server return tempfails (4xx) for new messages, while keeping current sessions active - wait until current requests are satisfied (I keep a dictionary of current pending messages) - shut the whole thing down What is the best solution for this use case? It's not really clear to me how to catch SIGTERM and handle pending requests *before* the underlying services start to shutdown (i.e. even addSystemEventTrigger('before', 'shutdown', callable) is called too late for my needs). Thank you very much for your help! Fabio
On 10:52 am, sangiovanni@nweb.it wrote:
Hello list,
I need to implement a graceful shutdown procedure for a twistd application. The application is made up of two services: an internet.TCPClient and an internet.TCPServer. They're glued together with a MultiService instance, which is in turn set to have 'application' as parent. The server and the client work together, making a proxy (SMTP server and AMQP client).
My goal is the following: - intercept a SIGTERM signal - 'block' on the server side: since it's SMTP I get this by setting a variable that makes the server return tempfails (4xx) for new messages, while keeping current sessions active - wait until current requests are satisfied (I keep a dictionary of current pending messages) - shut the whole thing down
This is exactly what before shutdown triggers are for. Alternatively, use the higher-level API and implement `stopService` on one of your services. Either way, return a `Deferred` that only fires when you're satisfied it is time for shutdown to proceed. You said before shutdown triggers are too late but you didn't say why. I think that's based on a misunderstanding - but if not, then explain why it doesn't work for your scenario. Jean-Paul
What is the best solution for this use case? It's not really clear to me how to catch SIGTERM and handle pending requests *before* the underlying services start to shutdown (i.e. even addSystemEventTrigger('before', 'shutdown', callable) is called too late for my needs).
Thank you very much for your help!
Fabio
On Thu, Sep 4, 2014 at 2:02 PM,
You said before shutdown triggers are too late but you didn't say why. I think that's based on a misunderstanding - but if not, then explain why it doesn't work for your scenario.
Hi, thanks for your reply.
I've tried the following:
def sleep(secs):
log.msg('from within trigger')
d = defer.Deferred()
reactor.callLater(secs, d.callback, None)
return d
reactor.addSystemEventTrigger('before', 'shutdown', sleep, 10)
This is what I can see in the logs:
Sep 4 14:25:06 prepyproxy01 proxy [4924]: [-] Received SIGTERM, shutting
down.
Sep 4 14:25:06 prepyproxy01 proxy [4924]: [-] from within trigger
Sep 4 14:25:06 prepyproxy01 proxy [4924]:
[TwistedProtocolConnection,client]
On 12:36 pm, sangiovanni@nweb.it wrote:
On Thu, Sep 4, 2014 at 2:02 PM,
wrote: You said before shutdown triggers are too late but you didn't say why. I think that's based on a misunderstanding - but if not, then explain why it doesn't work for your scenario.
Hi, thanks for your reply.
I've tried the following:
def sleep(secs): log.msg('from within trigger') d = defer.Deferred() reactor.callLater(secs, d.callback, None) return d
reactor.addSystemEventTrigger('before', 'shutdown', sleep, 10)
All 'before' trigger are run concurrently. If you're using `Application` then your `sleep` trigger runs concurrently with the application's `stopService` trigger (because `Application` has its stopService added as another 'before' shutdown' trigger alongside yours). If you want to delay your application shutdown, you need to cooperate a little more closely with it. Either attach your application shutdown code as a callback to the sleep Deferred or move the sleep into the stopService implementation of one of the services on your application and trigger the remaining stopService calls (eg the stopService call on the MultiService you mentioned) when the sleep Deferred fires there. Jean-Paul
Hi,
I just wanted to give an update about the way I got it, in case this could
be useful for somebody.
I essentially inherited from service.MultiService like this:
class GracefulMultiService(service.MultiService):
def stopService(self):
d = defer.succeed(None)
d.addCallback(self._wait_for_your_cleanup_code_to_complete)
d.addCallback(lambda _: self)
d.addCallback(service.MultiService.stopService)
return d
@defer.inlineCallbacks
def _wait_for_your_cleanup_code_to_complete(self):
[...]
yield sleep(10) # cleanup code example
def sleep(secs):
d = defer.Deferred()
reactor.callLater(secs, d.callback, None)
return d
Thanks again for your help!
On Thu, Sep 4, 2014 at 2:55 PM,
On 12:36 pm, sangiovanni@nweb.it wrote:
On Thu, Sep 4, 2014 at 2:02 PM,
wrote: You said before shutdown triggers are too late but you didn't say why. I think that's based on a misunderstanding - but if not, then explain why it doesn't work for your scenario.
Hi, thanks for your reply.
I've tried the following:
def sleep(secs): log.msg('from within trigger') d = defer.Deferred() reactor.callLater(secs, d.callback, None) return d
reactor.addSystemEventTrigger('before', 'shutdown', sleep, 10)
All 'before' trigger are run concurrently. If you're using `Application` then your `sleep` trigger runs concurrently with the application's `stopService` trigger (because `Application` has its stopService added as another 'before' shutdown' trigger alongside yours).
If you want to delay your application shutdown, you need to cooperate a little more closely with it. Either attach your application shutdown code as a callback to the sleep Deferred or move the sleep into the stopService implementation of one of the services on your application and trigger the remaining stopService calls (eg the stopService call on the MultiService you mentioned) when the sleep Deferred fires there.
Jean-Paul
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- *Fabio Sangiovanni | System Integrator* *T* +39 0372 24525* F* +39 0372 800725 Via Dei Comizi Agrari 10, 26100 Cremona - Italia Milano - Cremona - San Francisco MailUp® La soluzione per l'invio di email e SMS mailup.it http://www.mailup.it
participants (2)
-
exarkun@twistedmatrix.com
-
Fabio Sangiovanni