Hi! I'm trying to implement graceful shutdown of a HTTP server and I am unsure of the preferred way of implementing it. I'm attempting to do this by adding a system event trigger for "before" "shutdown" that should stop accepting new requests and wait for current ones to finish. My problem is that though my requests has fired their notifyFinish() deferreds the data has not been written to the client. Calling reactor.getWriters() shows me that there are active writers who are probably writing/flushing data to the client. What is the preferred way of waiting for the requests to finish and finish writing the response to the client? Here is example code for my own attempt: from twisted.internet import reactor, defer from twisted.application import internet, service from twisted.web.server import Site, NOT_DONE_YET from twisted.web.resource import Resource class SlowResource(Resource): isLeaf = True waiting_requests = [] def notify_no_more_waiting(self): if not self.waiting_requests: return defer.succeed(None) return defer.gatherResults(self.waiting_requests, consumeErrors=True) \ .addBoth(lambda ign: None) def write_result(self, request): request.write('{}') request.finish() def render_GET(self, request): reactor.callLater(5, self.write_result, request) d = request.notifyFinish() self.waiting_requests.append(d) d.addBoth(lambda ign: self.waiting_requests.remove(d)) return NOT_DONE_YET slow_resource = SlowResource() site = Site(slow_resource) application = service.Application("MyApp") server = internet.TCPServer(8080, site) server.setServiceParent(application) def wait_for_writers(): d = defer.Deferred() def check_writers(): if len(reactor.getWriters()) > 0: reactor.callLater(0.1, check_writers) else: d.callback(None) check_writers() return d @defer.inlineCallbacks def graceful_shutdown(): yield server.stopService() yield slow_resource.notify_no_more_waiting() #yield wait_for_writers() reactor.addSystemEventTrigger('before', 'shutdown', graceful_shutdown) Uncommenting the yield in graceful_shutdown gives me my intended behaviour but I don't like the solution since I have no idea if the writers there have anything to do with writing the results. They could be any writer writing anything I assume. Running this code with twistd -ny graceful.tac and curling http://localhost:8080 and then issuing a SIGINT to the twistd process results in an curl: (52) Empty reply from server. -- /Jonas
I didn't read over your code, but this is something that I've independently implemented. You can see my implementation at <https://github.com/habnabit/polecat/blob/master/polecat.py>. The Site subclass does some other monitoring-related things, but you should be able to tease out just the graceful shutdown parts if that's all you want. Polecat needs some work and a lot of documentation, but for now, the way to do a graceful shutdown is to make a PolecatSite and then addSystemEventTrigger('before', 'shutdown', yourPolecatSite.gracefullyStopActiveClients).
Thanks for the link. Looking at your code I found the magic lines I was missing. Implementing my own HTTPChannel was the key to success. Now I don't have to poll reactor.getWriters() to see if there are unfinished writers. I ended up with something like this: class ApiHTTPChannel(HTTPChannel): _connection_lost = None def notifyConnectionLost(self): if self._connection_lost is not None: return self._connection_lost return defer.succeed(None) def connectionMade(self): HTTPChannel.connectionMade(self) self._connection_lost = defer.Deferred() def connectionLost(self, reason): HTTPChannel.connectionLost(self, reason) self._connection_lost.callback(None) Thanks On 28 August 2013 18:09, Aaron Gallagher <_@habnab.it> wrote:
I didn't read over your code, but this is something that I've independently implemented. You can see my implementation at <https://github.com/habnabit/polecat/blob/master/polecat.py>. The Site subclass does some other monitoring-related things, but you should be able to tease out just the graceful shutdown parts if that's all you want.
Polecat needs some work and a lot of documentation, but for now, the way to do a graceful shutdown is to make a PolecatSite and then addSystemEventTrigger('before', 'shutdown', yourPolecatSite.gracefullyStopActiveClients).
-- /Jonas
participants (2)
-
Aaron Gallagher
-
Jonas Lindmark