[Twisted-Python] Remote PB calls in ApplicationService.stopService
![](https://secure.gravatar.com/avatar/a82a913ec2f6cf578731da60b0802d26.jpg?s=120&d=mm&r=g)
I had two programs running; one was a twistd -y instance, the other was a simple script that received objects over Perspective Broker and printed them out. The -y'able application consists of one Service: class MyService(app.ApplicationService): def somePBRemoteCall(self, event): d = pb.getObjectAt(self.host, self.port, self.timeout) d.addCallback(self.foundCatcher, event) def foundCatcher(self, obj, event): d = obj.callRemote('catch', event) def stopService(self): somePBRemoteCall(i_am_dying_event) app.ApplicationService.stopService(self) This isn't the code verbatim, but this is basically what it's doing. I want the receiving side to be informed when the sending side is about to stop. On the sending side, this results in 'Connection failed'. On the receiving side, the following traceback appears. Could somebody more familiar with the internals of Perspective Broker explain what's going wrong, or how I might change my approach? ---snip--- Traceback (most recent call last): File "sandbox/test-catcher.py", line 15, in ? reactor.run() File "/usr/lib/python2.2/site-packages/twisted/internet/default.py", line 122, in run self.mainLoop() File "/usr/lib/python2.2/site-packages/twisted/internet/default.py", line 133, in mainLoop self.doIteration(t) File "/usr/lib/python2.2/site-packages/twisted/internet/default.py", line 472, in doSelect _logrun(selectable, _drdw, selectable, method, dict) --- <exception caught here> --- File "/usr/lib/python2.2/site-packages/twisted/python/log.py", line 65, in callWithLogger callWithContext({"system": lp}, func, *args, **kw) File "/usr/lib/python2.2/site-packages/twisted/python/log.py", line 52, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/lib/python2.2/site-packages/twisted/python/context.py", line 32, in callWithContext return func(*args,**kw) File "/usr/lib/python2.2/site-packages/twisted/internet/default.py", line 495, in _doReadOrWrite selectable.connectionLost(f) File "/usr/lib/python2.2/site-packages/twisted/internet/tcp.py", line 253, in connectionLost protocol.connectionLost(reason) File "/usr/lib/python2.2/site-packages/twisted/spread/pb.py", line 563, in connectionLost for lobj in self.remotelyCachedObjects.values(): exceptions.AttributeError: Broker instance has no attribute 'remotelyCachedObjects' ---snip--- -- Alex Levy WWW: http://mesozoic.geecs.org "Never let your sense of morals prevent you from doing what is right." -- Salvor Hardin, Isaac Asimov's _Foundation_
![](https://secure.gravatar.com/avatar/0f15c04b6acde258bd27586371ae94b1.jpg?s=120&d=mm&r=g)
Alex Levy <mesozoic@polynode.com> writes:
There are a couple of problems interacting here. One is that you're sending out data as the application is shutting down, which means that many of the usual assumptions (about the request eventually being sent over the wire, about a response eventually coming back) are not guaranteed to remain true. A second is that, currently, all PB calls require acknowledgement, even if you expect the answer to be a simple 'None' (like the None that's returned implicitly when you fall off the end of a Python function). There are hooks in place to allow a call to tell the receiver that no answer is expected (or will be accepted), allowing that end to skip the response phase, but those hooks are disabled because there is no way to the caller to know whether you're going to add a callback to the Deferred or not (especially because that callback could be added *after* the request has been sent). So for now, the far end always acknowleges every single PB call. Or at least it tries to. I suspect you're seeing the following sequence of events: reactor.stop() 'shutdown' system event trigger is fired, callbacks are run app._beforeShutDown is run as a 'before' trigger on the 'shutdown' event stopService() is run on all Services callRemote(aaaaaarrgh) remote_aaaaaarrgh is serialized and queued, maybe sent, maybe not app._afterShutDown is run as the 'after' trigger, does app.save() reactor exits program exits all sockets are closed The serialized remote call may or may not get sent: if the transport's .write method tries to write some data immediately, then it will probably get out. But the connection will be dropped soon afterwards, probably before the far end has completed sending an answer. Given that PB is about acknowleged method invocations, it will probably be cleaner to try to delay shutdown briefly while you get your "I am dying" messages through. Fortunately, both .stopService and the before-shutdown event trigger that drives it have a useful property: you can return a Deferred from them and the rest of the shutdown process will be stalled until that Deferred is fired. So just do this: def stopService(self): d = somePBRemoteCall(i_am_dying_event) app.ApplicationService.stopService(self) return d (or do the superclass call first, it doesn't matter) This will send out your shutdown messages but then return to the main reactor loop, waiting for the messages to be acknowleged before proceeding with the rest of shutdown (app.save() and exiting the reactor). BTW: I think the exception you're seeing is a bug, and results from the connectionLost messages being invoked twice on the same Broker. I think this is an edge case that can only happen when a sender does something weird. hope that helps, -Brian
![](https://secure.gravatar.com/avatar/3477a9de290ec6d77129af504faa1c0b.jpg?s=120&d=mm&r=g)
On Sat, 02 Aug 2003, Brian Warner <warner@lothar.com> wrote:
BTW, currently that feature is not in the new Application code I'm working on. I could add it, but when we discussed it I didn't know of anybody using that feature. It seems a bit too dangerous to allow a buggy remote side to delay our shutdown by a long amount of time [depending on PB timeouts]. I humbly suggest that an architecture which depends on being able to send "one last message" is probably flawed. -- Moshe Zadka -- http://moshez.org/ Buffy: I don't like you hanging out with someone that... short. Riley: Yeah, a lot of young people nowadays are experimenting with shortness. Agile Programming Language -- http://www.python.org/
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Saturday, August 2, 2003, at 05:55 AM, Moshe Zadka wrote:
There are a couple of different use-cases here that need to be separated; for example, shutting down an active thread pool requires waiting, otherwise you will deadlock - and yes, this is due to the fact that OS threads are a basically flawed concept, but we've been over that one before :). Sometimes it's also useful to be able to send out a descriptive message before dying, like, "The service is going down! Please hurry up." I admit that the deferred-from-stopService is probably not the best way to handle this, though; there should be a different canonical way to shut down a Twisted server that allows those sorts of messages to be sent. However, as long as the deferred in the latter case always comes with the timeout, I think the use cases are about equivalent. Thoughts?
![](https://secure.gravatar.com/avatar/0f15c04b6acde258bd27586371ae94b1.jpg?s=120&d=mm&r=g)
Alex Levy <mesozoic@polynode.com> writes:
There are a couple of problems interacting here. One is that you're sending out data as the application is shutting down, which means that many of the usual assumptions (about the request eventually being sent over the wire, about a response eventually coming back) are not guaranteed to remain true. A second is that, currently, all PB calls require acknowledgement, even if you expect the answer to be a simple 'None' (like the None that's returned implicitly when you fall off the end of a Python function). There are hooks in place to allow a call to tell the receiver that no answer is expected (or will be accepted), allowing that end to skip the response phase, but those hooks are disabled because there is no way to the caller to know whether you're going to add a callback to the Deferred or not (especially because that callback could be added *after* the request has been sent). So for now, the far end always acknowleges every single PB call. Or at least it tries to. I suspect you're seeing the following sequence of events: reactor.stop() 'shutdown' system event trigger is fired, callbacks are run app._beforeShutDown is run as a 'before' trigger on the 'shutdown' event stopService() is run on all Services callRemote(aaaaaarrgh) remote_aaaaaarrgh is serialized and queued, maybe sent, maybe not app._afterShutDown is run as the 'after' trigger, does app.save() reactor exits program exits all sockets are closed The serialized remote call may or may not get sent: if the transport's .write method tries to write some data immediately, then it will probably get out. But the connection will be dropped soon afterwards, probably before the far end has completed sending an answer. Given that PB is about acknowleged method invocations, it will probably be cleaner to try to delay shutdown briefly while you get your "I am dying" messages through. Fortunately, both .stopService and the before-shutdown event trigger that drives it have a useful property: you can return a Deferred from them and the rest of the shutdown process will be stalled until that Deferred is fired. So just do this: def stopService(self): d = somePBRemoteCall(i_am_dying_event) app.ApplicationService.stopService(self) return d (or do the superclass call first, it doesn't matter) This will send out your shutdown messages but then return to the main reactor loop, waiting for the messages to be acknowleged before proceeding with the rest of shutdown (app.save() and exiting the reactor). BTW: I think the exception you're seeing is a bug, and results from the connectionLost messages being invoked twice on the same Broker. I think this is an edge case that can only happen when a sender does something weird. hope that helps, -Brian
![](https://secure.gravatar.com/avatar/3477a9de290ec6d77129af504faa1c0b.jpg?s=120&d=mm&r=g)
On Sat, 02 Aug 2003, Brian Warner <warner@lothar.com> wrote:
BTW, currently that feature is not in the new Application code I'm working on. I could add it, but when we discussed it I didn't know of anybody using that feature. It seems a bit too dangerous to allow a buggy remote side to delay our shutdown by a long amount of time [depending on PB timeouts]. I humbly suggest that an architecture which depends on being able to send "one last message" is probably flawed. -- Moshe Zadka -- http://moshez.org/ Buffy: I don't like you hanging out with someone that... short. Riley: Yeah, a lot of young people nowadays are experimenting with shortness. Agile Programming Language -- http://www.python.org/
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Saturday, August 2, 2003, at 05:55 AM, Moshe Zadka wrote:
There are a couple of different use-cases here that need to be separated; for example, shutting down an active thread pool requires waiting, otherwise you will deadlock - and yes, this is due to the fact that OS threads are a basically flawed concept, but we've been over that one before :). Sometimes it's also useful to be able to send out a descriptive message before dying, like, "The service is going down! Please hurry up." I admit that the deferred-from-stopService is probably not the best way to handle this, though; there should be a different canonical way to shut down a Twisted server that allows those sorts of messages to be sent. However, as long as the deferred in the latter case always comes with the timeout, I think the use cases are about equivalent. Thoughts?
participants (4)
-
Alex Levy
-
Brian Warner
-
Glyph Lefkowitz
-
Moshe Zadka