Mailman 3 January 2012 - Twisted

[Twisted-Python] PB and FilePager
by Jonathan Stoppani 03 Jan '12

03 Jan '12

Hi there, I encountered a problem while dealing with file transfers using PB and t.s.u.FilePager. The sscce illustrates the problem: --------- server.py -------------------------------------------------- from twisted.spread import pb from twisted.internet import reactor ROOT = '/tmp/' class ImageCollector(pb.Referenceable): def __init__(self, fd): self.fd = fd def remote_dummy(self): pass def remote_gotPage(self, page): self.fd.write(page) def remote_endedPaging(self): print "Completed" self.fd.close() class VurmController(pb.Root): def remote_createImage(self, imageId): return ImageCollector(open(ROOT + imageId, 'w')) reactor.listenTCP(8789, pb.PBServerFactory(VurmController())) reactor.run() ---------------------------------------------------------------------- --------- client.py -------------------------------------------------- import sys from twisted.spread import pb from twisted.internet import reactor, defer from twisted.spread import util FILE = 'myfile.something' factory = pb.PBClientFactory() reactor.connectTCP("localhost", 8789, factory) d = factory.getRootObject() def createImage(controller): return controller.callRemote('createImage', 'image-id') d.addCallback(createImage) def sendImage(ctl): print ctl.callRemote('dummy') ############# LINE 19 ############# d = defer.Deferred() util.FilePager(ctl, open(FILE), callback=lambda: d.callback(None)) return d d.addCallback(sendImage) def done(_): print "Transfer completed" #reactor.callLater(1, reactor.stop) reactor.stop() d.addCallback(done) reactor.run() ---------------------------------------------------------------------- I'm trying to *upload* a file from the client to the server by calling a method on the server which returns a collector to be used with a FilePager instance. When run with the line client.py:19 commented out, the script blocks before sending out any file chunks, if the line is uncommented (effectively calling the 'dummy' method remotely), everything works as expected. Digging around in the sources, I found out that t.p.u.Pager registers itself to the collector's pb broker as a pageProducer. When resumeProducing is called on the broker, it asks the FilePager instance for the next page, which leads to the invocation of the following method on FilePager:: def sendNextPage(self): """ Get the first chunk read and send it to collector. """ if not self.chunks: return val = self.chunks.pop(0) self.producer.resumeProducing() self.collector.callRemote("gotPage", val) As the t.b.FileSender producer does not yet had a chance to write something to the FilePager, the method returns straight away without sending anything. As anything was sent, the Broker.resumeProducing method is never called again, and thus the FilePager.sendNextPage neither. The problem is caused by the calling chain of the FilePager constructor: FilePager.__init__() -> Pager.__init__() -> broker.registerPageProducer(FilePager) -> transport.registerProducer(broker) -> broker.resumeProducing() -> FilePager.sendNextPage() -> <no data, don't send anything> -> FilePager.startProducing(fd) -> FileSender().beginFileTransfer(fd, FilePager) -> FilePager.registerProducer(FileSender) -> FileSender.resumeProducing() -> FilePager.write(data) -> <store data, wait for sendNextPage to be called> Simply inverting these two method calls (Pager.__init__ and FilePager.startProducing) solves the problem. The same problem does not appear when calling the 'dummy' method (client.py:19) because the data for the method call is waiting to be sent out by the reactor and the transport does not call 'resumeProducing' on the broker until the next iteration. This allows the call to FilePager.startProducing to complete before sendNextPage is ever called on it. Does this sound correct? It seems only strange to me that nobody else had this problem previously. If no errors on my side are found, I'll submit this as a ticket. Cheers, Jonathan

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 31 Dec '11

31 Dec '11

1 0

Re: [Twisted-Python] uhhvvccvv*n.hnjjjjjjjjjjjjjuppiikijk klklbjkjjjmkjlmoklinkojkopojollohz o ooz ouj lmp oji ljk ljk koo :Twisted-Pyt ho n Di gest , Vol 93, Issue 27
by sykadul＠astalavista.com 31 Dec '11

31 Dec '11

Ladies and gentleman, I will be unplugged from my email until the 17th of January. In the mean time here's a video of a bunny opening your mail http://www.youtube.com/watch?v=LMyaRmTwdKs Your mail will not be forwarded and I will contact you when I come back, alternatively you can contact one of the other administrators or email info(a)astalavista.com Merry christmas and a happy new year! Best regards, Sykadul

3 2

Re: [Twisted-Python] uhhvvccvv*n.hnjjjjjjjjjjjjjuppiikijk klklbjkjjjmkjlmoklinkojkopojollohzo ooz ouj lmp oji ljk ljk koo :Twisted-Pytho n Di gest , Vol 93, Issue 27
by sykadul＠astalavista.com 31 Dec '11

31 Dec '11

Ladies and gentleman, I will be unplugged from my email until the 17th of January. In the mean time here's a video of a bunny opening your mail http://www.youtube.com/watch?v=LMyaRmTwdKs Your mail will not be forwarded and I will contact you when I come back, alternatively you can contact one of the other administrators or email info(a)astalavista.com Merry christmas and a happy new year! Best regards, Sykadul

1 0

Re: [Twisted-Python] uhhvvccvv*n.hnjjjjjjjjjjjjjuppiikijk klklbjkjjjmkjlmoklinkojkopojollohzoooz ouj lmp oji ljk ljk koo :Twisted-Python Di gest , Vol 93, Issue 27
by sykadul＠astalavista.com 31 Dec '11

31 Dec '11

Ladies and gentleman, I will be unplugged from my email until the 17th of January. In the mean time here's a video of a bunny opening your mail http://www.youtube.com/watch?v=LMyaRmTwdKs Your mail will not be forwarded and I will contact you when I come back, alternatively you can contact one of the other administrators or email info(a)astalavista.com Merry christmas and a happy new year! Best regards, Sykadul

1 0

Re: [Twisted-Python] uhhvvccvv*n.hnjjjjjjjjjjjjjuppiikijk klklbjkjjjmkjlmoklinkojkopojollohzooozouj lmp oji ljk ljk koo :Twisted-Python Digest , Vol 93, Issue 27
by sykadul＠astalavista.com 31 Dec '11

31 Dec '11

Ladies and gentleman, I will be unplugged from my email until the 17th of January. In the mean time here's a video of a bunny opening your mail http://www.youtube.com/watch?v=LMyaRmTwdKs Your mail will not be forwarded and I will contact you when I come back, alternatively you can contact one of the other administrators or email info(a)astalavista.com Merry christmas and a happy new year! Best regards, Sykadul

1 0

Re: [Twisted-Python] uhhvvccvv*n.hnjjjjjjjjjjjjjuppiikijk klklbjkjjjmkjlmoklinkojkopojollohzooozoujlmp oji ljk ljk koo :Twisted-Python Digest, Vol 93, Issue 27
by sykadul＠astalavista.com 31 Dec '11

31 Dec '11

Ladies and gentleman, I will be unplugged from my email until the 17th of January. In the mean time here's a video of a bunny opening your mail http://www.youtube.com/watch?v=LMyaRmTwdKs Your mail will not be forwarded and I will contact you when I come back, alternatively you can contact one of the other administrators or email info(a)astalavista.com Merry christmas and a happy new year! Best regards, Sykadul

1 0

Re: [Twisted-Python] uhhvvccvv*n.hnjjjjjjjjjjjjjuppiikijk klklbjkjjjmkjlmoklinkojkopojollohzooozoujlmpoji ljk ljk koo :Twisted-Python Digest, Vol 93, Issue 27
by sykadul＠astalavista.com 31 Dec '11

31 Dec '11

Ladies and gentleman, I will be unplugged from my email until the 17th of January. In the mean time here's a video of a bunny opening your mail http://www.youtube.com/watch?v=LMyaRmTwdKs Your mail will not be forwarded and I will contact you when I come back, alternatively you can contact one of the other administrators or email info(a)astalavista.com Merry christmas and a happy new year! Best regards, Sykadul

1 0

Re: [Twisted-Python] uhhvvccvv*n.hnjjjjjjjjjjjjjuppiikijk klklbjkjjjmkjlmoklinkojkopojollohzooozoujlmpojiljk ljk koo :Twisted-Python Digest, Vol 93, Issue 27
by sykadul＠astalavista.com 31 Dec '11

31 Dec '11

Ladies and gentleman, I will be unplugged from my email until the 17th of January. In the mean time here's a video of a bunny opening your mail http://www.youtube.com/watch?v=LMyaRmTwdKs Your mail will not be forwarded and I will contact you when I come back, alternatively you can contact one of the other administrators or email info(a)astalavista.com Merry christmas and a happy new year! Best regards, Sykadul

1 0

Re: [Twisted-Python] uhhvvccvv*n.hnjjjjjjjjjjjjjuppiikijk klklbjkjjjmkjlmoklinkojkopojollohzooozoujlmpojiljkljk koo :Twisted-Python Digest, Vol 93, Issue 27
by sykadul＠astalavista.com 31 Dec '11

31 Dec '11

Ladies and gentleman, I will be unplugged from my email until the 17th of January. In the mean time here's a video of a bunny opening your mail http://www.youtube.com/watch?v=LMyaRmTwdKs Your mail will not be forwarded and I will contact you when I come back, alternatively you can contact one of the other administrators or email info(a)astalavista.com Merry christmas and a happy new year! Best regards, Sykadul

1 0