[Twisted-Python] PB and FilePager
Hi there,
I encountered a problem while dealing with file transfers using PB and t.s.u.FilePager.
The sscce illustrates the problem:
--------- server.py --------------------------------------------------
from twisted.spread import pb
from twisted.internet import reactor
ROOT = '/tmp/'
class ImageCollector(pb.Referenceable):
def __init__(self, fd):
self.fd = fd
def remote_dummy(self):
pass
def remote_gotPage(self, page):
self.fd.write(page)
def remote_endedPaging(self):
print "Completed"
self.fd.close()
class VurmController(pb.Root):
def remote_createImage(self, imageId):
return ImageCollector(open(ROOT + imageId, 'w'))
reactor.listenTCP(8789, pb.PBServerFactory(VurmController()))
reactor.run()
----------------------------------------------------------------------
--------- client.py --------------------------------------------------
import sys
from twisted.spread import pb
from twisted.internet import reactor, defer
from twisted.spread import util
FILE = 'myfile.something'
factory = pb.PBClientFactory()
reactor.connectTCP("localhost", 8789, factory)
d = factory.getRootObject()
def createImage(controller):
return controller.callRemote('createImage', 'image-id')
d.addCallback(createImage)
def sendImage(ctl):
print ctl.callRemote('dummy') ############# LINE 19 #############
d = defer.Deferred()
util.FilePager(ctl, open(FILE), callback=lambda: d.callback(None))
return d
d.addCallback(sendImage)
def done(_):
print "Transfer completed"
#reactor.callLater(1, reactor.stop)
reactor.stop()
d.addCallback(done)
reactor.run()
----------------------------------------------------------------------
I'm trying to *upload* a file from the client to the server by calling a method on the
server which returns a collector to be used with a FilePager instance.
When run with the line client.py:19 commented out, the script blocks before sending
out any file chunks, if the line is uncommented (effectively calling the 'dummy' method
remotely), everything works as expected.
Digging around in the sources, I found out that t.p.u.Pager registers itself to the
collector's pb broker as a pageProducer. When resumeProducing is called on the broker,
it asks the FilePager instance for the next page, which leads to the invocation of
the following method on FilePager::
def sendNextPage(self):
"""
Get the first chunk read and send it to collector.
"""
if not self.chunks:
return
val = self.chunks.pop(0)
self.producer.resumeProducing()
self.collector.callRemote("gotPage", val)
As the t.b.FileSender producer does not yet had a chance to write something to the
FilePager, the method returns straight away without sending anything.
As anything was sent, the Broker.resumeProducing method is never called again, and
thus the FilePager.sendNextPage neither.
The problem is caused by the calling chain of the FilePager constructor:
FilePager.__init__()
-> Pager.__init__()
-> broker.registerPageProducer(FilePager)
-> transport.registerProducer(broker)
-> broker.resumeProducing()
-> FilePager.sendNextPage()
->
On 27 Jun 2011, 08:52 pm, jonathan@stoppani.name wrote:
Hi there,
I encountered a problem while dealing with file transfers using PB and t.s.u.FilePager.
This looks like http://twistedmatrix.com/trac/ticket/3445 which has been open for a while (and is still open). The fix does seem straight-forward - keep track of the paused/unpaused state so new chunks can be written right away. Note that reversing the order of the __init__ and startProducing calls probably isn't the right solution, as that would re-introduce http://twistedmatrix.com/trac/ticket/3023. Anyone want to take a shot at this ticket? Jean-Paul
participants (2)
-
exarkun@twistedmatrix.com
-
Jonathan Stoppani