[Twisted-Python] PB and FilePager

June 27, 2011

      Hi there,

I encountered a problem while dealing with file transfers using PB and t.s.u.FilePager.

The sscce illustrates the problem:

--------- server.py --------------------------------------------------
from twisted.spread import pb
from twisted.internet import reactor

ROOT = '/tmp/'

class ImageCollector(pb.Referenceable): 
def __init__(self, fd):
    self.fd = fd

def remote_dummy(self):
    pass

def remote_gotPage(self, page):
    self.fd.write(page)

def remote_endedPaging(self):
    print "Completed"
    self.fd.close()

class VurmController(pb.Root):
def remote_createImage(self, imageId):
    return ImageCollector(open(ROOT + imageId, 'w'))

reactor.listenTCP(8789, pb.PBServerFactory(VurmController()))
reactor.run()
----------------------------------------------------------------------

--------- client.py --------------------------------------------------
import sys

from twisted.spread import pb
from twisted.internet import reactor, defer
from twisted.spread import util

FILE = 'myfile.something'

factory = pb.PBClientFactory()
reactor.connectTCP("localhost", 8789, factory)

d = factory.getRootObject()

def createImage(controller):
return controller.callRemote('createImage', 'image-id')
d.addCallback(createImage)

def sendImage(ctl):
print ctl.callRemote('dummy') ############# LINE 19 #############
d = defer.Deferred()
util.FilePager(ctl, open(FILE), callback=lambda: d.callback(None))
return d
d.addCallback(sendImage)

def done(_):
print "Transfer completed"
#reactor.callLater(1, reactor.stop)
reactor.stop()
d.addCallback(done)

reactor.run()
----------------------------------------------------------------------

I'm trying to *upload* a file from the client to the server by calling a method on the
server which returns a collector to be used with a FilePager instance.

When run with the line client.py:19 commented out, the script blocks before sending
out any file chunks, if the line is uncommented (effectively calling the 'dummy' method
remotely), everything works as expected.

Digging around in the sources, I found out that t.p.u.Pager registers itself to the
collector's pb broker as a pageProducer. When resumeProducing is called on the broker,
it asks the FilePager instance for the next page, which leads to the invocation of
the following method on FilePager::

def sendNextPage(self):
    """
    Get the first chunk read and send it to collector.
    """
    if not self.chunks:
        return

    val = self.chunks.pop(0)
    self.producer.resumeProducing()
    self.collector.callRemote("gotPage", val)

As the t.b.FileSender producer does not yet had a chance to write something to the
FilePager, the method returns straight away without sending anything.

As anything was sent, the Broker.resumeProducing method is never called again, and
thus the FilePager.sendNextPage neither.

The problem is caused by the calling chain of the FilePager constructor:

FilePager.__init__()
-> Pager.__init__()
-> broker.registerPageProducer(FilePager)
  -> transport.registerProducer(broker)
    -> broker.resumeProducing()
      -> FilePager.sendNextPage()
        -> <no data, don't send anything>
-> FilePager.startProducing(fd)
-> FileSender().beginFileTransfer(fd, FilePager)
  -> FilePager.registerProducer(FileSender)
    -> FileSender.resumeProducing()
      -> FilePager.write(data)
        -> <store data, wait for sendNextPage to be called>

Simply inverting these two method calls (Pager.__init__ and FilePager.startProducing)
solves the problem.

The same problem does not appear when calling the 'dummy' method (client.py:19)
because the data for the method call is waiting to be sent out by the reactor and
the transport does not call 'resumeProducing' on the broker until the next iteration.

This allows the call to FilePager.startProducing to complete before sendNextPage is
ever called on it.

Does this sound correct? It seems only strange to me that nobody else had this problem
previously. If no errors on my side are found, I'll submit this as a ticket.

Cheers,
Jonathan

[Twisted-Python] PB and FilePager

Jonathan Stoppani