Nicola Larosa:
In my experience, sending big files over PB takes way too much time. This is due to the serialization-deserialization process involved. Paging avoids blocking, which is good, but it still takes much more than sending the files as-is.
Justin Mazzola Paluska:
How bad is the slow down? Or, to ask the question another way, how much CPU will the process actually take?
100% CPU for all the time it takes. Serialization is CPU- and memory-intensive.
You could send the files over an HTTP connection, avoiding the serialization overhead. Setting up HTTP clients and servers is very easy in Twisted, as you surely know.
Indeed! Using HTTP is appealing because it is closer to just stuffing bits in a socket from an efficiency standpoint. Is there a way of passing a RemoteReference to an HTTP server?
I don't think so. There's no overlapping that I know of, between PB and HTTP.
Or is the best thing to do just use the PB to send a URL to the DEST server?
That's what I was hinting at, yes. Of course you should separately take care of any required authentication, authorization and encryption on the HTTP connection.