On Wed, May 17, 2006 at 08:41:21AM +0200, Nicola Larosa wrote:
In my experience, sending big files over PB takes way too much time. This is due to the serialization-deserialization process involved. Paging avoids blocking, which is good, but it still takes much more than sending the files as-is.
How bad is the slow down? Or, to ask the question another way, how much CPU will the process actually take? I ask because the machines in question are being used as file servers for streaming applications like video and whatnot, so having Twisted suck up all of the CPU may disrupt the other streams. (We're not using Twisted to actually serve the files, but rather as the application framework for various web- and GUI- based management clients. The copy data feature would be the first time we're moving lots of data with Twisted.)
- Finally, should I be doing something completely different? Normally, outside of my application, I'd just use rsync, scp, or some such. However, the users of this application don't know how to use these tools. I can't spawn these programs without getting into authentication issues between the machines. Doing this within Twisted seems like a good idea because the machines are already authenticated to each other through PB, but I could be wrong.
You could send the files over an HTTP connection, avoiding the serialization overhead. Setting up HTTP clients and servers is very easy in Twisted, as you surely know.
Indeed! Using HTTP is appealing because it is closer to just stuffing bits in a socket from an efficiency standpoint. Is there a way of passing a RemoteReference to an HTTP server? Or is the best thing to do just use the PB to send a URL to the DEST server? --Justin