FIle transfer over network - with Pyro?

[exarkun at twistedmatrix.com]

On 06:58 pm, strombrg at gmail.com wrote:
>On Jun 3, 10:47 am, Nathan Huesken <pyt... at lonely-star.org> wrote:
>>Hi,
>>
>>I am writing a network application which needs from time to time do
>>file transfer (I am writing the server as well as the client).
>>For simple network messages, I use pyro because it is very 
>>comfortable.
>>But I suspect, that doing a file transfer is very inefficient over
>>pyro, am I right (the files are pretty big)?
>>
>>I somehow need to ensure, that the client requesting a file transfer 
>>is
>>the same client getting the file. So some sort of authentication is
>>needed.
>>
>>What library would you use to do the file transfer?
>>Regards,
>>Nathan
>
>I've never used Pyro, but for a fast network file transfer in Python,
>I'd probably use the socket module directly, with a cache oblivious
>algorithm:
>   http://en.wikipedia.org/wiki/Cache-oblivious_algorithm
>
>It doesn't use sockets, it uses files, but I recently did a Python
>progress meter application that uses a cache oblivious algorithm that
>can get over 5 gigabits/second throughput (that's without the network
>in the picture, though if it were used on 10 Gig-E with a suitable
>transport it could probably do nearly that), on a nearly-modern PC
>running Ubuntu with 2 cores  It's at:
>   http://stromberg.dnsalias.org/~strombrg/gprog/ .

This seems needlessly complicated.  Do you have a hard drive that can 
deliver 5 gigabits/second to your application?  More than likely not.

A more realistic answer is probably to use something based on HTTP. 
This solves a number of real-world problems, like the exact protocol to 
use over the network, and detecting network issues which cause the 
transfer to fail.  It also has the benefit that there's plenty of 
libraries already written to help you out.

Jean-Paul