Fastest way to write huge files

saswat at saswat at
Fri Aug 29 03:52:20 CEST 2008

On Aug 28, 5:35 pm, "Mohamed Yousef" <harrr... at> wrote:
> Hello ,
> let's say , I'm moving large files through network between devices
> what is the fastest way to do this ?
> what i came up with :-
> 1) using regular file operations with an in memory limit of 4MB which
> when filled written to disk and re-filled again
> 2) using memory mapped files in the following way , map first 20MB of
> file and after finishing them flush and map next 20MB
> ------
> which of them should be faster in writting to disk ? and less memory consuming
> or is there faster ways ?
> Thanks,
> Regards ,
> Mohamed Yousef

In the send side :

You may want to use sendfile and here is python package for same

In the receive side :

You may want to use splice (Linux 2.6.17 onwards) - Just read from a
network fd and write to a file fd

I just assumed you are using Linux.:-)


More information about the Python-list mailing list