Fastest way to write huge files

Larry Bates larry.bates at vitalEsafe.com
Sat Aug 30 15:00:08 CEST 2008


Mohamed Yousef wrote:
>> If connection is over Internet via HTTP the connection speed is so slow in
>> relation to the speed of your CPU that it doesn't really matter.
> this is not always true , espicially when using a localhost or a local
> netwtork server
> the problem is the increase in cpu and memory usage make it a painful bug
> such that downloading a big file (say 1 GB ) would introduce a big cpu
> usage (already tested)
> 
> Thanks ,
> 
> Regards,
> Mohamed Yousef

I disagree.  There is very little overhead in file writing if you stream your 
writes to disk in small blocks as they arrive via HTTP.  Don't wait for the 
entire 1Gb to arrive and then write it.  Python can write small to a file 
blazingly fast with normal file writing I/O. Any extra CPU overhead you may see 
is probably due to reading the entire 1Gb into memory and seeing swapping to 
disk as you exhaust main memory.  Interleaving your HTTP reading with file 
writing should be very fast.

-Larry



More information about the Python-list mailing list