Creating huge data in very less time.

Tim Chase python.list at tim.thechases.com
Tue Mar 31 12:33:30 CEST 2009


venutaurus539 at gmail.com wrote:
> On Mar 31, 1:15 pm, Steven D'Aprano
>> The fastest HDDs can reach about 125 MB per second under
>> ideal circumstances, so that will take at least 8 seconds
>> per 1GB file or 8000 seconds in total.
> 
> That time is reasonable. 

You did catch the bit about "the *fastest* HDDs" (my emphasis). 
Unless you've got some massive RAID or Gig-E/Fiberchanel SAN, you 
likely don't have these ideal conditions.  Additionally, I've 
seen "125MB/sec" as the read speeds -- sustained write speeds are 
often lower.  Under more real-world testing, you'll likely get 
throughput closer to 30-70MB/sec.  Call that roughly half the 
throughput, and you're up to 16,000 seconds, or about 4.5hr.  A 
far cry from the "few minutes of time" you first mentioned...

And this doesn't take into consideration the OS overhead of the 
filesystem type.  Some filesystem types are optimized for large 
sequential access, while others work better with smaller files. 
You then have things like OS permission overhead, directory-path 
overhead, and other disk I/O going on at the same time.

-tkc








More information about the Python-list mailing list