[FEEDBACK] Is this script efficient...is there a better way?

David LeBlanc whisper at oz.net
Thu Sep 12 05:31:44 EDT 2002

> Have you actually tested this? I used to believe that larger buffers are
> always better for performance.
> Wrong.
> I ran some test on Linux for the effect of buffer size on file reading
> speed and the results were very interesting. I started with a buffer size
> of 32 bytes, tested file I/O throughput and increased it
> logarithmically by
> about 2% for each step.  As expected, the time to read 1mb improved as the
> buffer size increased until it hit a minimum for a buffer size of around
> 4-8k (the graph is very noisy so it's hard to tell) and then rose back up
> to a value that is 10-20% worse for buffer sizes of 32-64k and remained
> more-or-less constant for anything higher.
> This performance curve may be a result of the CPU cache or the OS
> architecture.
> The chunk size used by xreadlines is 8k which is about the optimum
> value (at least for linux).
> 	Oren

Disk cluster size? Fragmentation?

Dave LeBlanc
Seattle, WA USA

More information about the Python-list mailing list