[FEEDBACK] Is this script efficient...is there a better way?
whisper at oz.net
Thu Sep 12 11:31:44 CEST 2002
> Have you actually tested this? I used to believe that larger buffers are
> always better for performance.
> I ran some test on Linux for the effect of buffer size on file reading
> speed and the results were very interesting. I started with a buffer size
> of 32 bytes, tested file I/O throughput and increased it
> logarithmically by
> about 2% for each step. As expected, the time to read 1mb improved as the
> buffer size increased until it hit a minimum for a buffer size of around
> 4-8k (the graph is very noisy so it's hard to tell) and then rose back up
> to a value that is 10-20% worse for buffer sizes of 32-64k and remained
> more-or-less constant for anything higher.
> This performance curve may be a result of the CPU cache or the OS
> The chunk size used by xreadlines is 8k which is about the optimum
> value (at least for linux).
Disk cluster size? Fragmentation?
Seattle, WA USA
More information about the Python-list