[Python-3000] Comment on iostack library

[Guido van Rossum]

On 8/29/06, Talin <talin at acm.org> wrote:
> I've been thinking more about the iostack proposal. Right now, a typical
> file handle consists of 3 "layers" - one representing the backing store
> (file, memory, network, etc.), one for adding buffering, and one
> representing the program-level API for reading strings, bytes, decoded
> text, etc.
>
> I wonder if it wouldn't be better to cut that down to two. Specifically,
> I would like to suggest eliminating the buffering layer.
>
> My reasoning is fairly straightforward: Most file system handles,
> network handles and other operating system handles already support
> buffering, and they do a far better job of it than we can. The handles
> that don't support buffering are memory streams - which don't need
> buffering anyway.
>
> Of course, it would make sense for Python to provide its own buffering
> implementation if we were going to always use the lowest-level i/o API
> provided by the operating system, but I can't see why we would want to
> do that. The OS knows how to allocate an optimal buffer, using
> information such as the block size of the filesystem, whereas trying to
> achieve this same level of functionality in the Python standard library
> would be needlessly complex IMHO.

I'm not sure I follow.

We *definitely* don't want to use stdio -- it's not part of the OS
anyway, and has some annoying quirks like not giving you any insight
in how it is using the buffer, nor changing the buffer size on the
fly, and crashing when you switch read and write calls.

So given that, how would you implement readline()? Reading one byte at
a time until you've got the \n is definitely way too slow given the
constant overhead of system calls.

Regarding optimal buffer size, I've never seen a program for which 8K
wasn't optimal. Larger buffers simply don't pay off.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)