[Python-ideas] BufferedIO and detach

[Nick Coghlan]

On Mon, Mar 4, 2013 at 4:44 PM, Robert Collins
<robertc at robertcollins.net> wrote:
> Some variations I can think of...
>
> The buffer_only flag I suggested, on read_into, read1, read etc.
>
> Have detach return the buffered data as you suggest - that would be
> incompatible unless we stash it on the raw object somewhere, or do
> something along those lines.
>
> A read0 - analogous to read1, returns data from the buffer, but
> guarantees no underlying calls.
>
> I think exposing the buffer more explicitly is a good principle,
> independent of whether we change detach or not.

As Guido noted, you actually have multiple layers of buffering to
contend with - for a text stream, you may have already decoded
characters and partially decoded data in the codec's internal buffer,
in addition to any data in the IO buffer. That's actually one of the
interesting problems with supporting a "set_encoding()" method on IO
streams (see http://bugs.python.org/issue15216).

How does the following API sound for your purposes? (this is based on
what set_encoding() effectively has to do under the hood):

    BufferedReader:

        def push_data(binary_data):
            """Prepends contents of 'data' to the internal buffer"""

        def clear_buffer():
            """Clears the internal buffer and returns the previous
content as a bytes object"""

    TextIOWrapper:

        def push_data(char_data, binary_data=b""):
            """Prepends contents of 'data' to the internal buffer. If
binary_data is provided, it is pushed into the underlying IO buffered
reader. Raises UnsupportedOperation if the underlying stream has no
"push_data" method."""

        def clear_buffer():
            """Clears the internal buffers and returns the previous
content as a (char_data, binary_data) pair. The binary data includes
any data that was queued inside the codec, as well as the contents of
the underlying IO buffer"""

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia