[Python-ideas] BufferedIO and detach

[Robert Collins]

On 4 March 2013 22:59, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Le Mon, 4 Mar 2013 19:44:27 +1300,
> Robert Collins
> <robertc at robertcollins.net> a écrit :
>>
>> Yes exactly. A little more context on how I came to ask the question.
>> I wanted to accumulate all input on an arbitrary stream within 5ms,
>> without blocking for longer. Using raw IO + select, its possible to
>> loop, reading one byte at a time. The io module doesn't have an API
>> (that I could find) for putting an existing stream into non-blocking
>> mode, so reading a larger amount and taking what is returned isn't
>> viable.
>
> What do you mean exactly by that?

Just what I said. I'll happily try to rephrase. What bit was unclear?

>> However, without raw I/O, select() will timeout because it consults
>> the underlying file descriptor bypassing the buffer. So - the only
>> reason to want raw I/O is to be able to use select reliably.
>
> That's a pretty good reason actually. Raw I/O is exactly for those
> cases. Non-blocking buffered I/O is a hard conceptual problem:
> http://bugs.python.org/issue13322

Sure, it can get tricky to reason about. But - the whole point of
libraries like io is to encapsulate common solutions to tricky things
- so that we don't have a hundred incompatible not-quite-the-same
layers sitting on top. Right now select + BufferedIOReader is plain
buggy regardless of non-blocking or not... ; I'd like to fix that -
for instance, if select consulted the buffer somehow and returned
immediately if the buffer had data, that would be an improvement (as
select doesn't say *how much* data can be read).

>> An
>> alternative would be being able to drain the buffer with no underlying
>> I/O calls at all, then use select + read1, then rinse and repeat.
>
> Have you tried peek()?

Per http://docs.python.org/3.2/library/io.html#io.BufferedReader.peek
peek may cause I/O. Only one, but still you cannot control it.

-Rob

-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Cloud Services