[Web-SIG] WSGI: allowing short reads

[Robert Collins]

On 28 September 2014 00:00, Antoine Pitrou <antoine at python.org> wrote:
>
> Hi,
>
> Robert Collins <robertc at ...> writes:
>>
>> https://github.com/python-web-sig/wsgi-ng/issues/5
>>
>> tl;dr - we don't specify whether read(size) has to return size bytes
>> or just not more than size, today. the IO library is clear that
>> read(n) returns up to n, and also offers read1 that guarantees only
>> one read call.
>
> I think you've got things mixed up. There are two different things in
> "the IO library" (which is really the 3.x IO stack):
>
> * buffered binary I/O has read(n) and read1(n):
>   - read(n) will block until n bytes are received (except for non-blocking
>     fds)
>   - read1(n) will issue at most one system call and can return fewer than
>     n bytes
>
> * raw binary I/O only has read(n):
>   - read(n) will issue at most one system call and can return fewer than
>     n bytes
>
> So, depending on which layer you are placing yourself on, one or the
> other of your statements is wrong.

Ugh. Thanks!

> That said, it would be reasonable for WSGI to expose the raw I/O layer,
> IMHO. "Prettifying" libraries can wrap it inside a BufferedReader if
> they like.

We often can't expose the real underlying socket, but having the
semantics of raw binary I/O for the file-like things we do expose
makes more sense to me.

> Note that I once proposed generalized prefetching on I/O streams, but
> it was rejected:
> https://mail.python.org/pipermail/python-ideas/2010-September/008179.html
> (skip to the prefetch() proposal)
>
> It was in the context of improving streamed unpickling, which is
> a problem a bit similar - but less horrible - to JSON unserializing;
> since then, the problem was solved in a different way by adding a
> framing layer to pickle protocol 4 :-).

BufferedReader solves this well though, doesn't it?

-Rob

-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud