Frederick Reeve wrote:

The other point still stands though. I would like to see peek changed. I am willing to write and submit changes but don't want to unless others agree this is a good idea. So I put forth the implementation at the bottom of this email. If its bad or you don't see the point I may try to clarify but if nobody things its good, please just tell me I'm waisting your time, and I will go away. I also apologize my last email was so long.

peek(n): If n is less than 0, None, or not set; return buffer contents with out advancing stream position. If the buffer is empty read a full chunk and return the buffer. Otherwise return exactly n bytes up to _chunk size_(not contents) with out advancing the stream position. If the buffer contents is less than n, buffer an additional chunk from the "raw" stream before hand. If EOF is encountered during any raw read then return as much as we can up to n. (maybe I should write that in code form??)

I would phrase this suggestion as users having a reasonable expectation that the following invariant should hold for a buffered stream: f.peek(n) == f.read(n) Since the latter method will perform as many reads of the underlying stream as necessary to reach the requested number of bytes (or EOF), then so should the former. However, the default value for n for peek() should remain at 1 to remain consistent with the current documented behaviour. If this invariant was implemented, I would also suggest adding a "peek1" method to parallel "read1". Note that the current behaviour I get from Python 3.1 is for it to return the *entire* buffer, no matter what number I pass to it: (current Py3k head)

...

...

f = open('setup.py', 'rb') len(f.peek(10)) 4096 len(f.peek(1)) 4096 len(f.peek(4095)) 4096 len(f.peek(10095)) 4096

That's an outright bug - I've promoted an existing issue about this [1] to a release blocker and sent it to Benjamin to have another look at. Cheers, Nick. [1] http://bugs.python.org/issue5811 -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

Cameron Simpson wrote:

On 13Jun2009 12:24, Nick Coghlan wrote: | I would phrase this suggestion as users having a reasonable expectation | that the following invariant should hold for a buffered stream: | | f.peek(n) == f.read(n) | | Since the latter method will perform as many reads of the underlying | stream as necessary to reach the requested number of bytes (or EOF), | then so should the former.

I disagree. If that were that case, why have peek() at all? I realise that it doesn't move the logical position, but it does mean that peek(huge_number) imposes a requirement to grow the stream buffer arbitrarily.

A peek that does at most one raw read has the advantage that it can pick up data outside the buffer but lurking in the OS buffer, yet to be obtained. Those data are free, if they're present. (Of course, if they're absent peek() wil still block).

Note my suggestion later that if the above invariant were to be adopted then a peek1() method should be added to parallel read1(). However, from what Benjamin has said, a more likely invariant is going to be: preview = f.peek(n) f.read(n).startswith(preview) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

Antoine Pitrou wrote:

The original docstring for peek() says:

...we do at most one raw read to satisfy it.

In that light, I'm not sure it's a bug

It may be behaving according to the docs, but is that behaviour useful? Seems to me that if you're asking for n bytes, then it's because you're doing some kind of parsing that requires lookahead, and nothing less than n bytes will do. I think it would be more useful if the "at most one raw read" part were dropped. That would give it the kind of deterministic behaviour generally expected when dealing with buffered streams. -- Greg

On 14Jun2009 12:33, Greg Ewing wrote:

Antoine Pitrou wrote:

...

The original docstring for peek() says:

...we do at most one raw read to satisfy it.

In that light, I'm not sure it's a bug

It may be behaving according to the docs, but is that behaviour useful?

Seems to me that if you're asking for n bytes, then it's because you're doing some kind of parsing that requires lookahead, and nothing less than n bytes will do.

I think it would be more useful if the "at most one raw read" part were dropped. That would give it the kind of deterministic behaviour generally expected when dealing with buffered streams.

Is it possible to access the buffer? I see nothing in the docs. People seem to want peek() to be "read() without moving the read offset", which it almost seems to be. Nick and Greg both want it to really be that, and thus do enough raw reads to get "n" bytes; Nick wants a peek1() like read1(), too. It has a pleasing feel to me, too. But ... For myself, I'd expect more often to want to see if there's stuff in the buffer _without_ doing any raw reads at all. A peek0(n), if you will: Read and return up to n bytes without calling on the raw stream. It feels like peek is trying to span both extremes and doesn't satisfy either really well. If peek gets enhanced to act like read in terms of the amount of data returned, should there not be a facility to examine buffered data without raw reads? Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ Being on a Beemer and not having a wave returned by a Sportster is like having a clipper ship's hailing not returned by an orphaned New Jersey solid waste barge. - OTL

2009/6/14 Cameron Simpson :

On 14Jun2009 15:16, I wrote: | Is it possible to access the buffer? I see nothing in the docs.

I've just found getvalue() in IOBase. Forget I said anything. It seems to be my day for that kind of post:-(

Where are you seeing this? Only BytesIO and StringIO have a getvalue() method. -- Regards, Benjamin

Cameron Simpson wrote:

For myself, I'd expect more often to want to see if there's stuff in the buffer _without_ doing any raw reads at all.

What uses do you have in mind for that? -- Greg

Cameron Simpson wrote:

It seems like whenever I want to do some kind of opportunistic but non-blocking stuff with a remote service

Do you actually do this with buffered streams? I find it's better to steer well clear of buffered I/O objects when doing non-blocking stuff, because they don't play well with other things like select(). Anyhow, I wouldn't be opposed to having a way of looking into the buffer, but it should be a separate API -- preferably with a better name than peek0(), which gives no clue at all about what it does differently from peek(). -- Greg

Cameron Simpson wrote:

On 16Jun2009 11:21, Greg Ewing wrote:

...

Cameron Simpson wrote:

...

It seems like whenever I want to do some kind of opportunistic but non-blocking stuff with a remote service Do you actually do this with buffered streams?

Sure, in C, python and perl quite happily. In some circumstances. Provided you're careful about when to fflush() it can all go quite smoothly. It's certainly not applicable to everything.

...

I find it's better to steer well clear of buffered I/O objects when doing non-blocking stuff, because they don't play well with other things like select().

Ah, the non-blockingness. Well that's the rub. I normally avoid non-blocking requirements by using threads, so that the thread gathering from the stream can block. Then the consumer can poll a Queue from the worker thread, etc.

I really don't like select(/poll/epoll etc) much; aside from select's scaling issues with lots of files (hence poll/epoll) there are high performance things where having an event loop using select is a good way to go, but I generally prefer using threads and/or generators to make the code clear (single flow of control, single task for the block of code, etc) if there's no reason not to.

...

Anyhow, I wouldn't be opposed to having a way of looking into the buffer, but it should be a separate API -- preferably with a better name than peek0(), which gives no clue at all about what it does differently from peek().

Indeed, though arguably read1() is a lousy name too, on the same basis. My itch is that peek() _feels_ like it should be "look into the buffer" but actually can block and/or change the buffer.

Can block, but not if you don't want it too. You might just want to see what, if anything, is currently available, up to n bytes.

Cameron Simpson wrote:

On 16Jun2009 02:18, MRAB wrote:

...

...

My itch is that peek() _feels_ like it should be "look into the buffer" but actually can block and/or change the buffer.

Can block, but not if you don't want it too. You might just want to see what, if anything, is currently available, up to n bytes.

Am I missing something?

In the face of an _empty_ buffer (which I can't tell from outside) how do I prevent peek() blocking? More generally, if I go peek(n) and if n > bytes_in_buffer_right_now and the raw stream would block if a raw read is done?

My concerns would go away if I could probe the buffer content size; then I could ensure peek(n) chose n <= the content size. If that's not enough, my problem - I can choose to read-and-block or go away and come back later.

I was thinking along the lines of: def peek(self, size=None, block=True) If 'block' is True then return 'size' bytes, unless the end of the file/stream is reached; if 'block' is False then return up to 'size' bytes, without blocking. The blocking form might impose a limit to how much can be peeked (the maximum size of the buffer), or it might enlarge the buffer as necessary.

MRAB wrote:

I was thinking along the lines of: def peek(self, size=None, block=True) If 'block' is True then return 'size' bytes, unless the end of the file/stream is reached; if 'block' is False then return up to 'size' bytes, without blocking....

I tend to prefer zero-ish defaults, how about: def peek(self, size=None, nonblocking=False) We still don't have "at most one read" code, but something a bit like data = obj.peek(size=desired, nonblocking=True) if len(data) < desired: data = obj.peek(size=wanted, nonblocking=False) might suffice. --Scott David Daniels Scott.Daniels@Acm.Org

Scott David Daniels writes:

MRAB wrote:

...

I was thinking along the lines of: def peek(self, size=None, block=True) If 'block' is True then return 'size' bytes, unless the end of the file/stream is reached; if 'block' is False then return up to 'size' bytes, without blocking....

I tend to prefer zero-ish defaults, how about: def peek(self, size=None, nonblocking=False)

Since blocking and non-blocking are already used to refer to different types of raw streams, another verb should be found for this option. Antoine.

Cameron Simpson wrote:

I normally avoid non-blocking requirements by using threads, so that the thread gathering from the stream can block.

If you have a thread dedicated to reading from that stream, then I don't see why you need to peek into the buffer. Just have it loop reading a packet at a time and put each completed packet in the queue. If several packets arrive at once, it'll loop around that many times before blocking.

arguably read1() is a lousy name too, on the same basis.

Certainly.

My itch is that peek() _feels_ like it should be "look into the buffer" but actually can block and/or change the buffer.

My problem with the idea of looking into the buffer is that it crosses levels of abstraction. A buffered stream is supposed to behave the same way as a deterministic non-buffered stream, with the buffer being an internal optimisation detail that doesn't exist as far as the outside world is concerned. Sometimes it's pragmatic to break the abstraction, but it should be made very obvious when you're doing that. So I'd prefer something like peek_buffer() to make it perfectly clear what's being done. Anything else such as peek() that doesn't explicitly mention the buffer should fit into the abstraction properly. -- Greg

On 17Jun2009 10:55, Greg Ewing wrote:

Cameron Simpson wrote:

...

I normally avoid non-blocking requirements by using threads, so that the thread gathering from the stream can block.

If you have a thread dedicated to reading from that stream, then I don't see why you need to peek into the buffer. Just have it loop reading a packet at a time and put each completed packet in the queue. If several packets arrive at once, it'll loop around that many times before blocking.

Yes, this is true. But people not using threads, or at any rate not dedicating a thread to the reading task, don't have such luxury. Are we disputing the utility of being able to ask "how much might I read/peek without blocking"? Or disputing the purpose of peek, which feels to me like it should/might ask that question, but doesn't. [...]

...

My itch is that peek() _feels_ like it should be "look into the buffer" but actually can block and/or change the buffer.

My problem with the idea of looking into the buffer is that it crosses levels of abstraction. A buffered stream is supposed to behave the same way as a deterministic non-buffered stream, with the buffer being an internal optimisation detail that doesn't exist as far as the outside world is concerned.

Sometimes it's pragmatic to break the abstraction, but it should be made very obvious when you're doing that. So I'd prefer something like peek_buffer() to make it perfectly clear what's being done.

Anything else such as peek() that doesn't explicitly mention the buffer should fit into the abstraction properly.

It's perfectly possible, even reasonable, to avoid talking about the buffer at all. I'd be happy not to mention the buffer. For example, one can readily imagine the buffered stream being capable of querying its input raw stream if there's "available now" data. The raw stream can sometimes know if a read of a given size would block, or be able to ask what size read will not block. As a concrete example, the UNIX FIONREAD ioctl can generally query a file descriptor for instantly-available data (== in the OS buffer). I've also used UNIXen where your can fstat() a pipe and use the st_size field to test for available unread data in the pipe buffer. Raw streams which can't do that would return 0 (== can't guarentee any non-blocking data) unless the stream itself also had a buffer of its own and it wasn't empty. So I would _want_ the spec for available_data() (new lousy name) to talk about "data available without blocking", allowing the implementation to use data in the IO buffer and/or to query the raw stream, etc. Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ For those who understand, NO explanation is needed, for those who don't understand, NO explanation will be given! - Davey D

Even if you don't mention it explicitly, its existence shows through in the fact that there is an arbitrary limit on the amount you can peek ahead, and that limit needs to be documented so that people can write correct programs.

This is true of both kinds of peeking, so I concede that they both break the abstraction.

However I think the non-blocking peek breaks it more than the blocking one, because it also brings non-deterministic behaviour. It depends on the point of view. For example, someone is writing a program that must read from any kind of file descriptor and generate the derivation tree of the text read

Greg Ewing wrote: based on some context-free grammar. The problem is that the chosen method to accomplish it would read 2 symbols (bytes) ahead and this guy is using peek() to grab these 2 bytes. The program will seem to work correctly most of the time, but on the 4095th byte read, he would grab 1 byte at most using peek() (despite the existence of say 10k bytes ahead). The blocking definition of peek() would create this hard-to-spot bug. Thus, for him, this behavior would seem non-deterministic. On the other hand, if someone is conscious about the number of raw reads, he would definitely be willing to look into the documentation for any parameters that match his special needs. That's why the non-blocking behavior should be the default one while the blocking behavior should be accessible by choice.

Greg Ewing wrote:

That's exactly why I think the blocking version should keep reading until the requested number of bytes is available (or the buffer is full or EOF occurs). Do you mean that the blocking version should keep waiting for new bytes until they show up? This would not be acceptable, since the program would hang forever most of the time (no changes to the buffer would ever occur in this situation when there's only the main thread running).

Am I understanding this correctly: * The blocking version would not do any raw reads. * The non-blocking version would do.

Frederick Reeve writes:

peek(n): If n is less than 0, None, or not set; return buffer contents with out advancing stream position. If the buffer is empty read a full chunk and return the buffer. Otherwise return exactly n bytes up to _chunk size_(not contents) with out advancing the stream position. If the buffer contents is less than n, buffer an additional chunk from the "raw" stream before hand. If EOF is encountered during any raw read then return as much as we can up to n. (maybe I should write that in code form??)

This proposal looks reasonable to me. Please note that it's too late for 3.1 anyway - we're in release candidate phase. Once you have a patch, you can post it on the bug tracker. Regards Antoine.