On Thu, Oct 16, 2014 at 4:34 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:

On Thu, 16 Oct 2014 03:54:32 +0300 Paul Sokolovsky <pmiscml@gmail.com> wrote:

...

Hello,

io.RawIOBase.read() is well specified for behavior in case it immediately gets a would-block condition: "If the object is in non-blocking mode and no bytes are available, None is returned." (https://docs.python.org/3/library/io.html#io.RawIOBase.read).

However, nothing is said about such condition for io.IOBase.readline(), which is mixin method in a base class, default implementation of which thus would use io.RawIOBase.read(). Looking at 3.4.0 source, iobase.c: iobase_readline() has:

b = _PyObject_CallMethodId(self, &PyId_read, "n", nreadahead); [...] if (!PyBytes_Check(b)) { PyErr_Format(PyExc_IOError, "read() should have returned a bytes object, " "not '%.200s'", Py_TYPE(b)->tp_name);

I.e. it's not even ready to receive legitimate return value of None from read(). I didn't try to write a testcase though, so may be missing something.

So, how readline() should behave in this case, and can that be specified in the Library Reference?

Well, the problem is that it's not obvious how to implement such methods in a non-blocking context.

Let's says some data is received but there isn't a complete line. Should readline() return just that data (an incomplete line)? That breaks the API's contract. Should readline() buffer the incomplete line and keep it for the next readline() call? But then the internal buffer becomes unbounded: perhaps there is no new line in the next 4GB of incoming data...

And besides, raw I/O objects *shouldn't* have an internal buffer. That's the role of the buffered I/O layer.

Well, occasionally this occurs, and I think it's reasonable for readline() to deal with it. The argument about a 4 GB buffer is irrelevant -- this can happen with a blocking underlying stream too. I think that at the point where the readline() function says to itself "I need more data" it should ask the underlying stream for data. If that returns an empty string, meaning EOF, readline() is satisfied and return whatever it has buffered (even if it's empty). If that returns some bytes containing a newline, readline() is satisfied, returns the data up to that point, and buffers the rest (if any). If the underlying stream returns None, I think it makes sense for readline() to return None too -- attempting to read more will just turn into a busy-wait loop, and that's the opposite of what should happen. You may argue that the caller of readline() doesn't expect this. Sure. But in the end, if the stream is unbuffered and the caller isn't prepared for that, the caller will always get in trouble. Maybe it'll treat the None as EOF. That's fine -- it would be the same if it was calling read() on the underlying stream and it got None (the EOF signalling is the same in both cases). At least, by being prepared for the None from the underlying read() in the readline() code, someone who knows what they are doing can use readline() on a non-blocking stream -- when they receive None they will have to ask their selector (or whatever they use) to wait for the underlying FD and then they can try again. (Alternatively, we could raise BlockingIOError, which is that the OS level read() raises if there's no data immediately available on a non-blocking FD; but it seems that streams have already gotten a convention of returning None instead, so I think that should be propagated up the stack.) Oh, BTW, I tested this a little bit. Currently readline() returns an empty string (or empty bytes, depending on which level you use) when the stream is nonblocking. I think returning None makes muck more sense. -- --Guido van Rossum (python.org/~guido)

Hello, On Thu, 16 Oct 2014 13:34:06 +0200 Antoine Pitrou <solipsis@pitrou.net> wrote:

On Thu, 16 Oct 2014 03:54:32 +0300 Paul Sokolovsky <pmiscml@gmail.com> wrote:

...

Hello,

io.RawIOBase.read() is well specified for behavior in case it immediately gets a would-block condition: "If the object is in non-blocking mode and no bytes are available, None is returned." (https://docs.python.org/3/library/io.html#io.RawIOBase.read).

However, nothing is said about such condition for io.IOBase.readline(), which is mixin method in a base class, default implementation of which thus would use io.RawIOBase.read(). Looking at 3.4.0 source, iobase.c: iobase_readline() has:

b = _PyObject_CallMethodId(self, &PyId_read, "n", nreadahead); [...] if (!PyBytes_Check(b)) { PyErr_Format(PyExc_IOError, "read() should have returned a bytes object, " "not '%.200s'", Py_TYPE(b)->tp_name);

I.e. it's not even ready to receive legitimate return value of None from read(). I didn't try to write a testcase though, so may be missing something.

So, how readline() should behave in this case, and can that be specified in the Library Reference?

Well, the problem is that it's not obvious how to implement such methods in a non-blocking context.

Let's says some data is received but there isn't a complete line. Should readline() return just that data (an incomplete line)? That breaks the API's contract. Should readline() buffer the incomplete line and keep it for the next readline() call? But then the internal buffer becomes unbounded: perhaps there is no new line in the next 4GB of incoming data...

And besides, raw I/O objects *shouldn't* have an internal buffer. That's the role of the buffered I/O layer.

Yes, sure, readline() is defined on io.IOBase which is underspecified for buffered-ness, so should have behavior which can be implemented for both buffered and unbuffered case. You're right also in saying that readline on non-blocking stream can't work always the same way as blocking version, and that it "breaks the API's contract". But it should be possible to extend that contract for non-blocking readline() in pretty natural way: 1) An invariant of readline() is that it doesn't modify stream data, it just segments it. So, readline() + write() looped until EOF will produce the same result as read(N) + write(). Non-blocking readline() will still satisfy this. 2) Even with blocking readline(), it can return a string not ending with end-of-line character(s). For blocking readline, this may happen with just the last line of a stream, with non-blocking, it may happen for any call. The point is that even with blocking readline(), the caller should be ready to check that a line satisfies its "complete line" criteria, for non-blocking case, it's just will be different set of criteria and actions to satisfy them. I guess, defining non-blocking readline() in such way is better then let it be underspecified whether it's supported or not (and if yes, then how), or prohibit it. -- Best regards, Paul mailto:pmiscml@gmail.com