[Python-Dev] lifting of prohibition against readlines inside a "for line in file" in Py3?

[Guido van Rossum]

On Wed, Feb 18, 2009 at 6:38 PM,  <rdmurray at bitdance.com> wrote:
> On Wed, 18 Feb 2009 at 21:25, Antoine Pitrou wrote:
>>
>> Nick Coghlan <ncoghlan <at> gmail.com> writes:
>>>
>>> I *think* the 2.x system had an internal buffer that was used by the
>>> file iterator, but not by the file methods. With the new IO stack for
>>> 3.0, there is now a common buffer shared by all the file operations
>>> (including iteration).
>>>
>>> However, given that the lifting of the restriction is currently
>>> undocumented, I wouldn't want to see a commitment to keeping it lifted
>>> until we know that it won't cause any problems for the io-in-c rewrite
>>> for 3.1 (hopefully someone with more direct involvement with that
>>> rewrite will chime in, since they'll know a lot more about it than I do).
>>
>> As you said, there is no special buffering for the file iterator in 3.x,
>> which
>> means the restriction could be lifted (actually there is nothing relying
>> on this
>> restriction in the current code, except perhaps the "telling" flag in
>> TextIOWrapper).
>
> Currently I have python (2.x) code that uses 'readline' instead of 'for
> x in myfile' in order to avoid the 'for' buffering (ie: be presented
> with the next line as soon as it is written to the other end of a pipe,
> instead of waiting for the buffer to fill).  Does "no special buffering"
> mean that 'for' doesn't use a read-ahead buffer in p3k, or that readline
> does use such a buffer, because the latter could make my code break
> unexpectedly when porting to p3k.

Have a look at the code in io.py (class TextIOWrapper):

http://svn.python.org/view/python/branches/py3k/Lib/io.py?view=log

I believe it doesn't force reading ahead more than necessary. If a
single low-level read() returns enough data to satisfy the __next__()
or readline() (or it can be satisfied from data already buffered) then
it won't force reading more.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)