lifting of prohibition against readlines inside a "for line in file" in Py3?
In Digest Vol. 67, Issue 52 (13 Feb 2009) I pointed out that Python 2's prohibition against performing readlines on a file being iterated over appears to have been lifted in Python 3. I asked if this was intentional and whether it should be add to the "What's New" documentation. I also expressed muy surprise that "for line in fil"'s can be nested using the same fil in both Python 2 and 3. I presented an example for each point and some and further comments. I didn't get any response. Is this the wrong list for the question? Did appropriate responders assume another would respond? I want to reraise this because lifting of that prohibition is a quite significant change in the behavior from Python 2. Even if it ws a bug in Python 2, or the side effect of other changes in Python 3, it should at least be documented prominently. True, no-one's code will be affected by lifting a prohibition against something their code couldn't have done, but the new behavior offers significant flexibility in writing "for line in fil" iterations in that it allows recognizing the beginning of a sequence of lines that should be considered a unified "chunk" and allows the loop to do readlines to handle the rest of the chunk. Some of these can be handled by just nesting a second "for line in fil" inside a conditional inside the outer iteration but some are better handled by individual readlines. I'd appreciate comments -- especially a redirection to a different list, if this one isn't appropriate for my query.
Mitchell L Model wrote:
I didn't get any response. Is this the wrong list for the question? Did appropriate responders assume another would respond?
Probably the latter (I know I left it to those that had more to do with the IO stack rewrite). This is definitely the right list for the question.
I want to reraise this because lifting of that prohibition is a quite significant change in the behavior from Python 2. Even if it ws a bug in Python 2, or the side effect of other changes in Python 3, it should at least be documented prominently. True, no-one's code will be affected by lifting a prohibition against something their code couldn't have done, but the new behavior offers significant flexibility in writing "for line in fil" iterations in that it allows recognizing the beginning of a sequence of lines that should be considered a unified "chunk" and allows the loop to do readlines to handle the rest of the chunk. Some of these can be handled by just nesting a second "for line in fil" inside a conditional inside the outer iteration but some are better handled by individual readlines.
I'd appreciate comments -- especially a redirection to a different list, if this one isn't appropriate for my query.
I *think* the 2.x system had an internal buffer that was used by the file iterator, but not by the file methods. With the new IO stack for 3.0, there is now a common buffer shared by all the file operations (including iteration). However, given that the lifting of the restriction is currently undocumented, I wouldn't want to see a commitment to keeping it lifted until we know that it won't cause any problems for the io-in-c rewrite for 3.1 (hopefully someone with more direct involvement with that rewrite will chime in, since they'll know a lot more about it than I do). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Hello, Nick Coghlan <ncoghlan <at> gmail.com> writes:
I *think* the 2.x system had an internal buffer that was used by the file iterator, but not by the file methods. With the new IO stack for 3.0, there is now a common buffer shared by all the file operations (including iteration).
However, given that the lifting of the restriction is currently undocumented, I wouldn't want to see a commitment to keeping it lifted until we know that it won't cause any problems for the io-in-c rewrite for 3.1 (hopefully someone with more direct involvement with that rewrite will chime in, since they'll know a lot more about it than I do).
As you said, there is no special buffering for the file iterator in 3.x, which means the restriction could be lifted (actually there is nothing relying on this restriction in the current code, except perhaps the "telling" flag in TextIOWrapper). Regards Antoine.
On Wed, 18 Feb 2009 at 21:25, Antoine Pitrou wrote:
Nick Coghlan <ncoghlan <at> gmail.com> writes:
I *think* the 2.x system had an internal buffer that was used by the file iterator, but not by the file methods. With the new IO stack for 3.0, there is now a common buffer shared by all the file operations (including iteration).
However, given that the lifting of the restriction is currently undocumented, I wouldn't want to see a commitment to keeping it lifted until we know that it won't cause any problems for the io-in-c rewrite for 3.1 (hopefully someone with more direct involvement with that rewrite will chime in, since they'll know a lot more about it than I do).
As you said, there is no special buffering for the file iterator in 3.x, which means the restriction could be lifted (actually there is nothing relying on this restriction in the current code, except perhaps the "telling" flag in TextIOWrapper).
Currently I have python (2.x) code that uses 'readline' instead of 'for x in myfile' in order to avoid the 'for' buffering (ie: be presented with the next line as soon as it is written to the other end of a pipe, instead of waiting for the buffer to fill). Does "no special buffering" mean that 'for' doesn't use a read-ahead buffer in p3k, or that readline does use such a buffer, because the latter could make my code break unexpectedly when porting to p3k. --RDM
On Wed, Feb 18, 2009 at 6:38 PM, <rdmurray@bitdance.com> wrote:
On Wed, 18 Feb 2009 at 21:25, Antoine Pitrou wrote:
Nick Coghlan <ncoghlan <at> gmail.com> writes:
I *think* the 2.x system had an internal buffer that was used by the file iterator, but not by the file methods. With the new IO stack for 3.0, there is now a common buffer shared by all the file operations (including iteration).
However, given that the lifting of the restriction is currently undocumented, I wouldn't want to see a commitment to keeping it lifted until we know that it won't cause any problems for the io-in-c rewrite for 3.1 (hopefully someone with more direct involvement with that rewrite will chime in, since they'll know a lot more about it than I do).
As you said, there is no special buffering for the file iterator in 3.x, which means the restriction could be lifted (actually there is nothing relying on this restriction in the current code, except perhaps the "telling" flag in TextIOWrapper).
Currently I have python (2.x) code that uses 'readline' instead of 'for x in myfile' in order to avoid the 'for' buffering (ie: be presented with the next line as soon as it is written to the other end of a pipe, instead of waiting for the buffer to fill). Does "no special buffering" mean that 'for' doesn't use a read-ahead buffer in p3k, or that readline does use such a buffer, because the latter could make my code break unexpectedly when porting to p3k.
Have a look at the code in io.py (class TextIOWrapper): http://svn.python.org/view/python/branches/py3k/Lib/io.py?view=log I believe it doesn't force reading ahead more than necessary. If a single low-level read() returns enough data to satisfy the __next__() or readline() (or it can be satisfied from data already buffered) then it won't force reading more. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
On Wed, 18 Feb 2009 at 20:31, Guido van Rossum wrote:
On Wed, Feb 18, 2009 at 6:38 PM, <rdmurray@bitdance.com> wrote:
On Wed, 18 Feb 2009 at 21:25, Antoine Pitrou wrote:
Nick Coghlan <ncoghlan <at> gmail.com> writes:
I *think* the 2.x system had an internal buffer that was used by the file iterator, but not by the file methods. With the new IO stack for 3.0, there is now a common buffer shared by all the file operations (including iteration).
However, given that the lifting of the restriction is currently undocumented, I wouldn't want to see a commitment to keeping it lifted until we know that it won't cause any problems for the io-in-c rewrite for 3.1 (hopefully someone with more direct involvement with that rewrite will chime in, since they'll know a lot more about it than I do).
As you said, there is no special buffering for the file iterator in 3.x, which means the restriction could be lifted (actually there is nothing relying on this restriction in the current code, except perhaps the "telling" flag in TextIOWrapper).
Currently I have python (2.x) code that uses 'readline' instead of 'for x in myfile' in order to avoid the 'for' buffering (ie: be presented with the next line as soon as it is written to the other end of a pipe, instead of waiting for the buffer to fill). Does "no special buffering" mean that 'for' doesn't use a read-ahead buffer in p3k, or that readline does use such a buffer, because the latter could make my code break unexpectedly when porting to p3k.
Have a look at the code in io.py (class TextIOWrapper):
http://svn.python.org/view/python/branches/py3k/Lib/io.py?view=log
I believe it doesn't force reading ahead more than necessary. If a single low-level read() returns enough data to satisfy the __next__() or readline() (or it can be satisfied from data already buffered) then it won't force reading more.
Hmm. I'm not sure I'm reading the code right, but it looks from the docstrings like TextIOWrapper expects to read from a BufferedIOBase object, whose doc string contains this comment: If the argument is positive, and the underlying raw stream is not 'interactive', multiple raw reads may be issued to satisfy the byte count (unless EOF is reached first). But for interactive raw streams (XXX and for pipes?), at most one raw read will be issued, and a short result does not imply that EOF is imminent. Since the 'pipe' comment is an XXX, it is not clear that my use case is covered. However, the actual implementation of readinto seems to only call 'read' once, so as long as the 'read' of the subclass returns whatever bytes are available, then it looks good to me :) Since TextIOWrapper is careful to call 'read1' on the wrapped buffer object, and the one place that 'read1' has a docstring clearly indicates that it does at most one read and returns whatever data is ready, it seems that the _intent_ of the code is as you expressed. I'm a python programmer first, and my C is pretty rusty, so I'm not sure if I'm up to looking through the new C code to see how this got translated. I'm thinking that both my use case (and in my case 'for' should now work for me) and the OP's are the way it is intended to work, but documentation of this seems like it would be a good idea. Since the OP doesn't seem to have opened a ticket, I did so: http://bugs.python.org/issue5323. As I said there, I'm willing to work on doc and test patches if this is the behavior the io library is required to have in 3.x. --RDM
Hello, <rdmurray <at> bitdance.com> writes:
Since the 'pipe' comment is an XXX, it is not clear that my use case is covered. However, the actual implementation of readinto seems to only call 'read' once, so as long as the 'read' of the subclass returns whatever bytes are available, then it looks good to me :)
The only thing I can say is that it should work as you expect it to. But it would be great if you could give it a try and share your observations with us. We haven't had many people report on real-world uses of the new IO library (although we could optimistically deduce from it that people aren't having any problems, except for the speed issue). Regards Antoine.
Mitchell L Model wrote:
In Digest Vol. 67, Issue 52 (13 Feb 2009) I pointed out that Python 2's prohibition against performing readlines on a file being iterated over appears to have been lifted in Python 3. I asked if this was intentional and whether it should be add to the "What's New" documentation. I also expressed muy surprise that "for line in fil"'s can be nested using the same fil in both Python 2 and 3. I presented an example for each point and some and further comments.
If I understand, you are asking whether the new behavior is an undocumented new feature that you can use or an error that may be reverted. It is possible that it is an accident and there is no answer yet decided. I suspect your original query got lost in the shuffle. If you do not get an answer this time, file an issue on the tracker bugs.python.org but do not select whether it is a behavior or doc issue. At least, it will stay open until resolved. tjr
Terry Reedy wrote:
I suspect your original query got lost in the shuffle. If you do not get an answer this time, file an issue on the tracker bugs.python.org but do not select whether it is a behavior or doc issue. At least, it will stay open until resolved.
Filing a tracker issue is probably a good idea regardless. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
participants (6)
-
Antoine Pitrou -
Guido van Rossum -
Mitchell L Model -
Nick Coghlan -
rdmurray＠bitdance.com -
Terry Reedy