[Tutor] f.readlines(size)

[Nancy Pham-Nguyen]

Hi Peter,
Thanks a lot for pointing out the itertools.islice, itertools.izip_longest, and experimenting with readlines(size).I'm using Python 2 (you got me, a newbie learning readlines from Google's Python page).
I used the itertools.islice and it worked like a charm. izip requires so many arguments if you want to read multiple lines at a time (for a large file).
I notice now that my file was too small and so readlines(size) read the whole file. I increased my file size to 20,000 of those lines and it did read chunk of lines. The buffersize doesn't work the way it's explained but on some approximate bounds as you wrote.On my system, the number of lines read doesn't change for buffer size up to 8,192 (~2**8), same number of lines was read for buffer sizes between 8,193 and 16,374, then 16,375 to ....  E.g. If the size I specified is in one of the range/bound, a certain number of lines will be read.
Nancy

      From: Peter Otten <__peter__ at web.de>
 To: tutor at python.org 
 Sent: Tuesday, June 6, 2017 12:36 AM
 Subject: Re: [Tutor] f.readlines(size)
   
Nancy Pham-Nguyen wrote:

> Hi,

Hi Nancy, the only justification for the readlines() method is to serve as a 
trap to trick newbies into writing scripts that consume more memory than 
necessary. While the size argument offers a way around that, there are still 
next to no use cases for readlines.

Iterating over a file directly is a very common operation and a lot of work 
to make it efficient was spent on it. Use it whenever possible.

To read groups of lines consider

# last chunk may be shorter
with open(FILENAME) as f:
    while True:
        chunk = list(itertools.islice(f, 3))
        if not chunk:
            break
        process_lines(chunk)

or 

# last chunk may be filled with None values
with open(FILENAME) as f:
    for chunk in itertools.zip_longest(f, f, f): # Py2: izip_longest
        process_lines(chunk)

In both cases you will get chunks of three lines, the only difference being 
the handling of the last chunk.

> I'm trying to understand the optional size argument in file.readlines
> method. The help(file) shows: |  readlines(...) |      readlines([size])
> -> list of strings, each a line from the file. |      |      Call
> readline() repeatedly and return a list of the lines so read. |      The
> optional size argument, if given, is an approximate bound on the |    
> total number of bytes in the lines returned. From the
> documentation:f.readlines() returns a list containing all the lines of
> data in the file. If given an optional parameter sizehint, it reads that
> many bytes from the file and enough more to complete a line, and returns
> the lines from that. This is often used to allow efficient reading of a
> large file by lines, but without having to load the entire file in memory.
> Only complete lines will be returned. I wrote the function below to try
> it, thinking that it would print multiple times, 3 lines at a time, but it
> printed all in one shot, just like when I din't specify the optional
> argument. Could someone explain what I've missed? See input file and
> output below. Thanks,Nancy 

> def readLinesWithSize():
>    # bufsize = 65536
>    bufsize = 45      
>    with open('input.txt') as f:        while True:        
>        # print len(f.readlines(bufsize))  # this will print 33          
> print            
> lines = f.readlines(bufsize)            print lines    
>        if not lines:                break            for line in lines:
>                pass      readLinesWithSize() Output:

This seems to be messed up a little by a "helpful" email client. Therefore 
I'll give my own:

$ cat readlines_demo.py
LINESIZE=32
with open("tmp.txt", "w") as f:
    for i in range(30):
        f.write("{:02} {}\n".format(i, "x"*(LINESIZE-4)))

BUFSIZE = LINESIZE*3-1
print("bufsize", BUFSIZE)

with open("tmp.txt", "r") as f:
    while True:
        chunk = f.readlines(BUFSIZE)
        if not chunk:
            break
        print(sum(map(len, chunk)), "bytes:", chunk)
$ python3 readlines_demo.py
bufsize 95
96 bytes: ['00 xxxxxxxxxxxxxxxxxxxxxxxxxxxx\n', '01 
xxxxxxxxxxxxxxxxxxxxxxxxxxxx\n', '02 xxxxxxxxxxxxxxxxxxxxxxxxxxxx\n']
96 bytes: ['03 xxxxxxxxxxxxxxxxxxxxxxxxxxxx\n', '04 
xxxxxxxxxxxxxxxxxxxxxxxxxxxx\n', '05 xxxxxxxxxxxxxxxxxxxxxxxxxxxx\n']
96 bytes: ['06 xxxxxxxxxxxxxxxxxxxxxxxxxxxx\n', '07 
xxxxxxxxxxxxxxxxxxxxxxxxxxxx\n', '08 xxxxxxxxxxxxxxxxxxxxxxxxxxxx\n']
...

So in Python 3 this does what you expect, readlines() stops collecting more 
lines once the total number of bytes exceeds those specified.

"""
readlines(...) method of _io.TextIOWrapper instance
    Return a list of lines from the stream.
    
    hint can be specified to control the number of lines read: no more
    lines will be read if the total size (in bytes/characters) of all
    lines so far exceeds hint.
"""

In Python 2 the docstring is a little vague

"""
The optional size argument, if given, is an *approximate* *bound* on the
total number of bytes in the lines returned.
"""

(emphasis mine) and it seems that small size values which defeat the goal of 
making the operation efficient are ignored:

$ python readlines_demo.py
('bufsize', 95)
(960, 'bytes:', ['00 xxxxxxxxxxxxxxxxxxxxxxxxxxxx\n', '01 
xxxxxxxxxxxxxxxxxxxxxxxxxxxx\n', '28 xxxxxxxxxxxxxxxxxxxxxxxxxxxx\n', '29
...
 xxxxxxxxxxxxxxxxxxxxxxxxxxxx\n'])

Playing around a bit on my system the minimum value with an effect seems to 
be about 2**13, but I haven't consulted the readlines source code to verify.

_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor