[Tutor] f.readlines(size)

[Cameron Simpson]

On 05Jun2017 21:04, Nancy Pham-Nguyen <nancyphng at sbcglobal.net> wrote:
>I'm trying to understand the optional size argument in file.readlines method. 
>The help(file) shows:
> |  readlines(...) |      readlines([size]) -> list of strings, each a line 
>from the file. |       |      Call readline() repeatedly and return a list of 
>the lines so read. |      The optional size argument, if given, is an 
>approximate bound on the |      total number of bytes in the lines returned.
>From the documentation:f.readlines() returns a list containing all the lines of data in the file.
>If given an optional parameter sizehint, it reads that many bytes from the 
>file
>and enough more to complete a line, and returns the lines from that.
>This is often used to allow efficient reading of a large file by lines,
>but without having to load the entire file in memory. Only complete lines
>will be returned.
>I wrote the function below to try it, thinking that it would print multiple times, 3 lines at a time, but it printed all in one shot, just like when I din't specify the optional argument. Could someone explain what I've missed? See input file and output below.

I'm using this to test:

  from __future__ import print_function
  import sys
  lines = sys.stdin.readlines(1023)
  print(len(lines))
  print(sum(len(_) for _ in lines))
  print(repr(lines))

I've fed it a 41760 byte input (the size isn't important except that it needs 
to be "big enough"). The output starts like this:

  270
  8243

and then the line listing. That 8243 looks interesting, being close to 8192, a 
power of 2. The documentation you quote says:

  The optional size argument, if given, is an approximate bound on the total 
  number of bytes in the lines returned. [...] it reads that many bytes from 
  the file and enough more to complete a line, and returns the lines from that.

It looks to me like readlines uses the sizehint somewhat liberally; the purpose 
as described in the doco is to read input efficiently without using an 
unbounded amount of memory. Imagine feeding readlines() a terabyte input file, 
without the sizehint. It would try to pull it all into memory. With the 
sizehint you get a simple form of batching of the input into smallish groups of 
lines.

I would say, from my experiments here, that the underlying I/O is doing 8192 
byte reads from the file as the default buffer. So although I've asked for 1023 
bytes, readlines says something like: I want at least 1023 bytes; the I/O 
system loads 8192 bytes because that is its normal read size, then readlines 
picks up all the buffer. It does this so as to gather as many lines as readily 
available. It then asks for more data to complete the last line. The last line 
of my readlines() result is:

  %.class: %.java %.class-prereqs : $(("%.class-prereqs" G?<P)).class

which is 68 bytes long including the newline character. 8192 + 68 = 8260, just 
over the 8243 bytes of "complete lines" I got back.

So this sizehint is just a clue, and does not change the behaviour of the 
underlying I/O. It just prevents readlines() reading the entire file.

If you want tighter control, may I suggest iterating over the file like this:

  for line in sys.stdin:
    ... do stuff with the line ...

This also does not change the underlying I/O buffer size, but it does let you 
gather only the lines you want: you can count line lengths or numbers or 
whatever criteria you find useful if you want to stop be fore the end of the 
file.

Cheers,
Cameron Simpson <cs at zip.com.au>