[Tutor] search list with regex

[Steven D'Aprano]

Alan Gauld wrote:
> 
> "Elwin Estle" <chrysalis_reborn at yahoo.com> wrote
> 
>> parse various text files and my standard method is to
>> slurp the whole thing up into a string variable, then
>> break it up into a list that I can then work on
> 
> If you read it with readlines() Python will do all of
> that for you...


Very true, and it's a good tool to have... but an even better tool to 
have in your tool box is to learn about lazy processing. Often, you 
don't need to slurp the whole file into one big list. Any time you can 
process each line independently of the others, you should consider lazy 
processing: read one line, process it, and move on to the next.

Instead of:

fp = open(filename)
lines = fp.lines()  # could be huge!
for line in lines:
     process(line)


this is faster and more efficient:

fp = open(filename)
for line in fp:
     process(line)


It doesn't read all the lines in one go, so it can handle much bigger files.



-- 
Steven