[Tutor] list comprehension, testing for multiple conditions

[Pete O'Connell]

Thanks Peter. This looks like what I need:

with open(fileName) as lines:
    wanted = [line.strip("\n") for line in lines if "vn" not in line
and "vt" not in line and line != "\n"]

Cheers

And in response to Allan's suggestion. I can see using a generator in
a situation where the if statements were more numerous and complex. I
am sure that will come in handy.

Thanks


On Wed, Aug 22, 2012 at 7:06 PM, Peter Otten <__peter__ at web.de> wrote:
> Pete O'Connell wrote:
>
>> Hi I am trying to parse a text file and create a list of all the lines
>> that don't include: "vn", "vt" or are empty. I want to make this as
>> fast as possible because I will be parsing many files each containing
>> thousands of lines. I though I would give list comprehensions a try.
>> The last 3 lines of the code below have three list comprehensions that
>> I would like to combine into 1 but I am not sure how to do that.
>> Any tips would be greatly appreciated
>>
>> pete
>>
>> #start############################################################
>> fileName = '/usr/home/poconnell/Desktop/objCube.obj'
>> theFileOpened = open(fileName,'r')
>> theTextAsList = theFileOpened.readlines()
>
> If you have a file with 1,000,000 lines you have now a list of 1,000,000
> strings of which perhaps 1,000 match your criteria. You are squandering
> memory. Rule of thumb: never use readlines(), iterate over the file
> directly.
>
>> theTextAsListStripped = []
>> for aLine in theTextAsList:
>>
>>     theTextAsListStripped.append(aLine.strip("\n"))
>>
>> theTextAsListNoVn = [x for x in theTextAsListStripped if "vn" not in x]
>> theTextAsListNoVnOrVt = [x for x in theTextAsListNoVn if "vt" not in x]
>> theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListNoVn if x !=
>> ""]
>
> I think that should be
>
> theTextAsListNoVnOrVtOrEmptyLine = [x for x in theTextAsListNoVnOrVt if x !=
> ""]
>
> You can combine the three if clauses or add them all to one list-comp:
>
> with open(filename) as lines:
>     wanted = [line.strip("\n") for line in lines
>                   if "vn" not in line and "vt" not in line and line != "\n"]
>
>
> You can even have multiple if clauses in one list-comp (but that is rarely
> used):
>
> with open(filename) as lines:
>     wanted = [line.strip("\n") for line
>                   if "vn" not in line
>                   if "vt" not in x
>                   if line != "\n"]
>
> While your problem is simple enough to combine all filters into one list-
> comp some problems are not. You can then prevent the intermediate lists from
> materializing by using generator expressions. The result minimizes memory
> consumption, too, and should be (almost) as fast. For example:
>
> with open(filename) as lines:
>     # use gen-exps to remove empty and whitespace-only lines
>     stripped = (line.strip() for line in lines)
>     nonempty = (line for line in stripped if line)
>
>     wanted = [line for line in nonempty
>                   if "vt" not in line and "vn" not in line]
>
>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor



-- 
-