speed question, reading csv using takewhile() and dropwhile()

[John Posner]

On 2/19/2010 3:02 PM, MRAB wrote:
> Is this any better?
>
> def read_data_file(filename):
>     reader = csv.reader(open(filename, "U"),delimiter='\t')
>     data = []
>     for row in reader:
>         if '[MASKS]' in row:
>             break
>         data.append(row)

As noted in another thread recently, you can save time by *not* looking 
up the "append" method of the list object each time through the FOR loop:

      data = []
      app_method = data.append
      for row in reader:
          if '[MASKS]' in row:
              break
          app_method(row)

Similarly in the rest of the code. This technique improved performance 
about 31% in this test:

#--------------------
import timeit
tt = timeit.repeat("for i in xrange(1000000): mylist.append(i)",
                    "mylist=[]",
                    number=25)
print "look up append() method each time:", min(tt)

tt = timeit.repeat("for i in xrange(1000000): app(i)",
                    "mylist=[]; app = mylist.append",
                    number=25)
print "look up append() method just once:", min(tt)
#--------------------

output:

look up append() method each time: 8.45481741783
look up append() method just once: 5.84429637887

-John