speeding up reading files (possibly with cython)

Carl Banks pavlovevidence at gmail.com
Sun Mar 8 12:16:17 CET 2009

On Mar 7, 3:06 pm, per <perfr... at gmail.com> wrote:
> hi all,
> i have a program that essentially loops through a textfile file thats
> about 800 MB in size containing tab separated data... my program
> parses this file and stores its fields in a dictionary of lists.

When building a very large structure like you're doing, the cyclic
garbage collector can be a bottleneck.  Try disabling the cyclic
garbage collector before building the large dictionary, and re-
enabling it afterwards.

import gc
    for line in file:
        split_values = line.strip().split('\t')
        # do stuff with split_values

Carl Banks

More information about the Python-list mailing list