speeding up reading files (possibly with cython)

S Arrowsmith sion at paintbox.UUCP
Mon Mar 9 14:55:57 CET 2009

Carl Banks  <pavlovevidence at gmail.com> wrote:
>When building a very large structure like you're doing, the cyclic
>garbage collector can be a bottleneck.  Try disabling the cyclic
>garbage collector before building the large dictionary, and re-
>enabling it afterwards.
>import gc
>    for line in file:
>        split_values =3D line.strip().split('\t')
>        # do stuff with split_values
>    gc.enable()

Completely untested, but if you find yourself doing that a lot,

import gc
from contextlib import contextmanager

def no_gc():

with no_gc():
     for line in file:
	 # ... etc.

be worth considering?


   under construction

More information about the Python-list mailing list