speeding up reading files (possibly with cython)

[S Arrowsmith]

Carl Banks  <pavlovevidence at gmail.com> wrote:
>When building a very large structure like you're doing, the cyclic
>garbage collector can be a bottleneck.  Try disabling the cyclic
>garbage collector before building the large dictionary, and re-
>enabling it afterwards.
>
>import gc
>gc.disable()
>try:
>    for line in file:
>        split_values =3D line.strip().split('\t')
>        # do stuff with split_values
>finally:
>    gc.enable()

Completely untested, but if you find yourself doing that a lot,
might:

import gc
from contextlib import contextmanager

@contextmanager
def no_gc():
    gc.disable()
    yield
    gc.enable()

with no_gc():
     for line in file:
	 # ... etc.

be worth considering?

-- 
\S

   under construction