Program very slow to finish

Emile van Sebille emile at
Sat Nov 3 17:49:51 CET 2001

"Fred" <fredNo at> wrote in message
news:3BE41CBE.94AED22 at
> I'm dealing with about 100Gb of data that I first just need to
> characterize.  So, since the slowest part will be simply reading the
> data, I'm testing various languages and methods on a 10 line subset, a
> 32 Mb subset and a 750 Mb subset.
> The following python program prints out the results in about 30 seconds,
> however it doesn't finish for another minute with the 32 Mb set of
> data!  At first I thought it was stuck and killed it; however I finally
> let it run on the smaller data set and all was well.
> Is this a garbage collection issue?  Is there a better way to count the
> individual values than dictionaries?  I put the sys.exit call in while
> trying to figure out what was happening but it didn't make a difference.

I'd suspect that python is cleaning house before exiting.  Try exiting with
os._exit or os.abort.


Emile van Sebille
emile at


More information about the Python-list mailing list