Program very slow to finish
fredNo at nospamco.com
Sun Nov 4 21:49:16 CET 2001
Good call. os_exit(0) gives:
$ time python unitest.py < 841123.dat
Total lines read: 316480
B: 158229 Value: 20191822207L Max/Min: 9101400 12
S: 158251 Value: 20193690457L Max/Min: 9101400 12
Number of acc: 56568
Number of tra: 64704
Number of stk: 388
Emile van Sebille wrote:
> "Fred" <fredNo at nospamco.com> wrote in message
> news:3BE41CBE.94AED22 at nospamco.com...
> > I'm dealing with about 100Gb of data that I first just need to
> > characterize. So, since the slowest part will be simply reading the
> > data, I'm testing various languages and methods on a 10 line subset, a
> > 32 Mb subset and a 750 Mb subset.
> > The following python program prints out the results in about 30 seconds,
> > however it doesn't finish for another minute with the 32 Mb set of
> > data! At first I thought it was stuck and killed it; however I finally
> > let it run on the smaller data set and all was well.
> > Is this a garbage collection issue? Is there a better way to count the
> > individual values than dictionaries? I put the sys.exit call in while
> > trying to figure out what was happening but it didn't make a difference.
> I'd suspect that python is cleaning house before exiting. Try exiting with
> os._exit or os.abort.
> Emile van Sebille
> emile at fenx.com
More information about the Python-list