memory usage multi value hash

Terry Reedy tjreedy at udel.edu
Thu Apr 14 19:28:47 CEST 2011


On 4/14/2011 12:55 PM, Peter Otten wrote:

> I don't expect that it matters much, but you don't need to sort your data if
> you use a dictionary anyway:

Which means that one can build the dict line by line, as each is read, 
instead of reading the entire file into memory. So it does matter for 
intermediate memory use.

> import csv
> import sys
>
> infile, outfile = sys.argv[1:]
>
> d = {}
> with open(infile, "rb") as instream:
>      for key, value in csv.reader(instream, delimiter=';'):
>          d.setdefault(key, [key]).append(value)
>
> with open(outfile, "wb") as outstream:
>      csv.writer(outstream).writerows(d.itervalues())


-- 
Terry Jan Reedy




More information about the Python-list mailing list