memory usage multi value hash

Terry Reedy tjreedy at
Thu Apr 14 13:28:47 EDT 2011

On 4/14/2011 12:55 PM, Peter Otten wrote:

> I don't expect that it matters much, but you don't need to sort your data if
> you use a dictionary anyway:

Which means that one can build the dict line by line, as each is read, 
instead of reading the entire file into memory. So it does matter for 
intermediate memory use.

> import csv
> import sys
> infile, outfile = sys.argv[1:]
> d = {}
> with open(infile, "rb") as instream:
>      for key, value in csv.reader(instream, delimiter=';'):
>          d.setdefault(key, [key]).append(value)
> with open(outfile, "wb") as outstream:
>      csv.writer(outstream).writerows(d.itervalues())

Terry Jan Reedy

More information about the Python-list mailing list