Merge/append CSV files with different headers

Mark Lawrence breamoreboy at
Mon Mar 24 19:24:10 CET 2014

On 24/03/2014 17:50, Vincent Davis wrote:
> I have several csv file I need to append (vertically). They have
> different but overlapping headers. For example;
> file1 headers ['a', 'b', 'c']
> file2 headers ['d', 'e']
> file3 headers ['c', 'd']
> Is there a better way than this
> import csv
> def merge_csv(fileList, newFileName):
>      allHeaders = set([])
>      for afile in fileList:
>          with open(afile, 'rb') as csvfilesin:
>              eachheader = csv.reader(csvfilesin, delimiter=',').next()
>              allHeaders.update(eachheader)
>      print(allHeaders)
>      with open(newFileName, 'wb') as csvfileout:
>          outfile = csv.DictWriter(csvfileout, allHeaders)
>          outfile.writeheader()
>          for afile in fileList:
>              print('***'+afile)
>              with open(afile, 'rb') as csvfilesin:
>                  rows = csv.DictReader(csvfilesin, delimiter=',')
>                  for r in rows:
>                      print(allHeaders.issuperset(r.keys()))
>                      outfile.writerow(r)
> Vincent Davis

I haven't looked too hard at your code but guess you could simplify it 
by using the fieldnames parameter to the DictReader class or making use 
of the Sniffer class.  See

My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.

Mark Lawrence

This email is free from viruses and malware because avast! Antivirus protection is active.

More information about the Python-list mailing list