[New-bugs-announce] [issue29405] improve csv.Sniffer().sniff() behavior

Milt Epstein report at bugs.python.org
Tue Jan 31 19:26:25 EST 2017

New submission from Milt Epstein:

I'm trying to use csv.Sniffer().sniff(sample_data) to determine the delimiter on a number of input files.  Through some trial and error, many "Could not determine delimiter" errors, and analyzing how this routine works/behaves, I settled on sample_data being some number of lines of the input file, particularly 30.  This value seems to allow the routine to work more frequently, although not always, particularly on short input files.

I realize the way this routine works is somewhat idiosyncratic, and it won't be so easy to improve it generally, but there's one simple change that occurred to me that would help in some cases.  Currently the function _guess_delimiter() in csv.py contains the following lines:

            # build a list of possible delimiters
            modeList = modes.items()
            total = float(chunkLength * iteration)

So total is increased by chunkLength on each iteration.  The problem occurs when total becomes greater than the length of sample_data, that is, the iteration would go beyond the end of sample_data.  That reading is handled fine, it's truncated at the end of sample_data, but total is needlessly set too high.  My suggested change is to add the following two lines after the above:

            if total > len(data):
                total = float(len(data))

components: Library (Lib)
messages: 286570
nosy: mepstein
priority: normal
severity: normal
status: open
title: improve csv.Sniffer().sniff() behavior
type: behavior
versions: Python 3.5

Python tracker <report at bugs.python.org>

More information about the New-bugs-announce mailing list