[Csv] Re: [Python-Dev] csv module TODO list

[M.-A. Lemburg]

Andrew McNamara wrote:
>>The CSV format is often used for exchanging large data files, not just for
>>spreadsheet output.
>>
>>My experience: files with over a million rows are not uncommon. FWIW, no
>>Unicode.
> 
> Matches my experience also, but I suspect we both live in English speaking
> countries. Elsewhere in the world, the ratios could be reversed.

Hmm, wasn't XML intended to replace CSV (among other formats) for
exchanging tons of data ;-)

As I mentioned before, there's no such thing as the one fits all
general CSV parser or writer.

If Unicode CSV data is not common enough, you might want to provide
a solution based on a UTF-8 string encoding - a decoder could
convert the input stream to UTF-8, you then process that data
using the existing CSV parser and then convert it back to Unicode
in the .next() method.

So far, I've only ever used Unicode CSV data for exchange with
Asian language spreadsheets.

> There has also been some suggestion that the native string type in Python
> will become Unicode at some point in the future.

Indeed :-)

>>My (jaundiced, but based on experience) viewpoint on newlines inside
>>quoted strings:
>>
>>Prob (spreadsheet file with newlines inside data fields) = 0.001
>>
>>Prob (some programmer has not quoted their quotes properly) = 0.999
>>
>>Hence I suggest an option to specify this as a bug.
> 
> I agree. What makes this extra exciting at the moment is that the CSV
> module will happily sit there slurping the whole file into memory trying
> to match a stray quote (of course, I only noticed this when trying to
> read a multi-gigabyte file).

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 05 2005)
 >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
 >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
 >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::