[ python-Bugs-1606092 ] csv module broken for unicode

[SourceForge.net]

Bugs item #1606092, was opened at 2006-11-30 15:46
Message generated for change (Comment added) made by lemburg
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1606092&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Unicode
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: JettLogic (jettlogic)
>Assigned to: Nobody/Anonymous (nobody)
Summary: csv module broken for unicode

Initial Comment:
The csv module does not accept data to write/read as anything other than ascii-or-utf-8 str, and the do-it-yourself example in the Python 2.5 Manual to write in another encoding is extremely clunky: 

1) convert unicode to utf-8
2) use csv on utf-8 with cStringIO output
3) convert utf-8 to unicode
4) convert unicode to target encoding (may be utf-8...)

So clunky as to be a bug - csv clearly can't handle unicode at all.  The module functions are in dire need of either accepting unicode objects (letting the output stream worry about the encoding, like codecs.StreamWriter), or at the very least accepting data directly in a target encoding instead of roundabout utf-8.

To read another encoding is a bit less onerous than writing:

1) wrap file to return utf-8
2) use csv, getting utf-8 output
3) convert utf-8 to unicode object

Anyone willing to fix the csv module?

----------------------------------------------------------------------

>Comment By: M.-A. Lemburg (lemburg)
Date: 2006-12-03 13:54

Message:
Logged In: YES 
user_id=38388
Originator: NO

It should be easy to provide a wrapper class which implements the above in
plain Python.

However, if noone volunteers to write such code, it's not going to happen.


I've found that the builtin csv module is not flexible enough to deal with
the often broken CSV data you typically find in practice, so perhaps adding
a pure Python implementation which works with Unicode might prove to be a
better approach.

Unassigning the report, since I don't have time for this.


----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2006-12-03 11:34

Message:
Logged In: YES 
user_id=21627
Originator: NO

Are you willing to fix it?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1606092&group_id=5470