[Python-Dev] csv module and universal newlines

[Andrew McNamara]

This item, from the TODO list, has been bugging me for a while:

>* Reader and universal newlines don't interact well, reader doesn't
>  honour Dialect's lineterminator setting. All outstanding bug id's
>  (789519, 944890, 967934 and 1072404) are related to this - it's 
>  a difficult problem and further discussion is needed.

The csv parser consumes lines from an iterator, but it also has it's own
idea of end-of-line conventions, which are currently only used by the
writer, not the reader, which is a source of much confusion. The writer,
by default, also attempts to emit a \r\n sequence, which results in more
confusion unless the file is opened in binary mode.

I'm looking for suggestions for how we can mitigate these problems
(without breaking things for existing users).

The standard file iterator includes the end-of-line characters in the
returned string. One potentional solution is, then, to ignore the line
chunking done by the file iterator, and logically concatenate the source
lines until the csv parser's idea of lineterminator is seen - but this
defeats negates the benefits of using an iterator.

Another option might be to provide a new interface that relies on a
file-like object being supplied. The lineterminator character would only
be used with this interface, with the current interface falling back to
using only \n. Rather a drastic solution.

Any other ideas?

-- 
Andrew McNamara, Senior Developer, Object Craft
http://www.object-craft.com.au/