[New-bugs-announce] [issue17090] io.TextIOWrapper does not handle UTF-8 encoded streams correctly

Christoph Rauch report at bugs.python.org
Thu Jan 31 16:28:27 CET 2013


New submission from Christoph Rauch:

I have uncovered a strange behavior in io.TextIOWrapper which I think is a bug.

#!/usr/bin/env python
# encoding: utf-8

import csv 
import io
                                                                                                                                                                                                              
raw_file = io.FileIO('utf-8-encoded.csv', 'rb')
stream = io.BufferedReader(raw_file)
stream = io.TextIOWrapper(stream, encoding="UTF-8")
reader = csv.reader(stream, delimiter=";")

cells = 0 

for row in reader:
    # Cells should contain 4 Unicode characters.
    assert all([len(cell.decode('utf-8')) == 4 for cell in row]), row 
    cells += len(row)

assert cells == 210, cells

This produces a not very useful:

Traceback (most recent call last):
  File "utf8-textio-test.py", line 15, in <module>
    for row in reader:
UnicodeEncodeError: 'ascii' codec can't encode characters in position 1-4: ordinal not in range(128)

The only way to let it *not* crash is to set encoding to ascii and errors to ignore, but this clears out all the characters with ord>128, clearly not useful as well, so I hope this behavior is not intended.

I appended a file with which to test this problem.

----------
components: IO
files: utf-8-encoded.csv
messages: 181028
nosy: Christoph.Rauch
priority: normal
severity: normal
status: open
title: io.TextIOWrapper does not handle UTF-8 encoded streams correctly
type: behavior
versions: Python 2.7
Added file: http://bugs.python.org/file28922/utf-8-encoded.csv

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue17090>
_______________________________________


More information about the New-bugs-announce mailing list