[New-bugs-announce] [issue17090] io.TextIOWrapper does not handle UTF-8 encoded streams correctly
Christoph Rauch
report at bugs.python.org
Thu Jan 31 16:28:27 CET 2013
New submission from Christoph Rauch:
I have uncovered a strange behavior in io.TextIOWrapper which I think is a bug.
#!/usr/bin/env python
# encoding: utf-8
import csv
import io
raw_file = io.FileIO('utf-8-encoded.csv', 'rb')
stream = io.BufferedReader(raw_file)
stream = io.TextIOWrapper(stream, encoding="UTF-8")
reader = csv.reader(stream, delimiter=";")
cells = 0
for row in reader:
# Cells should contain 4 Unicode characters.
assert all([len(cell.decode('utf-8')) == 4 for cell in row]), row
cells += len(row)
assert cells == 210, cells
This produces a not very useful:
Traceback (most recent call last):
File "utf8-textio-test.py", line 15, in <module>
for row in reader:
UnicodeEncodeError: 'ascii' codec can't encode characters in position 1-4: ordinal not in range(128)
The only way to let it *not* crash is to set encoding to ascii and errors to ignore, but this clears out all the characters with ord>128, clearly not useful as well, so I hope this behavior is not intended.
I appended a file with which to test this problem.
----------
components: IO
files: utf-8-encoded.csv
messages: 181028
nosy: Christoph.Rauch
priority: normal
severity: normal
status: open
title: io.TextIOWrapper does not handle UTF-8 encoded streams correctly
type: behavior
versions: Python 2.7
Added file: http://bugs.python.org/file28922/utf-8-encoded.csv
_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue17090>
_______________________________________
More information about the New-bugs-announce
mailing list