[New-bugs-announce] [issue18268] ElementTree.fromstring non-deterministically gives unicode text data

Brendan O'Connor report at bugs.python.org
Wed Jun 19 23:21:54 CEST 2013

New submission from Brendan O'Connor:

(This is Python 2.7 so I'm using string vs unicode terminology.)

When I use ElementTree.fromstring(), and use the .text field on nodes, the value is usually a string object, but in rare cases it's a unicode object.  I'm parsing many XML documents of newspaper text [1]; on one subset of the data, out of 5 million nodes, ~200 of them have a unicode object for the .text field.

I think this is all related to http://bugs.python.org/issue11033 but I can't figure out how, exactly.  I'm passing in strings to ElementTree.fromstring() like you're supposed to.

The workaround is to defensively convert the .text value to unicode [3].

[1] data is http://www.ldc.upenn.edu/Catalog/catalogEntry.jsp?catalogId=LDC2012T21

[2] my processing code is https://github.com/brendano/gigaword_conversion/blob/master/annogw2justsent.py


def convert_to_unicode(mystr):
    if isinstance(mystr, unicode):
        return mystr
    if isinstance(mystr, str):
        return mystr.decode('utf8')

messages: 191496
nosy: Brendan.OConnor
priority: normal
severity: normal
status: open
title: ElementTree.fromstring non-deterministically gives unicode text data
type: behavior
versions: Python 2.7

Python tracker <report at bugs.python.org>

More information about the New-bugs-announce mailing list