[Python-Dev] issue2180 and using 'tokenize' with Python 3 'str's

[Meador Inge]

Hi All,

I was going through some of the open issues related to 'tokenize' and ran
across 'issue2180'.  The reproduction case for this issue is along the lines
of:

 >>> tokenize.tokenize(io.StringIO("if 1:\n \\\n #hey\n print 1").readline)

but, with 'py3k' I get:

    >>> tokenize.tokenize(io.StringIO("if 1:\n  \\\n  #hey\n  print
1").readline)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/Users/minge/Code/python/py3k/Lib/tokenize.py", line 360, in
tokenize
        encoding, consumed = detect_encoding(readline)
      File "/Users/minge/Code/python/py3k/Lib/tokenize.py", line 316, in
detect_encoding
        if first.startswith(BOM_UTF8):
    TypeError: Can't convert 'bytes' object to str implicitly

which, as seen in the trace, is because the 'detect_encoding' function in
'Lib/tokenize.py' searches for 'BOM_UTF8' (a 'bytes' object) in the string
to tokenize 'first' (a 'str' object).  It seems to me that strings should
still be able to be tokenized, but maybe I am missing something.

Is the implementation of 'detect_encoding' correct in how it attempts to
determine an encoding or should I open an issue for this?

---
Meador
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100927/c154b63b/attachment.html>