[New-bugs-announce] [issue10095] Support undecodable filenames in the parser API
report at bugs.python.org
Thu Oct 14 01:52:58 CEST 2010
New submission from STINNER Victor <victor.stinner at haypocalc.com>:
It looks like the parser API (eg. PyParser_ParseFileFlagsEx, PyParser_ASTFromFile) expects utf-8 filename: err_input() decodes the filename from utf-8. But
Example in a non-ascii directory (/home/SHARE/SVN/py3kéŁ) and an ascii locale:
$ LANG= ./python -c "import inspect"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/SHARE/SVN/py3k\xe9\u0141/Lib/inspect.py", line 1
SyntaxError: encoding problem: with BOM
The problem occurs in fp_setreadl(): this function reopens the file with the right encoding. But to open the file, the bytes filename is decoded from utf-8 (in strict mode), whereas the filename (in my example) contains surrogates and utf-8 in strict mode rejects surrogates.
To support undecodable filenames in the parser API, we have two solutions:
* Use the filesystem encoding with surrogateescape (PyUnicode_EncodeFSDefault, PyUnicode_DecodeFSDefault)
* Use utf-8 in another mode: surrogateescape or surrogatepass
The parser API has many public functions, and we have to consider the compatibility with Python 3.1.
See also #9713 and #8611.
components: Interpreter Core, Unicode
title: Support undecodable filenames in the parser API
versions: Python 3.2
Python tracker <report at bugs.python.org>
More information about the New-bugs-announce