[issue14811] decoding_fgets() truncates long lines and fails with a SyntaxError("Non-UTF-8 code starting with...")

[STINNER Victor]

STINNER Victor <victor.stinner at gmail.com> added the comment:

> Function decoding_fgets (Parser/tokenizer.c) reads line in buffer
> of fixed size 8192 (line truncated to size 8191) and then fails
> because line is cut in the middle of a multibyte UTF-8 character.

It looks like BUFSIZ is much smaller than 8192 on Windows: it's maybe only 1024 bytes.

Attached patch detects when a line is truncated (longer than the internal buffer).

A better solution is maybe to reallocate the buffer if the string is longer than the buffer (write a universal fgets which allocates the buffer while the line is read). Most functions parsing Python source code uses a dynamic buffer. For example "import module" now reads the whole file content before parsing it (see FileLoader.get_data() in Lib/importlib/_bootstrap.py).

At least, we should use a longer buffer on Windows (ex: use 8192 on all platforms?).

I only found two functions parsing the a Python file line by line: PyRun_InteractiveOneFlags() and PyRun_FileExFlags(). There are many variant of these functions (ex: PyRun_InteractiveOne and PyRun_File). These functions are part of the C Python API and used by programs to execute Python code when Python is embeded in a program.

PS: As noticed by Serhiy Storchaka, the bug is not specific to Windows. It's just that the internal buffer is much smaller on Windows.

----------
components: +Interpreter Core -Windows
keywords: +patch
nosy: +haypo
title: Syntax error on long UTF-8 lines -> decoding_fgets() truncates long lines and fails with a SyntaxError("Non-UTF-8 code starting with...")
Added file: http://bugs.python.org/file25605/detect_truncate.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue14811>
_______________________________________