[issue26000] Crash in Tokenizer - Heap-use-after-free

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan <tir.karthi at gmail.com> added the comment:

Thanks William for the information. I can reproduce this on 3.5.6. I was able to bisect this down to
#31852 that deals with similar cases and fixed with commit 690c36f2f1085145d364a89bfed5944dd2470308.

$ cpython git:(master) git checkout 690c36f2f1085145d364a89bfed5944dd2470308
HEAD is now at 690c36f2f1 [3.6] bpo-31852: Fix segfault caused by using the async soft keyword (GH-4122)
$ cpython git:(690c36f2f1) git clean -xdf && ./configure --with-pydebug && make -s -j4
$ cpython git:(690c36f2f1) ./python.exe ../backups/vuln.py
  File "../backups/vuln.py", line 2
SyntaxError: Non-UTF-8 code starting with '\xef' in file ../backups/vuln.py on line 2, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details
$ cpython git:(690c36f2f1) ./python.exe ../backups/vuln2.py
  File "../backups/vuln2.py", line 3
SyntaxError: Non-UTF-8 code starting with '\xdd' in file ../backups/vuln2.py on line 3, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details

# Reproduce the crash

➜  cpython git:(690c36f2f1) git checkout 690c36f2f1085145d364a89bfed5944dd2470308~1
Previous HEAD position was 690c36f2f1 [3.6] bpo-31852: Fix segfault caused by using the async soft keyword (GH-4122)
HEAD is now at 2702380870 bpo-31304: Update starmap_async documentation. (GH-4168) (GH-4177)
➜  cpython git:(2702380870) make
➜  cpython git:(2702380870) ./python.exe ../backups/vuln2.py
Assertion failed: (!PyErr_Occurred()), function PyObject_Call, file Objects/abstract.c, line 2247.
^[[A[2]    71701 abort      ./python.exe ../backups/vuln2.py
➜  cpython git:(2702380870) ./python.exe ../backups/vuln.py
Assertion failed: (!PyErr_Occurred()), function PyObject_Call, file Objects/abstract.c, line 2247.
[2]    71712 abort      ./python.exe ../backups/vuln.py

It doesn't affect master, 3.7.0 and v3.6.4+ . Since 3.5 is in security mode and was not backported to 3.5 in the linked ticket. I propose to close this ticket and reopen a separate one with Larry added to it if the fix needs an explicit backport to 3.5.6 on priority.


Thanks

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue26000>
_______________________________________