[New-bugs-announce] [issue25222] 3.5.0 regression - Fatal Python error: Cannot recover from stack overflow

[Bernie Hackett]

New submission from Bernie Hackett:

While running PyMongo's test suite against python 3.5.0 the interpreter inconsistently aborts when we test encoding a recursive data structure like:

evil = {}
evil['evil'] = evil

The test that triggers this was added to test the use of Py_EnterRecursiveCall in PyMongo's C extensions and passes without issues on all previous CPython releases back to 2.4.x (the oldest version PyMongo supports). The interesting thing about this abort is that it only occurs when testing PyMongo *without* its C extensions. The stacktrace looks like this:

test_bad_encode (test_bson.TestBSON) ... Exception ignored in: <bound method PeriodicExecutor.close of <pymongo.periodic
_executor.PeriodicExecutor object at 0x00000000046AB0F0>>
RecursionError: maximum recursion depth exceeded while calling a Python object
Fatal Python error: Cannot recover from stack overflow.
 
Thread 0x00000b6c (most recent call first):
  File "C:\10gen\mongo-python-driver\pymongo\periodic_executor.py", line 105 in _run
  File "C:\Python35\lib\threading.py", line 871 in run
  File "C:\Python35\lib\threading.py", line 923 in _bootstrap_inner
  File "C:\Python35\lib\threading.py", line 891 in _bootstrap
 
Thread 0x00000690 (most recent call first):
  File "C:\Python35\lib\threading.py", line 297 in wait
  File "C:\10gen\mongo-python-driver\pymongo\thread_util.py", line 199 in wait
  File "C:\10gen\mongo-python-driver\pymongo\periodic_executor.py", line 110 in _run
  File "C:\Python35\lib\threading.py", line 871 in run
  File "C:\Python35\lib\threading.py", line 923 in _bootstrap_inner
  File "C:\Python35\lib\threading.py", line 891 in _bootstrap
 
Thread 0x00000900 (most recent call first):
  File "C:\Python35\lib\threading.py", line 297 in wait
  File "C:\10gen\mongo-python-driver\pymongo\thread_util.py", line 199 in wait
  File "C:\10gen\mongo-python-driver\pymongo\periodic_executor.py", line 110 in _run
  File "C:\Python35\lib\threading.py", line 871 in run
  File "C:\Python35\lib\threading.py", line 923 in _bootstrap_inner
  File "C:\Python35\lib\threading.py", line 891 in _bootstrap
 
Current thread 0x00000a20 (most recent call first):
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 692 in _element_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in _encode_mapping
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in _encode_list
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 692 in _element_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in _encode_mapping
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in _encode_list
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 692 in _element_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in _encode_mapping
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in _encode_list
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 692 in _element_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in _encode_mapping
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in _encode_list
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 692 in _element_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in _encode_mapping
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in _encode_list
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 692 in _element_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in _encode_mapping
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in _encode_list
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 692 in _element_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in _encode_mapping
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in _encode_list
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 692 in _element_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in _encode_mapping
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in _encode_list
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 692 in _element_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in _encode_mapping
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in _encode_list
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 692 in _element_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in _encode_mapping
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in _encode_list
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 692 in _element_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in _encode_mapping
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in _encode_list
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 692 in _element_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in _encode_mapping
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in _encode_list
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 692 in _element_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in _encode_mapping
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in _encode_list
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 692 in _element_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 433 in _encode_mapping
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in <listcomp>
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 462 in _encode_list
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 653 in _name_value_to_bson
  File "C:\10gen\mongo-python-driver\bson\__init__.py", line 692 in _element_to_bson

My current working theory is if RecursionError is raised during a thread switch the recursion depth is deemed too great to recover from. Maybe the GIL is no longer being held during some call to _Py_CheckRecursiveCall? This failure looks suspiciously similar to issue22971 given that, in that ticket, the failure occurs while or immediately after running test_thread.

The failure has been observed on 64 and 32bit Windows 7 VMs as well as a 64bit Amazon Linux instance. Sadly, I haven't yet been able to devise a reproduction that doesn't involve running PyMongo's test suite.

----------
components: Interpreter Core
messages: 251453
nosy: behackett
priority: normal
severity: normal
status: open
title: 3.5.0 regression - Fatal Python error: Cannot recover from stack overflow
versions: Python 3.5

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue25222>
_______________________________________