[New-bugs-announce] [issue41631] _ast module: get_global_ast_state() doesn't work with Mercurial lazy import

[STINNER Victor]

New submission from STINNER Victor <vstinner at python.org>:

Building Mercurial with Python 3.9.0rc1 fails with the error:

SystemError: <built-in function compile> returned NULL without setting an error

The problem comes from the PyAST_Check() function. This function calls get_global_ast_state() which gets the state of the _ast module. If the module is not imported yet, it is imported.

The problem is that Mercurial monkey-patches the __import__() builtin function to implement lazy imports. get_global_ast_state() calls PyImport_Import() which returns a Mercurial _LazyModule(__name__='_ast', ...) object. Calling get_ast_state() (PyModule_GetState()) on it is unsafe since it's not the _ast extension module, but another module (which has no state, PyModule_GetState() returns NULL).

https://bugzilla.redhat.com/show_bug.cgi?id=1871992#c1

--

The _ast extension module was modified multiple times recently:

* September 2019: The extension was converted to PEP 384 (stable ABI): bpo-38113, commit ac46eb4ad6662cf6d771b20d8963658b2186c48c
* July 2020: The extension was converted to PEP 489 (multiphase init): bpo-41194, commit b1cc6ba73a51d5cc3aeb113b5e7378fb50a0e20a
* (and bugfixes: see bpo-41194 and bpo-41204)

I did the PEP 489 change to fix a regression caused by the first change (PEP 384), two bugs in fact:

* bpo-41194 Python 3.9.0b3 crash on compile() in PyAST_Check() when the _ast module is loaded more than once
* bpo-41261: 3.9-dev SEGV in object_recursive_isinstance in ast.literal_eval

----------
components: Library (Lib)
messages: 375881
nosy: vstinner
priority: release blocker
severity: normal
status: open
title: _ast module: get_global_ast_state() doesn't work with Mercurial lazy import
versions: Python 3.10, Python 3.9

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue41631>
_______________________________________