[New-bugs-announce] [issue43228] Regression in function builtins

[STINNER Victor]

New submission from STINNER Victor <vstinner at python.org>:

The bpo-42990 introduced a regression with the following commit:

commit d6c33fbd346765c6a8654dccacb2338006bf2b47
Author: Mark Shannon <mark at hotpy.org>
Date:   Fri Jan 29 13:24:55 2021 +0000

    bpo-42990: Introduce 'frame constructor' struct to simplify API for PyEval_CodeEval and friends (GH-24298)
    
    * Introduce 'frame constructor' to simplify API for frame creation
    
    * Embed struct using a macro to conform to PEP 7


The commit adds a new PyFunctionObject.func_builtins member which is not constructed properly in the following code.

The reproduce the issue, use attached cloudpickle_bug.py reproducer:

./python -m venv env
./env/bin/python -m pip install cloudpickle
./env/bin/python cloudpickle_bug.py

Current output:
---
3
Traceback (most recent call last):
  File "/home/vstinner/python/master/cloudpickle_bug.py", line 10, in <module>
    print(func2(text))
  File "/home/vstinner/python/master/cloudpickle_bug.py", line -1, in func
NameError: name 'len' is not defined
---

Output before this commit:
---
3
3
---

pickle.loads() calls PyFunction_NewWithQualName() with globals={'__package__': None, '__name__': '__main__', '__file__': '/home/vstinner/python/master/cloudpickle_bug.py'}. In that case, _PyEval_BuiltinsFromGlobals() creates the {'None': None} builtin dictionary and stores it in PyFunctionObject.func_builtins.

=> func2 builtins is {'None': None}


Before the commit, calling func2() got the builtins dictionary from frame_get_builtins(PyFrameObject *back, PyObject *globals). Now it's get from the PyFunctionObject.func_builtins member which is not initialized properly.


If you want to see where PyFunction_NewWithQualName() is called, you can put a break point in _PyEval_BuiltinsFromGlobals(), at the code path creating the {'None': None} dictionary.

vstinner at apu$ gdb -args ./env/bin/python -i cloudpickle_bug.py
(...)
(gdb) b frameobject.c:1190
Breakpoint 1 at 0x6543fa: file Objects/frameobject.c, line 1190.

(gdb) run
(...)
3

Breakpoint 1, _PyEval_BuiltinsFromGlobals (globals={'__package__': None, '__name__': '__main__', '__file__': '/home/vstinner/python/master/cloudpickle_bug.py'})
    at Objects/frameobject.c:1190
1190	        builtins = PyDict_New();

(gdb) where
#0  _PyEval_BuiltinsFromGlobals (globals={'__package__': None, '__name__': '__main__', '__file__': '/home/vstinner/python/master/cloudpickle_bug.py'})
    at Objects/frameobject.c:1190
#1  0x0000000000444191 in PyFunction_NewWithQualName (code=<code at remote 0x7fffea3db2b0>, 
    globals={'__package__': None, '__name__': '__main__', '__file__': '/home/vstinner/python/master/cloudpickle_bug.py'}, qualname=0x0) at Objects/funcobject.c:47
#2  0x0000000000444314 in PyFunction_New (code=<code at remote 0x7fffea3db2b0>, 
    globals={'__package__': None, '__name__': '__main__', '__file__': '/home/vstinner/python/master/cloudpickle_bug.py'}) at Objects/funcobject.c:87
#3  0x0000000000445707 in func_new_impl (type=0x850300 <PyFunction_Type>, code=0x7fffea3db2b0, 
    globals={'__package__': None, '__name__': '__main__', '__file__': '/home/vstinner/python/master/cloudpickle_bug.py'}, name=None, defaults=None, closure=None)
    at Objects/funcobject.c:576
#4  0x000000000044542b in func_new (type=0x850300 <PyFunction_Type>, 
    args=(<code at remote 0x7fffea3db2b0>, {'__package__': None, '__name__': '__main__', '__file__': '/home/vstinner/python/master/cloudpickle_bug.py'}, None, None, None), kwargs=0x0) at Objects/clinic/funcobject.c.h:73
#5  0x0000000000490f4a in type_call (type=0x850300 <PyFunction_Type>, 
    args=(<code at remote 0x7fffea3db2b0>, {'__package__': None, '__name__': '__main__', '__file__': '/home/vstinner/python/master/cloudpickle_bug.py'}, None, None, None), kwds=0x0) at Objects/typeobject.c:1039
#6  0x0000000000431332 in _PyObject_Call (tstate=0x8a0030, callable=<type at remote 0x850300>, 
    args=(<code at remote 0x7fffea3db2b0>, {'__package__': None, '__name__': '__main__', '__file__': '/home/vstinner/python/master/cloudpickle_bug.py'}, None, None, None), kwargs=0x0) at Objects/call.c:305
#7  0x00000000004316ab in PyObject_CallObject (callable=<type at remote 0x850300>, 
    args=(<code at remote 0x7fffea3db2b0>, {'__package__': None, '__name__': '__main__', '__file__': '/home/vstinner/python/master/cloudpickle_bug.py'}, None, None, None)) at Objects/call.c:399
#8  0x00007fffea4fdcc8 in load_reduce (self=0x7fffea5c8b90) at /home/vstinner/python/master/Modules/_pickle.c:6785
#9  0x00007fffea4fe743 in load (self=0x7fffea5c8b90) at /home/vstinner/python/master/Modules/_pickle.c:6933
#10 0x00007fffea50015a in _pickle_loads_impl (module=<module at remote 0x7fffea378d10>, 
    data=b'\x80\x05\x95\xdf\x01\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\r_builtin_type\x94\x93\x94\x8c\nLambdaType\x94\x85\x94R\x94(h\x02\x8c\x08CodeType\x94\x85\x94R\x94(K\x01K\x00K\x00K\x01K\x02KCC\x08t\x00|\x00\x83\x01S\x00\x94N\x85\x94\x8c\x03len\x94\x85\x94\x8c\x01s\x94\x85\x94\x8c//home/vstinner/python/master/cloudpickle_bug.py\x94\x8c\x04func\x94K\x04C\x02\x00\x01\x94))t\x94R\x94}\x94(\x8c\x0b__package__\x94N\x8c\x08__name__\x94\x8c\x08__main__\x94\x8c\x08__file__\x94h\x0fuNNNt\x94R\x94\x8c\x1ccloudpickle.cloudpickle_fast\x94\x8c\x12_function_setstate\x94\x93\x94h\x1a}\x94}\x94(h\x16h\x10\x8c\x0c__qualname__\x94h\x10\x8c\x0f__annotations__\x94}\x94\x8c\x0e__kwdefaults__\x94N\x8c\x0c__defaults__\x94N\x8c\n__module__\x94h\x17\x8c\x07__doc__\x94N\x8c\x0b__closure__\x94N\x8c\x17_cloudpickle_submodules\x94]\x94\x8c\x0b__globals__\x94}\x94u\x86\x94\x86R0.', fix_imports=1, encoding=0x7fffea502473 "ASCII", errors=0x7fffea502479 "strict", buffers=0x0)
(...)


(gdb) py-bt
Traceback (most recent call first):
  <built-in method loads of module object at remote 0x7fffea378d10>
  File "/home/vstinner/python/master/cloudpickle_bug.py", line 9, in <module>
    func2 = pickle.loads(cloudpickle.dumps(func))

----------
components: Interpreter Core
files: cloudpickle_bug.py
messages: 387011
nosy: Mark.Shannon, vstinner
priority: normal
severity: normal
status: open
title: Regression in function builtins
versions: Python 3.10
Added file: https://bugs.python.org/file49810/cloudpickle_bug.py

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue43228>
_______________________________________