Mailman 3 April 2025 - Python-checkins

dict: Remove redundant incref of immortal object Py_EMPTY_KEYS (GH-133200)
by methane April 30, 2025

April 30, 2025

https://github.com/python/cpython/commit/011979132648d50f83d4506d768dca24de… commit: 011979132648d50f83d4506d768dca24de47c8c6 branch: main author: Mae Hood <22501070+biglizards(a)users.noreply.github.com> committer: methane <songofacandy(a)gmail.com> date: 2025-05-01T08:39:26+09:00 summary: dict: Remove redundant incref of immortal object Py_EMPTY_KEYS (GH-133200) files: M Objects/dictobject.c diff --git a/Objects/dictobject.c b/Objects/dictobject.c index 00658a8ac35bcf..59b0cf1ce7d422 100644 --- a/Objects/dictobject.c +++ b/Objects/dictobject.c @@ -4851,7 +4851,8 @@ dict_new(PyTypeObject *type, PyObject *args, PyObject *kwds) d->ma_used = 0; d->_ma_watcher_tag = 0; - dictkeys_incref(Py_EMPTY_KEYS); + // We don't inc ref empty keys because they're immortal + assert((Py_EMPTY_KEYS)->dk_refcnt == _Py_DICT_IMMORTAL_INITIAL_REFCNT); d->ma_keys = Py_EMPTY_KEYS; d->ma_values = NULL; ASSERT_CONSISTENT(d);

1 0

gh-132775: Add _PyPickle_GetXIData() (gh-133107)
by ericsnowcurrently April 30, 2025

April 30, 2025

https://github.com/python/cpython/commit/cb35c11d82efd2959bda0397abcc1719bf… commit: cb35c11d82efd2959bda0397abcc1719bf6bb0cb branch: main author: Eric Snow <ericsnowcurrently(a)gmail.com> committer: ericsnowcurrently <ericsnowcurrently(a)gmail.com> date: 2025-04-30T17:34:05-06:00 summary: gh-132775: Add _PyPickle_GetXIData() (gh-133107) There's some extra complexity due to making sure we we get things right when handling functions and classes defined in the __main__ module. This is also reflected in the tests, including the addition of extra functions in test.support.import_helper. files: M Include/internal/pycore_crossinterp.h M Lib/test/support/import_helper.py M Lib/test/test_crossinterp.py M Modules/_testinternalcapi.c M Python/crossinterp.c diff --git a/Include/internal/pycore_crossinterp.h b/Include/internal/pycore_crossinterp.h index 4b7446a1f40ccf..4b4617fdbcb2ad 100644 --- a/Include/internal/pycore_crossinterp.h +++ b/Include/internal/pycore_crossinterp.h @@ -171,6 +171,13 @@ PyAPI_FUNC(_PyBytes_data_t *) _PyBytes_GetXIDataWrapped( xid_newobjfunc, _PyXIData_t *); +// _PyObject_GetXIData() for pickle +PyAPI_DATA(PyObject *) _PyPickle_LoadFromXIData(_PyXIData_t *); +PyAPI_FUNC(int) _PyPickle_GetXIData( + PyThreadState *, + PyObject *, + _PyXIData_t *); + // _PyObject_GetXIData() for marshal PyAPI_FUNC(PyObject *) _PyMarshal_ReadObjectFromXIData(_PyXIData_t *); PyAPI_FUNC(int) _PyMarshal_GetXIData( diff --git a/Lib/test/support/import_helper.py b/Lib/test/support/import_helper.py index 42cfe9cfa8cb72..edb734d294f287 100644 --- a/Lib/test/support/import_helper.py +++ b/Lib/test/support/import_helper.py @@ -1,6 +1,7 @@ import contextlib import _imp import importlib +import importlib.machinery import importlib.util import os import shutil @@ -332,3 +333,110 @@ def ensure_lazy_imports(imported_module, modules_to_block): ) from .script_helper import assert_python_ok assert_python_ok("-S", "-c", script) + + +(a)contextlib.contextmanager +def module_restored(name): + """A context manager that restores a module to the original state.""" + missing = object() + orig = sys.modules.get(name, missing) + if orig is None: + mod = importlib.import_module(name) + else: + mod = type(sys)(name) + mod.__dict__.update(orig.__dict__) + sys.modules[name] = mod + try: + yield mod + finally: + if orig is missing: + sys.modules.pop(name, None) + else: + sys.modules[name] = orig + + +def create_module(name, loader=None, *, ispkg=False): + """Return a new, empty module.""" + spec = importlib.machinery.ModuleSpec( + name, + loader, + origin='<import_helper>', + is_package=ispkg, + ) + return importlib.util.module_from_spec(spec) + + +def _ensure_module(name, ispkg, addparent, clearnone): + try: + mod = orig = sys.modules[name] + except KeyError: + mod = orig = None + missing = True + else: + missing = False + if mod is not None: + # It was already imported. + return mod, orig, missing + # Otherwise, None means it was explicitly disabled. + + assert name != '__main__' + if not missing: + assert orig is None, (name, sys.modules[name]) + if not clearnone: + raise ModuleNotFoundError(name) + del sys.modules[name] + # Try normal import, then fall back to adding the module. + try: + mod = importlib.import_module(name) + except ModuleNotFoundError: + if addparent and not clearnone: + addparent = None + mod = _add_module(name, ispkg, addparent) + return mod, orig, missing + + +def _add_module(spec, ispkg, addparent): + if isinstance(spec, str): + name = spec + mod = create_module(name, ispkg=ispkg) + spec = mod.__spec__ + else: + name = spec.name + mod = importlib.util.module_from_spec(spec) + sys.modules[name] = mod + if addparent is not False and spec.parent: + _ensure_module(spec.parent, True, addparent, bool(addparent)) + return mod + + +def add_module(spec, *, parents=True): + """Return the module after creating it and adding it to sys.modules. + + If parents is True then also create any missing parents. + """ + return _add_module(spec, False, parents) + + +def add_package(spec, *, parents=True): + """Return the module after creating it and adding it to sys.modules. + + If parents is True then also create any missing parents. + """ + return _add_module(spec, True, parents) + + +def ensure_module_imported(name, *, clearnone=True): + """Return the corresponding module. + + If it was already imported then return that. Otherwise, try + importing it (optionally clear it first if None). If that fails + then create a new empty module. + + It can be helpful to combine this with ready_to_import() and/or + isolated_modules(). + """ + if sys.modules.get(name) is not None: + mod = sys.modules[name] + else: + mod, _, _ = _force_import(name, False, True, clearnone) + return mod diff --git a/Lib/test/test_crossinterp.py b/Lib/test/test_crossinterp.py index 5ebb78b0ea9e3b..32d6fd4e94bf1b 100644 --- a/Lib/test/test_crossinterp.py +++ b/Lib/test/test_crossinterp.py @@ -1,3 +1,6 @@ +import contextlib +import importlib +import importlib.util import itertools import sys import types @@ -9,7 +12,7 @@ _interpreters = import_helper.import_module('_interpreters') from _interpreters import NotShareableError - +from test import _code_definitions as code_defs from test import _crossinterp_definitions as defs @@ -21,6 +24,88 @@ if (isinstance(o, type) and n not in ('DynamicClassAttribute', '_GeneratorWrapper'))] +DEFS = defs +with open(code_defs.__file__) as infile: + _code_defs_text = infile.read() +with open(DEFS.__file__) as infile: + _defs_text = infile.read() + _defs_text = _defs_text.replace('from ', '# from ') +DEFS_TEXT = f""" +####################################### +# from {code_defs.__file__} + +{_code_defs_text} + +####################################### +# from {defs.__file__} + +{_defs_text} +""" +del infile, _code_defs_text, _defs_text + + +def load_defs(module=None): + """Return a new copy of the test._crossinterp_definitions module. + + The module's __name__ matches the "module" arg, which is either + a str or a module. + + If the "module" arg is a module then the just-loaded defs are also + copied into that module. + + Note that the new module is not added to sys.modules. + """ + if module is None: + modname = DEFS.__name__ + elif isinstance(module, str): + modname = module + module = None + else: + modname = module.__name__ + # Create the new module and populate it. + defs = import_helper.create_module(modname) + defs.__file__ = DEFS.__file__ + exec(DEFS_TEXT, defs.__dict__) + # Copy the defs into the module arg, if any. + if module is not None: + for name, value in defs.__dict__.items(): + if name.startswith('_'): + continue + assert not hasattr(module, name), (name, getattr(module, name)) + setattr(module, name, value) + return defs + + +(a)contextlib.contextmanager +def using___main__(): + """Make sure __main__ module exists (and clean up after).""" + modname = '__main__' + if modname not in sys.modules: + with import_helper.isolated_modules(): + yield import_helper.add_module(modname) + else: + with import_helper.module_restored(modname) as mod: + yield mod + + +(a)contextlib.contextmanager +def temp_module(modname): + """Create the module and add to sys.modules, then remove it after.""" + assert modname not in sys.modules, (modname,) + with import_helper.isolated_modules(): + yield import_helper.add_module(modname) + + +(a)contextlib.contextmanager +def missing_defs_module(modname, *, prep=False): + assert modname not in sys.modules, (modname,) + if prep: + with import_helper.ready_to_import(modname, DEFS_TEXT): + yield modname + else: + with import_helper.isolated_modules(): + yield modname + class _GetXIDataTests(unittest.TestCase): @@ -32,52 +117,49 @@ def get_xidata(self, obj, *, mode=None): def get_roundtrip(self, obj, *, mode=None): mode = self._resolve_mode(mode) - xid =_testinternalcapi.get_crossinterp_data(obj, mode) + return self._get_roundtrip(obj, mode) + + def _get_roundtrip(self, obj, mode): + xid = _testinternalcapi.get_crossinterp_data(obj, mode) return _testinternalcapi.restore_crossinterp_data(xid) - def iter_roundtrip_values(self, values, *, mode=None): + def assert_roundtrip_identical(self, values, *, mode=None): mode = self._resolve_mode(mode) for obj in values: with self.subTest(obj): - xid = _testinternalcapi.get_crossinterp_data(obj, mode) - got = _testinternalcapi.restore_crossinterp_data(xid) - yield obj, got - - def assert_roundtrip_identical(self, values, *, mode=None): - for obj, got in self.iter_roundtrip_values(values, mode=mode): - # XXX What about between interpreters? - self.assertIs(got, obj) + got = self._get_roundtrip(obj, mode) + self.assertIs(got, obj) def assert_roundtrip_equal(self, values, *, mode=None, expecttype=None): - for obj, got in self.iter_roundtrip_values(values, mode=mode): - self.assertEqual(got, obj) - self.assertIs(type(got), - type(obj) if expecttype is None else expecttype) - -# def assert_roundtrip_equal_not_identical(self, values, *, -# mode=None, expecttype=None): -# mode = self._resolve_mode(mode) -# for obj in values: -# cls = type(obj) -# with self.subTest(obj): -# got = self._get_roundtrip(obj, mode) -# self.assertIsNot(got, obj) -# self.assertIs(type(got), type(obj)) -# self.assertEqual(got, obj) -# self.assertIs(type(got), -# cls if expecttype is None else expecttype) -# -# def assert_roundtrip_not_equal(self, values, *, mode=None, expecttype=None): -# mode = self._resolve_mode(mode) -# for obj in values: -# cls = type(obj) -# with self.subTest(obj): -# got = self._get_roundtrip(obj, mode) -# self.assertIsNot(got, obj) -# self.assertIs(type(got), type(obj)) -# self.assertNotEqual(got, obj) -# self.assertIs(type(got), -# cls if expecttype is None else expecttype) + mode = self._resolve_mode(mode) + for obj in values: + with self.subTest(obj): + got = self._get_roundtrip(obj, mode) + self.assertEqual(got, obj) + self.assertIs(type(got), + type(obj) if expecttype is None else expecttype) + + def assert_roundtrip_equal_not_identical(self, values, *, + mode=None, expecttype=None): + mode = self._resolve_mode(mode) + for obj in values: + with self.subTest(obj): + got = self._get_roundtrip(obj, mode) + self.assertIsNot(got, obj) + self.assertIs(type(got), + type(obj) if expecttype is None else expecttype) + self.assertEqual(got, obj) + + def assert_roundtrip_not_equal(self, values, *, + mode=None, expecttype=None): + mode = self._resolve_mode(mode) + for obj in values: + with self.subTest(obj): + got = self._get_roundtrip(obj, mode) + self.assertIsNot(got, obj) + self.assertIs(type(got), + type(obj) if expecttype is None else expecttype) + self.assertNotEqual(got, obj) def assert_not_shareable(self, values, exctype=None, *, mode=None): mode = self._resolve_mode(mode) @@ -95,6 +177,363 @@ def _resolve_mode(self, mode): return mode +class PickleTests(_GetXIDataTests): + + MODE = 'pickle' + + def test_shareable(self): + self.assert_roundtrip_equal([ + # singletons + None, + True, + False, + # bytes + *(i.to_bytes(2, 'little', signed=True) + for i in range(-1, 258)), + # str + 'hello world', + '你好世界', + '', + # int + sys.maxsize, + -sys.maxsize - 1, + *range(-1, 258), + # float + 0.0, + 1.1, + -1.0, + 0.12345678, + -0.12345678, + # tuple + (), + (1,), + ("hello", "world", ), + (1, True, "hello"), + ((1,),), + ((1, 2), (3, 4)), + ((1, 2), (3, 4), (5, 6)), + ]) + # not shareable using xidata + self.assert_roundtrip_equal([ + # int + sys.maxsize + 1, + -sys.maxsize - 2, + 2**1000, + # tuple + (0, 1.0, []), + (0, 1.0, {}), + (0, 1.0, ([],)), + (0, 1.0, ({},)), + ]) + + def test_list(self): + self.assert_roundtrip_equal_not_identical([ + [], + [1, 2, 3], + [[1], (2,), {3: 4}], + ]) + + def test_dict(self): + self.assert_roundtrip_equal_not_identical([ + {}, + {1: 7, 2: 8, 3: 9}, + {1: [1], 2: (2,), 3: {3: 4}}, + ]) + + def test_set(self): + self.assert_roundtrip_equal_not_identical([ + set(), + {1, 2, 3}, + {frozenset({1}), (2,)}, + ]) + + # classes + + def assert_class_defs_same(self, defs): + # Unpickle relative to the unchanged original module. + self.assert_roundtrip_identical(defs.TOP_CLASSES) + + instances = [] + for cls, args in defs.TOP_CLASSES.items(): + if cls in defs.CLASSES_WITHOUT_EQUALITY: + continue + instances.append(cls(*args)) + self.assert_roundtrip_equal_not_identical(instances) + + # these don't compare equal + instances = [] + for cls, args in defs.TOP_CLASSES.items(): + if cls not in defs.CLASSES_WITHOUT_EQUALITY: + continue + instances.append(cls(*args)) + self.assert_roundtrip_not_equal(instances) + + def assert_class_defs_other_pickle(self, defs, mod): + # Pickle relative to a different module than the original. + for cls in defs.TOP_CLASSES: + assert not hasattr(mod, cls.__name__), (cls, getattr(mod, cls.__name__)) + self.assert_not_shareable(defs.TOP_CLASSES) + + instances = [] + for cls, args in defs.TOP_CLASSES.items(): + instances.append(cls(*args)) + self.assert_not_shareable(instances) + + def assert_class_defs_other_unpickle(self, defs, mod, *, fail=False): + # Unpickle relative to a different module than the original. + for cls in defs.TOP_CLASSES: + assert not hasattr(mod, cls.__name__), (cls, getattr(mod, cls.__name__)) + + instances = [] + for cls, args in defs.TOP_CLASSES.items(): + with self.subTest(cls): + setattr(mod, cls.__name__, cls) + xid = self.get_xidata(cls) + inst = cls(*args) + instxid = self.get_xidata(inst) + instances.append( + (cls, xid, inst, instxid)) + + for cls, xid, inst, instxid in instances: + with self.subTest(cls): + delattr(mod, cls.__name__) + if fail: + with self.assertRaises(NotShareableError): + _testinternalcapi.restore_crossinterp_data(xid) + continue + got = _testinternalcapi.restore_crossinterp_data(xid) + self.assertIsNot(got, cls) + self.assertNotEqual(got, cls) + + gotcls = got + got = _testinternalcapi.restore_crossinterp_data(instxid) + self.assertIsNot(got, inst) + self.assertIs(type(got), gotcls) + if cls in defs.CLASSES_WITHOUT_EQUALITY: + self.assertNotEqual(got, inst) + elif cls in defs.BUILTIN_SUBCLASSES: + self.assertEqual(got, inst) + else: + self.assertNotEqual(got, inst) + + def assert_class_defs_not_shareable(self, defs): + self.assert_not_shareable(defs.TOP_CLASSES) + + instances = [] + for cls, args in defs.TOP_CLASSES.items(): + instances.append(cls(*args)) + self.assert_not_shareable(instances) + + def test_user_class_normal(self): + self.assert_class_defs_same(defs) + + def test_user_class_in___main__(self): + with using___main__() as mod: + defs = load_defs(mod) + self.assert_class_defs_same(defs) + + def test_user_class_not_in___main___with_filename(self): + with using___main__() as mod: + defs = load_defs('__main__') + assert defs.__file__ + mod.__file__ = defs.__file__ + self.assert_class_defs_not_shareable(defs) + + def test_user_class_not_in___main___without_filename(self): + with using___main__() as mod: + defs = load_defs('__main__') + defs.__file__ = None + mod.__file__ = None + self.assert_class_defs_not_shareable(defs) + + def test_user_class_not_in___main___unpickle_with_filename(self): + with using___main__() as mod: + defs = load_defs('__main__') + assert defs.__file__ + mod.__file__ = defs.__file__ + self.assert_class_defs_other_unpickle(defs, mod) + + def test_user_class_not_in___main___unpickle_without_filename(self): + with using___main__() as mod: + defs = load_defs('__main__') + defs.__file__ = None + mod.__file__ = None + self.assert_class_defs_other_unpickle(defs, mod, fail=True) + + def test_user_class_in_module(self): + with temp_module('__spam__') as mod: + defs = load_defs(mod) + self.assert_class_defs_same(defs) + + def test_user_class_not_in_module_with_filename(self): + with temp_module('__spam__') as mod: + defs = load_defs(mod.__name__) + assert defs.__file__ + # For now, we only address this case for __main__. + self.assert_class_defs_not_shareable(defs) + + def test_user_class_not_in_module_without_filename(self): + with temp_module('__spam__') as mod: + defs = load_defs(mod.__name__) + defs.__file__ = None + self.assert_class_defs_not_shareable(defs) + + def test_user_class_module_missing_then_imported(self): + with missing_defs_module('__spam__', prep=True) as modname: + defs = load_defs(modname) + # For now, we only address this case for __main__. + self.assert_class_defs_not_shareable(defs) + + def test_user_class_module_missing_not_available(self): + with missing_defs_module('__spam__') as modname: + defs = load_defs(modname) + self.assert_class_defs_not_shareable(defs) + + def test_nested_class(self): + eggs = defs.EggsNested() + with self.assertRaises(NotShareableError): + self.get_roundtrip(eggs) + + # functions + + def assert_func_defs_same(self, defs): + # Unpickle relative to the unchanged original module. + self.assert_roundtrip_identical(defs.TOP_FUNCTIONS) + + def assert_func_defs_other_pickle(self, defs, mod): + # Pickle relative to a different module than the original. + for func in defs.TOP_FUNCTIONS: + assert not hasattr(mod, func.__name__), (cls, getattr(mod, func.__name__)) + self.assert_not_shareable(defs.TOP_FUNCTIONS) + + def assert_func_defs_other_unpickle(self, defs, mod, *, fail=False): + # Unpickle relative to a different module than the original. + for func in defs.TOP_FUNCTIONS: + assert not hasattr(mod, func.__name__), (cls, getattr(mod, func.__name__)) + + captured = [] + for func in defs.TOP_FUNCTIONS: + with self.subTest(func): + setattr(mod, func.__name__, func) + xid = self.get_xidata(func) + captured.append( + (func, xid)) + + for func, xid in captured: + with self.subTest(func): + delattr(mod, func.__name__) + if fail: + with self.assertRaises(NotShareableError): + _testinternalcapi.restore_crossinterp_data(xid) + continue + got = _testinternalcapi.restore_crossinterp_data(xid) + self.assertIsNot(got, func) + self.assertNotEqual(got, func) + + def assert_func_defs_not_shareable(self, defs): + self.assert_not_shareable(defs.TOP_FUNCTIONS) + + def test_user_function_normal(self): +# self.assert_roundtrip_equal(defs.TOP_FUNCTIONS) + self.assert_func_defs_same(defs) + + def test_user_func_in___main__(self): + with using___main__() as mod: + defs = load_defs(mod) + self.assert_func_defs_same(defs) + + def test_user_func_not_in___main___with_filename(self): + with using___main__() as mod: + defs = load_defs('__main__') + assert defs.__file__ + mod.__file__ = defs.__file__ + self.assert_func_defs_not_shareable(defs) + + def test_user_func_not_in___main___without_filename(self): + with using___main__() as mod: + defs = load_defs('__main__') + defs.__file__ = None + mod.__file__ = None + self.assert_func_defs_not_shareable(defs) + + def test_user_func_not_in___main___unpickle_with_filename(self): + with using___main__() as mod: + defs = load_defs('__main__') + assert defs.__file__ + mod.__file__ = defs.__file__ + self.assert_func_defs_other_unpickle(defs, mod) + + def test_user_func_not_in___main___unpickle_without_filename(self): + with using___main__() as mod: + defs = load_defs('__main__') + defs.__file__ = None + mod.__file__ = None + self.assert_func_defs_other_unpickle(defs, mod, fail=True) + + def test_user_func_in_module(self): + with temp_module('__spam__') as mod: + defs = load_defs(mod) + self.assert_func_defs_same(defs) + + def test_user_func_not_in_module_with_filename(self): + with temp_module('__spam__') as mod: + defs = load_defs(mod.__name__) + assert defs.__file__ + # For now, we only address this case for __main__. + self.assert_func_defs_not_shareable(defs) + + def test_user_func_not_in_module_without_filename(self): + with temp_module('__spam__') as mod: + defs = load_defs(mod.__name__) + defs.__file__ = None + self.assert_func_defs_not_shareable(defs) + + def test_user_func_module_missing_then_imported(self): + with missing_defs_module('__spam__', prep=True) as modname: + defs = load_defs(modname) + # For now, we only address this case for __main__. + self.assert_func_defs_not_shareable(defs) + + def test_user_func_module_missing_not_available(self): + with missing_defs_module('__spam__') as modname: + defs = load_defs(modname) + self.assert_func_defs_not_shareable(defs) + + def test_nested_function(self): + self.assert_not_shareable(defs.NESTED_FUNCTIONS) + + # exceptions + + def test_user_exception_normal(self): + self.assert_roundtrip_not_equal([ + defs.MimimalError('error!'), + ]) + self.assert_roundtrip_equal_not_identical([ + defs.RichError('error!', 42), + ]) + + def test_builtin_exception(self): + msg = 'error!' + try: + raise Exception + except Exception as exc: + caught = exc + special = { + BaseExceptionGroup: (msg, [caught]), + ExceptionGroup: (msg, [caught]), +# UnicodeError: (None, msg, None, None, None), + UnicodeEncodeError: ('utf-8', '', 1, 3, msg), + UnicodeDecodeError: ('utf-8', b'', 1, 3, msg), + UnicodeTranslateError: ('', 1, 3, msg), + } + exceptions = [] + for cls in EXCEPTION_TYPES: + args = special.get(cls) or (msg,) + exceptions.append(cls(*args)) + + self.assert_roundtrip_not_equal(exceptions) + + class MarshalTests(_GetXIDataTests): MODE = 'marshal' @@ -444,22 +883,12 @@ def test_module(self): ]) def test_class(self): - self.assert_not_shareable([ - defs.Spam, - defs.SpamOkay, - defs.SpamFull, - defs.SubSpamFull, - defs.SubTuple, - defs.EggsNested, - ]) - self.assert_not_shareable([ - defs.Spam(), - defs.SpamOkay(), - defs.SpamFull(1, 2, 3), - defs.SubSpamFull(1, 2, 3), - defs.SubTuple([1, 2, 3]), - defs.EggsNested(), - ]) + self.assert_not_shareable(defs.CLASSES) + + instances = [] + for cls, args in defs.CLASSES.items(): + instances.append(cls(*args)) + self.assert_not_shareable(instances) def test_builtin_type(self): self.assert_not_shareable([ diff --git a/Modules/_testinternalcapi.c b/Modules/_testinternalcapi.c index 4bfe88f2cf920c..812737e294fcb7 100644 --- a/Modules/_testinternalcapi.c +++ b/Modules/_testinternalcapi.c @@ -1939,6 +1939,11 @@ get_crossinterp_data(PyObject *self, PyObject *args, PyObject *kwargs) goto error; } } + else if (strcmp(mode, "pickle") == 0) { + if (_PyPickle_GetXIData(tstate, obj, xidata) != 0) { + goto error; + } + } else if (strcmp(mode, "marshal") == 0) { if (_PyMarshal_GetXIData(tstate, obj, xidata) != 0) { goto error; diff --git a/Python/crossinterp.c b/Python/crossinterp.c index 753d784a503467..a9f9b78562917e 100644 --- a/Python/crossinterp.c +++ b/Python/crossinterp.c @@ -3,6 +3,7 @@ #include "Python.h" #include "marshal.h" // PyMarshal_WriteObjectToString() +#include "osdefs.h" // MAXPATHLEN #include "pycore_ceval.h" // _Py_simple_func #include "pycore_crossinterp.h" // _PyXIData_t #include "pycore_initconfig.h" // _PyStatus_OK() @@ -10,6 +11,155 @@ #include "pycore_typeobject.h" // _PyStaticType_InitBuiltin() +static Py_ssize_t +_Py_GetMainfile(char *buffer, size_t maxlen) +{ + // We don't expect subinterpreters to have the __main__ module's + // __name__ set, but proceed just in case. + PyThreadState *tstate = _PyThreadState_GET(); + PyObject *module = _Py_GetMainModule(tstate); + if (_Py_CheckMainModule(module) < 0) { + return -1; + } + Py_ssize_t size = _PyModule_GetFilenameUTF8(module, buffer, maxlen); + Py_DECREF(module); + return size; +} + + +static PyObject * +import_get_module(PyThreadState *tstate, const char *modname) +{ + PyObject *module = NULL; + if (strcmp(modname, "__main__") == 0) { + module = _Py_GetMainModule(tstate); + if (_Py_CheckMainModule(module) < 0) { + assert(_PyErr_Occurred(tstate)); + return NULL; + } + } + else { + module = PyImport_ImportModule(modname); + if (module == NULL) { + return NULL; + } + } + return module; +} + + +static PyObject * +runpy_run_path(const char *filename, const char *modname) +{ + PyObject *run_path = PyImport_ImportModuleAttrString("runpy", "run_path"); + if (run_path == NULL) { + return NULL; + } + PyObject *args = Py_BuildValue("(sOs)", filename, Py_None, modname); + if (args == NULL) { + Py_DECREF(run_path); + return NULL; + } + PyObject *ns = PyObject_Call(run_path, args, NULL); + Py_DECREF(run_path); + Py_DECREF(args); + return ns; +} + + +static PyObject * +pyerr_get_message(PyObject *exc) +{ + assert(!PyErr_Occurred()); + PyObject *args = PyException_GetArgs(exc); + if (args == NULL || args == Py_None || PyObject_Size(args) < 1) { + return NULL; + } + if (PyUnicode_Check(args)) { + return args; + } + PyObject *msg = PySequence_GetItem(args, 0); + Py_DECREF(args); + if (msg == NULL) { + PyErr_Clear(); + return NULL; + } + if (!PyUnicode_Check(msg)) { + Py_DECREF(msg); + return NULL; + } + return msg; +} + +#define MAX_MODNAME (255) +#define MAX_ATTRNAME (255) + +struct attributeerror_info { + char modname[MAX_MODNAME+1]; + char attrname[MAX_ATTRNAME+1]; +}; + +static int +_parse_attributeerror(PyObject *exc, struct attributeerror_info *info) +{ + assert(exc != NULL); + assert(PyErr_GivenExceptionMatches(exc, PyExc_AttributeError)); + int res = -1; + + PyObject *msgobj = pyerr_get_message(exc); + if (msgobj == NULL) { + return -1; + } + const char *err = PyUnicode_AsUTF8(msgobj); + + if (strncmp(err, "module '", 8) != 0) { + goto finally; + } + err += 8; + + const char *matched = strchr(err, '\''); + if (matched == NULL) { + goto finally; + } + Py_ssize_t len = matched - err; + if (len > MAX_MODNAME) { + goto finally; + } + (void)strncpy(info->modname, err, len); + info->modname[len] = '\0'; + err = matched; + + if (strncmp(err, "' has no attribute '", 20) != 0) { + goto finally; + } + err += 20; + + matched = strchr(err, '\''); + if (matched == NULL) { + goto finally; + } + len = matched - err; + if (len > MAX_ATTRNAME) { + goto finally; + } + (void)strncpy(info->attrname, err, len); + info->attrname[len] = '\0'; + err = matched + 1; + + if (strlen(err) > 0) { + goto finally; + } + res = 0; + +finally: + Py_DECREF(msgobj); + return res; +} + +#undef MAX_MODNAME +#undef MAX_ATTRNAME + + /**************/ /* exceptions */ /**************/ @@ -287,6 +437,308 @@ _PyObject_GetXIData(PyThreadState *tstate, } +/* pickle C-API */ + +struct _pickle_context { + PyThreadState *tstate; +}; + +static PyObject * +_PyPickle_Dumps(struct _pickle_context *ctx, PyObject *obj) +{ + PyObject *dumps = PyImport_ImportModuleAttrString("pickle", "dumps"); + if (dumps == NULL) { + return NULL; + } + PyObject *bytes = PyObject_CallOneArg(dumps, obj); + Py_DECREF(dumps); + return bytes; +} + + +struct sync_module_result { + PyObject *module; + PyObject *loaded; + PyObject *failed; +}; + +struct sync_module { + const char *filename; + char _filename[MAXPATHLEN+1]; + struct sync_module_result cached; +}; + +static void +sync_module_clear(struct sync_module *data) +{ + data->filename = NULL; + Py_CLEAR(data->cached.module); + Py_CLEAR(data->cached.loaded); + Py_CLEAR(data->cached.failed); +} + + +struct _unpickle_context { + PyThreadState *tstate; + // We only special-case the __main__ module, + // since other modules behave consistently. + struct sync_module main; +}; + +static void +_unpickle_context_clear(struct _unpickle_context *ctx) +{ + sync_module_clear(&ctx->main); +} + +static struct sync_module_result +_unpickle_context_get_module(struct _unpickle_context *ctx, + const char *modname) +{ + if (strcmp(modname, "__main__") == 0) { + return ctx->main.cached; + } + else { + return (struct sync_module_result){ + .failed = PyExc_NotImplementedError, + }; + } +} + +static struct sync_module_result +_unpickle_context_set_module(struct _unpickle_context *ctx, + const char *modname) +{ + struct sync_module_result res = {0}; + struct sync_module_result *cached = NULL; + const char *filename = NULL; + if (strcmp(modname, "__main__") == 0) { + cached = &ctx->main.cached; + filename = ctx->main.filename; + } + else { + res.failed = PyExc_NotImplementedError; + goto finally; + } + + res.module = import_get_module(ctx->tstate, modname); + if (res.module == NULL) { + res.failed = _PyErr_GetRaisedException(ctx->tstate); + assert(res.failed != NULL); + goto finally; + } + + if (filename == NULL) { + Py_CLEAR(res.module); + res.failed = PyExc_NotImplementedError; + goto finally; + } + res.loaded = runpy_run_path(filename, modname); + if (res.loaded == NULL) { + Py_CLEAR(res.module); + res.failed = _PyErr_GetRaisedException(ctx->tstate); + assert(res.failed != NULL); + goto finally; + } + +finally: + if (cached != NULL) { + assert(cached->module == NULL); + assert(cached->loaded == NULL); + assert(cached->failed == NULL); + *cached = res; + } + return res; +} + + +static int +_handle_unpickle_missing_attr(struct _unpickle_context *ctx, PyObject *exc) +{ + // The caller must check if an exception is set or not when -1 is returned. + assert(!_PyErr_Occurred(ctx->tstate)); + assert(PyErr_GivenExceptionMatches(exc, PyExc_AttributeError)); + struct attributeerror_info info; + if (_parse_attributeerror(exc, &info) < 0) { + return -1; + } + + // Get the module. + struct sync_module_result mod = _unpickle_context_get_module(ctx, info.modname); + if (mod.failed != NULL) { + // It must have failed previously. + return -1; + } + if (mod.module == NULL) { + mod = _unpickle_context_set_module(ctx, info.modname); + if (mod.failed != NULL) { + return -1; + } + assert(mod.module != NULL); + } + + // Bail out if it is unexpectedly set already. + if (PyObject_HasAttrString(mod.module, info.attrname)) { + return -1; + } + + // Try setting the attribute. + PyObject *value = NULL; + if (PyDict_GetItemStringRef(mod.loaded, info.attrname, &value) <= 0) { + return -1; + } + assert(value != NULL); + int res = PyObject_SetAttrString(mod.module, info.attrname, value); + Py_DECREF(value); + if (res < 0) { + return -1; + } + + return 0; +} + +static PyObject * +_PyPickle_Loads(struct _unpickle_context *ctx, PyObject *pickled) +{ + PyObject *loads = PyImport_ImportModuleAttrString("pickle", "loads"); + if (loads == NULL) { + return NULL; + } + PyObject *obj = PyObject_CallOneArg(loads, pickled); + if (ctx != NULL) { + while (obj == NULL) { + assert(_PyErr_Occurred(ctx->tstate)); + if (!PyErr_ExceptionMatches(PyExc_AttributeError)) { + // We leave other failures unhandled. + break; + } + // Try setting the attr if not set. + PyObject *exc = _PyErr_GetRaisedException(ctx->tstate); + if (_handle_unpickle_missing_attr(ctx, exc) < 0) { + // Any resulting exceptions are ignored + // in favor of the original. + _PyErr_SetRaisedException(ctx->tstate, exc); + break; + } + Py_CLEAR(exc); + // Retry with the attribute set. + obj = PyObject_CallOneArg(loads, pickled); + } + } + Py_DECREF(loads); + return obj; +} + + +/* pickle wrapper */ + +struct _pickle_xid_context { + // __main__.__file__ + struct { + const char *utf8; + size_t len; + char _utf8[MAXPATHLEN+1]; + } mainfile; +}; + +static int +_set_pickle_xid_context(PyThreadState *tstate, struct _pickle_xid_context *ctx) +{ + // Set mainfile if possible. + Py_ssize_t len = _Py_GetMainfile(ctx->mainfile._utf8, MAXPATHLEN); + if (len < 0) { + // For now we ignore any exceptions. + PyErr_Clear(); + } + else if (len > 0) { + ctx->mainfile.utf8 = ctx->mainfile._utf8; + ctx->mainfile.len = (size_t)len; + } + + return 0; +} + + +struct _shared_pickle_data { + _PyBytes_data_t pickled; // Must be first if we use _PyBytes_FromXIData(). + struct _pickle_xid_context ctx; +}; + +PyObject * +_PyPickle_LoadFromXIData(_PyXIData_t *xidata) +{ + PyThreadState *tstate = _PyThreadState_GET(); + struct _shared_pickle_data *shared = + (struct _shared_pickle_data *)xidata->data; + // We avoid copying the pickled data by wrapping it in a memoryview. + // The alternative is to get a bytes object using _PyBytes_FromXIData(). + PyObject *pickled = PyMemoryView_FromMemory( + (char *)shared->pickled.bytes, shared->pickled.len, PyBUF_READ); + if (pickled == NULL) { + return NULL; + } + + // Unpickle the object. + struct _unpickle_context ctx = { + .tstate = tstate, + .main = { + .filename = shared->ctx.mainfile.utf8, + }, + }; + PyObject *obj = _PyPickle_Loads(&ctx, pickled); + Py_DECREF(pickled); + _unpickle_context_clear(&ctx); + if (obj == NULL) { + PyObject *cause = _PyErr_GetRaisedException(tstate); + assert(cause != NULL); + _set_xid_lookup_failure( + tstate, NULL, "object could not be unpickled", cause); + Py_DECREF(cause); + } + return obj; +} + + +int +_PyPickle_GetXIData(PyThreadState *tstate, PyObject *obj, _PyXIData_t *xidata) +{ + // Pickle the object. + struct _pickle_context ctx = { + .tstate = tstate, + }; + PyObject *bytes = _PyPickle_Dumps(&ctx, obj); + if (bytes == NULL) { + PyObject *cause = _PyErr_GetRaisedException(tstate); + assert(cause != NULL); + _set_xid_lookup_failure( + tstate, NULL, "object could not be pickled", cause); + Py_DECREF(cause); + return -1; + } + + // If we had an "unwrapper" mechnanism, we could call + // _PyObject_GetXIData() on the bytes object directly and add + // a simple unwrapper to call pickle.loads() on the bytes. + size_t size = sizeof(struct _shared_pickle_data); + struct _shared_pickle_data *shared = + (struct _shared_pickle_data *)_PyBytes_GetXIDataWrapped( + tstate, bytes, size, _PyPickle_LoadFromXIData, xidata); + Py_DECREF(bytes); + if (shared == NULL) { + return -1; + } + + // If it mattered, we could skip getting __main__.__file__ + // when "__main__" doesn't show up in the pickle bytes. + if (_set_pickle_xid_context(tstate, &shared->ctx) < 0) { + _xidata_clear(xidata); + return -1; + } + + return 0; +} + + /* marshal wrapper */ PyObject *

1 0

GH-125515: Remove two unused error branches. (#133181)
by freakboy3742 April 30, 2025

April 30, 2025

https://github.com/python/cpython/commit/6c522debc218d441756bf631abe8ec8d6c… commit: 6c522debc218d441756bf631abe8ec8d6c6f1c45 branch: main author: Russell Keith-Magee <russell(a)keith-magee.com> committer: freakboy3742 <russell(a)keith-magee.com> date: 2025-05-01T06:21:57+08:00 summary: GH-125515: Remove two unused error branches. (#133181) Remove two unused error branches in the generated bytecode handling. files: M Python/bytecodes.c M Python/generated_cases.c.h M Python/opcode_targets.h diff --git a/Python/bytecodes.c b/Python/bytecodes.c index b30fa0899106de..496a53dbabebd4 100644 --- a/Python/bytecodes.c +++ b/Python/bytecodes.c @@ -5332,18 +5332,6 @@ dummy_func( assert(tstate->tracing || eval_breaker == FT_ATOMIC_LOAD_UINTPTR_ACQUIRE(_PyFrame_GetCode(frame)->_co_instrumentation_version)); } - label(pop_4_error) { - stack_pointer -= 4; - assert(WITHIN_STACK_BOUNDS()); - goto error; - } - - label(pop_3_error) { - stack_pointer -= 3; - assert(WITHIN_STACK_BOUNDS()); - goto error; - } - label(pop_2_error) { stack_pointer -= 2; assert(WITHIN_STACK_BOUNDS()); diff --git a/Python/generated_cases.c.h b/Python/generated_cases.c.h index 89bfc069080a95..6841fbcc22e5a4 100644 --- a/Python/generated_cases.c.h +++ b/Python/generated_cases.c.h @@ -12230,20 +12230,6 @@ JUMP_TO_LABEL(error); #endif /* Py_TAIL_CALL_INTERP */ /* BEGIN LABELS */ - LABEL(pop_4_error) - { - stack_pointer -= 4; - assert(WITHIN_STACK_BOUNDS()); - JUMP_TO_LABEL(error); - } - - LABEL(pop_3_error) - { - stack_pointer -= 3; - assert(WITHIN_STACK_BOUNDS()); - JUMP_TO_LABEL(error); - } - LABEL(pop_2_error) { stack_pointer -= 2; diff --git a/Python/opcode_targets.h b/Python/opcode_targets.h index debaab98f00a8e..8c2dfb46077083 100644 --- a/Python/opcode_targets.h +++ b/Python/opcode_targets.h @@ -260,8 +260,6 @@ static void *opcode_targets[256] = { #else /* Py_TAIL_CALL_INTERP */ static py_tail_call_funcptr INSTRUCTION_TABLE[256]; -Py_PRESERVE_NONE_CC static PyObject *_TAIL_CALL_pop_4_error(TAIL_CALL_PARAMS); -Py_PRESERVE_NONE_CC static PyObject *_TAIL_CALL_pop_3_error(TAIL_CALL_PARAMS); Py_PRESERVE_NONE_CC static PyObject *_TAIL_CALL_pop_2_error(TAIL_CALL_PARAMS); Py_PRESERVE_NONE_CC static PyObject *_TAIL_CALL_pop_1_error(TAIL_CALL_PARAMS); Py_PRESERVE_NONE_CC static PyObject *_TAIL_CALL_error(TAIL_CALL_PARAMS);

1 0

gh-133153: Use rlcompleter for pdb's interact command (#133176)
by gaogaotiantian April 30, 2025

April 30, 2025

https://github.com/python/cpython/commit/327f5ff9fa4291e66079c61c77b273cb95… commit: 327f5ff9fa4291e66079c61c77b273cb953c302f branch: main author: Tian Gao <gaogaotiantian(a)hotmail.com> committer: gaogaotiantian <gaogaotiantian(a)hotmail.com> date: 2025-04-30T18:19:13-04:00 summary: gh-133153: Use rlcompleter for pdb's interact command (#133176) files: A Misc/NEWS.d/next/Library/2025-04-29-23-20-52.gh-issue-133153.M-w9yC.rst M Lib/pdb.py M Lib/test/test_pdb.py diff --git a/Lib/pdb.py b/Lib/pdb.py index e2c7468c50c354..343cf4404d7f8c 100644 --- a/Lib/pdb.py +++ b/Lib/pdb.py @@ -1157,6 +1157,22 @@ def completedefault(self, text, line, begidx, endidx): state += 1 return matches + @contextmanager + def _enable_rlcompleter(self, ns): + try: + import readline + except ImportError: + yield + return + + try: + old_completer = readline.get_completer() + completer = Completer(ns) + readline.set_completer(completer.complete) + yield + finally: + readline.set_completer(old_completer) + # Pdb meta commands, only intended to be used internally by pdb def _pdbcmd_print_frame_status(self, arg): @@ -2242,9 +2258,10 @@ def do_interact(self, arg): contains all the (global and local) names found in the current scope. """ ns = {**self.curframe.f_globals, **self.curframe.f_locals} - console = _PdbInteractiveConsole(ns, message=self.message) - console.interact(banner="*pdb interact start*", - exitmsg="*exit from pdb interact command*") + with self._enable_rlcompleter(ns): + console = _PdbInteractiveConsole(ns, message=self.message) + console.interact(banner="*pdb interact start*", + exitmsg="*exit from pdb interact command*") def do_alias(self, arg): """alias [name [command]] @@ -2749,14 +2766,18 @@ def _read_reply(self): self.error(f"Ignoring invalid message from client: {msg}") def _complete_any(self, text, line, begidx, endidx): - if begidx == 0: - return self.completenames(text, line, begidx, endidx) - - cmd = self.parseline(line)[0] - if cmd: - compfunc = getattr(self, "complete_" + cmd, self.completedefault) - else: + # If we're in 'interact' mode, we need to use the default completer + if self._interact_state: compfunc = self.completedefault + else: + if begidx == 0: + return self.completenames(text, line, begidx, endidx) + + cmd = self.parseline(line)[0] + if cmd: + compfunc = getattr(self, "complete_" + cmd, self.completedefault) + else: + compfunc = self.completedefault return compfunc(text, line, begidx, endidx) def cmdloop(self, intro=None): diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py index ae84fe3ce7d65a..be365a5a3ddeec 100644 --- a/Lib/test/test_pdb.py +++ b/Lib/test/test_pdb.py @@ -4855,6 +4855,34 @@ def func(): self.assertIn(b'4', output) self.assertNotIn(b'Error', output) + def test_interact_completion(self): + script = textwrap.dedent(""" + value = "speci" + import pdb; pdb.Pdb().set_trace() + """) + + # Enter interact mode + input = b"interact\n" + # Should fail to complete 'display' because that's a pdb command + input += b"disp\t\n" + # 'value' should still work + input += b"val\t + 'al'\n" + # Let's define a function to test <tab> + input += b"def f():\n" + input += b"\treturn 42\n" + input += b"\n" + input += b"f() * 2\n" + # Exit interact mode + input += b"exit()\n" + # continue + input += b"c\n" + + output = run_pty(script, input) + + self.assertIn(b"'disp' is not defined", output) + self.assertIn(b'special', output) + self.assertIn(b'84', output) + def load_tests(loader, tests, pattern): from test import test_pdb diff --git a/Misc/NEWS.d/next/Library/2025-04-29-23-20-52.gh-issue-133153.M-w9yC.rst b/Misc/NEWS.d/next/Library/2025-04-29-23-20-52.gh-issue-133153.M-w9yC.rst new file mode 100644 index 00000000000000..c609fa698dca49 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2025-04-29-23-20-52.gh-issue-133153.M-w9yC.rst @@ -0,0 +1 @@ +Do not complete :mod:`pdb` commands in ``interact`` mode of :mod:`pdb`.

1 0

gh-133213: Add tests for `string.templatelib.TemplateIter` (#133215)
by sobolevn April 30, 2025

April 30, 2025

https://github.com/python/cpython/commit/0e21ed7c09c687d62d6bf054022e66bccd… commit: 0e21ed7c09c687d62d6bf054022e66bccd1fa2bc branch: main author: sobolevn <mail(a)sobolevn.me> committer: sobolevn <mail(a)sobolevn.me> date: 2025-04-30T22:38:25+03:00 summary: gh-133213: Add tests for `string.templatelib.TemplateIter` (#133215) Co-authored-by: Adam Turner <9087854+AA-Turner(a)users.noreply.github.com> files: M Lib/test/test_string/test_templatelib.py diff --git a/Lib/test/test_string/test_templatelib.py b/Lib/test/test_string/test_templatelib.py index 5221ca673b24ec..5b9490c2be6de0 100644 --- a/Lib/test/test_string/test_templatelib.py +++ b/Lib/test/test_string/test_templatelib.py @@ -1,5 +1,6 @@ import pickle import unittest +from collections.abc import Iterator, Iterable from string.templatelib import Template, Interpolation from test.test_string._support import TStringBaseCase, fstring @@ -125,5 +126,28 @@ def test_pickle_interpolation(self): self.assertEqual(unpickled.format_spec, interpolation.format_spec) +class TemplateIterTests(unittest.TestCase): + def test_abc(self): + self.assertIsInstance(iter(t''), Iterable) + self.assertIsInstance(iter(t''), Iterator) + + def test_final(self): + TemplateIter = type(iter(t'')) + with self.assertRaisesRegex(TypeError, 'is not an acceptable base type'): + class Sub(TemplateIter): ... + + def test_iter(self): + x = 1 + res = list(iter(t'abc {x} yz')) + + self.assertEqual(res[0], 'abc ') + self.assertIsInstance(res[1], Interpolation) + self.assertEqual(res[1].value, 1) + self.assertEqual(res[1].expression, 'x') + self.assertEqual(res[1].conversion, None) + self.assertEqual(res[1].format_spec, '') + self.assertEqual(res[2], ' yz') + + if __name__ == '__main__': unittest.main()

1 0

gh-132775: Add _PyCode_GetVarCounts() (gh-133128)
by ericsnowcurrently April 30, 2025

April 30, 2025

https://github.com/python/cpython/commit/94b4fcd806e7b692955173d309ea3b70a1… commit: 94b4fcd806e7b692955173d309ea3b70a193ad96 branch: main author: Eric Snow <ericsnowcurrently(a)gmail.com> committer: ericsnowcurrently <ericsnowcurrently(a)gmail.com> date: 2025-04-30T18:19:20Z summary: gh-132775: Add _PyCode_GetVarCounts() (gh-133128) This helper is useful in a variety of ways, including in demonstrating how the different counts relate to one another. It will be used in a later change to help identify if a function is "stateless", meaning it doesn't have any free vars or globals. Note that a majority of this change is tests. files: M Include/cpython/funcobject.h M Include/internal/pycore_code.h M Lib/test/test_code.py M Modules/_testinternalcapi.c M Objects/codeobject.c diff --git a/Include/cpython/funcobject.h b/Include/cpython/funcobject.h index 598cd330bc9ca9..18249b95befe65 100644 --- a/Include/cpython/funcobject.h +++ b/Include/cpython/funcobject.h @@ -97,6 +97,11 @@ static inline PyObject* PyFunction_GET_GLOBALS(PyObject *func) { } #define PyFunction_GET_GLOBALS(func) PyFunction_GET_GLOBALS(_PyObject_CAST(func)) +static inline PyObject* PyFunction_GET_BUILTINS(PyObject *func) { + return _PyFunction_CAST(func)->func_builtins; +} +#define PyFunction_GET_BUILTINS(func) PyFunction_GET_BUILTINS(_PyObject_CAST(func)) + static inline PyObject* PyFunction_GET_MODULE(PyObject *func) { return _PyFunction_CAST(func)->func_module; } diff --git a/Include/internal/pycore_code.h b/Include/internal/pycore_code.h index 635d2b24f4bdff..9b02e2934aa49e 100644 --- a/Include/internal/pycore_code.h +++ b/Include/internal/pycore_code.h @@ -565,6 +565,57 @@ extern int _Py_ClearUnusedTLBC(PyInterpreterState *interp); #endif +typedef struct { + int total; + struct co_locals_counts { + int total; + struct { + int total; + int numposonly; + int numposorkw; + int numkwonly; + int varargs; + int varkwargs; + } args; + int numpure; + struct { + int total; + // numargs does not contribute to locals.total. + int numargs; + int numothers; + } cells; + struct { + int total; + int numpure; + int numcells; + } hidden; + } locals; + int numfree; // nonlocal + struct co_unbound_counts { + int total; + struct { + int total; + int numglobal; + int numbuiltin; + int numunknown; + } globals; + int numattrs; + int numunknown; + } unbound; +} _PyCode_var_counts_t; + +PyAPI_FUNC(void) _PyCode_GetVarCounts( + PyCodeObject *, + _PyCode_var_counts_t *); +PyAPI_FUNC(int) _PyCode_SetUnboundVarCounts( + PyThreadState *, + PyCodeObject *, + _PyCode_var_counts_t *, + PyObject *globalnames, + PyObject *attrnames, + PyObject *globalsns, + PyObject *builtinsns); + PyAPI_FUNC(int) _PyCode_ReturnsOnlyNone(PyCodeObject *); diff --git a/Lib/test/test_code.py b/Lib/test/test_code.py index 7cf09ee7847dc1..1b6dfe7c7890ad 100644 --- a/Lib/test/test_code.py +++ b/Lib/test/test_code.py @@ -777,6 +777,236 @@ def test_local_kinds(self): kinds = _testinternalcapi.get_co_localskinds(func.__code__) self.assertEqual(kinds, expected) + @unittest.skipIf(_testinternalcapi is None, "missing _testinternalcapi") + def test_var_counts(self): + self.maxDiff = None + def new_var_counts(*, + posonly=0, + posorkw=0, + kwonly=0, + varargs=0, + varkwargs=0, + purelocals=0, + argcells=0, + othercells=0, + freevars=0, + globalvars=0, + attrs=0, + unknown=0, + ): + nargvars = posonly + posorkw + kwonly + varargs + varkwargs + nlocals = nargvars + purelocals + othercells + if isinstance(globalvars, int): + globalvars = { + 'total': globalvars, + 'numglobal': 0, + 'numbuiltin': 0, + 'numunknown': globalvars, + } + else: + g_numunknown = 0 + if isinstance(globalvars, dict): + numglobal = globalvars['numglobal'] + numbuiltin = globalvars['numbuiltin'] + size = 2 + if 'numunknown' in globalvars: + g_numunknown = globalvars['numunknown'] + size += 1 + assert len(globalvars) == size, globalvars + else: + assert not isinstance(globalvars, str), repr(globalvars) + try: + numglobal, numbuiltin = globalvars + except ValueError: + numglobal, numbuiltin, g_numunknown = globalvars + globalvars = { + 'total': numglobal + numbuiltin + g_numunknown, + 'numglobal': numglobal, + 'numbuiltin': numbuiltin, + 'numunknown': g_numunknown, + } + unbound = globalvars['total'] + attrs + unknown + return { + 'total': nlocals + freevars + unbound, + 'locals': { + 'total': nlocals, + 'args': { + 'total': nargvars, + 'numposonly': posonly, + 'numposorkw': posorkw, + 'numkwonly': kwonly, + 'varargs': varargs, + 'varkwargs': varkwargs, + }, + 'numpure': purelocals, + 'cells': { + 'total': argcells + othercells, + 'numargs': argcells, + 'numothers': othercells, + }, + 'hidden': { + 'total': 0, + 'numpure': 0, + 'numcells': 0, + }, + }, + 'numfree': freevars, + 'unbound': { + 'total': unbound, + 'globals': globalvars, + 'numattrs': attrs, + 'numunknown': unknown, + }, + } + + import test._code_definitions as defs + funcs = { + defs.spam_minimal: new_var_counts(), + defs.spam_full: new_var_counts( + posonly=2, + posorkw=2, + kwonly=2, + varargs=1, + varkwargs=1, + purelocals=4, + globalvars=3, + attrs=1, + ), + defs.spam: new_var_counts( + posorkw=1, + ), + defs.spam_N: new_var_counts( + posorkw=1, + purelocals=1, + ), + defs.spam_C: new_var_counts( + posorkw=1, + purelocals=1, + argcells=1, + othercells=1, + ), + defs.spam_NN: new_var_counts( + posorkw=1, + purelocals=1, + ), + defs.spam_NC: new_var_counts( + posorkw=1, + purelocals=1, + argcells=1, + othercells=1, + ), + defs.spam_CN: new_var_counts( + posorkw=1, + purelocals=1, + argcells=1, + othercells=1, + ), + defs.spam_CC: new_var_counts( + posorkw=1, + purelocals=1, + argcells=1, + othercells=1, + ), + defs.eggs_nested: new_var_counts( + posorkw=1, + ), + defs.eggs_closure: new_var_counts( + posorkw=1, + freevars=2, + ), + defs.eggs_nested_N: new_var_counts( + posorkw=1, + purelocals=1, + ), + defs.eggs_nested_C: new_var_counts( + posorkw=1, + purelocals=1, + argcells=1, + freevars=2, + ), + defs.eggs_closure_N: new_var_counts( + posorkw=1, + purelocals=1, + freevars=2, + ), + defs.eggs_closure_C: new_var_counts( + posorkw=1, + purelocals=1, + argcells=1, + othercells=1, + freevars=2, + ), + defs.ham_nested: new_var_counts( + posorkw=1, + ), + defs.ham_closure: new_var_counts( + posorkw=1, + freevars=3, + ), + defs.ham_C_nested: new_var_counts( + posorkw=1, + ), + defs.ham_C_closure: new_var_counts( + posorkw=1, + freevars=4, + ), + } + assert len(funcs) == len(defs.FUNCTIONS), (len(funcs), len(defs.FUNCTIONS)) + for func in defs.FUNCTIONS: + with self.subTest(func): + expected = funcs[func] + counts = _testinternalcapi.get_code_var_counts(func.__code__) + self.assertEqual(counts, expected) + + def func_with_globals_and_builtins(): + mod1 = _testinternalcapi + mod2 = dis + mods = (mod1, mod2) + checks = tuple(callable(m) for m in mods) + return callable(mod2), tuple(mods), list(mods), checks + + func = func_with_globals_and_builtins + with self.subTest(f'{func} code'): + expected = new_var_counts( + purelocals=4, + globalvars=5, + ) + counts = _testinternalcapi.get_code_var_counts(func.__code__) + self.assertEqual(counts, expected) + + with self.subTest(f'{func} with own globals and builtins'): + expected = new_var_counts( + purelocals=4, + globalvars=(2, 3), + ) + counts = _testinternalcapi.get_code_var_counts(func) + self.assertEqual(counts, expected) + + with self.subTest(f'{func} without globals'): + expected = new_var_counts( + purelocals=4, + globalvars=(0, 3, 2), + ) + counts = _testinternalcapi.get_code_var_counts(func, globalsns={}) + self.assertEqual(counts, expected) + + with self.subTest(f'{func} without both'): + expected = new_var_counts( + purelocals=4, + globalvars=5, + ) + counts = _testinternalcapi.get_code_var_counts(func, globalsns={}, + builtinsns={}) + self.assertEqual(counts, expected) + + with self.subTest(f'{func} without builtins'): + expected = new_var_counts( + purelocals=4, + globalvars=(2, 0, 3), + ) + counts = _testinternalcapi.get_code_var_counts(func, builtinsns={}) + self.assertEqual(counts, expected) + def isinterned(s): return s is sys.intern(('_' + s + '_')[1:-1]) diff --git a/Modules/_testinternalcapi.c b/Modules/_testinternalcapi.c index 4301dfc2803f4a..4bfe88f2cf920c 100644 --- a/Modules/_testinternalcapi.c +++ b/Modules/_testinternalcapi.c @@ -999,6 +999,172 @@ get_co_localskinds(PyObject *self, PyObject *arg) return kinds; } +static PyObject * +get_code_var_counts(PyObject *self, PyObject *_args, PyObject *_kwargs) +{ + PyThreadState *tstate = _PyThreadState_GET(); + PyObject *codearg; + PyObject *globalnames = NULL; + PyObject *attrnames = NULL; + PyObject *globalsns = NULL; + PyObject *builtinsns = NULL; + static char *kwlist[] = {"code", "globalnames", "attrnames", "globalsns", + "builtinsns", NULL}; + if (!PyArg_ParseTupleAndKeywords(_args, _kwargs, + "O|OOO!O!:get_code_var_counts", kwlist, + &codearg, &globalnames, &attrnames, + &PyDict_Type, &globalsns, &PyDict_Type, &builtinsns)) + { + return NULL; + } + if (PyFunction_Check(codearg)) { + if (globalsns == NULL) { + globalsns = PyFunction_GET_GLOBALS(codearg); + } + if (builtinsns == NULL) { + builtinsns = PyFunction_GET_BUILTINS(codearg); + } + codearg = PyFunction_GET_CODE(codearg); + } + else if (!PyCode_Check(codearg)) { + PyErr_SetString(PyExc_TypeError, + "argument must be a code object or a function"); + return NULL; + } + PyCodeObject *code = (PyCodeObject *)codearg; + + _PyCode_var_counts_t counts = {0}; + _PyCode_GetVarCounts(code, &counts); + if (_PyCode_SetUnboundVarCounts( + tstate, code, &counts, globalnames, attrnames, + globalsns, builtinsns) < 0) + { + return NULL; + } + +#define SET_COUNT(DICT, STRUCT, NAME) \ + do { \ + PyObject *count = PyLong_FromLong(STRUCT.NAME); \ + int res = PyDict_SetItemString(DICT, #NAME, count); \ + Py_DECREF(count); \ + if (res < 0) { \ + goto error; \ + } \ + } while (0) + + PyObject *locals = NULL; + PyObject *args = NULL; + PyObject *cells = NULL; + PyObject *hidden = NULL; + PyObject *unbound = NULL; + PyObject *globals = NULL; + PyObject *countsobj = PyDict_New(); + if (countsobj == NULL) { + return NULL; + } + SET_COUNT(countsobj, counts, total); + + // locals + locals = PyDict_New(); + if (locals == NULL) { + goto error; + } + if (PyDict_SetItemString(countsobj, "locals", locals) < 0) { + goto error; + } + SET_COUNT(locals, counts.locals, total); + + // locals.args + args = PyDict_New(); + if (args == NULL) { + goto error; + } + if (PyDict_SetItemString(locals, "args", args) < 0) { + goto error; + } + SET_COUNT(args, counts.locals.args, total); + SET_COUNT(args, counts.locals.args, numposonly); + SET_COUNT(args, counts.locals.args, numposorkw); + SET_COUNT(args, counts.locals.args, numkwonly); + SET_COUNT(args, counts.locals.args, varargs); + SET_COUNT(args, counts.locals.args, varkwargs); + + // locals.numpure + SET_COUNT(locals, counts.locals, numpure); + + // locals.cells + cells = PyDict_New(); + if (cells == NULL) { + goto error; + } + if (PyDict_SetItemString(locals, "cells", cells) < 0) { + goto error; + } + SET_COUNT(cells, counts.locals.cells, total); + SET_COUNT(cells, counts.locals.cells, numargs); + SET_COUNT(cells, counts.locals.cells, numothers); + + // locals.hidden + hidden = PyDict_New(); + if (hidden == NULL) { + goto error; + } + if (PyDict_SetItemString(locals, "hidden", hidden) < 0) { + goto error; + } + SET_COUNT(hidden, counts.locals.hidden, total); + SET_COUNT(hidden, counts.locals.hidden, numpure); + SET_COUNT(hidden, counts.locals.hidden, numcells); + + // numfree + SET_COUNT(countsobj, counts, numfree); + + // unbound + unbound = PyDict_New(); + if (unbound == NULL) { + goto error; + } + if (PyDict_SetItemString(countsobj, "unbound", unbound) < 0) { + goto error; + } + SET_COUNT(unbound, counts.unbound, total); + SET_COUNT(unbound, counts.unbound, numattrs); + SET_COUNT(unbound, counts.unbound, numunknown); + + // unbound.globals + globals = PyDict_New(); + if (globals == NULL) { + goto error; + } + if (PyDict_SetItemString(unbound, "globals", globals) < 0) { + goto error; + } + SET_COUNT(globals, counts.unbound.globals, total); + SET_COUNT(globals, counts.unbound.globals, numglobal); + SET_COUNT(globals, counts.unbound.globals, numbuiltin); + SET_COUNT(globals, counts.unbound.globals, numunknown); + +#undef SET_COUNT + + Py_DECREF(locals); + Py_DECREF(args); + Py_DECREF(cells); + Py_DECREF(hidden); + Py_DECREF(unbound); + Py_DECREF(globals); + return countsobj; + +error: + Py_DECREF(countsobj); + Py_XDECREF(locals); + Py_XDECREF(args); + Py_XDECREF(cells); + Py_XDECREF(hidden); + Py_XDECREF(unbound); + Py_XDECREF(globals); + return NULL; +} + static PyObject * jit_enabled(PyObject *self, PyObject *arg) { @@ -2120,6 +2286,8 @@ static PyMethodDef module_functions[] = { {"code_returns_only_none", code_returns_only_none, METH_O, NULL}, {"get_co_framesize", get_co_framesize, METH_O, NULL}, {"get_co_localskinds", get_co_localskinds, METH_O, NULL}, + {"get_code_var_counts", _PyCFunction_CAST(get_code_var_counts), + METH_VARARGS | METH_KEYWORDS, NULL}, {"jit_enabled", jit_enabled, METH_NOARGS, NULL}, #ifdef _Py_TIER2 {"add_executor_dependency", add_executor_dependency, METH_VARARGS, NULL}, diff --git a/Objects/codeobject.c b/Objects/codeobject.c index bf24a4af445356..d643eb9fd61ae9 100644 --- a/Objects/codeobject.c +++ b/Objects/codeobject.c @@ -1690,6 +1690,241 @@ PyCode_GetFreevars(PyCodeObject *code) } +static int +identify_unbound_names(PyThreadState *tstate, PyCodeObject *co, + PyObject *globalnames, PyObject *attrnames, + PyObject *globalsns, PyObject *builtinsns, + struct co_unbound_counts *counts) +{ + // This function is inspired by inspect.getclosurevars(). + // It would be nicer if we had something similar to co_localspluskinds, + // but for co_names. + assert(globalnames != NULL); + assert(PySet_Check(globalnames)); + assert(PySet_GET_SIZE(globalnames) == 0 || counts != NULL); + assert(attrnames != NULL); + assert(PySet_Check(attrnames)); + assert(PySet_GET_SIZE(attrnames) == 0 || counts != NULL); + assert(globalsns == NULL || PyDict_Check(globalsns)); + assert(builtinsns == NULL || PyDict_Check(builtinsns)); + assert(counts == NULL || counts->total == 0); + Py_ssize_t len = Py_SIZE(co); + for (int i = 0; i < len; i++) { + _Py_CODEUNIT inst = _Py_GetBaseCodeUnit(co, i); + if (inst.op.code == LOAD_ATTR) { + PyObject *name = PyTuple_GET_ITEM(co->co_names, inst.op.arg>>1); + if (counts != NULL) { + if (PySet_Contains(attrnames, name)) { + if (_PyErr_Occurred(tstate)) { + return -1; + } + continue; + } + counts->total += 1; + counts->numattrs += 1; + } + if (PySet_Add(attrnames, name) < 0) { + return -1; + } + } + else if (inst.op.code == LOAD_GLOBAL) { + PyObject *name = PyTuple_GET_ITEM(co->co_names, inst.op.arg>>1); + if (counts != NULL) { + if (PySet_Contains(globalnames, name)) { + if (_PyErr_Occurred(tstate)) { + return -1; + } + continue; + } + counts->total += 1; + counts->globals.total += 1; + counts->globals.numunknown += 1; + if (globalsns != NULL && PyDict_Contains(globalsns, name)) { + if (_PyErr_Occurred(tstate)) { + return -1; + } + counts->globals.numglobal += 1; + counts->globals.numunknown -= 1; + } + if (builtinsns != NULL && PyDict_Contains(builtinsns, name)) { + if (_PyErr_Occurred(tstate)) { + return -1; + } + counts->globals.numbuiltin += 1; + counts->globals.numunknown -= 1; + } + } + if (PySet_Add(globalnames, name) < 0) { + return -1; + } + } + } + return 0; +} + + +void +_PyCode_GetVarCounts(PyCodeObject *co, _PyCode_var_counts_t *counts) +{ + // Count the locals, cells, and free vars. + struct co_locals_counts locals = {0}; + int numfree = 0; + PyObject *kinds = co->co_localspluskinds; + Py_ssize_t numlocalplusfree = PyBytes_GET_SIZE(kinds); + for (int i = 0; i < numlocalplusfree; i++) { + _PyLocals_Kind kind = _PyLocals_GetKind(co->co_localspluskinds, i); + if (kind & CO_FAST_FREE) { + assert(!(kind & CO_FAST_LOCAL)); + assert(!(kind & CO_FAST_HIDDEN)); + assert(!(kind & CO_FAST_ARG)); + numfree += 1; + } + else { + // Apparently not all non-free vars a CO_FAST_LOCAL. + assert(kind); + locals.total += 1; + if (kind & CO_FAST_ARG) { + locals.args.total += 1; + if (kind & CO_FAST_ARG_VAR) { + if (kind & CO_FAST_ARG_POS) { + assert(!(kind & CO_FAST_ARG_KW)); + assert(!locals.args.varargs); + locals.args.varargs = 1; + } + else { + assert(kind & CO_FAST_ARG_KW); + assert(!locals.args.varkwargs); + locals.args.varkwargs = 1; + } + } + else if (kind & CO_FAST_ARG_POS) { + if (kind & CO_FAST_ARG_KW) { + locals.args.numposorkw += 1; + } + else { + locals.args.numposonly += 1; + } + } + else { + assert(kind & CO_FAST_ARG_KW); + locals.args.numkwonly += 1; + } + if (kind & CO_FAST_CELL) { + locals.cells.total += 1; + locals.cells.numargs += 1; + } + // Args are never hidden currently. + assert(!(kind & CO_FAST_HIDDEN)); + } + else { + if (kind & CO_FAST_CELL) { + locals.cells.total += 1; + locals.cells.numothers += 1; + if (kind & CO_FAST_HIDDEN) { + locals.hidden.total += 1; + locals.hidden.numcells += 1; + } + } + else { + locals.numpure += 1; + if (kind & CO_FAST_HIDDEN) { + locals.hidden.total += 1; + locals.hidden.numpure += 1; + } + } + } + } + } + assert(locals.args.total == ( + co->co_argcount + co->co_kwonlyargcount + + !!(co->co_flags & CO_VARARGS) + + !!(co->co_flags & CO_VARKEYWORDS))); + assert(locals.args.numposonly == co->co_posonlyargcount); + assert(locals.args.numposonly + locals.args.numposorkw == co->co_argcount); + assert(locals.args.numkwonly == co->co_kwonlyargcount); + assert(locals.cells.total == co->co_ncellvars); + assert(locals.args.total + locals.numpure == co->co_nlocals); + assert(locals.total + locals.cells.numargs == co->co_nlocals + co->co_ncellvars); + assert(locals.total + numfree == co->co_nlocalsplus); + assert(numfree == co->co_nfreevars); + + // Get the unbound counts. + assert(PyTuple_GET_SIZE(co->co_names) >= 0); + struct co_unbound_counts unbound = { + .total = (int)PyTuple_GET_SIZE(co->co_names), + // numglobal and numattrs can be set later + // with _PyCode_SetUnboundVarCounts(). + .numunknown = (int)PyTuple_GET_SIZE(co->co_names), + }; + + // "Return" the result. + *counts = (_PyCode_var_counts_t){ + .total = locals.total + numfree + unbound.total, + .locals = locals, + .numfree = numfree, + .unbound = unbound, + }; +} + +int +_PyCode_SetUnboundVarCounts(PyThreadState *tstate, + PyCodeObject *co, _PyCode_var_counts_t *counts, + PyObject *globalnames, PyObject *attrnames, + PyObject *globalsns, PyObject *builtinsns) +{ + int res = -1; + PyObject *globalnames_owned = NULL; + PyObject *attrnames_owned = NULL; + + // Prep the name sets. + if (globalnames == NULL) { + globalnames_owned = PySet_New(NULL); + if (globalnames_owned == NULL) { + goto finally; + } + globalnames = globalnames_owned; + } + else if (!PySet_Check(globalnames)) { + _PyErr_Format(tstate, PyExc_TypeError, + "expected a set for \"globalnames\", got %R", globalnames); + goto finally; + } + if (attrnames == NULL) { + attrnames_owned = PySet_New(NULL); + if (attrnames_owned == NULL) { + goto finally; + } + attrnames = attrnames_owned; + } + else if (!PySet_Check(attrnames)) { + _PyErr_Format(tstate, PyExc_TypeError, + "expected a set for \"attrnames\", got %R", attrnames); + goto finally; + } + + // Fill in unbound.globals and unbound.numattrs. + struct co_unbound_counts unbound = {0}; + if (identify_unbound_names( + tstate, co, globalnames, attrnames, globalsns, builtinsns, + &unbound) < 0) + { + goto finally; + } + assert(unbound.numunknown == 0); + assert(unbound.total <= counts->unbound.total); + assert(counts->unbound.numunknown == counts->unbound.total); + unbound.numunknown = counts->unbound.total - unbound.total; + unbound.total = counts->unbound.total; + counts->unbound = unbound; + res = 0; + +finally: + Py_XDECREF(globalnames_owned); + Py_XDECREF(attrnames_owned); + return res; +} + + /* Here "value" means a non-None value, since a bare return is identical * to returning None explicitly. Likewise a missing return statement * at the end of the function is turned into "return None". */

1 0

GH-114809: Add support for macOS multi-arch builds with the JIT enabled (#131751)
by savannahostrowski April 30, 2025

April 30, 2025

https://github.com/python/cpython/commit/26c0248b54b6b2a5df51dd3da8c0ebb1b2… commit: 26c0248b54b6b2a5df51dd3da8c0ebb1b2958bc4 branch: main author: Savannah Ostrowski <savannahostrowski(a)gmail.com> committer: savannahostrowski <savannahostrowski(a)gmail.com> date: 2025-04-30T11:03:57-07:00 summary: GH-114809: Add support for macOS multi-arch builds with the JIT enabled (#131751) Co-authored-by: Ronald Oussoren <ronaldoussoren(a)mac.com> Co-authored-by: Brandt Bucher <brandtbucher(a)microsoft.com> files: A Misc/NEWS.d/next/Core_and_Builtins/2025-03-26-04-55-25.gh-issue-114809.8rNyT7.rst M .github/workflows/jit.yml M .gitignore M Tools/jit/_targets.py M Tools/jit/build.py M configure M configure.ac diff --git a/.github/workflows/jit.yml b/.github/workflows/jit.yml index e08d3c8bfd61a8..af460f4264932c 100644 --- a/.github/workflows/jit.yml +++ b/.github/workflows/jit.yml @@ -113,7 +113,7 @@ jobs: find /usr/local/bin -lname '*/Library/Frameworks/Python.framework/*' -delete brew install llvm@${{ matrix.llvm }} export SDKROOT="$(xcrun --show-sdk-path)" - ./configure --enable-experimental-jit ${{ matrix.debug && '--with-pydebug' || '' }} + ./configure --enable-experimental-jit --enable-universalsdk --with-universal-archs=universal2 ${{ matrix.debug && '--with-pydebug' || '' }} make all --jobs 4 ./python.exe -m test --multiprocess 0 --timeout 4500 --verbose2 --verbose3 diff --git a/.gitignore b/.gitignore index 360934041c7d38..2a6f249275c32e 100644 --- a/.gitignore +++ b/.gitignore @@ -138,7 +138,7 @@ Tools/unicode/data/ # hendrikmuhs/ccache-action@v1 /.ccache /cross-build/ -/jit_stencils.h +/jit_stencils*.h /platform /profile-clean-stamp /profile-run-stamp diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2025-03-26-04-55-25.gh-issue-114809.8rNyT7.rst b/Misc/NEWS.d/next/Core_and_Builtins/2025-03-26-04-55-25.gh-issue-114809.8rNyT7.rst new file mode 100644 index 00000000000000..19d92c33bc6d21 --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2025-03-26-04-55-25.gh-issue-114809.8rNyT7.rst @@ -0,0 +1 @@ +Add support for macOS multi-arch builds with the JIT enabled diff --git a/Tools/jit/_targets.py b/Tools/jit/_targets.py index f7ea159884c5a4..e0ac735a20f691 100644 --- a/Tools/jit/_targets.py +++ b/Tools/jit/_targets.py @@ -25,6 +25,7 @@ CPYTHON = TOOLS.parent PYTHON_EXECUTOR_CASES_C_H = CPYTHON / "Python" / "executor_cases.c.h" TOOLS_JIT_TEMPLATE_C = TOOLS_JIT / "template.c" +ASYNCIO_RUNNER = asyncio.Runner() _S = typing.TypeVar("_S", _schema.COFFSection, _schema.ELFSection, _schema.MachOSection) _R = typing.TypeVar( @@ -35,6 +36,7 @@ @dataclasses.dataclass class _Target(typing.Generic[_S, _R]): triple: str + condition: str _: dataclasses.KW_ONLY alignment: int = 1 args: typing.Sequence[str] = () @@ -188,7 +190,12 @@ async def _build_stencils(self) -> dict[str, _stencils.StencilGroup]: return stencil_groups def build( - self, out: pathlib.Path, *, comment: str = "", force: bool = False + self, + out: pathlib.Path, + *, + comment: str = "", + force: bool = False, + stencils_h: str = "jit_stencils.h", ) -> None: """Build jit_stencils.h in the given directory.""" if not self.stable: @@ -197,14 +204,14 @@ def build( outline = "=" * len(warning) print("\n".join(["", outline, warning, request, outline, ""])) digest = f"// {self._compute_digest(out)}\n" - jit_stencils = out / "jit_stencils.h" + jit_stencils = out / stencils_h if ( not force and jit_stencils.exists() and jit_stencils.read_text().startswith(digest) ): return - stencil_groups = asyncio.run(self._build_stencils()) + stencil_groups = ASYNCIO_RUNNER.run(self._build_stencils()) jit_stencils_new = out / "jit_stencils.h.new" try: with jit_stencils_new.open("w") as file: @@ -512,10 +519,12 @@ def get_target(host: str) -> _COFF | _ELF | _MachO: """Build a _Target for the given host "triple" and options.""" target: _COFF | _ELF | _MachO if re.fullmatch(r"aarch64-apple-darwin.*", host): - target = _MachO(host, alignment=8, prefix="_") + condition = "defined(__aarch64__) && defined(__APPLE__)" + target = _MachO(host, condition, alignment=8, prefix="_") elif re.fullmatch(r"aarch64-pc-windows-msvc", host): args = ["-fms-runtime-lib=dll", "-fplt"] - target = _COFF(host, alignment=8, args=args) + condition = "defined(_M_ARM64)" + target = _COFF(host, condition, alignment=8, args=args) elif re.fullmatch(r"aarch64-.*-linux-gnu", host): args = [ "-fpic", @@ -523,22 +532,27 @@ def get_target(host: str) -> _COFF | _ELF | _MachO: # was required to disable them. "-mno-outline-atomics", ] - target = _ELF(host, alignment=8, args=args) + condition = "defined(__aarch64__) && defined(__linux__)" + target = _ELF(host, condition, alignment=8, args=args) elif re.fullmatch(r"i686-pc-windows-msvc", host): args = [ "-DPy_NO_ENABLE_SHARED", # __attribute__((preserve_none)) is not supported "-Wno-ignored-attributes", ] - target = _COFF(host, args=args, prefix="_") + condition = "defined(_M_IX86)" + target = _COFF(host, condition, args=args, prefix="_") elif re.fullmatch(r"x86_64-apple-darwin.*", host): - target = _MachO(host, prefix="_") + condition = "defined(__x86_64__) && defined(__APPLE__)" + target = _MachO(host, condition, prefix="_") elif re.fullmatch(r"x86_64-pc-windows-msvc", host): args = ["-fms-runtime-lib=dll"] - target = _COFF(host, args=args) + condition = "defined(_M_X64)" + target = _COFF(host, condition, args=args) elif re.fullmatch(r"x86_64-.*-linux-gnu", host): args = ["-fno-pic", "-mcmodel=medium", "-mlarge-data-threshold=0"] - target = _ELF(host, args=args) + condition = "defined(__x86_64__) && defined(__linux__)" + target = _ELF(host, condition, args=args) else: raise ValueError(host) return target diff --git a/Tools/jit/build.py b/Tools/jit/build.py index a8cb0f67c36363..4d1e484b6838eb 100644 --- a/Tools/jit/build.py +++ b/Tools/jit/build.py @@ -11,7 +11,10 @@ comment = f"$ {shlex.join([pathlib.Path(sys.executable).name] + sys.argv)}" parser = argparse.ArgumentParser(description=__doc__) parser.add_argument( - "target", type=_targets.get_target, help="a PEP 11 target triple to compile for" + "target", + nargs="+", + type=_targets.get_target, + help="a PEP 11 target triple to compile for", ) parser.add_argument( "-d", "--debug", action="store_true", help="compile for a debug build of Python" @@ -23,6 +26,22 @@ "-v", "--verbose", action="store_true", help="echo commands as they are run" ) args = parser.parse_args() - args.target.debug = args.debug - args.target.verbose = args.verbose - args.target.build(pathlib.Path.cwd(), comment=comment, force=args.force) + for target in args.target: + target.debug = args.debug + target.force = args.force + target.verbose = args.verbose + target.build( + pathlib.Path.cwd(), + comment=comment, + stencils_h=f"jit_stencils-{target.triple}.h", + force=args.force, + ) + + with open("jit_stencils.h", "w") as fp: + for idx, target in enumerate(args.target): + fp.write(f"#{'if' if idx == 0 else 'elif'} {target.condition}\n") + fp.write(f'#include "jit_stencils-{target.triple}.h"\n') + + fp.write("#else\n") + fp.write('#error "unexpected target"\n') + fp.write("#endif\n") diff --git a/configure b/configure index 58c4a646c934db..1ec91021a38b66 100755 --- a/configure +++ b/configure @@ -900,6 +900,8 @@ LDSHARED SHLIB_SUFFIX DSYMUTIL_PATH DSYMUTIL +JIT_STENCILS_H +REGEN_JIT_COMMAND UNIVERSAL_ARCH_FLAGS WASM_STDLIB WASM_ASSETS_DIR @@ -928,8 +930,6 @@ LLVM_AR PROFILE_TASK DEF_MAKE_RULE DEF_MAKE_ALL_RULE -JIT_STENCILS_H -REGEN_JIT_COMMAND ABI_THREAD ABIFLAGS LN @@ -1092,13 +1092,13 @@ with_pydebug with_trace_refs enable_pystats with_assertions -enable_experimental_jit enable_optimizations with_lto enable_bolt with_strict_overflow enable_safety enable_slower_safety +enable_experimental_jit with_dsymutil with_address_sanitizer with_memory_sanitizer @@ -1823,9 +1823,6 @@ Optional Features: --disable-gil enable experimental support for running without the GIL (default is no) --enable-pystats enable internal statistics gathering (default is no) - --enable-experimental-jit[=no|yes|yes-off|interpreter] - build the experimental just-in-time compiler - (default is no) --enable-optimizations enable expensive, stable optimizations (PGO, etc.) (default is no) --enable-bolt enable usage of the llvm-bolt post-link optimizer @@ -1834,6 +1831,9 @@ Optional Features: no performance overhead --enable-slower-safety enable usage of the security compiler options with performance overhead + --enable-experimental-jit[=no|yes|yes-off|interpreter] + build the experimental just-in-time compiler + (default is no) --enable-loadable-sqlite-extensions support loadable extensions in the sqlite3 module, see Doc/library/sqlite3.rst (default is no) @@ -8367,51 +8367,6 @@ else printf "%s\n" "no" >&6; } fi -# Check for --enable-experimental-jit: -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for --enable-experimental-jit" >&5 -printf %s "checking for --enable-experimental-jit... " >&6; } -# Check whether --enable-experimental-jit was given. -if test ${enable_experimental_jit+y} -then : - enableval=$enable_experimental_jit; -else case e in #( - e) enable_experimental_jit=no ;; -esac -fi - -case $enable_experimental_jit in - no) jit_flags=""; tier2_flags="" ;; - yes) jit_flags="-D_Py_JIT"; tier2_flags="-D_Py_TIER2=1" ;; - yes-off) jit_flags="-D_Py_JIT"; tier2_flags="-D_Py_TIER2=3" ;; - interpreter) jit_flags=""; tier2_flags="-D_Py_TIER2=4" ;; - interpreter-off) jit_flags=""; tier2_flags="-D_Py_TIER2=6" ;; # Secret option - *) as_fn_error $? "invalid argument: --enable-experimental-jit=$enable_experimental_jit; expected no|yes|yes-off|interpreter" "$LINENO" 5 ;; -esac -if ${tier2_flags:+false} : -then : - -else case e in #( - e) as_fn_append CFLAGS_NODIST " $tier2_flags" ;; -esac -fi -if ${jit_flags:+false} : -then : - -else case e in #( - e) as_fn_append CFLAGS_NODIST " $jit_flags" - REGEN_JIT_COMMAND="\$(PYTHON_FOR_REGEN) \$(srcdir)/Tools/jit/build.py $host" - JIT_STENCILS_H="jit_stencils.h" - if test "x$Py_DEBUG" = xtrue -then : - as_fn_append REGEN_JIT_COMMAND " --debug" -fi ;; -esac -fi - - -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $tier2_flags $jit_flags" >&5 -printf "%s\n" "$tier2_flags $jit_flags" >&6; } - # Enable optimization flags @@ -10706,42 +10661,50 @@ fi UNIVERSAL_ARCH_FLAGS="-arch ppc -arch i386" LIPO_32BIT_FLAGS="" ARCH_RUN_32BIT="" + ARCH_TRIPLES=`echo {ppc,i386}-apple-darwin` ;; 64-bit) UNIVERSAL_ARCH_FLAGS="-arch ppc64 -arch x86_64" LIPO_32BIT_FLAGS="" ARCH_RUN_32BIT="true" + ARCH_TRIPLES=`echo {ppc64,x86_64}-apple-darwin` ;; all) UNIVERSAL_ARCH_FLAGS="-arch i386 -arch ppc -arch ppc64 -arch x86_64" LIPO_32BIT_FLAGS="-extract ppc7400 -extract i386" ARCH_RUN_32BIT="/usr/bin/arch -i386 -ppc" + ARCH_TRIPLES=`echo {i386,ppc,ppc64,x86_64}-apple-darwin` ;; universal2) UNIVERSAL_ARCH_FLAGS="-arch arm64 -arch x86_64" LIPO_32BIT_FLAGS="" LIPO_INTEL64_FLAGS="-extract x86_64" ARCH_RUN_32BIT="true" + ARCH_TRIPLES=`echo {aarch64,x86_64}-apple-darwin` ;; intel) UNIVERSAL_ARCH_FLAGS="-arch i386 -arch x86_64" LIPO_32BIT_FLAGS="-extract i386" ARCH_RUN_32BIT="/usr/bin/arch -i386" + ARCH_TRIPLES=`echo {i386,x86_64}-apple-darwin` ;; intel-32) UNIVERSAL_ARCH_FLAGS="-arch i386" LIPO_32BIT_FLAGS="" ARCH_RUN_32BIT="" + ARCH_TRIPLES=i386-apple-darwin ;; intel-64) UNIVERSAL_ARCH_FLAGS="-arch x86_64" LIPO_32BIT_FLAGS="" ARCH_RUN_32BIT="true" + ARCH_TRIPLES=x86_64-apple-darwin ;; 3-way) UNIVERSAL_ARCH_FLAGS="-arch i386 -arch ppc -arch x86_64" LIPO_32BIT_FLAGS="-extract ppc7400 -extract i386" ARCH_RUN_32BIT="/usr/bin/arch -i386 -ppc" + ARCH_TRIPLES=`echo {i386,ppc,x86_64}-apple-darwin` ;; *) as_fn_error $? "proper usage is --with-universal-arch=universal2|32-bit|64-bit|all|intel|3-way" "$LINENO" 5 @@ -10858,6 +10821,51 @@ else case e in #( esac fi +# Check for --enable-experimental-jit: +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking for --enable-experimental-jit" >&5 +printf %s "checking for --enable-experimental-jit... " >&6; } +# Check whether --enable-experimental-jit was given. +if test ${enable_experimental_jit+y} +then : + enableval=$enable_experimental_jit; +else case e in #( + e) enable_experimental_jit=no ;; +esac +fi + +case $enable_experimental_jit in + no) jit_flags=""; tier2_flags="" ;; + yes) jit_flags="-D_Py_JIT"; tier2_flags="-D_Py_TIER2=1" ;; + yes-off) jit_flags="-D_Py_JIT"; tier2_flags="-D_Py_TIER2=3" ;; + interpreter) jit_flags=""; tier2_flags="-D_Py_TIER2=4" ;; + interpreter-off) jit_flags=""; tier2_flags="-D_Py_TIER2=6" ;; # Secret option + *) as_fn_error $? "invalid argument: --enable-experimental-jit=$enable_experimental_jit; expected no|yes|yes-off|interpreter" "$LINENO" 5 ;; +esac +if ${tier2_flags:+false} : +then : + +else case e in #( + e) as_fn_append CFLAGS_NODIST " $tier2_flags" ;; +esac +fi +if ${jit_flags:+false} : +then : + +else case e in #( + e) as_fn_append CFLAGS_NODIST " $jit_flags" + REGEN_JIT_COMMAND="\$(PYTHON_FOR_REGEN) \$(srcdir)/Tools/jit/build.py ${ARCH_TRIPLES:-$host}" + JIT_STENCILS_H="jit_stencils.h" + if test "x$Py_DEBUG" = xtrue +then : + as_fn_append REGEN_JIT_COMMAND " --debug" +fi ;; +esac +fi + + +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $tier2_flags $jit_flags" >&5 +printf "%s\n" "$tier2_flags $jit_flags" >&6; } + case "$ac_cv_cc_name" in mpicc) CFLAGS_NODIST="$CFLAGS_NODIST" diff --git a/configure.ac b/configure.ac index ee2cabbf7cd268..16367b3ea53c27 100644 --- a/configure.ac +++ b/configure.ac @@ -1802,41 +1802,6 @@ else AC_MSG_RESULT([no]) fi -# Check for --enable-experimental-jit: -AC_MSG_CHECKING([for --enable-experimental-jit]) -AC_ARG_ENABLE([experimental-jit], - [AS_HELP_STRING([--enable-experimental-jit@<:@=no|yes|yes-off|interpreter@:>@], - [build the experimental just-in-time compiler (default is no)])], - [], - [enable_experimental_jit=no]) -case $enable_experimental_jit in - no) jit_flags=""; tier2_flags="" ;; - yes) jit_flags="-D_Py_JIT"; tier2_flags="-D_Py_TIER2=1" ;; - yes-off) jit_flags="-D_Py_JIT"; tier2_flags="-D_Py_TIER2=3" ;; - interpreter) jit_flags=""; tier2_flags="-D_Py_TIER2=4" ;; - interpreter-off) jit_flags=""; tier2_flags="-D_Py_TIER2=6" ;; # Secret option - *) AC_MSG_ERROR( - [invalid argument: --enable-experimental-jit=$enable_experimental_jit; expected no|yes|yes-off|interpreter]) ;; -esac -AS_VAR_IF([tier2_flags], - [], - [], - [AS_VAR_APPEND([CFLAGS_NODIST], [" $tier2_flags"])]) -AS_VAR_IF([jit_flags], - [], - [], - [AS_VAR_APPEND([CFLAGS_NODIST], [" $jit_flags"]) - AS_VAR_SET([REGEN_JIT_COMMAND], - ["\$(PYTHON_FOR_REGEN) \$(srcdir)/Tools/jit/build.py $host"]) - AS_VAR_SET([JIT_STENCILS_H], ["jit_stencils.h"]) - AS_VAR_IF([Py_DEBUG], - [true], - [AS_VAR_APPEND([REGEN_JIT_COMMAND], [" --debug"])], - [])]) -AC_SUBST([REGEN_JIT_COMMAND]) -AC_SUBST([JIT_STENCILS_H]) -AC_MSG_RESULT([$tier2_flags $jit_flags]) - # Enable optimization flags AC_SUBST([DEF_MAKE_ALL_RULE]) AC_SUBST([DEF_MAKE_RULE]) @@ -2652,42 +2617,50 @@ AS_VAR_IF([ac_cv_gcc_compat], [yes], [ UNIVERSAL_ARCH_FLAGS="-arch ppc -arch i386" LIPO_32BIT_FLAGS="" ARCH_RUN_32BIT="" + ARCH_TRIPLES=`echo {ppc,i386}-apple-darwin` ;; 64-bit) UNIVERSAL_ARCH_FLAGS="-arch ppc64 -arch x86_64" LIPO_32BIT_FLAGS="" ARCH_RUN_32BIT="true" + ARCH_TRIPLES=`echo {ppc64,x86_64}-apple-darwin` ;; all) UNIVERSAL_ARCH_FLAGS="-arch i386 -arch ppc -arch ppc64 -arch x86_64" LIPO_32BIT_FLAGS="-extract ppc7400 -extract i386" ARCH_RUN_32BIT="/usr/bin/arch -i386 -ppc" + ARCH_TRIPLES=`echo {i386,ppc,ppc64,x86_64}-apple-darwin` ;; universal2) UNIVERSAL_ARCH_FLAGS="-arch arm64 -arch x86_64" LIPO_32BIT_FLAGS="" LIPO_INTEL64_FLAGS="-extract x86_64" ARCH_RUN_32BIT="true" + ARCH_TRIPLES=`echo {aarch64,x86_64}-apple-darwin` ;; intel) UNIVERSAL_ARCH_FLAGS="-arch i386 -arch x86_64" LIPO_32BIT_FLAGS="-extract i386" ARCH_RUN_32BIT="/usr/bin/arch -i386" + ARCH_TRIPLES=`echo {i386,x86_64}-apple-darwin` ;; intel-32) UNIVERSAL_ARCH_FLAGS="-arch i386" LIPO_32BIT_FLAGS="" ARCH_RUN_32BIT="" + ARCH_TRIPLES=i386-apple-darwin ;; intel-64) UNIVERSAL_ARCH_FLAGS="-arch x86_64" LIPO_32BIT_FLAGS="" ARCH_RUN_32BIT="true" + ARCH_TRIPLES=x86_64-apple-darwin ;; 3-way) UNIVERSAL_ARCH_FLAGS="-arch i386 -arch ppc -arch x86_64" LIPO_32BIT_FLAGS="-extract ppc7400 -extract i386" ARCH_RUN_32BIT="/usr/bin/arch -i386 -ppc" + ARCH_TRIPLES=`echo {i386,ppc,x86_64}-apple-darwin` ;; *) AC_MSG_ERROR([proper usage is --with-universal-arch=universal2|32-bit|64-bit|all|intel|3-way]) @@ -2778,6 +2751,41 @@ AS_VAR_IF([ac_cv_gcc_compat], [yes], [ esac ]) +# Check for --enable-experimental-jit: +AC_MSG_CHECKING([for --enable-experimental-jit]) +AC_ARG_ENABLE([experimental-jit], + [AS_HELP_STRING([--enable-experimental-jit@<:@=no|yes|yes-off|interpreter@:>@], + [build the experimental just-in-time compiler (default is no)])], + [], + [enable_experimental_jit=no]) +case $enable_experimental_jit in + no) jit_flags=""; tier2_flags="" ;; + yes) jit_flags="-D_Py_JIT"; tier2_flags="-D_Py_TIER2=1" ;; + yes-off) jit_flags="-D_Py_JIT"; tier2_flags="-D_Py_TIER2=3" ;; + interpreter) jit_flags=""; tier2_flags="-D_Py_TIER2=4" ;; + interpreter-off) jit_flags=""; tier2_flags="-D_Py_TIER2=6" ;; # Secret option + *) AC_MSG_ERROR( + [invalid argument: --enable-experimental-jit=$enable_experimental_jit; expected no|yes|yes-off|interpreter]) ;; +esac +AS_VAR_IF([tier2_flags], + [], + [], + [AS_VAR_APPEND([CFLAGS_NODIST], [" $tier2_flags"])]) +AS_VAR_IF([jit_flags], + [], + [], + [AS_VAR_APPEND([CFLAGS_NODIST], [" $jit_flags"]) + AS_VAR_SET([REGEN_JIT_COMMAND], + ["\$(PYTHON_FOR_REGEN) \$(srcdir)/Tools/jit/build.py ${ARCH_TRIPLES:-$host}"]) + AS_VAR_SET([JIT_STENCILS_H], ["jit_stencils.h"]) + AS_VAR_IF([Py_DEBUG], + [true], + [AS_VAR_APPEND([REGEN_JIT_COMMAND], [" --debug"])], + [])]) +AC_SUBST([REGEN_JIT_COMMAND]) +AC_SUBST([JIT_STENCILS_H]) +AC_MSG_RESULT([$tier2_flags $jit_flags]) + case "$ac_cv_cc_name" in mpicc) CFLAGS_NODIST="$CFLAGS_NODIST"

1 0

[3.13] Apply 'mod' role to typing module (GH-133201) (#133217)
by AA-Turner April 30, 2025

April 30, 2025

https://github.com/python/cpython/commit/4f006ffdb9c87c54cea9226c23fb1b9712… commit: 4f006ffdb9c87c54cea9226c23fb1b97120a01a9 branch: 3.13 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: AA-Turner <9087854+AA-Turner(a)users.noreply.github.com> date: 2025-04-30T16:58:19Z summary: [3.13] Apply 'mod' role to typing module (GH-133201) (#133217) Apply 'mod' role to typing module (GH-133201) (cherry picked from commit 2b67db7ce3d73c9ad905125f9a602657afe9be9f) Co-authored-by: Rafael Fontenelle <rffontenelle(a)users.noreply.github.com> Co-authored-by: Adam Turner <9087854+AA-Turner(a)users.noreply.github.com> files: M Doc/library/typing.rst diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index bba7114fc1ff33..ee51f81846e9b3 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -665,7 +665,7 @@ through a simple assignment:: User-defined generics for parameter expressions are also supported via parameter specification variables in the form ``[**P]``. The behavior is consistent with type variables' described above as parameter specification variables are -treated by the typing module as a specialized type variable. The one exception +treated by the :mod:`!typing` module as a specialized type variable. The one exception to this is that a list of types can be used to substitute a :class:`ParamSpec`:: >>> class Z[T, **P]: ... # T is a TypeVar; P is a ParamSpec @@ -706,7 +706,7 @@ are intended primarily for static type checking. A user-defined generic class can have ABCs as base classes without a metaclass conflict. Generic metaclasses are not supported. The outcome of parameterizing -generics is cached, and most types in the typing module are :term:`hashable` and +generics is cached, and most types in the :mod:`!typing` module are :term:`hashable` and comparable for equality. @@ -2685,7 +2685,7 @@ types. Protocols --------- -The following protocols are provided by the typing module. All are decorated +The following protocols are provided by the :mod:`!typing` module. All are decorated with :func:`@runtime_checkable <runtime_checkable>`. .. class:: SupportsAbs @@ -3384,7 +3384,7 @@ Deprecated aliases ------------------ This module defines several deprecated aliases to pre-existing -standard library classes. These were originally included in the typing +standard library classes. These were originally included in the :mod:`!typing` module in order to support parameterizing these generic classes using ``[]``. However, the aliases became redundant in Python 3.9 when the corresponding pre-existing classes were enhanced to support ``[]`` (see @@ -3397,7 +3397,7 @@ interpreter for these aliases. If at some point it is decided to remove these deprecated aliases, a deprecation warning will be issued by the interpreter for at least two releases -prior to removal. The aliases are guaranteed to remain in the typing module +prior to removal. The aliases are guaranteed to remain in the :mod:`!typing` module without deprecation warnings until at least Python 3.14. Type checkers are encouraged to flag uses of the deprecated types if the

1 0

Apply 'mod' role to typing module (#133201)
by AA-Turner April 30, 2025

April 30, 2025

https://github.com/python/cpython/commit/2b67db7ce3d73c9ad905125f9a602657af… commit: 2b67db7ce3d73c9ad905125f9a602657afe9be9f branch: main author: Rafael Fontenelle <rffontenelle(a)users.noreply.github.com> committer: AA-Turner <9087854+AA-Turner(a)users.noreply.github.com> date: 2025-04-30T16:52:03Z summary: Apply 'mod' role to typing module (#133201) Co-authored-by: Adam Turner <9087854+AA-Turner(a)users.noreply.github.com> files: M Doc/library/typing.rst diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index a0beed4a8c77fb..acf235818edc5b 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -665,7 +665,7 @@ through a simple assignment:: User-defined generics for parameter expressions are also supported via parameter specification variables in the form ``[**P]``. The behavior is consistent with type variables' described above as parameter specification variables are -treated by the typing module as a specialized type variable. The one exception +treated by the :mod:`!typing` module as a specialized type variable. The one exception to this is that a list of types can be used to substitute a :class:`ParamSpec`:: >>> class Z[T, **P]: ... # T is a TypeVar; P is a ParamSpec @@ -706,7 +706,7 @@ are intended primarily for static type checking. A user-defined generic class can have ABCs as base classes without a metaclass conflict. Generic metaclasses are not supported. The outcome of parameterizing -generics is cached, and most types in the typing module are :term:`hashable` and +generics is cached, and most types in the :mod:`!typing` module are :term:`hashable` and comparable for equality. @@ -2787,7 +2787,7 @@ types. Protocols --------- -The following protocols are provided by the typing module. All are decorated +The following protocols are provided by the :mod:`!typing` module. All are decorated with :func:`@runtime_checkable <runtime_checkable>`. .. class:: SupportsAbs @@ -3531,7 +3531,7 @@ Deprecated aliases ------------------ This module defines several deprecated aliases to pre-existing -standard library classes. These were originally included in the typing +standard library classes. These were originally included in the :mod:`!typing` module in order to support parameterizing these generic classes using ``[]``. However, the aliases became redundant in Python 3.9 when the corresponding pre-existing classes were enhanced to support ``[]`` (see @@ -3544,7 +3544,7 @@ interpreter for these aliases. If at some point it is decided to remove these deprecated aliases, a deprecation warning will be issued by the interpreter for at least two releases -prior to removal. The aliases are guaranteed to remain in the typing module +prior to removal. The aliases are guaranteed to remain in the :mod:`!typing` module without deprecation warnings until at least Python 3.14. Type checkers are encouraged to flag uses of the deprecated types if the

1 0

[3.13] Remove redundant ``--keep-going`` when running Sphinx (GH-133156) (#133214)
by AA-Turner April 30, 2025

April 30, 2025

https://github.com/python/cpython/commit/110bec88bf09311a24447241076b16e649… commit: 110bec88bf09311a24447241076b16e6492a3768 branch: 3.13 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: AA-Turner <9087854+AA-Turner(a)users.noreply.github.com> date: 2025-04-30T17:44:00+01:00 summary: [3.13] Remove redundant ``--keep-going`` when running Sphinx (GH-133156) (#133214) Remove redundant ``--keep-going`` when running Sphinx (GH-133156) (cherry picked from commit c78216e42c7548f148708fc08cfefbcda9b3ae01) Co-authored-by: Hugo van Kemenade <1324225+hugovk(a)users.noreply.github.com> files: M .github/workflows/reusable-docs.yml diff --git a/.github/workflows/reusable-docs.yml b/.github/workflows/reusable-docs.yml index 79c28223ac3706..657e0a6bf662f7 100644 --- a/.github/workflows/reusable-docs.yml +++ b/.github/workflows/reusable-docs.yml @@ -66,7 +66,7 @@ jobs: run: | set -Eeuo pipefail # Build docs with the nit-picky option; write warnings to file - make -C Doc/ PYTHON=../python SPHINXOPTS="--quiet --nitpicky --fail-on-warning --keep-going --warning-file sphinx-warnings.txt" html + make -C Doc/ PYTHON=../python SPHINXOPTS="--quiet --nitpicky --fail-on-warning --warning-file sphinx-warnings.txt" html - name: 'Check warnings' if: github.event_name == 'pull_request' run: | @@ -101,4 +101,4 @@ jobs: run: make -C Doc/ PYTHON=../python venv # Use "xvfb-run" since some doctest tests open GUI windows - name: 'Run documentation doctest' - run: xvfb-run make -C Doc/ PYTHON=../python SPHINXERRORHANDLING="--fail-on-warning --keep-going" doctest + run: xvfb-run make -C Doc/ PYTHON=../python SPHINXERRORHANDLING="--fail-on-warning" doctest

1 0