Mailman 3 February 2022 - capi-sig

New script: add Python 3.10 support to your C extensions without losing Python 3.6 support
by Victor Stinner 07 May '22

07 May '22

Hi, I wrote a new script which adds Python 3.10 support to your C extensions without losing Python 3.6 support: https://github.com/pythoncapi/pythoncapi_compat For example, it replaces "op->ob_type" with "Py_TYPE(op)" and replaces "frame->f_back" with "_PyFrame_GetBackBorrow(frame)". It relies on the pythoncapi_compat.h header file that I wrote to implement Python 3.9 and Python 3.10 on old Python versions. Examples: Py_NewRef() and PyThreadState_GetFrame(). The _PyFrame_GetBackBorrow() function doesn't exist in the Python C API, it's only provided by pythoncapi_compat.h to ease the migration of C extensions. I advise you to replace _PyFrame_GetBackBorrow() (borrowed reference) with PyFrame_GetBack() (strong reference). The PyFrame_GetBack() function was added to Python 3.9 and pythoncapi_compat.h provides it on older Python versions. This project is related to my PEP 620 "Hide implementation details from the C API" which tries to make the C API more abstract to later allow to implement new optimization in CPython and to make other Python implementations like PyPy faster when running C extensions. This project only targets extension modules written in C by using directly the "Python.h" API. I advise you to use Cython or HPy to no longer be bothered with incompatible C API changes at every Python release ;-) * https://cython.org/ * https://hpy.readthedocs.io/ I hope that my script will facilitate migration of C extensions to HPy. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

2 2

ANN: expy 0.6.5 released!
by Yingjie Lan 16 Apr '22

16 Apr '22

EXPY is an express way to extend Python! EXPY provides a way to extend python in an elegant way. For more information and a tutorial, see: http://expy.sourceforge.net/ What's new: 1. Correct treatment of __init__ method. 2. Give warnings of missing Py_INCREF on appropriate special type methods. 3. Documentation update. Cheers, Yingjie

3 3

Deallocating objects from previous run
by Victor Milovanov 31 Mar '22

31 Mar '22

Hi, TL;DR; Is it safe to Py_DecRef an object that was created before Py_Finalize after Py_Initialize was called to restart Python runtime? I am debugging 3.9+ support in https://github.com/pythonnet/pythonnet The issue I originally got is validate_list from gcmodule.c fails at assert(prev == GC_PREV(head)); In debugging this, I sprinkled my code with calls to validate_list, and found that the assertion fails in the following scenario: Py_Initialize(); ... PyObject* myObj = PyObject_Call(simple_class, empty_tuple, NULL); Py_Finalize(); ... Py_Initialize(); ... validate_list(gc.generations[2], collecting_clear_unreachable_clear); // OK Py_DecRef(myObj); validate_list(gc.generations[2], collecting_clear_unreachable_clear); // EXPLOSION Now I have not yet tried to narrow down the steps further, because I realised that I am unsure about the question in TL;DR;. E.g. is this use of myObj supported by embedding API in general, or is this expected, that after reinitializing the runtime Py_DecRef on old objects would break GC? In my scenario simple_class is defined in Python as inheriting from object, and only has two attributes set at runtime. One has value builtins.StopIteration, and the other one is an instance of traceback. Regards, Victor

4 3

PyFrameObject structure has been moved to the internal C API
by Victor Stinner 21 Mar '22

21 Mar '22

Hi, tl; dr the C API of PyFrameObject changed. If you have issues, please speak up! I moved the PyFrameObject structure to the internal C API headers. You should now use the public C API to access its members: * ``f_back``: use :c:func:`PyFrame_GetBack`. * ``f_blockstack``: removed. * ``f_builtins``: use ``PyObject_GetAttrString((PyObject*)frame, "f_builtins")``. * ``f_code``: use :c:func:`PyFrame_GetCode`. * ``f_gen``: removed. * ``f_globals``: use ``PyObject_GetAttrString((PyObject*)frame, "f_globals")``. * ``f_iblock``: removed. * ``f_lasti``: use ``PyObject_GetAttrString((PyObject*)frame, "f_lasti")``. Code using ``f_lasti`` with ``PyCode_Addr2Line()`` must use :c:func:`PyFrame_GetLineNumber` instead. * ``f_lineno``: use :c:func:`PyFrame_GetLineNumber` * ``f_locals``: use ``PyObject_GetAttrString((PyObject*)frame, "f_locals")``. * ``f_stackdepth``: removed. * ``f_state``: no public API (renamed to ``f_frame.f_state``). * ``f_trace``: no public API. * ``f_trace_lines``: use ``PyObject_GetAttrString((PyObject*)frame, "f_trace_lines")`` (it also be modified). * ``f_trace_opcodes``: use ``PyObject_GetAttrString((PyObject*)frame, "f_trace_opcodes")`` (it also be modified). * ``f_localsplus``: no public API (renamed to ``f_frame.localsplus``). * ``f_valuestack``: removed. See What's New in Python 3.11 for more details (I completed the doc, but it will only be updated online tomorrow, there is a cron task once per day): https://docs.python.org/dev/whatsnew/3.11.html#id2 You can use the pythoncapi_compat project to get PyFrame_GetBack() and PyFrame_GetCode() on Python 3.8 and older: https://pythoncapi-compat.readthedocs.io/ If there is no public C API to access the PyFrameObject members used by your project, please speak up! See for example my PR to use the public C API in coverage: https://github.com/nedbat/coveragepy/pull/1331 There is an on-going work for 2 years to add new getter functions for PyFrameObject: https://bugs.python.org/issue40421 In Python 3.10, there was no rush to make the structure internal. Things changed quickly in Python 3.11: a heavy rework on Python internals is on-going to optimize Python (especially ceval.c and how frames are created and used). For example, the _PyEval_EvalFrameDefault() function now gets a new InterpreterFrame structure, it no longer takes a PyFrameObject. Python 3.11 now creates Python frame objects on demand. Because of these changes, reading directly PyFrameObject.f_back is now an error since the member is not always filled, but it wasn't a compiler error. You must now call the public PyFrame_GetBack() function rather than accessing directly to the member. Since Python 2.3 (that's quite old ;-)), reading directly PyFrameObject.f_lineno is unsafe since it depends if the Python code is being traced or not: you must use PyFrame_GetLineNumber() instead. See also the issue for the longer rationale: https://bugs.python.org/issue46836 I know that these changes are inconvenient, but I'm here if you need my help for updating your project! Moreover, Python 3.11 is scheduled for October, so there are still a few months to adapt your project, and maybe add new getter functions. The best would be to add all required getter functions before Python 3.11 beta1 scheduled for the beginning of May (2022-05-06). Victor -- Night gathers, and now my watch begins. It shall not end until my death.

3 6

ANNOUNCE: PyBindGen 0.8
by Gustavo Carneiro 10 Mar '22

10 Mar '22

The other day I released PyBindGen 0.8. Main news is that it features a new experimental header file scanner based on pygccxml (i.e., it's similar to py++ in scope, if not in maturity, but does not use boost.pythonunderneath). == What == PyBindGen is a Python module that is geared to generating C/C++ code that binds a C/C++ library for Python. It does so without extensive use of either C++ templates or C pre-processor macros. It has modular handling of C/C++ types, and can be easily extended with Python plugins. The generated code is almost as clean as what a human programmer would write, and does not depend on any library or header files besides Python itself. == Where == https://launchpad.net/pybindgen/ == NEWS == - Support C++ instance attributes through getter/setter methods - Support functions as methods of C++ classes - Support the PyObject* type - Support unsigned int, C strings (char*) (from Mark Lee) - Add basic support for enum types - New experimental automatic module generator based on C/C++ header file scanner and annotations in comments, using pygccxml - Some bug fixes -- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit "The universe is always one step beyond logic." -- Frank Herbert

2 1

C API equivalent to abc.ABCMeta.register?
by Daniel Stutzbach 02 Mar '22

02 Mar '22

I'm the author of an extension module (blist) that provides a type that fits the MutableSequence API. Is there a canonical way for me to register the type as a MutableSequence from the C API? -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>

3 2

Replacing PySequence_Check / PyMapping_Check
by Aviram Hassan 13 Feb '22

13 Feb '22

Hi all, Tl;dr - As mentioned in issue https://bugs.python.org/issue5945, `PySequence_Check` and `PyMapping_Check` should be deprecated and discontinued. The reason for this is that they do not provide accurate results and can lead to unexpected behavior. (PyMapping_Check for a list will return True, and PySequence_Check for a mapping that doesn't subclass dict will also return true) This proposal is to have a process to deprecate those and introduce new functions instead. Thanks to the latest changes in the C-API, we now have flags set for classes that are sequences or mapping - those were created for the pattern matching functionality. This lets us have a path for clear and accurate determination of sequence or mapping. The use cases for the change are as follows: - C-API code that wishes to use the PySequence/PyMapping API and to guarantee correct behavior needs to check it is indeed a valid sequence/mapping - Static types casting from ambiguous “PyObject” to “PySequence” & “PyMapping” types as provided by PyO3 ( https://pyo3.rs/v0.15.1/conversions/tables.html#argument-types) The correct way to determine right now if an object is a sequence, is to check its `Py_TPFLAGS_SEQUENCE` flag and check if it’s a string, bytes or bytearray as they are the only sequences not having `TPFLAGS_SEQUENCE` per specification. This is cumbersome for most non-expert developers not aware of the nuances in the C-API. Having a C-API that provides this check out of the box can ease it for most. `PyMapping_Check` and `PySequence_Check` are still widely used and are **part of the stable ABI** even though core developers recommend avoiding those. This is very odd to stabilize and is an API that should be deprecated. Our suggested change is to add two new functions: `PyMapping_CheckNew` & `PySequence_CheckNew`. They will be implemented using the flags check and the str/bytes/bytearray check. Those functions will be 100% accurate and follow the existing documentations. We will do the following process to improve C-API: 1. Add these functions. 2. Add a deprecation notice on `PyMapping_Check` and `PySequence_Check` and remove all built-in usage of it. We will recommend the use of the new functions. 3. Drop the old functions. 4. Alias the new functions to `PyMapping_Check` and `PySequence_Check` and add a deprecation notice of `PyMapping_CheckNew` and `PySequence_CheckNew`. 5. Rename `PyMapping_CheckNew` and `PySequence_CheckNew` to `Mapping_Check` and `PySequence_Check`. The timeline will probably be long as it’s part of stable ABI, but in our opinion, the change is for the better. Caveats: - Sequences and Mappings will match the typing.Sequence and typing.Mapping abcs. Duck typing is further “discouraged”. We do not think of it as an issue since Python already has idioms and static type checkers that discourage this sort of duck typing. - C extensions skipping multiple minor Python versions will use the new code without their knowledge, which is not backwards compatible. This is an issue with any removal of a stable ABI or language function, and we consider it acceptable. Thanks to Bar Harel for helping me with the proposal. This is was initially discussed in this bpo: https://bugs.python.org/issue46376 Best regards, Aviram

6 6

capi-sig February 2022