Hi,
I wrote a new script which adds Python 3.10 support to your C
extensions without losing Python 3.6 support:
https://github.com/pythoncapi/pythoncapi_compat
For example, it replaces "op->ob_type" with "Py_TYPE(op)" and replaces
"frame->f_back" with "_PyFrame_GetBackBorrow(frame)".
It relies on the pythoncapi_compat.h header file that I wrote to
implement Python 3.9 and Python 3.10 on old Python versions. Examples:
Py_NewRef() and PyThreadState_GetFrame().
The _PyFrame_GetBackBorrow() function doesn't exist in the Python C
API, it's only provided by pythoncapi_compat.h to ease the migration
of C extensions. I advise you to replace _PyFrame_GetBackBorrow()
(borrowed reference) with PyFrame_GetBack() (strong reference). The
PyFrame_GetBack() function was added to Python 3.9 and
pythoncapi_compat.h provides it on older Python versions.
This project is related to my PEP 620 "Hide implementation details
from the C API" which tries to make the C API more abstract to later
allow to implement new optimization in CPython and to make other
Python implementations like PyPy faster when running C extensions.
This project only targets extension modules written in C by using
directly the "Python.h" API. I advise you to use Cython or HPy to no
longer be bothered with incompatible C API changes at every Python
release ;-)
* https://cython.org/
* https://hpy.readthedocs.io/
I hope that my script will facilitate migration of C extensions to HPy.
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
EXPY is an express way to extend Python!
EXPY provides a way to extend python in an elegant way. For more information and a tutorial, see: http://expy.sourceforge.net/
What's new:
1. Correct treatment of __init__ method.
2. Give warnings of missing Py_INCREF on
appropriate special type methods.
3. Documentation update.
Cheers,
Yingjie
Hi,
TL;DR; Is it safe to Py_DecRef an object that was created before Py_Finalize
after Py_Initialize was called to restart Python runtime?
I am debugging 3.9+ support in https://github.com/pythonnet/pythonnet
The issue I originally got is validate_list from gcmodule.c fails at
assert(prev == GC_PREV(head));
In debugging this, I sprinkled my code with calls to validate_list, and found
that the assertion fails in the following scenario:
Py_Initialize();
...
PyObject* myObj = PyObject_Call(simple_class, empty_tuple, NULL);
Py_Finalize();
...
Py_Initialize();
...
validate_list(gc.generations[2], collecting_clear_unreachable_clear); // OK
Py_DecRef(myObj);
validate_list(gc.generations[2], collecting_clear_unreachable_clear); // EXPLOSION
Now I have not yet tried to narrow down the steps further, because I realised
that I am unsure about the question in TL;DR;. E.g. is this use of myObj
supported by embedding API in general, or is this expected, that after
reinitializing the runtime Py_DecRef on old objects would break GC?
In my scenario simple_class is defined in Python as inheriting from object,
and only has two attributes set at runtime. One has value
builtins.StopIteration, and the other one is an instance of traceback.
Regards,
Victor
Hi,
tl; dr the C API of PyFrameObject changed. If you have issues, please speak up!
I moved the PyFrameObject structure to the internal C API headers. You
should now use the public C API to access its members:
* ``f_back``: use :c:func:`PyFrame_GetBack`.
* ``f_blockstack``: removed.
* ``f_builtins``: use ``PyObject_GetAttrString((PyObject*)frame,
"f_builtins")``.
* ``f_code``: use :c:func:`PyFrame_GetCode`.
* ``f_gen``: removed.
* ``f_globals``: use ``PyObject_GetAttrString((PyObject*)frame, "f_globals")``.
* ``f_iblock``: removed.
* ``f_lasti``: use ``PyObject_GetAttrString((PyObject*)frame, "f_lasti")``.
Code using ``f_lasti`` with ``PyCode_Addr2Line()`` must use
:c:func:`PyFrame_GetLineNumber` instead.
* ``f_lineno``: use :c:func:`PyFrame_GetLineNumber`
* ``f_locals``: use ``PyObject_GetAttrString((PyObject*)frame, "f_locals")``.
* ``f_stackdepth``: removed.
* ``f_state``: no public API (renamed to ``f_frame.f_state``).
* ``f_trace``: no public API.
* ``f_trace_lines``: use ``PyObject_GetAttrString((PyObject*)frame,
"f_trace_lines")``
(it also be modified).
* ``f_trace_opcodes``: use ``PyObject_GetAttrString((PyObject*)frame,
"f_trace_opcodes")``
(it also be modified).
* ``f_localsplus``: no public API (renamed to ``f_frame.localsplus``).
* ``f_valuestack``: removed.
See What's New in Python 3.11 for more details (I completed the doc,
but it will only be updated online tomorrow, there is a cron task once
per day):
https://docs.python.org/dev/whatsnew/3.11.html#id2
You can use the pythoncapi_compat project to get PyFrame_GetBack() and
PyFrame_GetCode() on Python 3.8 and older:
https://pythoncapi-compat.readthedocs.io/
If there is no public C API to access the PyFrameObject members used
by your project, please speak up!
See for example my PR to use the public C API in coverage:
https://github.com/nedbat/coveragepy/pull/1331
There is an on-going work for 2 years to add new getter functions for
PyFrameObject:
https://bugs.python.org/issue40421
In Python 3.10, there was no rush to make the structure internal.
Things changed quickly in Python 3.11: a heavy rework on Python
internals is on-going to optimize Python (especially ceval.c and how
frames are created and used). For example, the
_PyEval_EvalFrameDefault() function now gets a new InterpreterFrame
structure, it no longer takes a PyFrameObject. Python 3.11 now creates
Python frame objects on demand.
Because of these changes, reading directly PyFrameObject.f_back is now
an error since the member is not always filled, but it wasn't a
compiler error. You must now call the public PyFrame_GetBack()
function rather than accessing directly to the member.
Since Python 2.3 (that's quite old ;-)), reading directly
PyFrameObject.f_lineno is unsafe since it depends if the Python code
is being traced or not: you must use PyFrame_GetLineNumber() instead.
See also the issue for the longer rationale:
https://bugs.python.org/issue46836
I know that these changes are inconvenient, but I'm here if you need
my help for updating your project! Moreover, Python 3.11 is scheduled
for October, so there are still a few months to adapt your project,
and maybe add new getter functions. The best would be to add all
required getter functions before Python 3.11 beta1 scheduled for the
beginning of May (2022-05-06).
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
The other day I released PyBindGen 0.8. Main news is that it features a new
experimental header file scanner based on pygccxml (i.e., it's similar to
py++ in scope, if not in maturity, but does not use boost.pythonunderneath).
== What ==
PyBindGen is a Python module that is geared to generating C/C++ code that
binds a C/C++ library for Python. It does so without extensive use of either
C++ templates or C pre-processor macros. It has modular handling of C/C++
types, and can be easily extended with Python plugins. The generated code is
almost as clean as what a human programmer would write, and does not depend
on any library or header files besides Python itself.
== Where ==
https://launchpad.net/pybindgen/
== NEWS ==
- Support C++ instance attributes through getter/setter methods
- Support functions as methods of C++ classes
- Support the PyObject* type
- Support unsigned int, C strings (char*) (from Mark Lee)
- Add basic support for enum types
- New experimental automatic module generator based on C/C++
header file scanner and annotations in comments, using pygccxml
- Some bug fixes
--
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
"The universe is always one step beyond logic." -- Frank Herbert
I'm the author of an extension module (blist) that provides a type that fits
the MutableSequence API. Is there a canonical way for me to register the
type as a MutableSequence from the C API?
--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>
Hi all,
Tl;dr - As mentioned in issue https://bugs.python.org/issue5945,
`PySequence_Check` and `PyMapping_Check` should be deprecated and
discontinued. The reason for this is that they do not provide accurate
results and can lead to unexpected behavior. (PyMapping_Check for a list
will return True, and PySequence_Check for a mapping that doesn't subclass
dict will also return true)
This proposal is to have a process to deprecate those and introduce new
functions instead.
Thanks to the latest changes in the C-API, we now have flags set for
classes that are sequences or mapping - those were created for the pattern
matching functionality.
This lets us have a path for clear and accurate determination of sequence
or mapping.
The use cases for the change are as follows:
-
C-API code that wishes to use the PySequence/PyMapping API and to
guarantee correct behavior needs to check it is indeed a valid
sequence/mapping
-
Static types casting from ambiguous “PyObject” to “PySequence” &
“PyMapping” types as provided by PyO3 (
https://pyo3.rs/v0.15.1/conversions/tables.html#argument-types)
The correct way to determine right now if an object is a sequence, is to
check its `Py_TPFLAGS_SEQUENCE` flag and check if it’s a string, bytes or
bytearray as they are the only sequences not having `TPFLAGS_SEQUENCE` per
specification. This is cumbersome for most non-expert developers not aware
of the nuances in the C-API. Having a C-API that provides this check out of
the box can ease it for most.
`PyMapping_Check` and `PySequence_Check` are still widely used and are
**part of the stable ABI** even though core developers recommend avoiding
those. This is very odd to stabilize and is an API that should be
deprecated.
Our suggested change is to add two new functions: `PyMapping_CheckNew` &
`PySequence_CheckNew`. They will be implemented using the flags check and
the str/bytes/bytearray check. Those functions will be 100% accurate and
follow the existing documentations.
We will do the following process to improve C-API:
1.
Add these functions.
2.
Add a deprecation notice on `PyMapping_Check` and `PySequence_Check` and
remove all built-in usage of it. We will recommend the use of the new
functions.
3.
Drop the old functions.
4.
Alias the new functions to `PyMapping_Check` and `PySequence_Check` and
add a deprecation notice of `PyMapping_CheckNew` and `PySequence_CheckNew`.
5.
Rename `PyMapping_CheckNew` and `PySequence_CheckNew` to `Mapping_Check`
and `PySequence_Check`.
The timeline will probably be long as it’s part of stable ABI, but in our
opinion, the change is for the better.
Caveats:
-
Sequences and Mappings will match the typing.Sequence and typing.Mapping
abcs. Duck typing is further “discouraged”. We do not think of it as an
issue since Python already has idioms and static type checkers that
discourage this sort of duck typing.
-
C extensions skipping multiple minor Python versions will use the new
code without their knowledge, which is not backwards compatible. This is an
issue with any removal of a stable ABI or language function, and we
consider it acceptable.
Thanks to Bar Harel for helping me with the proposal.
This is was initially discussed in this bpo:
https://bugs.python.org/issue46376
Best regards,
Aviram