I'm the author of an extension module (blist) that provides a type that fits
the MutableSequence API. Is there a canonical way for me to register the
type as a MutableSequence from the C API?
--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>
Hi all,
Tl;dr - As mentioned in issue https://bugs.python.org/issue5945,
`PySequence_Check` and `PyMapping_Check` should be deprecated and
discontinued. The reason for this is that they do not provide accurate
results and can lead to unexpected behavior. (PyMapping_Check for a list
will return True, and PySequence_Check for a mapping that doesn't subclass
dict will also return true)
This proposal is to have a process to deprecate those and introduce new
functions instead.
Thanks to the latest changes in the C-API, we now have flags set for
classes that are sequences or mapping - those were created for the pattern
matching functionality.
This lets us have a path for clear and accurate determination of sequence
or mapping.
The use cases for the change are as follows:
-
C-API code that wishes to use the PySequence/PyMapping API and to
guarantee correct behavior needs to check it is indeed a valid
sequence/mapping
-
Static types casting from ambiguous “PyObject” to “PySequence” &
“PyMapping” types as provided by PyO3 (
https://pyo3.rs/v0.15.1/conversions/tables.html#argument-types)
The correct way to determine right now if an object is a sequence, is to
check its `Py_TPFLAGS_SEQUENCE` flag and check if it’s a string, bytes or
bytearray as they are the only sequences not having `TPFLAGS_SEQUENCE` per
specification. This is cumbersome for most non-expert developers not aware
of the nuances in the C-API. Having a C-API that provides this check out of
the box can ease it for most.
`PyMapping_Check` and `PySequence_Check` are still widely used and are
**part of the stable ABI** even though core developers recommend avoiding
those. This is very odd to stabilize and is an API that should be
deprecated.
Our suggested change is to add two new functions: `PyMapping_CheckNew` &
`PySequence_CheckNew`. They will be implemented using the flags check and
the str/bytes/bytearray check. Those functions will be 100% accurate and
follow the existing documentations.
We will do the following process to improve C-API:
1.
Add these functions.
2.
Add a deprecation notice on `PyMapping_Check` and `PySequence_Check` and
remove all built-in usage of it. We will recommend the use of the new
functions.
3.
Drop the old functions.
4.
Alias the new functions to `PyMapping_Check` and `PySequence_Check` and
add a deprecation notice of `PyMapping_CheckNew` and `PySequence_CheckNew`.
5.
Rename `PyMapping_CheckNew` and `PySequence_CheckNew` to `Mapping_Check`
and `PySequence_Check`.
The timeline will probably be long as it’s part of stable ABI, but in our
opinion, the change is for the better.
Caveats:
-
Sequences and Mappings will match the typing.Sequence and typing.Mapping
abcs. Duck typing is further “discouraged”. We do not think of it as an
issue since Python already has idioms and static type checkers that
discourage this sort of duck typing.
-
C extensions skipping multiple minor Python versions will use the new
code without their knowledge, which is not backwards compatible. This is an
issue with any removal of a stable ABI or language function, and we
consider it acceptable.
Thanks to Bar Harel for helping me with the proposal.
This is was initially discussed in this bpo:
https://bugs.python.org/issue46376
Best regards,
Aviram
You are receiving this email because we wish you to use our cost-effective IT services.
We are a China based Custom Software Application Development provider. We offer full cycle custom software programming services, from product idea, software development to support and enhancement. We employ a large pool of software engineers coming from different backgrounds. We are able to balance product development efforts and project duration to your business needs.
Core Services:
Custom Application Development
Custom Software Development
iPhone Application Development
iPad Application Development
Mobile Application Development
Android Mobile Application Development
Soft Product Development
Game Design & Development
Game Testing & Quality Assurance
Cloud Computing
Solutions:
CRM Software Application Development
ERP Software Application Development
Enterprise Portal Solution
Sales Force Automation Solution
Game Design & Development
Knowledge Management Solution
Workflow Management Solution
SharePoint Development Services
Microsoft Online Services
Microsoft Azure Services
Best regards,
Gary
ITVIATSA Software Development
Contact: ibsoftware(a)yeah.net
Pls send address to larryremove123(a)msn.com for remove
In bpo-42100, I added the private function _PyType_GetModuleByDef, which
allows module state access from slot methods (like tp_init or nb_add),
the main thing missing from PEP 573 (Module State Access from C
Extension Methods). The method is now used in 10 stdlib modules, without
known issues.
Since it was has optimized (thanks, Victor!), I haven't seen complaints
about performance issues. Sadly, it has a loop so it's much slower than
using C statics, but it's OK for many use cases.
I haven't found a better way to get per-module state for isolated
modules (and the stable ABI).
So, I'd like to drop the leading underscore, and add
PyType_GetModuleByDef directly to the limited API.
The function itself can be implemented using only limited API, though
it's a bit tricky to do so correctly (and our implementation uses
private speedups), so it's better if extension authors can use it as a
pre-made building block.
The fact that it can be implemented with only limited API means that
adding it should not add big burden for long-term maintenance in
CPython, nor for alternate implementations of the C-API.
Is anyone against? Does anyone feel this should be discussed more widely
than on capi-sig?
Hi,
I converted Py_TYPE() and Py_SIZE() macros to static inline functions
in the future Python 3.11. It's a backward incompatible change. For
example, "Py_TYPE(obj) = type;" must be replaced with
"Py_SET_TYPE(obj, type);".
You can use the upgrade_pythoncapi.py script of my pythoncapi_compat
project which does these changes for you: you just have to copy
pythoncapi_compat.h to your project. This header file provides new C
API functions like Py_NewRef() and Py_SET_TYPE() to old Python
versions, Python 2.7-3.11.
=> https://github.com/pythoncapi/pythoncapi_compat
I already converted Py_TYPE() and Py_SIZE() macros in Python 3.10, but
it broke too many C extensions and so I had to revert the change. In
the meanwhile, I updated many C extensions and created the
pythoncapi_compat project. For example, Cython and numpy have been
updated to use Py_SET_TYPE() and Py_SET_SIZE(). Mercurial and
immutables projects now use pythoncapi_compat.
I'm interested by feedback on my pythoncapi_compat project ;-)
Tell me if you need help to update your project for Python 3.11 C API changes:
https://docs.python.org/dev/whatsnew/3.11.html#c-api-changes
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
Hi,
TL;DR; Is it safe to Py_DecRef an object that was created before Py_Finalize
after Py_Initialize was called to restart Python runtime?
I am debugging 3.9+ support in https://github.com/pythonnet/pythonnet
The issue I originally got is validate_list from gcmodule.c fails at
assert(prev == GC_PREV(head));
In debugging this, I sprinkled my code with calls to validate_list, and found
that the assertion fails in the following scenario:
Py_Initialize();
...
PyObject* myObj = PyObject_Call(simple_class, empty_tuple, NULL);
Py_Finalize();
...
Py_Initialize();
...
validate_list(gc.generations[2], collecting_clear_unreachable_clear); // OK
Py_DecRef(myObj);
validate_list(gc.generations[2], collecting_clear_unreachable_clear); // EXPLOSION
Now I have not yet tried to narrow down the steps further, because I realised
that I am unsure about the question in TL;DR;. E.g. is this use of myObj
supported by embedding API in general, or is this expected, that after
reinitializing the runtime Py_DecRef on old objects would break GC?
In my scenario simple_class is defined in Python as inheriting from object,
and only has two attributes set at runtime. One has value
builtins.StopIteration, and the other one is an instance of traceback.
Regards,
Victor
Hi,
I would like to change the Python C API. I failed to write a single
document listing all constraints and proposing all changes that I
would to do. For example, my previous PEP 620 contains too many
changes and is too long.
Here is my attempt to focus on the bare minimum and (what I consider
as) the least controversial part: list current usages of C API and
constraints of these usages. This *informal* PEP should be the base of
future PEP changing the C API.
The current draft lives at:
https://github.com/vstinner/misc/blob/main/cpython/pep-c-api-next-level.rst
My PEP is based on HPy Next Level Manifesto written by Simon Cross:
https://github.com/hpyproject/hpy/wiki/c-api-next-level-manifesto
To reach most users of the C API, I cross-posted this email to
python-dev, capi-sig and hpy-dev.
+++++++++++++++++++++++++++++++++++++++++
Taking the Python C API to the Next Level
+++++++++++++++++++++++++++++++++++++++++
::
PEP: xxx
Title: Taking the Python C API to the Next Level
Author: Victor Stinner <vstinner(a)python.org>
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 28-Sep-2021
Python-Version: 3.11
While the C API is a key of the Python popularity, it causes multiple
subtle and complex issues. There are different ways to use the C API,
each usage has its own constraints, and some constraints are exclusive.
This document lists constraints but doesn't propose changes, it only
gives vague ideas how to solve some issues. More concrete C API changes
will require writing separated PEPs.
C extensions are a key component of the Python popularity
=========================================================
The Python popularity comes from its great programming language and from
its wide collection of modules freely available on PyPI. Many of the
most popular Python modules rely directly or indirectly on C extensions
written with the C API. The Python C API is a key component of the
Python popularity.
For example, the numpy project is now a common dependency on many
scientific projects and a large part of the project is written by hand
with the C API.
**Abandoning or removing the C API** is out of question. Years ago, the
incomplete C API support was the main drawback of PyPy, since PyPy only
supported a minority of C extensions.
Today, CPython still have a similar issue. **When Cython or numpy don't
support a new Python version** (because of incompatible C API changes),
many Python projects depending on them are cannot be installed,
especially during the development phase of the next Python version.
Backward compatibility and unmaintained C extensions
====================================================
One important property of the C API is the backward compatibility.
Developers expect that if their C extension works on Python 3.10, it
will work unmodified in Python 3.11: building the C extension with
Python 3.11 should be enough.
This property is even more important for unmaintained C extensions.
Sometimes, unmaintained just means that the only maintainer is busy or
overwhelmed for a few months. Sometimes, the project has no activity for
longer than 5 years.
When an incompatible change is introduced in the C API, like removing a
function or changing a function behavior, there is a **risk of breaking
an unknown number of C extensions**.
One option can be to update old C extensions when they are built on
recent Python versions, to adapt them to incompatible changes. This
conversion is non trivial and cannot handle all kinds of incompatible
changes.
Migration plan for incompatible changes
=======================================
There should be a **sensible migration path** for large C extensions
(e.g. numpy) when incompatible changes are introduced. Whenever
possible, it should be possible to write a **single code base** compatible
with old and new Python versions.
A **compatibility layer** can be maintained externally. Cython and
numpy have their own internal compatibility layer.
There should be a way to easily pick up common errors introduced by
migrating.
One practical way to **minimize the number of broken projects** is to
attempt to check in advance if an incompatible change is going to break
popular C extensions. For broken C extensions, propose a fix and wait
until a new release includes the fix, before introducing the change in
Python. Obviously, it doesn't solve the problem of less popular C
extensions and private C extensions.
Obtain the best possible performance
====================================
There are two main reasons for writing a C extension: implement a
function which cannot be written in pure Python, or write a **C
accelerator**: rewrite the 10% of an application in C where 90% of the
CPU time is spent. About the former use case, the intent is to obtain
the best possible performance. Tradeoffs are made with portability: it
is acceptable to only support a limited number of Python versions and to
only support a limited number of Python implementations (usually only
CPython).
Cython is a good example of accelerator. It is able to support a large
number of Python versions and multiple Python implementation with
compatibility layers and ``#ifdef``. The main drawback is that it is
common that Cython is **broken by incompatible changes made at each
Python release**. It happens because Cython relies on many
implementation details.
On the other side, the **limited C API** is a small as possible,
excludes implementation details on purpose, and provides a stable ABI.
Building a C extension with the limited C API only once produces a
binary wheel package usable on many Python versions, but each platform
still requires its own binary wheel package.
Emulating the current C API is inefficient
==========================================
The PyPy project is a Python implementation written from scratch, it was
not created as a CPython fork. It made many implementation choices
different than CPython: no reference counting, moving garbage collector,
JIT compiler, etc.
To support C extensions, PyPy emulates the Python C API in its cpyext
module. When the C API access an object, cpyext has to convert the PyPy
object to a CPython object (``PyObject``). CPython objects are less
efficient than PyPy objects with the PyPy JIT compiler and conversions
from PyPy objects to CPython objects are also inefficient. PyPy has to
reimplement every single detail of the CPython implementation to be as
much compatible as possible.
The C API exposes multiple implementation details:
* Reference counting, borrowed references, stealing references.
* Objects location in memory.
* Rely on pointers for object identity: Python 3.10 adds the ``Py_Is()``
function to solve this problem.
* Expose the memory layout of Python objects as part of the API.
* Expose static types.
* Implicit execution context.
* etc.
The C API of Python 3.10 is made of around 15 000 lines of C header
files, 1500 functions and 100 structures. Supporting the full C API is a
significant amount of work.
**Freezing the C API** for a few Python releases would help other Python
implementations to catch up with the latest Python version, but it
doesn't solve the efficiency problem. Moreover, it is common that adding
a new feature to Python requires to change the C API, even if it is just
to add new functions. Not adding new features to Python for a few Python
releases is out of question.
The C API prevents optimizing CPython
=====================================
It is challenging to evolve the C API to optimize CPython without
breaking the backward compatibility. Emulating the old C API is an
option, but it is inefficient.
If everything above is achievable -- and we believe it is! -- we'll
arrive in a wonderful new future where Python implementations can
experiment with all sorts of amazing new features:
* tracing garbage collectors;
* nurseries for short-lived objects;
* sub-interpreters with separate contexts;
* specialised implementations of lists;
* removing the GIL;
* avoiding the boxing of primitive types;
* just-in-time compilation;
* ... and many other things you can imagine that we haven't!
No one can guarantee that a particular new idea will work out, but
exposing fewer implementation details via the C API will make it
possible to try many new things.
--
Night gathers, and now my watch begins. It shall not end until my death.
Hi,
I'm trying to understand what is the best/safest/recommended way to
implement tp_dealloc on a heap type created by PyType_FromSpec.
The official docs don't say much on the topic. The only note I could find
is this:
> Finally, if the type is heap allocated (Py_TPFLAGS_HEAPTYPE), the
deallocator should decrement the reference count for its type object after
calling the type deallocator.
My doubts came after reading the source code. If I don't specify a
tp_dealloc, its default value depends on heap vs static types:
- for static types, the default is object_dealloc, which simply does a
tp->tp_free(self)
- for heap types created by PyType_FromSpecWithBases, the default
value[1] is subtype_dealloc, which seems to do a lot of complex logic which
I don't fully understand.
[1]
https://github.com/python/cpython/blob/main/Objects/typeobject.c#L3553-L3558
This means that if I create a heap type with a custom tp_dealloc, all the
logic implemented by subtype_dealloc will not be executed and that my type
will probably behave subtly differently. I also found BPO 26979 [2] where
Christian Tismer claims that "The default of PyType_FromSpec for tp_dealloc
is wrong!", but it seems that nothing has been done for that.
[2] https://bugs.python.org/issue26979
Another interesting data point is that PyType_FromSpec+tp_dealloc does not
seem to be used a lot in the wild. I tried to grep for Py_tp_dealloc in the
top4000 PyPI packages[3] and I found only a match, in Cython-generated code
[4]; but it's code which is behind an "#if
CYTHON_COMPILING_IN_LIMITED_API", which makes me to suspect which is not
actually used a lot in practice.
[3] https://github.com/hpyproject/top4000-pypi-packages
[4]
https://github.com/hpyproject/top4000-pypi-packages/blob/0cd919943a007f95f4…
So, back to my original problem:
1. Is the default value of tp_dealloc actually correct?
2. Is it actually possible to write a custom tp_dealloc which behaves
correctly?
3. If (2) is true, what is the simplest way to do it?
Cheers,
Antonio
