Hi Gustavo,
Le lundi 30 juillet 2018, Stefan Behnel python_capi@behnel.de a écrit :
Gustavo Carneiro schrieb am 30.07.2018 um 16:18:
What is the relation to PEP 384?
https://www.python.org/dev/peps/pep-0384/
The "stable ABI" is a restricted API/ABI that breaks backwards compatibility with all extension modules out there. If you write code against this API, then you can make it work in newer CPython versions without recompilation. This does not apply to any previously existing C extension, which means that any changes to the non-stable parts of the C-API would still break the world.
Stefan
Replacing macros accessing structures fields with function calls makes the generated machine code compatible with the stable ABI: http://pythoncapi.readthedocs.io/stable_abi.html
And: http://pythoncapi.readthedocs.io/new_api.html#hide-implementation-details
It means that C extensions using PyList_GET_ITEM() become compatible with new Python runtimes using the stable ABI, without having to modify their code.
The PEP 384 requires to replace PyList_GET_ITEM() with PyList_GetItem(). I would like to continue the work on this PEP, to allow to use the stable ABI on more C extensions.
Victor
Victor Stinner schrieb am 31.07.2018 um 00:04:
Le lundi 30 juillet 2018, Stefan Behnel a écrit :
Gustavo Carneiro schrieb am 30.07.2018 um 16:18:
What is the relation to PEP 384? https://www.python.org/dev/peps/pep-0384/
The "stable ABI" is a restricted API/ABI that breaks backwards compatibility with all extension modules out there. If you write code against this API, then you can make it work in newer CPython versions without recompilation. This does not apply to any previously existing C extension, which means that any changes to the non-stable parts of the C-API would still break the world.
Replacing macros accessing structures fields with function calls makes the generated machine code compatible with the stable ABI:
Except when you're using extension types.
Stefan
Hi Stefan,
2018-07-31 7:14 GMT+02:00 Stefan Behnel python_capi@behnel.de:
Replacing macros accessing structures fields with function calls makes the generated machine code compatible with the stable ABI:
Except when you're using extension types.
I'm not aware of that. Would you mind to elaborate?
Victor
On 31 Jul 2018, at 12:24, Victor Stinner vstinner@redhat.com wrote:
Hi Stefan,
2018-07-31 7:14 GMT+02:00 Stefan Behnel python_capi@behnel.de:
Replacing macros accessing structures fields with function calls makes the generated machine code compatible with the stable ABI:
Except when you're using extension types.
I'm not aware of that. Would you mind to elaborate?
With the regular API extension types are created by a (static) PyTypeObject structure, with the stable ABI you have to use a function based API to create and initialise the type. I expect that this is the major reason why there have been few attempts to migrate existing C extensions to the stable ABI.
Ronald
Hi,
2018-07-31 13:01 GMT+02:00 Ronald Oussoren ronaldoussoren@mac.com:
With the regular API extension types are created by a (static) PyTypeObject structure, with the stable ABI you have to use a function based API to create and initialise the type. I expect that this is the major reason why there have been few attempts to migrate existing C extensions to the stable ABI.
Oh, I see. I recall issues with PyTypeObject. I added a page: https://pythoncapi.readthedocs.io/type_object.html
And I proposed:
- PyTypeObject structure should become opaquet
- PyType_Ready() should be removed
https://pythoncapi.readthedocs.io/bad_api.html#pytype-ready-and-setting-dire...
Can PyType_FromSpec() be used for all cases? It seems like at least some types cannot be defined with it, since there is an additional PyType_FromSpecWithBases() private function.
Victor
On 31 Jul 2018, at 13:25, Victor Stinner vstinner@redhat.com wrote:
Hi,
2018-07-31 13:01 GMT+02:00 Ronald Oussoren ronaldoussoren@mac.com:
With the regular API extension types are created by a (static) PyTypeObject structure, with the stable ABI you have to use a function based API to create and initialise the type. I expect that this is the major reason why there have been few attempts to migrate existing C extensions to the stable ABI.
Oh, I see. I recall issues with PyTypeObject. I added a page: https://pythoncapi.readthedocs.io/type_object.html
And I proposed:
- PyTypeObject structure should become opaquet
- PyType_Ready() should be removed
https://pythoncapi.readthedocs.io/bad_api.html#pytype-ready-and-setting-dire...
Can PyType_FromSpec() be used for all cases? It seems like at least some types cannot be defined with it, since there is an additional PyType_FromSpecWithBases() private function.
I have never seriously looked at this API, but I’m pretty sure that it was the intention this API could replace the existing API.
That said, I have a use case where this API isn’t sufficient, but that’s because I’m doing stuff I shouldn’t be doing. I’ve created PEP 447 <https://www.python.org/dev/peps/pep-0447/ https://www.python.org/dev/peps/pep-0447/> to do away with that need, but haven’t had the time yet to finish the PEP and update the implementation to the current master.
Another, unrelated, use-case I have involves subclassing PyUnicode_Type in C code, and add a new slot to store a native pointer. That requires access to the C structure for Unicode objects (and replicating the code to initialise those structures). The primary reason for doing this is that it is currently not possible to have types that behave like strings when passing values to C code (that is, values other than str that can be used with the “s” format specifier for PyArg_Parse). This could probably be solved by having yet another tp_as* slot in PyType_Object, but I haven’t spent time on thinking this through yet.
BTW. PyArg_Parse* is also an API that leaks implementation details: The “O” specifier results in a borrowed reference, and “s” returns a pointer to internal storage.
Ronald
P.S. Both use cases are in PyObjC
2018-07-31 13:50 GMT+02:00 Ronald Oussoren ronaldoussoren@mac.com:
I have never seriously looked at this API, but I’m pretty sure that it was the intention this API could replace the existing API.
Ok, nice because it already exists :-)
That said, I have a use case where this API isn’t sufficient, but that’s because I’m doing stuff I shouldn’t be doing. I’ve created PEP 447 https://www.python.org/dev/peps/pep-0447/ to do away with that need, but haven’t had the time yet to finish the PEP and update the implementation to the current master.
Well, I'm sure that it's easy to find bunch of C extensions which do crazy stuff with the current C API and that cannot move to the C API.
Fine! The current C API is not removed and will stay available in the regular runtime! https://pythoncapi.readthedocs.io/runtimes.html#regular-python-usr-bin-pytho...
Hum, it seems like many people misunderstood my intent. My intent is not to force everybody to modify all their C extensions at once as we did with Python 3.0. No, my intent is to add an optional "new C API" that you can use if you would like to benefit of the stable ABI and maybe test experimental new runtimes.
Another, unrelated, use-case I have involves subclassing PyUnicode_Type in C code, and add a new slot to store a native pointer. That requires access to the C structure for Unicode objects (and replicating the code to initialise those structures). The primary reason for doing this is that it is currently not possible to have types that behave like strings when passing values to C code (that is, values other than str that can be used with the “s” format specifier for PyArg_Parse). This could probably be solved by having yet another tp_as* slot in PyType_Object, but I haven’t spent time on thinking this through yet.
Again, just keep your code unchanged and it will continue to run on the "regular runtime" of Python 3.8.
But for the long term, you should probably fine a more substainable solution. For example, does your beast work on PyPy?
BTW. PyArg_Parse* is also an API that leaks implementation details: The “O” specifier results in a borrowed reference, and “s” returns a pointer to internal storage.
Oh, I didn't notice. Ok, I added it to my list of "bad C API" :-)
Argument Clinic (AC) looks a way better technical solution to parse arguments. AC can easily be modified to increase/decrease the reference counter while calling the implementation of the function.
Hum, it seems like right now, AC doesn't touch the reference counter. I know that Armin Ringo from PyPy found many crashes caused by borrowed references. It's just a bad practice to rely on them!
But PyArg_ParseTuple(), the question is if we can rely on the fact that the caller is supposed to hold a strong reference. The problem is that from ceval.c to the final function, through functions like PyObject_Call(), everybody rely on the caller... and at the end, I'm not sure that *anyone* holds a strong reference...
My FASTCALL calling convention replaced a Python tuple with a C array: this change also reduces the number of INCREF/DECREF. Previously, the tuple had strong references to the Python objects. Now with the C array, we don't know. Maybe yes, maybe no...
Victor
2018-07-31 15:14 GMT+02:00 Andrew Svetlov andrew.svetlov@gmail.com:
IIRC stable API doesnt support tp_dict_offset and weakref_offset
Do you mean that you want to use them in your C types to make them faster?
To *get* the dict or weakref of a type, you don't need them, no?
Victor
I mean if I need to support __dict__ in C extension I should setup tp_dict_offset in a type structure to point on a dict field in an instance struct. Stable API doesnt provide such functionality.
We can add Py_tp_dictoffset typeslot but now it is absent. The same is for tp_weaklistoffset
On Tue, Jul 31, 2018, 16:37 Victor Stinner vstinner@redhat.com wrote:
2018-07-31 15:14 GMT+02:00 Andrew Svetlov andrew.svetlov@gmail.com:
IIRC stable API doesnt support tp_dict_offset and weakref_offset
Do you mean that you want to use them in your C types to make them faster?
To *get* the dict or weakref of a type, you don't need them, no?
Victor
IMHO the issue is that nobody tried seriously to use PyType_FromSpec(). Type.__dict__ and weak references are common features.
Yeah, we need to do something to add the stable ABI at the same feature level.
Victor
2018-07-31 17:38 GMT+02:00 Andrew Svetlov andrew.svetlov@gmail.com:
I mean if I need to support __dict__ in C extension I should setup tp_dict_offset in a type structure to point on a dict field in an instance struct. Stable API doesnt provide such functionality.
We can add Py_tp_dictoffset typeslot but now it is absent. The same is for tp_weaklistoffset
On Tue, Jul 31, 2018, 16:37 Victor Stinner vstinner@redhat.com wrote:
2018-07-31 15:14 GMT+02:00 Andrew Svetlov andrew.svetlov@gmail.com:
IIRC stable API doesnt support tp_dict_offset and weakref_offset
Do you mean that you want to use them in your C types to make them faster?
To *get* the dict or weakref of a type, you don't need them, no?
Victor
-- Thanks, Andrew Svetlov _______________________________________________ capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org
On 07/31/18 17:59, Victor Stinner wrote:
IMHO the issue is that nobody tried seriously to use PyType_FromSpec(). Type.__dict__ and weak references are common features.
Yeah, we need to do something to add the stable ABI at the same feature level.
I ran into the problem of missing dict/weaklist before, when working work on subinterpreter support/multi-phase init. The New C API initiative looks like a good place to collect these issues.
Victor
2018-07-31 17:38 GMT+02:00 Andrew Svetlov andrew.svetlov@gmail.com:
I mean if I need to support __dict__ in C extension I should setup tp_dict_offset in a type structure to point on a dict field in an instance struct. Stable API doesnt provide such functionality.
We can add Py_tp_dictoffset typeslot but now it is absent. The same is for tp_weaklistoffset
On Tue, Jul 31, 2018, 16:37 Victor Stinner vstinner@redhat.com wrote:
2018-07-31 15:14 GMT+02:00 Andrew Svetlov andrew.svetlov@gmail.com:
IIRC stable API doesnt support tp_dict_offset and weakref_offset
Do you mean that you want to use them in your C types to make them faster?
To *get* the dict or weakref of a type, you don't need them, no?
Victor
-- Thanks, Andrew Svetlov _______________________________________________ capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org
capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org
Yes, I'm completing my doc each time someone reports an issue :-)
https://pythoncapi.readthedocs.io/bad_api.html https://pythoncapi.readthedocs.io/type_object.html
Victor
2018-07-31 18:22 GMT+02:00 Petr Viktorin encukou@gmail.com:
On 07/31/18 17:59, Victor Stinner wrote:
IMHO the issue is that nobody tried seriously to use PyType_FromSpec(). Type.__dict__ and weak references are common features.
Yeah, we need to do something to add the stable ABI at the same feature level.
I ran into the problem of missing dict/weaklist before, when working work on subinterpreter support/multi-phase init. The New C API initiative looks like a good place to collect these issues.
Victor
2018-07-31 17:38 GMT+02:00 Andrew Svetlov andrew.svetlov@gmail.com:
I mean if I need to support __dict__ in C extension I should setup tp_dict_offset in a type structure to point on a dict field in an instance struct. Stable API doesnt provide such functionality.
We can add Py_tp_dictoffset typeslot but now it is absent. The same is for tp_weaklistoffset
On Tue, Jul 31, 2018, 16:37 Victor Stinner vstinner@redhat.com wrote:
2018-07-31 15:14 GMT+02:00 Andrew Svetlov andrew.svetlov@gmail.com:
IIRC stable API doesnt support tp_dict_offset and weakref_offset
Do you mean that you want to use them in your C types to make them faster?
To *get* the dict or weakref of a type, you don't need them, no?
Victor
-- Thanks, Andrew Svetlov _______________________________________________ capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org
capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org
capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org
On Tue, Jul 31, 2018 at 7:07 AM Victor Stinner vstinner@redhat.com wrote:
Hum, it seems like many people misunderstood my intent. My intent is not to force everybody to modify all their C extensions at once as we did with Python 3.0. No, my intent is to add an optional "new C API" that you can use if you would like to benefit of the stable ABI and maybe test experimental new runtimes.
That sounds great and I'm in favor. :) The idea of "experimental new runtimes" is intriguing. [1] The key point is that the new C-API will effectively be a compatibility layer between separate runtimes, right? So compliant extension modules will work in any of them.
A critical issue will be how we will phase out the "old" C-API. That will have to be resolved before we can ever integrate an "experimental" runtime back into the CPython code base.
FWIW, I had a great conversation with Travis Oliphant at PyCon this year about extension module compatibility (relative to subinterpreters). He had some good insights that I think apply here. The gist was that you can get away with some compatibility breakage if you're careful:
- small changes
- provide a good alternative to current usage
- work with commonly used extensions to make sure they don't break
One last thing: do you expect that extensions that do not use the new C-API will be increasingly penalized over time?
-eric
[1] I approve of efforts to solidify the concept of "runtime" for Python. :)
2018-07-31 17:38 GMT+02:00 Eric Snow ericsnowcurrently@gmail.com:
That sounds great and I'm in favor. :) The idea of "experimental new runtimes" is intriguing. [1]
It's described at: https://pythoncapi.readthedocs.io/runtimes.html
When it will be possible, I plan to experiment some of these ideas: https://pythoncapi.readthedocs.io/optimization_ideas.html
The key point is that the new C-API will effectively be a compatibility layer between separate runtimes, right?
C API compatibility may work, but it's weak. I want stronger guarantee: ABI compatibility (thing called "stable ABI").
If we can ABI compatibility, you can install a C extension once and use it with multiple runtimes, even runtimes of different Python versions.
So compliant extension modules will work in any of them.
I tried to describe the compatibility matrix at: https://pythoncapi.readthedocs.io/runtimes.html
A critical issue will be how we will phase out the "old" C-API.
I don't plan to remove the old C API.
That will have to be resolved before we can ever integrate an "experimental" runtime back into the CPython code base.
Technically, experimental runtimes can easily be forks of CPython.
In the best case, changes approved by everyones can become optional features merged into the master branch of CPython upstream, but only enabled by a compilation flag.
python3-exp would be CPython compiled with "./configure --with-experimantal-changes && make"
But I don't really care of putting experimental changes into upstream.
Currently, there is no such thing like multiple runtimes and so a change must be upstream or doesn't exist. There is no space for experimental change. I would like to allow people to experiment changes more easily.
FWIW, I had a great conversation with Travis Oliphant at PyCon this year about extension module compatibility (relative to subinterpreters). He had some good insights that I think apply here. The gist was that you can get away with some compatibility breakage if you're careful:
- small changes
My goal is to make the minimum changes to get a stable ABI: https://pythoncapi.readthedocs.io/bad_api.html
- provide a good alternative to current usage
My plan doesn't force anyone to move immediately to the new C API, and it includes solution for backward compatibility: https://pythoncapi.readthedocs.io/backward_compatibility.html
- work with commonly used extensions to make sure they don't break
That's also part of the plan. I plan to even propose changes to the most popular C extensions.
One last thing: do you expect that extensions that do not use the new C-API will be increasingly penalized over time?
If you don't switch to the new C API, you are stuck at the "Regular runtime". At the beginning, everybody will be stuck at this runtime. I don't see any reason to penalize our users. I expect that CPython will continue to be develop.
Said differently, I don't know what will be the status of C extensions in 5 or 10 years, and there is no idea to guess at this point.
Victor
-
Andrew Svetlov -
Eric Snow -
Petr Viktorin -
Ronald Oussoren -
Stefan Behnel -
Victor Stinner