Making PyInterpreterState an opaque type
Hi all, I've been working on the runtime lately, particularly focused on my multi-core Python project. One thing that would help simplify changes in this area is if PyInterpreterState were defined in Include/internal. This would mean the type would be opaque unless Py_BUILD_CORE were defined. The docs [1] already say none of the struct's fields are public. Furthermore, Victor already moved it into Include/cpython (i.e. not in the stable ABI) in November. Overall, the benefit of making internal types like this opaque is realized in our ability to change the internal details without further breaking C-API users. Realistically, there may be extension modules out there that use PyInterpreterState fields directly. They would break. I expect there to be few such modules and fixing them would not involve great effort. We'd have to add any missing accessor functions to the public C-API, which I see as a good thing. I have an issue [2] open for the change and a PR open. My PR already adds an entry to the porting section of the 3.8 What's New doc about dealing with PyInterpreterState. Anyway, I just wanted to see if there are any objections to making PyInterpreterState an opaque type outside of core use. -eric p.s. I'd like to do the same with PyThreadState, but that's a bit trickier [3] and not one of my immediate needs. :) [1] https://docs.python.org/3/c-api/init.html#c.PyInterpreterState [2] https://bugs.python.org/issue35886 [3] https://bugs.python.org/issue35949
On 2019-02-16 00:37, Eric Snow wrote:
One thing that would help simplify changes in this area is if PyInterpreterState were defined in Include/internal.
How would that help anything? I don't like the idea (in general, I'm not talking about PyInterpreterState specifically) that external modules should be second-class citizens compared to modules inside CPython. If you want to break the undocumented API, just break it. I don't mind. But I don't see why it's required to move the include to Include/internal for that.
On Sat, 16 Feb 2019 11:15:44 +0100 Jeroen Demeyer <J.Demeyer@UGent.be> wrote:
On 2019-02-16 00:37, Eric Snow wrote:
One thing that would help simplify changes in this area is if PyInterpreterState were defined in Include/internal.
How would that help anything? I don't like the idea (in general, I'm not talking about PyInterpreterState specifically) that external modules should be second-class citizens compared to modules inside CPython.
If you want to break the undocumented API, just break it. I don't mind. But I don't see why it's required to move the include to Include/internal for that.
This sounds like a reasonable design principle: decree the API non-stable and prone to breakage (it already is, anyway), don't hide it. It's true that in the PyInterpreterState case specifically, there doesn't seem much worthy of use by third-party libraries. Regards Antoine.
On 16Feb.2019 1332, Antoine Pitrou wrote:
On Sat, 16 Feb 2019 11:15:44 +0100 Jeroen Demeyer <J.Demeyer@UGent.be> wrote:
On 2019-02-16 00:37, Eric Snow wrote:
One thing that would help simplify changes in this area is if PyInterpreterState were defined in Include/internal.
How would that help anything? I don't like the idea (in general, I'm not talking about PyInterpreterState specifically) that external modules should be second-class citizens compared to modules inside CPython.
If you want to break the undocumented API, just break it. I don't mind. But I don't see why it's required to move the include to Include/internal for that.
This sounds like a reasonable design principle: decree the API non-stable and prone to breakage (it already is, anyway), don't hide it.
As I was chatting with Eric shortly before he posted this, I assume the idea would be to expose anything useful/necessary via a function. That at least removes the struct layout from the ABI, without removing functionality.
It's true that in the PyInterpreterState case specifically, there doesn't seem much worthy of use by third-party libraries.
Which seems to suggest that the answer to "which members are important to expose?" is "probably none". And that's possibly why Eric didn't mention it in his email :) This is mostly about being able to assign blame when things break, so I'm totally okay with extension modules that want to play with internals declaring Py_BUILD_CORE to get access to them (though I suspect that won't work out of the box - maybe we should have a Py_I_TOO_LIKE_TO_LIVE_DANGEROUSLY?). I like that we're taking (small) steps to reduce the size of our API. It helps balance out the growth and leaves us with a chance of one day being able to have an extension model that isn't as tied to C's ABI. Cheers, Steve
On Sat, 16 Feb 2019 14:34:46 -0800 Steve Dower <steve.dower@python.org> wrote:
On 16Feb.2019 1332, Antoine Pitrou wrote:
On Sat, 16 Feb 2019 11:15:44 +0100 Jeroen Demeyer <J.Demeyer@UGent.be> wrote:
On 2019-02-16 00:37, Eric Snow wrote:
One thing that would help simplify changes in this area is if PyInterpreterState were defined in Include/internal.
How would that help anything? I don't like the idea (in general, I'm not talking about PyInterpreterState specifically) that external modules should be second-class citizens compared to modules inside CPython.
If you want to break the undocumented API, just break it. I don't mind. But I don't see why it's required to move the include to Include/internal for that.
This sounds like a reasonable design principle: decree the API non-stable and prone to breakage (it already is, anyway), don't hide it.
As I was chatting with Eric shortly before he posted this, I assume the idea would be to expose anything useful/necessary via a function. That at least removes the struct layout from the ABI, without removing functionality.
Well, the ABI is allowed to break at each feature version (except for the "stable ABI" subset, which PyInterpreterState isn't part of), so I'm not sure that would change anything ;-)
It's true that in the PyInterpreterState case specifically, there doesn't seem much worthy of use by third-party libraries.
Which seems to suggest that the answer to "which members are important to expose?" is "probably none".
That sounds intuitive. But we don't know what kind of hacks some extension authors might do, for legitimate reasons... (perhaps some gevent-like framework needs access to the interpreter state?) Regards Antoine.
On Sat, Feb 16, 2019 at 3:47 PM Antoine Pitrou <solipsis@pitrou.net> wrote:
On Sat, 16 Feb 2019 14:34:46 -0800 Steve Dower <steve.dower@python.org> wrote:
Which seems to suggest that the answer to "which members are important to expose?" is "probably none".
That sounds intuitive. But we don't know what kind of hacks some extension authors might do, for legitimate reasons...
(perhaps some gevent-like framework needs access to the interpreter state?)
In those cases either we will expose accessor functions in the C-API or they can define Py_BUILD_CORE. -eric
On Tue, 19 Feb 2019 at 05:33, Eric Snow <ericsnowcurrently@gmail.com> wrote:
On Sat, Feb 16, 2019 at 3:47 PM Antoine Pitrou <solipsis@pitrou.net> wrote:
On Sat, 16 Feb 2019 14:34:46 -0800 Steve Dower <steve.dower@python.org> wrote:
Which seems to suggest that the answer to "which members are important to expose?" is "probably none".
That sounds intuitive. But we don't know what kind of hacks some extension authors might do, for legitimate reasons...
(perhaps some gevent-like framework needs access to the interpreter state?)
In those cases either we will expose accessor functions in the C-API or they can define Py_BUILD_CORE.
I really don't want us to ever get into a situation where we're actively encouraging third party projects to define Py_BUILD_CORE. If we decide we do want to go down a path like that, I'd instead prefer to see us define something more like "Py_FRAGILE_API" to make it clear that folks using those extra interfaces are tightly coupling themselves to a specific version of CPython, and are likely going to need to make changes when new versions are released. Even though we would probably *implement* that by having this snippet in one of our header files: #ifdef Py_FRAGILE_API #define Py_BUILD_CORE #endif I still think it would convey the concerns we have more clearly than simply telling people to define Py_BUILD_CORE would. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 19Feb2019 0555, Nick Coghlan wrote:
I really don't want us to ever get into a situation where we're actively encouraging third party projects to define Py_BUILD_CORE.
If we decide we do want to go down a path like that, I'd instead prefer to see us define something more like "Py_FRAGILE_API" to make it clear that folks using those extra interfaces are tightly coupling themselves to a specific version of CPython, and are likely going to need to make changes when new versions are released.
I mean, my suggestion of "Py_I_TOO_LIKE_TO_LIVE_DANGEROUSLY" was only a little bit tongue-in-cheek :) Maybe there's a good Monty Python reference we can use here? "Py_ITS_JUST_A_FLESH_WOUND" or "Py_THEN_WE_JUMP_OUT_OF_THE_RABBIT" Cheers, Steve
On Sat, Feb 16, 2019 at 3:34 PM Steve Dower <steve.dower@python.org> wrote:
On 16Feb.2019 1332, Antoine Pitrou wrote:
This sounds like a reasonable design principle: decree the API non-stable and prone to breakage (it already is, anyway), don't hide it.
As I was chatting with Eric shortly before he posted this, I assume the idea would be to expose anything useful/necessary via a function. That at least removes the struct layout from the ABI, without removing functionality.
It's true that in the PyInterpreterState case specifically, there doesn't seem much worthy of use by third-party libraries.
Which seems to suggest that the answer to "which members are important to expose?" is "probably none". And that's possibly why Eric didn't mention it in his email :)
This is mostly about being able to assign blame when things break, so I'm totally okay with extension modules that want to play with internals declaring Py_BUILD_CORE to get access to them (though I suspect that won't work out of the box - maybe we should have a Py_I_TOO_LIKE_TO_LIVE_DANGEROUSLY?).
I like that we're taking (small) steps to reduce the size of our API. It helps balance out the growth and leaves us with a chance of one day being able to have an extension model that isn't as tied to C's ABI.
Yeah, what Steve said. :) -eric
Steve Dower wrote on 2/16/19 14:34:>
This is mostly about being able to assign blame when things break, so I'm totally okay with extension modules that want to play with internals declaring Py_BUILD_CORE to get access to them (though I suspect that won't work out of the box - maybe we should have a Py_I_TOO_LIKE_TO_LIVE_DANGEROUSLY?).
Let's call it Py_POINTED_STICK of course! http://www.montypython.net/scripts/fruit.php -Barry
On 19Feb2019 1141, Barry Warsaw wrote:
Steve Dower wrote on 2/16/19 14:34:>
This is mostly about being able to assign blame when things break, so I'm totally okay with extension modules that want to play with internals declaring Py_BUILD_CORE to get access to them (though I suspect that won't work out of the box - maybe we should have a Py_I_TOO_LIKE_TO_LIVE_DANGEROUSLY?).
Let's call it Py_POINTED_STICK of course!
+1, and instead of "using the internal API" we can call it "coming at CPython with a banana" :D Cheers, Steve
On Tue, Feb 19, 2019 at 12:45 PM Steve Dower <steve.dower@python.org> wrote:
On 19Feb2019 1141, Barry Warsaw wrote:
Steve Dower wrote on 2/16/19 14:34:>
This is mostly about being able to assign blame when things break, so I'm totally okay with extension modules that want to play with internals declaring Py_BUILD_CORE to get access to them (though I suspect that won't work out of the box - maybe we should have a Py_I_TOO_LIKE_TO_LIVE_DANGEROUSLY?).
Let's call it Py_POINTED_STICK of course!
+1, and instead of "using the internal API" we can call it "coming at CPython with a banana" :D
I don't think now is the exact time to do this since how to even handle PEPs isn't really settled in this new steering council world, but eventually maybe a PEP to outlining the re-org, how to opt into what seems to be the 3 different layers of the C API for users, guidelines on how to decide what APIs go where, etc. might be warranted? Stuff is starting to get strewn about without a centralized thing to focus discussions around so probably having a PEP to focus around might help.
On 16/02/2019 23:34, Steve Dower wrote:
I like that we're taking (small) steps to reduce the size of our API.
I consider myself - an "outsider", so an "outsider's" view is that anything that makes it more clear about what is intended aka supported as the Python API is an improvement. Without clarity there is a chance (read risk) that someone starts using something and forces a long and difficult process to make it part of the official API or get acceptance that something never should have been done "that way". Shorter: promote clarity. IMHO: it is easier to move something from the 'internal' to public than v.v. and whenever there is not a compelling reason to not put something into some form of 'internal' - do it before not doing so bites you in a nasty way. My two (outsider) bits - :)
On Sat, Feb 16, 2019 at 3:16 AM Jeroen Demeyer <J.Demeyer@ugent.be> wrote:
On 2019-02-16 00:37, Eric Snow wrote:
One thing that would help simplify changes in this area is if PyInterpreterState were defined in Include/internal.
How would that help anything?
I'm talking just about changes in the runtime implementation. A lot of of the runtime-related API is defined in Include/internal. Relying on the public headers (i.e. Include/*) for internal runtime API can complicate changes there. I've run into this recently. Moving more internal API into the internal headers helps with that problem. Having distinct header files for the internal API is a relatively new thing (i.e. in the last year), which is why some of the internal API is still defined in the public header files.
I don't like the idea (in general, I'm not talking about PyInterpreterState specifically) that external modules should be second-class citizens compared to modules inside CPython.
If you want to break the undocumented API, just break it. I don't mind. But I don't see why it's required to move the include to Include/internal for that.
Keep in mind that the "internal" (or "private") API is intended for use exclusively in the runtime and in the builtin modules. Historically our approach to keeping API private was to use underscore prefixes and to leave them out of the documentation (along with guarding with "#ifndef Py_LIMITED_API"). However, this has lead to occasional confusion and breakage, and even to leaking things into the stable ABI that shouldn't have been. Lately we've been working on making the distinction between internal and public API (and stable ABI) more clear and less prone to accidental exposure. Victor has done a lot of work in this area. So I'd like to understand your objection. Is it with exposing some things only under Py_BUILD_CORE (i.e. when building Python itself)? Is it to having "private" C-API in general? Is it just to having separate include directories? -eric
On 2019-02-18 21:17, Eric Snow wrote:
Historically our approach to keeping API private was to use underscore prefixes and to leave them out of the documentation (along with guarding with "#ifndef Py_LIMITED_API"). However, this has lead to occasional confusion and breakage, and even to leaking things into the stable ABI that shouldn't have been. Lately we've been working on making the distinction between internal and public API (and stable ABI) more clear and less prone to accidental exposure. Victor has done a lot of work in this area.
So I'd like to understand your objection.
First of all, if everybody can actually #define Py_BUILD_CORE and get access to the complete API, I don't mind so much. But then it's important that this actually keeps working (i.e. that those headers will always be installed). Still, do we really need so many levels of API: (1) stable API (with #define Py_LIMITED_API) (2) public documented API (3) private undocumented API (the default exposed API) (4) internal API (with #define Py_BUILD_CORE) I would argue to fold (4) into (3). Applications using (3) already know that they are living dangerously by using private API. I'm afraid of hiding actually useful private macros under Py_BUILD_CORE. For example, Modules/_functoolsmodule.c and Modules/_json.c use API functions from (4). But if an API function is useful for implementing functools or json, then it's probably also useful for external extension modules: what if I want to implement something similar to functools or json, why shouldn't I be allowed to use those same API functions? For a very concrete example, was it really necessary to put _PyTuple_ITEMS in (4)? That's used in _functoolsmodule.c. Especially given that the very similar PySequence_Fast_ITEMS is in (2), that seems like a strange and arbitrary limiting choice. Jeroen.
On 18Feb.2019 1324, Jeroen Demeyer wrote:
Still, do we really need so many levels of API: (1) stable API (with #define Py_LIMITED_API) (2) public documented API (3) private undocumented API (the default exposed API) (4) internal API (with #define Py_BUILD_CORE)
I would argue to fold (4) into (3). Applications using (3) already know that they are living dangerously by using private API.
I agree completely. It's unfortunate we ended up in a state where the stable API was opt-in, but that's where we are now and we have to transition carefully. The ideal would be: * default to cross-version supported APIs (i.e. stable for all 3.*) * opt-in to current-version stable APIs (i.e. stable for all 3.7.*) * extra opt-in to unstable APIs (i.e. you are guaranteed to break one day without warning)
I'm afraid of hiding actually useful private macros under Py_BUILD_CORE. For example, Modules/_functoolsmodule.c and Modules/_json.c use API functions from (4). But if an API function is useful for implementing functools or json, then it's probably also useful for external extension modules: what if I want to implement something similar to functools or json, why shouldn't I be allowed to use those same API functions?
For a very concrete example, was it really necessary to put _PyTuple_ITEMS in (4)? That's used in _functoolsmodule.c. Especially given that the very similar PySequence_Fast_ITEMS is in (2), that seems like a strange and arbitrary limiting choice.
The reason to do this is that we can "guarantee" that we've fixed all users when we change the internal representation. Otherwise, the internal memory layout becomes part of the public ABI, which is what we want to fix. (PyTuple_GET_ITEM is just as problematic, FWIW.) If you always rebuild your extension for every micro version (3.x.y) of CPython, then sure, go ahead and use this. But you're by far into the minority of users/developers, and so we really don't want to optimise for this case when it's going to break the 90%+ of people who don't recompile everything all the time. Cheers, Steve
On 2019-02-19 04:04, Steve Dower wrote:
Otherwise, the internal memory layout becomes part of the public ABI
Of course, the ABI (not API) depends on the internal memory layout. Why is this considered a problem? If you want a fixed ABI, use API level (1) from my last post. If you want a fixed API but not ABI, use level (2). If you really want stuff to be broken at any time, use (3) or (4). This is why I don't see the need to make a difference between (3) and (4): neither of them makes any guarantees about stability.
On 2019-02-19 04:04, Steve Dower wrote:
On 18Feb.2019 1324, Jeroen Demeyer wrote:
For a very concrete example, was it really necessary to put _PyTuple_ITEMS in (4)? That's used in _functoolsmodule.c. Especially given that the very similar PySequence_Fast_ITEMS is in (2), that seems like a strange and arbitrary limiting choice.
The reason to do this is that we can "guarantee" that we've fixed all users when we change the internal representation.
I think that CPython should then at least "eat its own dog food" and don't use any of the internal functions/macros when implementing the stdlib. As I said before: if a function/macro is useful for implementing stdlib functionality like "functools" or "json", it's probably useful for external modules too.
On Tue, Feb 19, 2019 at 7:32 PM Jeroen Demeyer <J.Demeyer@ugent.be> wrote:
On 2019-02-19 04:04, Steve Dower wrote:
On 18Feb.2019 1324, Jeroen Demeyer wrote:
For a very concrete example, was it really necessary to put _PyTuple_ITEMS in (4)? That's used in _functoolsmodule.c. Especially given that the very similar PySequence_Fast_ITEMS is in (2), that seems like a strange and arbitrary limiting choice.
The reason to do this is that we can "guarantee" that we've fixed all users when we change the internal representation.
I think that CPython should then at least "eat its own dog food" and don't use any of the internal functions/macros when implementing the stdlib. As I said before: if a function/macro is useful for implementing stdlib functionality like "functools" or "json", it's probably useful for external modules too.
If we are perfect and we can design perfect APIs from start, I agree with you. But we should fix design mistake of new APIs. stdlibs are updated with Python itself. So changing internal APIs with micro version is OK. If Cython start using such internal APIs, external modules from PyPI will be broken when Python is upgraded. It feels nightmare to me. So having experimental APIs only for stdlibs makes sense to me. On the other hand, it makes sense to move _PyTuple_ITEMS to (3) or even (2). PyTuple_ITEMS(t) seems better than &PyTuple_GET_ITEM(t, 0). Regards, -- INADA Naoki <songofacandy@gmail.com>
Le mar. 19 févr. 2019 à 11:57, INADA Naoki <songofacandy@gmail.com> a écrit :
On the other hand, it makes sense to move _PyTuple_ITEMS to (3) or even (2). PyTuple_ITEMS(t) seems better than &PyTuple_GET_ITEM(t, 0).
Please don't use &PyTuple_GET_ITEM() or _PyTuple_ITEMS(). It prevents to use a more efficient storage for tuple. Something like: https://pythoncapi.readthedocs.io/optimization_ideas.html#specialized-list-f... PyPy already has the issue right now. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
Hi, On Tue, 19 Feb 2019 at 13:12, Victor Stinner <vstinner@redhat.com> wrote:
Please don't use &PyTuple_GET_ITEM() or _PyTuple_ITEMS(). It prevents to use a more efficient storage for tuple. Something like: https://pythoncapi.readthedocs.io/optimization_ideas.html#specialized-list-f...
PyPy already has the issue right now.
Just to clarify PyPy's point of view (or at least mine): 1. No, it no longer has this issue. You can misuse ``&PyTuple_GET_ITEM()`` freely with PyPy too. 2. This whole discussion is nice but is of little help to PyPy at this point. The performance hit comes mostly from emulating reference counting and non-movable objects. If the API was half the size and did not contain anything with irregular behavior, it would have made our job easier in the past, but now it's done---and it wouldn't have improved the performance of the result. A bientôt, Armin.
Just letting everyone know that I'm intending to restart this discussion over in capi-sig, as I feel like I've got an informational-PEP worth of "vision", "ideas" and "direction" and nomenclature for our C API (*not* talking about a rewrite, but the principles we should be following now... and would also want to follow in any rewrite ;) ). Nothing that should be extremely controversial. Well, perhaps. But let's at least get the stuff we already agree on into a PEP that we can use as a reference for guiding future work. I'll throw together an outline draft by email first, as I want to discuss the ideas right now rather than the grammar. Hopefully later this morning (next 3-4 hours). python-dev can expect (hope for) an informational PEP to return. If you're not currently on capi-sig, you can join it at https://mail.python.org/mailman3/lists/capi-sig.python.org/ Cheers, Steve
On Thu, Feb 21, 2019 at 6:01 AM Armin Rigo <armin.rigo@gmail.com> wrote:
Hi,
On Tue, 19 Feb 2019 at 13:12, Victor Stinner <vstinner@redhat.com> wrote:
Please don't use &PyTuple_GET_ITEM() or _PyTuple_ITEMS(). It prevents to use a more efficient storage for tuple. Something like:
https://pythoncapi.readthedocs.io/optimization_ideas.html#specialized-list-f...
PyPy already has the issue right now.
Just to clarify PyPy's point of view (or at least mine):
1. No, it no longer has this issue. You can misuse ``&PyTuple_GET_ITEM()`` freely with PyPy too.
2. This whole discussion is nice but is of little help to PyPy at this point. The performance hit comes mostly from emulating reference counting and non-movable objects. If the API was half the size and did not contain anything with irregular behavior, it would have made our job easier in the past, but now it's done---and it wouldn't have improved the performance of the result.
While it's unfortunate we start this conversation back when PyPy started to suffer through this so that we could try to make it easier for them, I don't want people to think trying to come up with a simpler FFI API eventually wouldn't be beneficial to other implementations (either ones that haven't reach Python 3 yet or have not even been written).
On 19Feb2019 0229, Jeroen Demeyer wrote:
On 2019-02-19 04:04, Steve Dower wrote:
On 18Feb.2019 1324, Jeroen Demeyer wrote:
For a very concrete example, was it really necessary to put _PyTuple_ITEMS in (4)? That's used in _functoolsmodule.c. Especially given that the very similar PySequence_Fast_ITEMS is in (2), that seems like a strange and arbitrary limiting choice.
The reason to do this is that we can "guarantee" that we've fixed all users when we change the internal representation.
I think that CPython should then at least "eat its own dog food" and don't use any of the internal functions/macros when implementing the stdlib. As I said before: if a function/macro is useful for implementing stdlib functionality like "functools" or "json", it's probably useful for external modules too.
I'm inclined to agree, but then I'm also one of the advocates for breaking out as much as possible of the stdlib into pip-installable modules, which would necessitate this :) There are certainly parts of the stdlib that are there to _enable_ the use of these features without exposing them - asyncio has some good examples of this. But the rest probably don't. If they really do, then we would have to define stable ways to get the same functionality (one example - numpy currently relies on being able to check the refcount to see if it equals 1, but we could easily provide a "Py_HasNoOtherReferences" function that would do the same thing and also allow us to one day move or remove reference counts without breaking numpy). That said, one of the criteria for "are you eligible to use the internal API" is "will users always have matched builds of this module" - for the standard library, the answer is yes, and so they can use that API. For third-party extension modules, the answer _might_ be yes, which means they _might_ be able to use the API. But both of those "mights" are outside of the control of the core development team, so we can't take that responsibility for you. The best we can do is make it easy to use the stable APIs, and make using the unstable APIs a deliberate choice so that those who do so are aware that they are now more responsible for their user's success than they thought they were. Cheers, Steve
On Mon, 18 Feb 2019 19:04:31 -0800 Steve Dower <steve.dower@python.org> wrote:
I'm afraid of hiding actually useful private macros under Py_BUILD_CORE. For example, Modules/_functoolsmodule.c and Modules/_json.c use API functions from (4). But if an API function is useful for implementing functools or json, then it's probably also useful for external extension modules: what if I want to implement something similar to functools or json, why shouldn't I be allowed to use those same API functions?
For a very concrete example, was it really necessary to put _PyTuple_ITEMS in (4)? That's used in _functoolsmodule.c. Especially given that the very similar PySequence_Fast_ITEMS is in (2), that seems like a strange and arbitrary limiting choice.
The reason to do this is that we can "guarantee" that we've fixed all users when we change the internal representation. Otherwise, the internal memory layout becomes part of the public ABI, which is what we want to fix. (PyTuple_GET_ITEM is just as problematic, FWIW.)
But PyTuple_GET_ITEM and PyList_GET_ITEM are important for performance, as are other performance-oriented macros.
If you always rebuild your extension for every micro version (3.x.y) of CPython, then sure, go ahead and use this.
Usually we would guarantee that API details don't change in bugfix versions (i.e. the 3.x.y -> 3.x.(y + 1) transition). Has that changed? That may turn out a big problem for several third-party extensions... Regards Antoine.
On Tue, 19 Feb 2019 at 20:41, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Mon, 18 Feb 2019 19:04:31 -0800 Steve Dower <steve.dower@python.org> wrote:
If you always rebuild your extension for every micro version (3.x.y) of CPython, then sure, go ahead and use this.
Usually we would guarantee that API details don't change in bugfix versions (i.e. the 3.x.y -> 3.x.(y + 1) transition). Has that changed? That may turn out a big problem for several third-party extensions...
This is the genuine technical difference between the three levels: * Py_BUILD_CORE -> no ABI stability guarantees at all * standard -> stable within a maintenance branch * Py_LIMITED_API -> stable across feature releases Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan schrieb am 19.02.19 um 15:00:
On Tue, 19 Feb 2019 at 20:41, Antoine Pitrou wrote:
On Mon, 18 Feb 2019 19:04:31 -0800 Steve Dower wrote:
If you always rebuild your extension for every micro version (3.x.y) of CPython, then sure, go ahead and use this.
Usually we would guarantee that API details don't change in bugfix versions (i.e. the 3.x.y -> 3.x.(y + 1) transition). Has that changed? That may turn out a big problem for several third-party extensions...
This is the genuine technical difference between the three levels:
* Py_BUILD_CORE -> no ABI stability guarantees at all * standard -> stable within a maintenance branch * Py_LIMITED_API -> stable across feature releases
I'm happy with this split, and i think this is how it should be. There is no reason (not withstanding critical bugs) to break the C-API within a maintenance (3.x) release series. Apart from the 3.5.[12] slip, CPython has proven very reliable in these guarantees. We can (or at least could) easily take care in Cython to enable version specific features and optimisations only from CPython alpha/beta releases on, and not when they should become available in later point releases, so that users can compile their code in, say, CPython 3.7.5 and it will work correctly in 3.7.1. We never cared about Py_BUILD_CORE (because that's obviously internal), and it's also not very likely that we will have a Py_LIMITED_API backend anywhere in the near future (although we would consider PRs for it that implement the support as an optional C compile time feature). What I would ask, though, and I think that's also Jeroen's request, is to be careful what you lock up behind Py_BUILD_CORE. Any new functionality should be available to extension modules by default, unless there is a good reason why it should remain internal. Usually, there is a reason why this functionality was added, and I doubt that there are many cases where these reasons are entirely internal to CPython. One thing that is not mentioned above are underscore private C-API functions. I imagine that they are a bit annoying for CPython itself because promoting them to public means renaming them, which is already a breaking change. But they are a clear marker for potential future breakage, which is good. Still, my experience so far suggests that they also fall under the "keep stable in maintenance branch" rule, which is even better. So, yeah, I'm happy with the status quo, and a bit worried about all the moving around of declarations and that scent of a sword of Damocles hanging over their potential confinement. IMHO, things should just be public and potentially marked as "unstable" to advertise a risk of breakage in a future CPython X.Y feature releases. Then it's up to the users to decide how much work they want to invest into keeping up with C-API changes vs. potentially sub-optimal but stable C-API usage. Stefan
On 19Feb2019 1212, Stefan Behnel wrote:
So, yeah, I'm happy with the status quo, and a bit worried about all the moving around of declarations and that scent of a sword of Damocles hanging over their potential confinement. IMHO, things should just be public and potentially marked as "unstable" to advertise a risk of breakage in a future CPython X.Y feature releases. Then it's up to the users to decide how much work they want to invest into keeping up with C-API changes vs. potentially sub-optimal but stable C-API usage.
Unfortunately, advertising a risk of breakage doesn't make the break less painful when it happens. We'd rather avoid that pain by preemptively breaking (at a major version update, e.g. 3.8) so that minor breaks (e.g. between 3.8.2 and 3.8.3) don't cause any problems at all. And if we preemptively break, then we can also preemptively add functions to cover what direct memory accesses previously used to do. And it's not up to the users - it's up to the package developers. Most of whom optimise for their own ease of life (as someone who supports Windows users, I'm well aware of where package developers cut painful corners ;) ). The only choice users get in the matter is whether they ever update Python, or if they switch to a language that is more respectful toward them. For what it's worth, the users I've been speaking to recently are *far* more concerned about being able to update Python without things breaking than they are about runtime performance. Cheers, Steve
Steve Dower schrieb am 19.02.19 um 21:40:
On 19Feb2019 1212, Stefan Behnel wrote:
Then it's up to the users to decide how much work they want to invest into keeping up with C-API changes vs. potentially sub-optimal but stable C-API usage. [...] And it's not up to the users - it's up to the package developers.
I meant "users" as in "users of the C-API", i.e. package developers. Stefan
Le mar. 19 févr. 2019 à 21:15, Stefan Behnel <stefan_ml@behnel.de> a écrit :
What I would ask, though, and I think that's also Jeroen's request, is to be careful what you lock up behind Py_BUILD_CORE. Any new functionality should be available to extension modules by default, unless there is a good reason why it should remain internal. Usually, there is a reason why this functionality was added, and I doubt that there are many cases where these reasons are entirely internal to CPython.
I think that we should have some rules here. One rule is that we should avoid APIs which allow to do something no possible in Python. That's an important rule for PyPy and other Python implementations. We cannot avoid such APIs completely, but they should be the exception, not the default. Another rule is to avoid stop adding new APIs which only exist for performance. For example, PyDict_GetItem() only exists for performance: PyObject_GetItem() can be used instead. There are multiple issues with writing "specialized code". For example, PyDict_GetItem() must only used if the type is exactly dict (PyDict_CheckExact). Otherwise, you change the Python semantics (don't respect overriden __getitem__). Each new API means more work for other Python implementations, but also more maintenance work for CPython. Premature optimization is the root of all evil. Most C extensions use premature optimization which are causing us a lot of troubles nowadays when we want to make the C API evolve and cause issues to PyPy which has issues to reimplement the C API on top of their different object model with a different GC. These are just proposals. Feel free to comment :-) Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On Thu, 21 Feb 2019 at 11:35, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Thu, 21 Feb 2019 12:13:51 +0100 Victor Stinner <vstinner@redhat.com> wrote:
Premature optimization is the root of all evil. Most C extensions use premature optimization
How do you know it's premature? Some extensions _are_ meant for speed.
Extensions that need to squeeze every bit of speed out of the C API are the exception rather than the rule. Making it easier for extension authors to naturally pick portable options seems reasonable to me. Gating the "fast, but unsafe" APIs behind some sort of "opt in" setting seems like a sensible approach. However I agree, making it *impossible* to get right down to the high-speed calls (with the understanding that you're typing yourself to CPython and need to carefully track internal changes) is not something we should do. Python's built an ecosystem around high-performance C extensions, and we should support that ecosystem. Paul
Le 21/02/2019 à 12:58, Paul Moore a écrit :
On Thu, 21 Feb 2019 at 11:35, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Thu, 21 Feb 2019 12:13:51 +0100 Victor Stinner <vstinner@redhat.com> wrote:
Premature optimization is the root of all evil. Most C extensions use premature optimization
How do you know it's premature? Some extensions _are_ meant for speed.
Extensions that need to squeeze every bit of speed out of the C API are the exception rather than the rule. Making it easier for extension authors to naturally pick portable options seems reasonable to me.
Actually, it would be interesting to have some kind of survey of C extensions (through random sampling? or popularity?) to find out why the developers had to write a C extension in the first place and what their concerns are. Intuitively there are three categories of C extensions: 1. extensions whose entire purpose is performance (this includes all scientific computing C extensions - including Numpy or Pandas -, but also C accelerators such as SQLAlchemy's or simplejson's) 2. extensions wrapping third-party APIs that are not performance-critical. If you are exposing a wrapper to e.g. the posix_spawn() system call, probably the wrapper performance isn't very important. 3. extensions wrapping third-party APIs that are performance-critical. For example, in a database wrapper, it's important that your native DB-to-Python and Python-to-native DB conversions are as fast as possible, because users are going to convert a *lot* of data. Note that category 2 may be taken care of by ctypes or cffi. Regards Antoine.
On Thu, 21 Feb 2019 at 12:12, Antoine Pitrou <antoine@python.org> wrote:
Actually, it would be interesting to have some kind of survey of C extensions (through random sampling? or popularity?) to find out why the developers had to write a C extension in the first place and what their concerns are.
Indeed. There's also embedding, where I suspect there's a much higher likelihood that performance isn't key. And in your survey, I'd split out "needs the Python/C interface to be fast" from "needs internal operations to be fast, but data transfer between C and Python isn't as critical". I suspect there's a lot of people who believe they are in the former category, but are actually in the latter... Paul
On Thu, Feb 21, 2019 at 9:23 PM Paul Moore <p.f.moore@gmail.com> wrote:
On Thu, 21 Feb 2019 at 12:12, Antoine Pitrou <antoine@python.org> wrote:
Actually, it would be interesting to have some kind of survey of C extensions (through random sampling? or popularity?) to find out why the developers had to write a C extension in the first place and what their concerns are.
Indeed. There's also embedding, where I suspect there's a much higher likelihood that performance isn't key.
And in your survey, I'd split out "needs the Python/C interface to be fast" from "needs internal operations to be fast, but data transfer between C and Python isn't as critical". I suspect there's a lot of people who believe they are in the former category, but are actually in the latter...
Paul
As my experience, I can't believe the majority of extensions are in category B. And when speed is matter, PyPy support is not mandatory. PyPy is fast enough with pure Python implementation. Python/C API doesn't make PyPy fast when creating or reading massive objects. These are my recent Python/C API usages. # msgpack msgpack is a serialization format like JSON, but in binary format. It contains pure Python implementation and Cython implementation. Cython implementation is for CPython. PyPy is fast enough with pure Python implementation. It is important to fast object access. It is like marshal in Python. So this is clearly in category A. # mysqlclient It is binding of libmysqlclient or libmariadbclient. In some case, query can return many small data and I need to convert them to Python object. So this library is basically in category B, but sometime in C. # MarkupSafe It is library which escape strings. It should handle massive small string chunks, because it is called from template engine. So this library is in category A. Note that MarkupSafe having pure Python version, like msgpack. While MarkupSafe is not optimized for PyPy yet, it's possible to add implementation for PyPy written in Python using PyPy's string builder [1]. [1] https://morepypy.blogspot.com/2011/10/speeding-up-json-encoding-in-pypy.html -- INADA Naoki <songofacandy@gmail.com>
Le jeu. 21 févr. 2019 à 12:36, Antoine Pitrou <solipsis@pitrou.net> a écrit :
On Thu, 21 Feb 2019 12:13:51 +0100 Victor Stinner <vstinner@redhat.com> wrote:
Premature optimization is the root of all evil. Most C extensions use premature optimization
How do you know it's premature? Some extensions _are_ meant for speed.
Sorry, I don't ask to stop optimizing C extension. I'm asking to stop to use low-level C API like PyTuple_GET_ITEM() if the modified code is the not the performance bottleneck. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On Thu, 21 Feb 2019 13:45:05 +0100 Victor Stinner <vstinner@redhat.com> wrote:
Le jeu. 21 févr. 2019 à 12:36, Antoine Pitrou <solipsis@pitrou.net> a écrit :
On Thu, 21 Feb 2019 12:13:51 +0100 Victor Stinner <vstinner@redhat.com> wrote:
Premature optimization is the root of all evil. Most C extensions use premature optimization
How do you know it's premature? Some extensions _are_ meant for speed.
Sorry, I don't ask to stop optimizing C extension. I'm asking to stop to use low-level C API like PyTuple_GET_ITEM() if the modified code is the not the performance bottleneck.
As long as some people need that API, you'll have to maintain it anyway, even if _less_ people use it. Ingesting lists and tuples as fast as possible is important for some use cases. I have worked personally on some of them (on e.g. Numba or PyArrow). Regards Antoine.
On 2019-02-21 12:53, Antoine Pitrou wrote:
On Thu, 21 Feb 2019 13:45:05 +0100 Victor Stinner <vstinner@redhat.com> wrote:
Le jeu. 21 févr. 2019 à 12:36, Antoine Pitrou <solipsis@pitrou.net> a écrit :
On Thu, 21 Feb 2019 12:13:51 +0100 Victor Stinner <vstinner@redhat.com> wrote:
Premature optimization is the root of all evil. Most C extensions use premature optimization
How do you know it's premature? Some extensions _are_ meant for speed.
Sorry, I don't ask to stop optimizing C extension. I'm asking to stop to use low-level C API like PyTuple_GET_ITEM() if the modified code is the not the performance bottleneck.
As long as some people need that API, you'll have to maintain it anyway, even if _less_ people use it. Ingesting lists and tuples as fast as possible is important for some use cases. I have worked personally on some of them (on e.g. Numba or PyArrow).
If I'm working with a dict, the first place I look is PyDict_*, and that leads me to PyDict_GetItem. The docs for PyDict_GetItem don't mention PyObject_GetItem. Perhaps, if PyObject_GetItem is recommended, it should say so, and similarly for other parts of the API.
Le lun. 18 févr. 2019 à 22:34, Jeroen Demeyer <J.Demeyer@ugent.be> a écrit :
First of all, if everybody can actually #define Py_BUILD_CORE and get access to the complete API, I don't mind so much. But then it's important that this actually keeps working (i.e. that those headers will always be installed).
Still, do we really need so many levels of API: (1) stable API (with #define Py_LIMITED_API) (2) public documented API (3) private undocumented API (the default exposed API) (4) internal API (with #define Py_BUILD_CORE)
It's not a matter of documentation. It's a matter of warranty provided to users. I would like to move towards (1) by default: only provide a stable API by default. IMHO most users will be just fine with this subset of the API. The borders between (2), (3) and (4) are unclear. I created Include/cpython/ which is not really a "private API" but more "CPython implementation details". A better definition of (1) would be "portable stable API" whereas (2)+(3) would be "CPython stable API". And (4) would be the unstable API. Summary: (1) Portable stable API (2) Portable CPython API (3) Unstable API ... The border between (2) and (3) is a "work-in-progress". I modified "make install" to install (3) as well: they are users of this API. Performance can be a good motivation: Cython for example. Debuggers and profiles really need to access to the lowest level of API usually because they can only *inspect* (structure fileds) but no *execute* code (call functions).
I'm afraid of hiding actually useful private macros under Py_BUILD_CORE.
Again, it's not a matter of usefulness. It's a matter of backward compatibility announced to users. I would like to make it more explicit that if you *opt in* for an unstable API, you are on your own. My intent is that in 5 years or 10 years, slowly, most C extensions will use (1) which will allow Python to experiment new optimizations, and should help PyPy (cpyext module) to become even more efficient. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
Hi Eric, IMHO the main blocker issue for any C API change is that nobody is able to measure the risk these changes. To better control the risk, I propose to select a list of popular C extensions, and build a CI to run their test suite on top of the development version of Python. Such CI wouldn't detect all backward incompatible changes. It wouldn't prevent us to merge backward incompatible changes. Some projects are already tested on the master branch of Python. My intent is to detect issues earlier, and if something goes wrong: discuss early to decide what to do. Fixing only some popular C extensions is one option. Another option is to provide some commands / hints to help maintainers of C extensions to adapt their code to the new C API. The other obvious option is to revert the change and maybe do it differently. Right now, it's too scary to walk in the dark. What I also would like to see is the creation of a group of people who work on the C API to discuss each change and test these changes properly. Victor Le sam. 16 févr. 2019 à 00:41, Eric Snow <ericsnowcurrently@gmail.com> a écrit :
Hi all,
I've been working on the runtime lately, particularly focused on my multi-core Python project. One thing that would help simplify changes in this area is if PyInterpreterState were defined in Include/internal. This would mean the type would be opaque unless Py_BUILD_CORE were defined.
The docs [1] already say none of the struct's fields are public. Furthermore, Victor already moved it into Include/cpython (i.e. not in the stable ABI) in November. Overall, the benefit of making internal types like this opaque is realized in our ability to change the internal details without further breaking C-API users.
Realistically, there may be extension modules out there that use PyInterpreterState fields directly. They would break. I expect there to be few such modules and fixing them would not involve great effort. We'd have to add any missing accessor functions to the public C-API, which I see as a good thing. I have an issue [2] open for the change and a PR open. My PR already adds an entry to the porting section of the 3.8 What's New doc about dealing with PyInterpreterState.
Anyway, I just wanted to see if there are any objections to making PyInterpreterState an opaque type outside of core use.
-eric
p.s. I'd like to do the same with PyThreadState, but that's a bit trickier [3] and not one of my immediate needs. :)
[1] https://docs.python.org/3/c-api/init.html#c.PyInterpreterState [2] https://bugs.python.org/issue35886 [3] https://bugs.python.org/issue35949 _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com
-- Night gathers, and now my watch begins. It shall not end until my death.
On 2019-02-21 12:18, Victor Stinner wrote:
What I also would like to see is the creation of a group of people who work on the C API to discuss each change and test these changes properly.
I don't think that we should "discuss each change", we should first have an overall plan. It doesn't make a lot of sense to take small steps if we have no clue where we're heading to. I am aware of https://pythoncapi.readthedocs.io/new_api.html but we should first make that into an accepted PEP.
participants (15)
-
Antoine Pitrou -
Antoine Pitrou -
Armin Rigo -
Barry Warsaw -
Brett Cannon -
Eric Snow -
INADA Naoki -
Jeroen Demeyer -
Michael -
MRAB -
Nick Coghlan -
Paul Moore -
Stefan Behnel -
Steve Dower -
Victor Stinner