Move the limited API to distinct headers.
(Sorry for all the posts related to the limited API. It's been a pain point for me lately.)
Would it be feasible to have distinct header files that contain only the limited API (perhaps in Include/limited)?
The files could be auto-generated to avoid duplicated code (like other files in the repo), though the nature of the limited API suggests the only future changes would be opt-in ones, which should be manual anyway. If there is no auto-generation then there should be a test to ensure the limited API matches the corresponding public API definitions exactly. Either way, Include/Python.h would have to be updated to switch to Include/limited if Py_LIMITED_API is defined. Also, FWIW, Include/limited would effectively fill the role of the manifest described in PEP 652. [1]
With Include/limited, would we still need Include/cpython? I suppose we could keep only public API in Include/*.h and keep "private-but-not-internal" API in Include/cpython (or rename it to Include/private or Include/unstable).
Finally, is the separation of headers by access "rings" worth it? On the one hand it makes the divisions explicit and obvious and helps avoid API leaking out accidentally. We'd had success in that regard with Include/internal and Include/cpython. On the other hand, the divisions mean related API is often spread out across multiple files (albeit with matching names), where otherwise those would all be in one file.
Anway, this is something that came to mind as I was writing up my other posts to this list.
-eric
[1] https://www.python.org/dev/peps/pep-0652/#stable-abi-manifest
There is a work-in-progress PR to document this organization: https://github.com/python/cpython/pull/24884
Victor
On Wed, Mar 17, 2021 at 8:03 PM Antoine Pitrou <antoine@python.org> wrote:
-- Night gathers, and now my watch begins. It shall not end until my death.
Le 17/03/2021 à 20:38, Victor Stinner a écrit :
There is a work-in-progress PR to document this organization: https://github.com/python/cpython/pull/24884
That's beside the point. Whether or not it's documented, it's still
cumbersome to navigate because you don't know upfront in which header
file you'll find the information you're looking for. So you end up
switching, for example, between Include/pystate.h,
Include/cpython/pystate.h and Include/internal/pycore_pystate.h. It
feels like a waste of time.
I think it would be much more convenient with a single header file per topic, and clearly-separated sections in the header, e.g.:
/* ------------------------------------------------------
- Stable ABI / limited API */
...
/* ------------------------------------------------------
- Public API */
...
/* ------------------------------------------------------
- CPython-specific API */
...
/* ------------------------------------------------------
- CPython internals, only for CPython core
- Use at your own risk */
...
Regards
Antoine.
On Thu, 18 Mar 2021 at 05:44, Antoine Pitrou <antoine@python.org> wrote:
Trying to guess where symbols are defined *is* a waste of time - one of the biggest reasons I switched to VSCode as my primary text editor was because the code base search was so good.
I think it would be much more convenient with a single header file per topic, and clearly-separated sections in the header, e.g.:
The problem with this approach is that you can't tell at a glance which of the API levels a symbol belongs to - you have to scroll up in the file to find the section header to work out which section it is in.
With the directory based layout, you don't even need to click on the search result to answer that question - the file path tells you.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
The main issue addressed by the new directory organization is previously, any new API magically became part of the limited C API by default. It was more a "deny list" approach, than an "allow list" approach.
Now, you must decide in which file you put a new API. You can still add it to the "wrong API" by mistake, but you first have to think about which file is the most appropriate.
In vim, I'm used to type ":grep -R Include FUNCTION" to find a function definition. I'm not convient, but I'm used to it. To find the function implementation, it's simpler: I put the cursor on function name and I type CTRL+] thanks to ctags ("make tags"), or I type :tag FUNCTION.
Victor
On 3/17/2021 6:00 PM, Eric Snow wrote:
Would it be feasible to have distinct header files that contain only the limited API (perhaps in Include/limited)?
I thought the idea of the current separation was for the limited API to be the "default", and the non-limited part to go into cpython/*.h. In which case, the problem is the lack of documentation (which Victor mentioned, but it clearly belongs in the header of each file as Antoine suggests).
I'm not personally too offended by separate or combined files, though I do prefer users to opt in via explicit includes rather than preprocessor definitions or transitive includes. But I even moreprefer not changing things too often.
Cheers, Steve
There is a work-in-progress PR to document this organization: https://github.com/python/cpython/pull/24884
Victor
On Wed, Mar 17, 2021 at 8:03 PM Antoine Pitrou <antoine@python.org> wrote:
-- Night gathers, and now my watch begins. It shall not end until my death.
Le 17/03/2021 à 20:38, Victor Stinner a écrit :
There is a work-in-progress PR to document this organization: https://github.com/python/cpython/pull/24884
That's beside the point. Whether or not it's documented, it's still
cumbersome to navigate because you don't know upfront in which header
file you'll find the information you're looking for. So you end up
switching, for example, between Include/pystate.h,
Include/cpython/pystate.h and Include/internal/pycore_pystate.h. It
feels like a waste of time.
I think it would be much more convenient with a single header file per topic, and clearly-separated sections in the header, e.g.:
/* ------------------------------------------------------
- Stable ABI / limited API */
...
/* ------------------------------------------------------
- Public API */
...
/* ------------------------------------------------------
- CPython-specific API */
...
/* ------------------------------------------------------
- CPython internals, only for CPython core
- Use at your own risk */
...
Regards
Antoine.
On Thu, 18 Mar 2021 at 05:44, Antoine Pitrou <antoine@python.org> wrote:
Trying to guess where symbols are defined *is* a waste of time - one of the biggest reasons I switched to VSCode as my primary text editor was because the code base search was so good.
I think it would be much more convenient with a single header file per topic, and clearly-separated sections in the header, e.g.:
The problem with this approach is that you can't tell at a glance which of the API levels a symbol belongs to - you have to scroll up in the file to find the section header to work out which section it is in.
With the directory based layout, you don't even need to click on the search result to answer that question - the file path tells you.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
The main issue addressed by the new directory organization is previously, any new API magically became part of the limited C API by default. It was more a "deny list" approach, than an "allow list" approach.
Now, you must decide in which file you put a new API. You can still add it to the "wrong API" by mistake, but you first have to think about which file is the most appropriate.
In vim, I'm used to type ":grep -R Include FUNCTION" to find a function definition. I'm not convient, but I'm used to it. To find the function implementation, it's simpler: I put the cursor on function name and I type CTRL+] thanks to ctags ("make tags"), or I type :tag FUNCTION.
Victor
On 3/17/2021 6:00 PM, Eric Snow wrote:
Would it be feasible to have distinct header files that contain only the limited API (perhaps in Include/limited)?
I thought the idea of the current separation was for the limited API to be the "default", and the non-limited part to go into cpython/*.h. In which case, the problem is the lack of documentation (which Victor mentioned, but it clearly belongs in the header of each file as Antoine suggests).
I'm not personally too offended by separate or combined files, though I do prefer users to opt in via explicit includes rather than preprocessor definitions or transitive includes. But I even moreprefer not changing things too often.
Cheers, Steve
participants (5)
-
Antoine Pitrou -
Eric Snow -
Nick Coghlan -
Steve Dower -
Victor Stinner