On 13Apr2020 1325, Paul Moore wrote:
Personally, I'd say that "recommended 3rd party tools" reads as saying "if you want a 3rd party tool to build extensions, these are good (and are a lot easier than using the raw C API)". That's a lot different than saying "we recommend that people writing C extensions do not use the raw C API, but use one of these tools instead".
Yeah, that's fair. But at the same time, saying anything more strong is an endorsement that we might have to withdraw at some point in the future (if the project we recommend implodes, for example).
Also, if we *are* going to push people away from the raw C API, then I think we should be recommending a particular tool (likely Cython) as what people writing their first extension (or wanting to switch from the raw C API for the first time) should use. Faced with the API docs, and a list of 3rd party options, I know that *I* am likely to say "yeah, leave that research for another day, I'll use what's in the docs in front of me for now". Also, if we are expecting to push people towards 3rd party tools, that seems to me to be a relatively significant shift in emphasis, and one we should be publicising more directly (via What's New, and blog postings / release announcements, etc.) In the absence of anything like that, I think it's quite reasonable for people to gravitate towards the traditional C API.
Right, except we haven't decided to do it yet. There's still a debate about whether the current third party tools are even sufficient (not to mention what "sufficient" means).
Having said all this, I *do* think that promoting some 3rd party tool (as I say, I suspect this would be Cython) as the recommended means of writing C extensions, is a reasonable approach to take. I just object to it happening "quietly" via changes like this which make it harder to use the raw C API, justifying themselves by saying "you shouldn't do that anyway".
Agreed, I'd rather be up front about it.
On a related but different note, what is the recommended policy (assuming it's not to use the C API) for embedding Python, and for exposing the embedding app to Python as a C extension? My standard example of this is the Vim interface to Python - see https://github.com/vim/vim/blob/master/src/if_python3.c. I originally wrote this back in the Python 1.5 days, so it's *very* old, and quite likely not how I'd write it now, even using the C API. But what's the recommendation for code like that in the face of these changes, and the suggestion that using 3rd party tools is the normal way to write C extensions?
I don't think any current 3rd party tools really help with embedding (I say that as a regular embedder, not as someone who skim-read their docs). In this case, you really do need low-level access to Python's thread and memory management, and the ability to interact directly with the rest of your application's data structures.
PyBind11 is the best I've used here - Cython insists on including all its boilerplate to make a complete module, which often is not what you want. But there's a lot of core things that need to be improved if embedding is going to get any better, as I've posted often enough. We can't rely on third-party tools here, yet.