Leading uderscore in C-API names (`_Py`): Private, or just a warning?
(Rather obviously, I'm not writing on behalf of the SC.)
The Steering Council's response to my PEP 689 (Unstable C API tier) opened a subtle question that I'd like to get some discussion on:
What does it mean if an API has a leading underscore (i.e. the _Py
prefix)?
It seems that currently, the underscore serves as a “warning” -- its users (and reviewers) should read the documentation when using such functions. (This might be more nuanced; I'll leave any details to other people so I won't misinterpret their views.)
I was surprised to learn that this is the status quo, and I don't think it's a good one. So I took what I thought is the status quo, tightened it, and I present it as a proposal:
## Proposal
Anything that starts with _Py
is a fully internal, private CPython
implementation detail. As a user, you can use it in a debugging session
but don't rely on it existing (or being compatible) in any different
CPython release. That said:
- We won't break this internal API for no reason, so if you use it now and have good tests you don't need stop using it right now.
- If you're using private API and don't see a public alternative, you
should contact CPython devs to
- see if we can add public API for the use case.
- let us know that someone's using the API, and we should be extra careful with it.
With this rule, you can simply grep your codebase for _Py
to see if
you're using API that can go away without warning.
I'm aware that there are currently many underscored names that are intended to be used. Under this proposal, we should eventually add non-underscores aliases for them to make their status clear. (That's a desired end state; there's no rush to get 100% there.)
## Underscore as a warning
The alternative (underscore means “warning”, read the documentation) is, IMO, unfortunate. The documentation for underscored functions is often missing, both for functions you can use and ones you shouldn't. Consider example questions like:
_PyCode_GetExtra
is mentioned in PEP 523, but not documented on docs.python.org. As a user, can I use it?_PyImport_AcquireLock
is mentioned in a StackOverflow answer, but not on docs.python.org. As a user, can I use it?- I find that (say)
_PyArg_UnpackStack
is no longer necessary in CPython. As a core dev, where do I need to look to see if I can remove it?
The issues these point out can't be fixed easily: there are hundreds of underscored functions exposed in the public headers, and no good way to prevent adding new ones (of either kind -- consumable or private).
Some API is even only exposed for technical reasons: _Py_NewRef
needs
to be exposed, even though we'd like users to never use it. (This
particular function is not too dangerous to use, but any macro or
static inline
function that's an implementation detail has the same
issue.)
But there's no way for CPython to mark something in a public API header
as private. Something that's undocumented on purpose is
indistinguishable from something we forgot to document. There's no way
to check a codebase for using internal API, short of manually checking
the docs of each function.
participants (3)
-
Marc-Andre Lemburg
-
Petr Viktorin
-
Victor Stinner