[Python-Dev] Are undocumented functions part of the stable ABI?

[Nick Coghlan]

On 6 April 2018 at 14:45, Guido van Rossum <guido at python.org> wrote:
> As you may guess from the silence, it may be hard to get a definitive answer
> to this question -- PEP 384's author has stopped actively participating in
> the Python community and I'm not sure if any core developers currently
> consider themselves to be the "guardians of the ABI".

It's only been a few days, and there are definitely some of us that
genuinely want the stable ABI to meet its guarantees (e.g. me, Steve
Dower, Eric Snow, Victor Stinner). It just isn't likely to come fully
into its own until more library developers are able to drop support
for Python 2.7.

> That said, from a quick skim of PEP 384 it seems to be quite strict in its
> position that anything not explicitly included by the PEP should be
> considered not part of the ABI, which makes me think that only a few
> PyCFunction related items are considered part of the ABI (searching for
> PyCFunction only finds three hits). OTOH it states that if Py_LIMITED_API is
> defined, all definitions that are not part of the ABI will be hidden. So
> from that (plus the source code) you should be able to tell which
> PyCFunction_* functions are part of the ABI -- again it does not appear the
> documentation status of a function matters for this rule.
>
> On the third hand, PEP 384 references a file "python3.def"
> (https://www.python.org/dev/peps/pep-0384/#id5) and that file contains
> several PyCFunction_* names. Maybe this is the hard rule you're looking for?
> Again, being documented is not a requirement.
>
> Another observation would be that (AFAICT) PEP 384 strictly forbids
> signature changes, but is mostly silent on semantics.

Right, PEP 384 is mainly focused on low level ABI compatibility:
whether or not the extension module loader will even be able to import
the module without getting segfaults due signature mismatches or
struct size changes.

Semantic questions are a bit different, as the question there isn't
"Will the module segfault?" it's "Will it work as expected?", and that
question applies regardless of whether you recompile it or not.

> On Wed, Apr 4, 2018 at 10:34 PM, Jeroen Demeyer <J.Demeyer at ugent.be> wrote:
>> The context is PEP 575. I guess my question is mostly about
>> PyCFunction_Check(). I will not be able to keep it 100% backwards compatible
>> simply because the goal of that PEP is precisely changing the classes of
>> some objects.
>>
>> Now the question is: am I allowed to change the implementation of
>> PyCFunction_Check()? If it's considered part of the stable ABI, then the
>> answer is immediately "no".

Changing macro definitions doesn't break the stable ABI, as long as
the *old* macro expansions still do the right thing.

So in the case of PEP 575, the following change would be OK (since the
old macro expansion would still work):

* add PyCFunction_CheckExact() to replace the current meaning of
PyCFunction_Check
* adjust PyCFunction_Check() to allow subclasses

In a lot of ways, you're actually better off than if these were
functions, as extension modules built against the stable ABI with an
old version won't even see the semantic change - they'll implicitly
keep PyCFunction_CheckExact semantics.

>> By the way, does anybody happen to know why the PyCFunction_* functions
>> are undocumented? Is it just an oversight in the docs or is it intentional?

Just an oversight as far as I am aware.

>> But regardless of the context, I think that the question "Are undocumented
>> functions part of the stable ABI?" should be answered in PEP 384.

We'd put any answer to that question in
https://docs.python.org/3/c-api/stable.html, rather than in PEP 384
(although it may make sense to link from the PEP back to the docs for
usage questions, for the benefit of folks that find the PEP first).

There isn't really a universal answer, though - the closest we get to
that is the fact that we default to "No, they're not part of the
stable ABI" when they have an underscore prefix (and hence aren't even
part of the public API), and "Yes, they are covered by the stable ABI"
otherwise (it may be a bug that they escaped into the stable ABI in
the first place, but once they're there, they're there).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia