[Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

[Nathaniel Smith]

On Mon, Nov 12, 2018 at 10:46 PM, Gregory P. Smith <greg at krypto.org> wrote:
>
> On Fri, Nov 9, 2018 at 5:50 PM Nathaniel Smith <njs at pobox.com> wrote:
>>
>> On Fri, Nov 9, 2018 at 4:30 PM, Victor Stinner <vstinner at redhat.com>
>> wrote:
>> > Ah, important points. I don't want to touch the current C API nor make
>> > it less efficient. And compatibility in both directions (current C API
>> > <=> new C API) is very important for me. There is no such plan as
>> > "Python 4" which would break the world and *force* everybody to
>> > upgrade to the new C API, or stay to Python 3 forever. No. The new C
>> > API must be an opt-in option, and current C API remains the default
>> > and not be changed.
>>
>> Doesn't this mean that you're just making the C API larger and more
>> complicated, rather than simplifying it? You cite some benefits
>> (tagged pointers, changing the layout of PyObject, making PyPy's life
>> easier), but I don't see how you can do any of those things so long as
>> the current C API remains supported.
[...]
> I'd love to get to a situation where the only valid ABI we support knows nothing about internal structs at all. Today, PyObject memory layout is exposed to the world and unchangable. :(
> This is a long process release wise (assume multiple stable releases go by before we could declare that).

It seems like the discussion so far is:

Victor: "I know people when people hear 'new API' they get scared and
think we're going to do a Python-3-like breaking transition, but don't
worry, we're never going to do that."
Nathaniel: "But then what does the new API add?"
Greg: "It lets us do a Python-3-like breaking transition!"

To make a new API work we need to *either* have some plan for how it
will produce benefits without a big breaking transition, *or* some
plan for how to make this kind of transition viable. These are both
super super hard questions -- that's why this discussion has been
dragging on for a decade now! But you do have to pick one or the other
:-).

> Experimentation with new internal implementations can begin once we have a new C API by explicitly breaking the old C API with-in such experiments (as is required for most anything interesting).  All code that is written to the new C API still works during this process, thus making the job of practical testing of such new VM internals easier.

So I think what you're saying is that your goal is to get a
new/better/shinier VM, and the plan to accomplish that is:

1. Define a new C API.
2. Migrate projects to the new C API.
3. Build a new VM that gets benefits from only supporting the new API.

This sounds exactly backwards to me?

If you define the new API before you build the VM, then no-one is
going to migrate, because why should they bother? You'd be asking
overworked third-party maintainers to do a bunch of work with no
benefit, except that maybe someday later something good might happen.

And if you define the new API first, then when you start building the
VM you're 100% guaranteed to discover that the new API isn't *quite*
right for the optimizations you want to do, and have to change it
again to make a new-new API. And then go back to the maintainers who
you did convince to put their neck out and do work on spec, and
explain that haha whoops actually they need to update their code
*again*.

There have been lots of Python VM projects at this point. They've
faced many challenges, but I don't think any have failed because there
just wasn't enough pure-Python code around to test the VM internals.
If I were trying to build a new Python VM, that's not even in the top
10 of issues I'd be worried about...

-n

-- 
Nathaniel J. Smith -- https://vorpus.org