On 18.03.2021 10:00, Ronald Oussoren wrote:

...

On 17 Mar 2021, at 19:26, M.-A. Lemburg mailto:mal@egenix.com> wrote:

On 17.03.2021 19:16, Eric Snow wrote:

...

On Wed, Mar 17, 2021 at 12:08 PM Barry Warsaw mailto:barry@python.org> wrote:

...

Perhaps an even deeper and more provocative question is whether extension writers should be writing against *any* C API?

Decoupling extensions from the concrete C API could give the interpreter more flexibility for internal change.

Good point!  I'm totally on board. :)  I suppose that would be a Python 4.0 event and we could have no stable ABI for 4.0.  (PEP 384 explicitly applies to Python 3.)

So drop the Python C API which has enabled creating such an elaborate software environment and established the basis for Python's popularity in so many industries ?

This would kill Python's popularity in the (data) science world and beyond, where performance is of essence.

Even with tools like Cython you still need the C API.

I’m fairly sure the proposal is to drop the *stable* ABI, not the entire C API.

Well, let's hope so :-)

I find the discussion and patching applied to the C API premature. We don't have a clear direction yet, but already start making assumptions about Python's future which may not hold.

People also forget that the GIL, which is motivating most of the changes at the moment, is needed not only by the Python interpreter to protect its own structures, but also by many C extensions, which are not written in a re-entrant way.

More so, importing an extension which links against an external library more than once into a process will most likely fail as well. The linker will only load the lib once and initialize it once, but the Python extension will still register with the lib once for each subinterpreter, which may well fail, unless the lib supports this and provide APIs to have multiple components register from within a single process. It's common to have static resources in C libraries, just like Python has/had static type objects, singletons, etc. and it's also common for C lib to not expect being registered with more than once.

IMO, the subinterpreters idea will only have limited use in practice (mostly for pure Python applications and perhaps some which only use Cython compiled optimizations). In today's world scaling up is much easier and more reliably had by running multiple Python processes and using inter-process communication for data exchange.

The ability to have processes crash, or be killed, and then restarted in case of problems results in much more reliable applications than any form of multi-threading. Even browsers on Windows have moved into this direction. That should tell us something, given that process creation on Windows is nowhere near as lean as thread creation.

Leaving the GIL in place and putting focus on making the inter-process data exchange as fast and efficient as possible would get us more fans than coming up with a solution which leaves behind hundreds of unusable C extensions.

E.g. being able to share a Python list or even dictionary between processes, without the need to recreate this in the resp. processes would result in much more efficiency, than being able to run more threaded Python code inside a single process.

Having a stable ABI makes it easier to have packages on PyPI available on day 1 of a new Python release. There’s currently a largish window between a new major Python release and the availability of packages on PyPI.  Creating wheels is a lot easier than it was in the past thanks to cloud CI/CD systems and supporting packages, but not all projects use those (and can use those).

That said, the current stable ABI might be too restrictive. Designing an API/ABI that can be kept stable while evolving the implementation is far from trivial, and our current ABI is not there yet. The delays in having wheels are largely an infrastructure problem, IMO.

The conda ecosystem has shown that it's well possible for a package infrastructure to automatically build packages for lots of different target systems.

The Raspberry Pi project pywheels has done the same for Raspis.

We don't need a stable ABI for this to happen.

What we need is stable Python C API, which doesn't require changes to the C extension code base for every single release, so that recompilation is indeed enough to get wheels for a new release -- at least in most cases.

This doesn't mean the Python C internals cannot change. It only means that Python has to ship with a compatibility layer which makes sure that the public API remains stable across at least a good number of Python releases.

The main reason for the stable ABI was to work around issues with getting extensions to compile on Windows in those days. Things have changed and this need is no longer important, since Windows compilers are readily available.

By having an automated infrastructure which can deal with handling the compilation step, we make things easier for C extension developers across the board and without having to restrict the C API to result in a stable ABI.

Personally, I think that we should all switch over to the conda infrastructure to get us there quickly. Others will disagree, of course, and rather continue reinventing wheels :-) That's fair.

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Experts (#1, Mar 18 2021)

...

...

Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/

Le 18/03/2021 à 15:33, Steve Dower a écrit :

On 18Mar2021 1026, M.-A. Lemburg wrote:

...

By having an automated infrastructure which can deal with handling the compilation step, we make things easier for C extension developers across the board and without having to restrict the C API to result in a stable ABI.

Personally, I think that we should all switch over to the conda infrastructure to get us there quickly. Others will disagree, of course, and rather continue reinventing wheels :-) That's fair.

By "conda infrastructure" I assume you mean conda-forge? So essentially standardise extension module (and package) building across the ecosystem, and establish a community that helps maintain this *apart* from the original package developers?

Not necessarily apart. I don't have a representative sample, but in my experience maintainers of conda-forge packages are often also upstream developers.

In other words, conda-forge has a rather loose model, it is not a heavily-curated distribution supervised by a well-identified organization or community that takes responsibility for the overall package set.

Regards

Antoine.

I'm not totally against this, FWIW, I just think it's unlikely that we'd get it to work (at least with the blessing of the upstream package developers, who always seem to insist on being in full control of their packages).

I would love to be convinced it could be done though :) What would it take? Am I just being too pessimistic?

Cheers, Steve

---

capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org https://mail.python.org/mailman3/lists/capi-sig.python.org/ Member address: antoine@python.org

On Wed, Mar 17, 2021 at 8:27 PM M.-A. Lemburg mal@egenix.com wrote:

Even with tools like Cython you still need the C API.

Agreed, although a layer of indirection can be helpful when making changes. For example, for HPy we are hoping to write an HPy backend for Cython at some point which generates C code that uses the HPy API instead of the existing C API. That is decidedly non-zero work, but will potentially port a large chunk of the C extension ecosystem in one go (with a fair bit of mopping up required, I'm sure).