On Thu, 18 Mar 2021 at 03:39, Eric Snow <ericsnowcurrently@gmail.com> wrote:

PEP 384 [1] introduced the limited API (AKA Py_LIMITED_API) and stable ABI. The main objective:

"Extension modules and applications embedding Python can work with

different feature releases as long as they restrict themselves to this stable ABI."

This goal is completely reasonable, particularly in 2009. Furthermore, even though there are things we would have done differently, in hindsight, the limited API mostly reaches that goal.

However, is it relevant in the current ecosystem? The burden on extension maintainers to build for each Python release is much lower now.

Is it really? conda-press never really eventuated (as far as I know), and getting any of the other CI multi-build projects up and running across multiple Python versions is still a major time investment. Being able to cut that complexity down to "build per target platform" rather than "build per target platform and supported Python version" is still a significant win.

At the same time, the maintenance burden of the current limited API (especially ABI compatibility) is relatively high, especially for our group of volunteers. It also makes some improvements harder. [2]

Well, no. The maintenance burden in and of itself is zero, since all you have to do to preserve the stable ABI is to not change the C API in the first place.

I'll pay the "makes some improvements harder" aspect, but subinterpreter based isolation is going to need a shared GIL backwards compatibility mode anyway for thread safety reasons, so also requiring it for abi3 compatibility reasons shouldn't be a significant overall increase in complexity.

The idea of defining a per-subinterpreter-locking-compatible stable ABI as abi4 would make sense, but that can wait until after per-subinterpreter-locking is a reality - it doesn't need to be defined up front.

So the question is: Is the stable ABI still worth it? If not, what would be the impact of abandoning it?

You can partially answer the second question for yourself by looking at PyPI's download data: https://packaging.python.org/guides/analyzing-pypi-package-downloads/#useful...

Running [1] gives me 93,512,997 abi3 wheel downloads in the past 30 days. I couldn't get the full wheel download count because I went over my free query quota on BigTable, but https://pypistats.org/top indicates that abi3 wheel download totals are in the same vicinity as the download volumes of the most popular Python packages (i.e. tens of millions per month).

Cheers, Nick

[1] #standardSQL SELECT COUNT(*) AS num_downloads FROM bigquery-public-data.pypi.file_downloads WHERE file.filename like "%-abi3-%" -- Only query the last 30 days of history AND DATE(timestamp) BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY) AND CURRENT_DATE()

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Thu, 18 Mar 2021 at 03:39, Eric Snow <ericsnowcurrently@gmail.com> wrote:

PEP 384 [1] introduced the limited API (AKA Py_LIMITED_API) and stable ABI. The main objective:

"Extension modules and applications embedding Python can work with

different feature releases as long as they restrict themselves to this stable ABI."

This goal is completely reasonable, particularly in 2009. Furthermore, even though there are things we would have done differently, in hindsight, the limited API mostly reaches that goal.

However, is it relevant in the current ecosystem? The burden on extension maintainers to build for each Python release is much lower now.

Is it really? conda-press never really eventuated (as far as I know), and getting any of the other CI multi-build projects up and running across multiple Python versions is still a major time investment. Being able to cut that complexity down to "build per target platform" rather than "build per target platform and supported Python version" is still a significant win.

At the same time, the maintenance burden of the current limited API (especially ABI compatibility) is relatively high, especially for our group of volunteers. It also makes some improvements harder. [2]

Well, no. The maintenance burden in and of itself is zero, since all you have to do to preserve the stable ABI is to not change the C API in the first place.

I'll pay the "makes some improvements harder" aspect, but subinterpreter based isolation is going to need a shared GIL backwards compatibility mode anyway for thread safety reasons, so also requiring it for abi3 compatibility reasons shouldn't be a significant overall increase in complexity.

The idea of defining a per-subinterpreter-locking-compatible stable ABI as abi4 would make sense, but that can wait until after per-subinterpreter-locking is a reality - it doesn't need to be defined up front.

So the question is: Is the stable ABI still worth it? If not, what would be the impact of abandoning it?

You can partially answer the second question for yourself by looking at PyPI's download data: https://packaging.python.org/guides/analyzing-pypi-package-downloads/#useful...

Running [1] gives me 93,512,997 abi3 wheel downloads in the past 30 days. I couldn't get the full wheel download count because I went over my free query quota on BigTable, but https://pypistats.org/top indicates that abi3 wheel download totals are in the same vicinity as the download volumes of the most popular Python packages (i.e. tens of millions per month).

Cheers, Nick

[1] #standardSQL SELECT COUNT(*) AS num_downloads FROM bigquery-public-data.pypi.file_downloads WHERE file.filename like "%-abi3-%" -- Only query the last 30 days of history AND DATE(timestamp) BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY) AND CURRENT_DATE()

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia