Py_FrozenMain and the stable ABI
I tried getting Py_FrozenMain from ctypes.pythonapi, but I get "undefined symbol". This is the only symbol from the Stable ABI that behaves this way. (It was re-added in bpo-42591.) So far, I haven't found what makes the symbol different, but I assume this is deliberate, since it happens on Windows as well (see a test PR). It seems to be included when Python is compiled as a shared library.
Is this something we want as part of the stable ABI? If we do, it would be good to always export it. AFAICS, Py_FrozenMain is only used with freeze.py for a custom Python build with frozen modules included.
$ ./python Python 3.11.0a0 (heads/pep652-ctypes:315d97b64aa, May 5 2021, 11:49:17) [GCC 11.1.1 20210428 (Red Hat 11.1.1-1)] on linux Type "help", "copyright", "credits" or "license" for more information.
import ctypes ctypes.pythonapi.Py_FrozenMain Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/pviktori/dev/cpython/Lib/ctypes/__init__.py", line 387, in __getattr__ func = self.__getitem__(name) File "/home/pviktori/dev/cpython/Lib/ctypes/__init__.py", line 392, in __getitem__ func = self._FuncPtr((name_or_ordinal, self)) AttributeError: ./python: undefined symbol: Py_FrozenMain
Hi Petr,
I reproduced the issue with Python is built without --enable-shared.
But when Python is built with --enable-shared, Py_FrozenMain() is exported as expected!
I created https://bugs.python.org/issue44133 and https://github.com/python/cpython/pull/26130 to fix the issue.
Victor
On Wed, May 5, 2021 at 3:18 PM Petr Viktorin <encukou@gmail.com> wrote:
I tried getting Py_FrozenMain from ctypes.pythonapi, but I get "undefined symbol". This is the only symbol from the Stable ABI that behaves this way. (It was re-added in bpo-42591.) So far, I haven't found what makes the symbol different, but I assume this is deliberate, since it happens on Windows as well (see a test PR). It seems to be included when Python is compiled as a shared library.
Is this something we want as part of the stable ABI? If we do, it would be good to always export it. AFAICS, Py_FrozenMain is only used with freeze.py for a custom Python build with frozen modules included.
$ ./python Python 3.11.0a0 (heads/pep652-ctypes:315d97b64aa, May 5 2021, 11:49:17) [GCC 11.1.1 20210428 (Red Hat 11.1.1-1)] on linux Type "help", "copyright", "credits" or "license" for more information.
import ctypes ctypes.pythonapi.Py_FrozenMain Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/pviktori/dev/cpython/Lib/ctypes/__init__.py", line 387, in __getattr__ func = self.__getitem__(name) File "/home/pviktori/dev/cpython/Lib/ctypes/__init__.py", line 392, in __getitem__ func = self._FuncPtr((name_or_ordinal, self)) AttributeError: ./python: undefined symbol: Py_FrozenMain
capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org https://mail.python.org/mailman3/lists/capi-sig.python.org/ Member address: vstinner@python.org
-- Night gathers, and now my watch begins. It shall not end until my death.
On 14 May 2021, at 19:05, Victor Stinner <vstinner@python.org> wrote:
Hi Petr,
I reproduced the issue with Python is built without --enable-shared.
But when Python is built with --enable-shared, Py_FrozenMain() is exported as expected!
I created https://bugs.python.org/issue44133 <https://bugs.python.org/issue44133> and https://github.com/python/cpython/pull/26130 <https://github.com/python/cpython/pull/26130> to fix the issue.
I guess exporting it for completeness sake is harmless, but I wonder how useful exporting the symbol from the python binary in a static build is.
AFAIK Py_FrozenMain is the main entry point for frozen binaries (linked to libpython). It does not seem to have other uses.
Ronald
—
Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/
On 14. 05. 21 20:07, Ronald Oussoren wrote:
On 14 May 2021, at 19:05, Victor Stinner <vstinner@python.org <mailto:vstinner@python.org>> wrote:
Hi Petr,
I reproduced the issue with Python is built without --enable-shared.
But when Python is built with --enable-shared, Py_FrozenMain() is exported as expected!
I createdhttps://bugs.python.org/issue44133 <https://bugs.python.org/issue44133>and https://github.com/python/cpython/pull/26130 <https://github.com/python/cpython/pull/26130>to fix the issue.
I guess exporting it for completeness sake is harmless, but I wonder how useful exporting the symbol from the python binary in a static build is.
AFAIK Py_FrozenMain is the main entry point for frozen binaries (linked to libpython). It does not seem to have other uses.
That's what I now think as well.
It's OK if Py_FrozenMain is exported, but also OK if it's missing (unless you're compiling a special build with frozen modules, which won't work at all without Py_FrozenMain).
Skipping any tests for it based on something like
hasattr(ctypes.pythonapi, 'Py_FrozenMain') should work fine.
I'll remove it from the Stable ABI list; there's still time for that in 3.10.
On 18.05.2021 11:16, Petr Viktorin wrote:
On 14. 05. 21 20:07, Ronald Oussoren wrote:
On 14 May 2021, at 19:05, Victor Stinner <vstinner@python.org <mailto:vstinner@python.org>> wrote:
Hi Petr,
I reproduced the issue with Python is built without --enable-shared.
But when Python is built with --enable-shared, Py_FrozenMain() is exported as expected!
I createdhttps://bugs.python.org/issue44133 <https://bugs.python.org/issue44133>and https://github.com/python/cpython/pull/26130 <https://github.com/python/cpython/pull/26130>to fix the issue.
I guess exporting it for completeness sake is harmless, but I wonder how useful exporting the symbol from the python binary in a static build is.
AFAIK Py_FrozenMain is the main entry point for frozen binaries (linked to libpython). It does not seem to have other uses.
That's what I now think as well.
That's the case, yes.
It's OK if Py_FrozenMain is exported, but also OK if it's missing (unless you're compiling a special build with frozen modules, which won't work at all without Py_FrozenMain).
Skipping any tests for it based on something like
hasattr(ctypes.pythonapi, 'Py_FrozenMain')should work fine.I'll remove it from the Stable ABI list; there's still time for that in 3.10.
If you are embedding a Python with frozen modules, you will need to call the Py_FrozenMain() API from your application's main() (or other entry point within your application).
Removing the export will break such applications. Removing the API from the stable ABI makes it impossible to use the stable ABI from such applications.
Is that intended ? Perhaps I'm missing something.
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, May 18 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On 18 May 2021, at 12:23, Marc-Andre Lemburg <mal@egenix.com> wrote:
On 18.05.2021 11:16, Petr Viktorin wrote:
On 14. 05. 21 20:07, Ronald Oussoren wrote:
On 14 May 2021, at 19:05, Victor Stinner <vstinner@python.org <mailto:vstinner@python.org>> wrote:
Hi Petr,
I reproduced the issue with Python is built without --enable-shared.
But when Python is built with --enable-shared, Py_FrozenMain() is exported as expected!
I createdhttps://bugs.python.org/issue44133 <https://bugs.python.org/issue44133>and https://github.com/python/cpython/pull/26130 <https://github.com/python/cpython/pull/26130>to fix the issue.
I guess exporting it for completeness sake is harmless, but I wonder how useful exporting the symbol from the python binary in a static build is.
AFAIK Py_FrozenMain is the main entry point for frozen binaries (linked to libpython). It does not seem to have other uses.
That's what I now think as well.
That's the case, yes.
It's OK if Py_FrozenMain is exported, but also OK if it's missing (unless you're compiling a special build with frozen modules, which won't work at all without Py_FrozenMain).
Skipping any tests for it based on something like
hasattr(ctypes.pythonapi, 'Py_FrozenMain')should work fine.I'll remove it from the Stable ABI list; there's still time for that in 3.10.
If you are embedding a Python with frozen modules, you will need to call the Py_FrozenMain() API from your application's main() (or other entry point within your application).
Removing the export will break such applications. Removing the API from the stable ABI makes it impossible to use the stable ABI from such applications.
Is that intended ? Perhaps I'm missing something.
The discussion is about having a Py_FrozenMain() ABI in a statically linked python executable. That executable cannot be used to create a frozen application because, as you say, those need a custom main function that calls Py_FrozenMain() and should therefore link to libpython (either dynamically or statically).
Ronald
-- Marc-Andre Lemburg eGenix.com <http://egenix.com/>
Professional Python Services directly from the Experts (#1, May 18 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ <https://www.egenix.com/> Python Product Development ... https://consulting.egenix.com/ <https://consulting.egenix.com/>
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com <http://egenix.com/> Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ <https://www.egenix.com/company/contact/> https://www.malemburg.com/ <https://www.malemburg.com/>
capi-sig mailing list -- capi-sig@python.org <mailto:capi-sig@python.org> To unsubscribe send an email to capi-sig-leave@python.org <mailto:capi-sig-leave@python.org> https://mail.python.org/mailman3/lists/capi-sig.python.org/ <https://mail.python.org/mailman3/lists/capi-sig.python.org/> Member address: ronaldoussoren@mac.com <mailto:ronaldoussoren@mac.com>
—
Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/
On 18.05.2021 13:02, Ronald Oussoren wrote:
If you are embedding a Python with frozen modules, you will need to call the Py_FrozenMain() API from your application's main() (or other entry point within your application).
Removing the export will break such applications. Removing the API from the stable ABI makes it impossible to use the stable ABI from such applications.
Is that intended ? Perhaps I'm missing something.
The discussion is about having a Py_FrozenMain() ABI in a statically linked python executable. That executable cannot be used to create a frozen application because, as you say, those need a custom main function that calls Py_FrozenMain() and should therefore link to libpython (either dynamically or statically).
I lost you there :-)
My understanding is that the ABI defines the exports of libpythonX.X.a and an executable using frozen modules would have to link against this lib. If you remove Py_FrozenMain() from the lib, this no longer works. If you remove Py_FrozenMain() from the stable ABI, your executable would not be able to reference it (when using the stable ABI).
Whether you statically or dynamically link your executable against the lib does not really matter.
Just to clarify: the executable will not be using the standard Python main() entry point.
Note that freeze.py works in exactly this way. It defines its own main(), which then calls Py_FrozenMain(). The frozen.o then gets linked statically at libpythonX.X.a and the set of frozen modules.
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, May 18 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On 18 May 2021, at 13:26, Marc-Andre Lemburg <mal@egenix.com> wrote:
On 18.05.2021 13:02, Ronald Oussoren wrote:
If you are embedding a Python with frozen modules, you will need to call the Py_FrozenMain() API from your application's main() (or other entry point within your application).
Removing the export will break such applications. Removing the API from the stable ABI makes it impossible to use the stable ABI from such applications.
Is that intended ? Perhaps I'm missing something.
The discussion is about having a Py_FrozenMain() ABI in a statically linked python executable. That executable cannot be used to create a frozen application because, as you say, those need a custom main function that calls Py_FrozenMain() and should therefore link to libpython (either dynamically or statically).
I lost you there :-)
My bad. I’ll try to be more clear ;-)
The issue that started this discussion thread is that a test that verifies that symbols in the stable ABI are present using ctypes fails when the python interpreter itself is statically linked. That’s because nothing in the interpreter itself references Py_FrozenMain() and hence the symbol doesn’t get included in a statically linked python interpreter.
That’s not really a problem because a frozen binary won’t use the statically linked interpreter anyway.
My understanding is that the ABI defines the exports of libpythonX.X.a and an executable using frozen modules would have to link against this lib. If you remove Py_FrozenMain() from the lib, this no longer works. If you remove Py_FrozenMain() from the stable ABI, your executable would not be able to reference it (when using the stable ABI).
Whether you statically or dynamically link your executable against the lib does not really matter.
Just to clarify: the executable will not be using the standard Python main() entry point.
Note that freeze.py works in exactly this way. It defines its own main(), which then calls Py_FrozenMain(). The frozen.o then gets linked statically at libpythonX.X.a and the set of frozen modules.
Agreed. And because of this Py_FrozenMain should not be removed from the stable ABI of libpython.
Ronald
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, May 18 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org https://mail.python.org/mailman3/lists/capi-sig.python.org/ Member address: ronaldoussoren@mac.com
—
Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/
On 18.05.2021 13:36, Ronald Oussoren wrote:
On 18 May 2021, at 13:26, Marc-Andre Lemburg <mal@egenix.com <mailto:mal@egenix.com>> wrote:
On 18.05.2021 13:02, Ronald Oussoren wrote:
If you are embedding a Python with frozen modules, you will need to call the Py_FrozenMain() API from your application's main() (or other entry point within your application).
Removing the export will break such applications. Removing the API from the stable ABI makes it impossible to use the stable ABI from such applications.
Is that intended ? Perhaps I'm missing something.
The discussion is about having a Py_FrozenMain() ABI in a statically linked python executable. That executable cannot be used to create a frozen application because, as you say, those need a custom main function that calls Py_FrozenMain() and should therefore link to libpython (either dynamically or statically).
I lost you there :-)
My bad. I’ll try to be more clear ;-)
The issue that started this discussion thread is that a test that verifies that symbols in the stable ABI are present using ctypes fails when the python interpreter itself is statically linked. That’s because nothing in the interpreter itself references Py_FrozenMain() and hence the symbol doesn’t get included in a statically linked python interpreter.
That’s not really a problem because a frozen binary won’t use the statically linked interpreter anyway.
Ok, that I understand :-) Thanks for the explanation.
Linker optimizations can indeed get in the way. ld has an option --whole-archive which usually guards against such optimizations.
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, May 18 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On 18. 05. 21 13:26, Marc-Andre Lemburg wrote:
On 18.05.2021 13:02, Ronald Oussoren wrote:
If you are embedding a Python with frozen modules, you will need to call the Py_FrozenMain() API from your application's main() (or other entry point within your application).
Removing the export will break such applications. Removing the API from the stable ABI makes it impossible to use the stable ABI from such applications.
Is that intended ? Perhaps I'm missing something.
The discussion is about having a Py_FrozenMain() ABI in a statically linked python executable. That executable cannot be used to create a frozen application because, as you say, those need a custom main function that calls Py_FrozenMain() and should therefore link to libpython (either dynamically or statically).
I lost you there :-)
My understanding is that the ABI defines the exports of libpythonX.X.a and an executable using frozen modules would have to link against this lib. If you remove Py_FrozenMain() from the lib, this no longer works. If you remove Py_FrozenMain() from the stable ABI, your executable would not be able to reference it (when using the stable ABI).
No, libpython is not limited to the stable ABI. python3.dll is, but that's a shared library.
Hm, I just found Py_ENABLE_SHARED. That sounds like the feature flag with which Py_FrozenMain should be defined/exported.
Whether you statically or dynamically link your executable against the lib does not really matter.
Just to clarify: the executable will not be using the standard Python main() entry point.
Note that freeze.py works in exactly this way. It defines its own main(), which then calls Py_FrozenMain(). The frozen.o then gets linked statically at libpythonX.X.a and the set of frozen modules.
On 18.05.2021 13:39, Petr Viktorin wrote:
On 18. 05. 21 13:26, Marc-Andre Lemburg wrote:
On 18.05.2021 13:02, Ronald Oussoren wrote:
If you are embedding a Python with frozen modules, you will need to call the Py_FrozenMain() API from your application's main() (or other entry point within your application).
Removing the export will break such applications. Removing the API from the stable ABI makes it impossible to use the stable ABI from such applications.
Is that intended ? Perhaps I'm missing something.
The discussion is about having a Py_FrozenMain() ABI in a statically linked python executable. That executable cannot be used to create a frozen application because, as you say, those need a custom main function that calls Py_FrozenMain() and should therefore link to libpython (either dynamically or statically).
I lost you there :-)
My understanding is that the ABI defines the exports of libpythonX.X.a and an executable using frozen modules would have to link against this lib. If you remove Py_FrozenMain() from the lib, this no longer works. If you remove Py_FrozenMain() from the stable ABI, your executable would not be able to reference it (when using the stable ABI).
No, libpython is not limited to the stable ABI. python3.dll is, but that's a shared library.
Hm, I just found Py_ENABLE_SHARED. That sounds like the feature flag with which Py_FrozenMain should be defined/exported.
Just to be clear: you can build the Python lib as a shared lib on Unix as well.
In fact, in many cases, you can turn an .a lib into an .so with just some linker magic, e.g.
https://entrenchant.blogspot.com/2009/03/linking-static-libraries-into-share...
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, May 18 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On 18. 05. 21 13:50, Marc-Andre Lemburg wrote:
On 18.05.2021 13:39, Petr Viktorin wrote:
On 18. 05. 21 13:26, Marc-Andre Lemburg wrote:
On 18.05.2021 13:02, Ronald Oussoren wrote:
If you are embedding a Python with frozen modules, you will need to call the Py_FrozenMain() API from your application's main() (or other entry point within your application).
Removing the export will break such applications. Removing the API from the stable ABI makes it impossible to use the stable ABI from such applications.
Is that intended ? Perhaps I'm missing something.
The discussion is about having a Py_FrozenMain() ABI in a statically linked python executable. That executable cannot be used to create a frozen application because, as you say, those need a custom main function that calls Py_FrozenMain() and should therefore link to libpython (either dynamically or statically).
I lost you there :-)
My understanding is that the ABI defines the exports of libpythonX.X.a and an executable using frozen modules would have to link against this lib. If you remove Py_FrozenMain() from the lib, this no longer works. If you remove Py_FrozenMain() from the stable ABI, your executable would not be able to reference it (when using the stable ABI).
No, libpython is not limited to the stable ABI. python3.dll is, but that's a shared library.
Hm, I just found Py_ENABLE_SHARED. That sounds like the feature flag with which Py_FrozenMain should be defined/exported.
Just to be clear: you can build the Python lib as a shared lib on Unix as well.
Sure, that's how I use Python :) But on Unix, that shared library exports all symbols, not just those in the Stable ABI.
In fact, in many cases, you can turn an .a lib into an .so with just some linker magic, e.g.
https://entrenchant.blogspot.com/2009/03/linking-static-libraries-into-share...
Configuring with --enable-shared sounds much safer.
On 18.05.2021 13:57, Petr Viktorin wrote:
On 18. 05. 21 13:50, Marc-Andre Lemburg wrote:
On 18.05.2021 13:39, Petr Viktorin wrote:
On 18. 05. 21 13:26, Marc-Andre Lemburg wrote:
On 18.05.2021 13:02, Ronald Oussoren wrote:
If you are embedding a Python with frozen modules, you will need to call the Py_FrozenMain() API from your application's main() (or other entry point within your application).
Removing the export will break such applications. Removing the API from the stable ABI makes it impossible to use the stable ABI from such applications.
Is that intended ? Perhaps I'm missing something.
The discussion is about having a Py_FrozenMain() ABI in a statically linked python executable. That executable cannot be used to create a frozen application because, as you say, those need a custom main function that calls Py_FrozenMain() and should therefore link to libpython (either dynamically or statically).
I lost you there :-)
My understanding is that the ABI defines the exports of libpythonX.X.a and an executable using frozen modules would have to link against this lib. If you remove Py_FrozenMain() from the lib, this no longer works. If you remove Py_FrozenMain() from the stable ABI, your executable would not be able to reference it (when using the stable ABI).
No, libpython is not limited to the stable ABI. python3.dll is, but that's a shared library.
Hm, I just found Py_ENABLE_SHARED. That sounds like the feature flag with which Py_FrozenMain should be defined/exported.
Just to be clear: you can build the Python lib as a shared lib on Unix as well.
Sure, that's how I use Python :) But on Unix, that shared library exports all symbols, not just those in the Stable ABI.
Right, but if you want to use the stable ABI, the compiler would not see the needed Py_FrozenMain() definition to link against -- if you remove it from the stable ABI.
So overall, it's probably not such a good idea to remove that particular API from the stable ABI list.
In fact, in many cases, you can turn an .a lib into an .so with just some linker magic, e.g.
https://entrenchant.blogspot.com/2009/03/linking-static-libraries-into-share...
Configuring with --enable-shared sounds much safer.
Of course, it's always better to build the lib as a shared lib :-)
Trying to turn a static lib into a shared one can result in issues with code not being relocatable, but it often works. IIRC, some Unixes can even dynamically link against .a libs directly (AIX I think was one of those, but I could be wrong).
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, May 18 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On 18. 05. 21 14:20, Marc-Andre Lemburg wrote:
On 18.05.2021 13:57, Petr Viktorin wrote:
On 18. 05. 21 13:50, Marc-Andre Lemburg wrote:
On 18.05.2021 13:39, Petr Viktorin wrote:
On 18. 05. 21 13:26, Marc-Andre Lemburg wrote:
On 18.05.2021 13:02, Ronald Oussoren wrote:
> If you are embedding a Python with frozen modules, you will need to > call the Py_FrozenMain() API from your application's main() (or other > entry point within your application). > > Removing the export will break such applications. Removing the API > from the stable ABI makes it impossible to use the stable ABI from > such applications. > > Is that intended ? Perhaps I'm missing something.
The discussion is about having a Py_FrozenMain() ABI in a statically linked python executable. That executable cannot be used to create a frozen application because, as you say, those need a custom main function that calls Py_FrozenMain() and should therefore link to libpython (either dynamically or statically).
I lost you there :-)
My understanding is that the ABI defines the exports of libpythonX.X.a and an executable using frozen modules would have to link against this lib. If you remove Py_FrozenMain() from the lib, this no longer works. If you remove Py_FrozenMain() from the stable ABI, your executable would not be able to reference it (when using the stable ABI).
No, libpython is not limited to the stable ABI. python3.dll is, but that's a shared library.
Hm, I just found Py_ENABLE_SHARED. That sounds like the feature flag with which Py_FrozenMain should be defined/exported.
Just to be clear: you can build the Python lib as a shared lib on Unix as well.
Sure, that's how I use Python :) But on Unix, that shared library exports all symbols, not just those in the Stable ABI.
Right, but if you want to use the stable ABI, the compiler would not see the needed Py_FrozenMain() definition to link against -- if you remove it from the stable ABI.
So overall, it's probably not such a good idea to remove that particular API from the stable ABI list.
Yeah, the more I dig, the more I'm convinced that Py_FrozenMain should be on the list.
That means it should always be available, though -- even in static builds. And the option to include it is compiler-specific:
- gcc needs --whole-archive
- clang seems to need -all_load
- not sure about MSVC
On 18.05.2021 15:08, Petr Viktorin wrote:
So overall, it's probably not such a good idea to remove that particular API from the stable ABI list.
Yeah, the more I dig, the more I'm convinced that Py_FrozenMain should be on the list.
That means it should always be available, though -- even in static builds. And the option to include it is compiler-specific:
- gcc needs --whole-archive
- clang seems to need -all_load
- not sure about MSVC
You can also try a trick: load the pointer to the API into a global variable (struct or array) together with at least one other API which does get used.
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, May 18 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On 18 May 2021, at 15:08, Petr Viktorin <encukou@gmail.com> wrote:
On 18. 05. 21 14:20, Marc-Andre Lemburg wrote:
On 18.05.2021 13:57, Petr Viktorin wrote:
On 18. 05. 21 13:50, Marc-Andre Lemburg wrote:
On 18.05.2021 13:39, Petr Viktorin wrote:
On 18. 05. 21 13:26, Marc-Andre Lemburg wrote:
On 18.05.2021 13:02, Ronald Oussoren wrote: >> If you are embedding a Python with frozen modules, you will need to >> call the Py_FrozenMain() API from your application's main() (or other >> entry point within your application). >> >> Removing the export will break such applications. Removing the API >> from the stable ABI makes it impossible to use the stable ABI from >> such applications. >> >> Is that intended ? Perhaps I'm missing something. > > The discussion is about having a Py_FrozenMain() ABI in a statically linked > python executable. That executable cannot be used to create a frozen > application > because, as you say, those need a custom main function that calls > Py_FrozenMain() and should therefore link to libpython (either dynamically or > statically).
I lost you there :-)
My understanding is that the ABI defines the exports of libpythonX.X.a and an executable using frozen modules would have to link against this lib. If you remove Py_FrozenMain() from the lib, this no longer works. If you remove Py_FrozenMain() from the stable ABI, your executable would not be able to reference it (when using the stable ABI).
No, libpython is not limited to the stable ABI. python3.dll is, but that's a shared library.
Hm, I just found Py_ENABLE_SHARED. That sounds like the feature flag with which Py_FrozenMain should be defined/exported.
Just to be clear: you can build the Python lib as a shared lib on Unix as well.
Sure, that's how I use Python :) But on Unix, that shared library exports all symbols, not just those in the Stable ABI.
Right, but if you want to use the stable ABI, the compiler would not see the needed Py_FrozenMain() definition to link against -- if you remove it from the stable ABI. So overall, it's probably not such a good idea to remove that particular API from the stable ABI list.
Yeah, the more I dig, the more I'm convinced that Py_FrozenMain should be on the list.
That means it should always be available, though -- even in static builds.
I don’t understand why this is so. Py_FrozenMain must only be present in the dynamic library, including the stable ABI on Windows (because AFAIK it already is). In a staticly linked python executable the symbol is not necessary because it will (and can) never be used.
And the option to include it is compiler-specific:
- gcc needs --whole-archive
- clang seems to need -all_load
- not sure about MSVC
That works, but a tool-independent way to force inclusion is to use the symbol from a source file that is already included. Or alternatively explicitly link the object file containing Py_FrozenMain() into the executable (for static builds).
Ronald —
Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/
On Wed, 19 May 2021, 12:00 am Ronald Oussoren via capi-sig, < capi-sig@python.org> wrote:
On 18 May 2021, at 15:08, Petr Viktorin <encukou@gmail.com> wrote:
On 18. 05. 21 14:20, Marc-Andre Lemburg wrote:
Right, but if you want to use the stable ABI, the compiler would not see the needed Py_FrozenMain() definition to link against -- if you remove it from the stable ABI. So overall, it's probably not such a good idea to remove that particular API from the stable ABI list.
Yeah, the more I dig, the more I'm convinced that Py_FrozenMain should be on the list.
That means it should always be available, though -- even in static builds.
I don’t understand why this is so. Py_FrozenMain must only be present in the dynamic library, including the stable ABI on Windows (because AFAIK it already is). In a staticly linked python executable the symbol is not necessary because it will (and can) never be used.
It's an edge case in the stable ABI guarantees - a module that uses the stable ABI should work from a statically linked Python, even if it does something odd like report the address of a symbol that isn't useful in a statically linked Python.
Cheers, Nick.
On 18. 05. 21 16:14, Nick Coghlan wrote:
On Wed, 19 May 2021, 12:00 am Ronald Oussoren via capi-sig, <capi-sig@python.org <mailto:capi-sig@python.org>> wrote:
> On 18 May 2021, at 15:08, Petr Viktorin <encukou@gmail.com <mailto:encukou@gmail.com>> wrote: > > > > On 18. 05. 21 14:20, Marc-Andre Lemburg wrote: >> Right, but if you want to use the stable ABI, the compiler would not see >> the needed Py_FrozenMain() definition to link against -- if you remove it >> from the stable ABI. >> So overall, it's probably not such a good idea to remove that >> particular API from the stable ABI list. > > Yeah, the more I dig, the more I'm convinced that Py_FrozenMain should be on the list. > > That means it should always be available, though -- even in static builds. I don’t understand why this is so. Py_FrozenMain must only be present in the dynamic library, including the stable ABI on Windows (because AFAIK it already is). In a staticly linked python executable the symbol is not necessary because it will (and can) never be used.It's an edge case in the stable ABI guarantees - a module that uses the stable ABI should work from a statically linked Python, even if it does something odd like report the address of a symbol that isn't useful in a statically linked Python.
Yes.
But -- this might not be possible on Windows. To use Py_FrozenMain on Windows, you need to provide definitions for the functions PyWinFreeze_ExeInit, PyWinFreeze_ExeTerm and PyInitFrozenExtensions.
Argh, researching this is turning up so many dead ends. Back to the beginning:
Py_FrozenMain was only added to python3dll.c in Python 3.10, in GH-23730. Adding it actually looks like an afterthought: IMO, adding an undocumented function to the stable ABI should be discussed a bit more, especially if it's as weird as this one. Victor, do you have any details here?
No one is using Py_FrozenMain with the stable ABI on Windows (unless they're testing 3.10). I doubt anyone needs it *in the Stable ABI* on other platforms either.
To reply to Marc-Andre's earlier question:
Removing the export will break such applications. Removing the API from the stable ABI makes it impossible to use the stable ABI from such applications.
Is that intended ? Perhaps I'm missing something.
Given that it was added in 3.10, removing it won't be a regression.
I'm definitely not proposing to remove Py_FrozenMain from the default, minor-version-specific ABI.
On 18.05.2021 17:07, Petr Viktorin wrote:
On 18. 05. 21 16:14, Nick Coghlan wrote:
On Wed, 19 May 2021, 12:00 am Ronald Oussoren via capi-sig, <capi-sig@python.org <mailto:capi-sig@python.org>> wrote:
> On 18 May 2021, at 15:08, Petr Viktorin <encukou@gmail.com <mailto:encukou@gmail.com>> wrote: > > > > On 18. 05. 21 14:20, Marc-Andre Lemburg wrote:
>> Right, but if you want to use the stable ABI, the compiler would not see >> the needed Py_FrozenMain() definition to link against -- if you remove it >> from the stable ABI. >> So overall, it's probably not such a good idea to remove that >> particular API from the stable ABI list. > > Yeah, the more I dig, the more I'm convinced that Py_FrozenMain should be on the list. > > That means it should always be available, though -- even in static builds.
I don’t understand why this is so. Py_FrozenMain must only be present in the dynamic library, including the stable ABI on Windows (because AFAIK it already is). In a staticly linked python executable the symbol is not necessary because it will (and can) never be used.
It's an edge case in the stable ABI guarantees - a module that uses the stable ABI should work from a statically linked Python, even if it does something odd like report the address of a symbol that isn't useful in a statically linked Python.
Yes.
But -- this might not be possible on Windows. To use Py_FrozenMain on Windows, you need to provide definitions for the functions PyWinFreeze_ExeInit, PyWinFreeze_ExeTerm and PyInitFrozenExtensions.
Argh, researching this is turning up so many dead ends. Back to the beginning:
Py_FrozenMain was only added to python3dll.c in Python 3.10, in GH-23730. Adding it actually looks like an afterthought: IMO, adding an undocumented function to the stable ABI should be discussed a bit more, especially if it's as weird as this one. Victor, do you have any details here?
No one is using Py_FrozenMain with the stable ABI on Windows (unless they're testing 3.10). I doubt anyone needs it *in the Stable ABI* on other platforms either.
To reply to Marc-Andre's earlier question:
Removing the export will break such applications. Removing the API from the stable ABI makes it impossible to use the stable ABI from such applications.
Is that intended ? Perhaps I'm missing something.
Given that it was added in 3.10, removing it won't be a regression.
I'm definitely not proposing to remove Py_FrozenMain from the default, minor-version-specific ABI.
I don't quite follow that logic. Not many people are using the stable ABI, that's clear, but that should be seen as a chance, not as proof that APIs are not used and thus not necessary.
If you're saying: "frozen applications should not use the stable ABI" that's fine, but it should be documented as such.
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, May 18 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On 18. 05. 21 17:17, Marc-Andre Lemburg wrote:
On 18.05.2021 17:07, Petr Viktorin wrote:
On 18. 05. 21 16:14, Nick Coghlan wrote:
On Wed, 19 May 2021, 12:00 am Ronald Oussoren via capi-sig, <capi-sig@python.org <mailto:capi-sig@python.org>> wrote:
> On 18 May 2021, at 15:08, Petr Viktorin <encukou@gmail.com <mailto:encukou@gmail.com>> wrote: > > > > On 18. 05. 21 14:20, Marc-Andre Lemburg wrote:
>> Right, but if you want to use the stable ABI, the compiler would not see >> the needed Py_FrozenMain() definition to link against -- if you remove it >> from the stable ABI. >> So overall, it's probably not such a good idea to remove that >> particular API from the stable ABI list. > > Yeah, the more I dig, the more I'm convinced that Py_FrozenMain should be on the list. > > That means it should always be available, though -- even in static builds.
I don’t understand why this is so. Py_FrozenMain must only be present in the dynamic library, including the stable ABI on Windows (because AFAIK it already is). In a staticly linked python executable the symbol is not necessary because it will (and can) never be used.
It's an edge case in the stable ABI guarantees - a module that uses the stable ABI should work from a statically linked Python, even if it does something odd like report the address of a symbol that isn't useful in a statically linked Python.
Yes.
But -- this might not be possible on Windows. To use Py_FrozenMain on Windows, you need to provide definitions for the functions PyWinFreeze_ExeInit, PyWinFreeze_ExeTerm and PyInitFrozenExtensions.
Argh, researching this is turning up so many dead ends. Back to the beginning:
Py_FrozenMain was only added to python3dll.c in Python 3.10, in GH-23730. Adding it actually looks like an afterthought: IMO, adding an undocumented function to the stable ABI should be discussed a bit more, especially if it's as weird as this one. Victor, do you have any details here?
No one is using Py_FrozenMain with the stable ABI on Windows (unless they're testing 3.10). I doubt anyone needs it *in the Stable ABI* on other platforms either.
To reply to Marc-Andre's earlier question:
Removing the export will break such applications. Removing the API from the stable ABI makes it impossible to use the stable ABI from such applications.
Is that intended ? Perhaps I'm missing something.
Given that it was added in 3.10, removing it won't be a regression.
I'm definitely not proposing to remove Py_FrozenMain from the default, minor-version-specific ABI.
I don't quite follow that logic. Not many people are using the stable ABI, that's clear, but that should be seen as a chance, not as proof that APIs are not used and thus not necessary.
If you're saying: "frozen applications should not use the stable ABI" that's fine, but it should be documented as such.
If you need something that's not part of the stable ABI yet, then you can't use it. That goes for any part of the stable ABI. It would certainly be good to make the Stable ABI complete, but that'll take time. In 3.10 are we finally adding a rigid definition of what the stable ABI even is :)
The missing piece can be added later, but if we add it now, removing it would be much harder.
On 18.05.2021 17:27, Petr Viktorin wrote:
I'm definitely not proposing to remove Py_FrozenMain from the default, minor-version-specific ABI.
I don't quite follow that logic. Not many people are using the stable ABI, that's clear, but that should be seen as a chance, not as proof that APIs are not used and thus not necessary.
If you're saying: "frozen applications should not use the stable ABI" that's fine, but it should be documented as such.
If you need something that's not part of the stable ABI yet, then you can't use it. That goes for any part of the stable ABI.
Exactly, which is why it should include a good set of the APIs to make it useful.
It would certainly be good to make the Stable ABI complete, but that'll take time. In 3.10 are we finally adding a rigid definition of what the stable ABI even is :)
The missing piece can be added later, but if we add it now, removing it would be much harder.
Didn't we just provide proof that this particular API is needed ? Why add it later ?
I never was a great fan of the stable ABI, but now do see a point since we're moving towards a new HPy based API.
The stable ABI could provide a path forward to have a basic set of APIs defined which can be made to work with HPy initially and then later extended further.
However, for this to work out, we'd have to have a better definition of what is supposed to go into the stable ABI. The PEPs don't provide much guidance on this and this discussion also doesn't clarify this.
Yes, the Py_FrozenMain() API is not used by many applications, but it is essential for a subset of applications. Is that good enough ?
Should the stable ABI be driven by "I need this to work" or rather "anything which doesn't expose internals is fine" ?
To give you an example:
When I wrote mxODBC, the database interface for Python in 1997, I found that Python was missing native object types for handling date/time values.
So I created mxDateTime, which exposes a C API for creating and working with date/time values. This became a standard for many years, until Python grew its own datetime module -- also exposing a C API.
Now, the stable ABI does not include this datetime C API, so for my purposes, it's not usable.
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, May 18 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On Wed, 19 May 2021, 2:23 am Marc-Andre Lemburg, <mal@egenix.com> wrote:
On 18.05.2021 17:27, Petr Viktorin wrote:
I'm definitely not proposing to remove Py_FrozenMain from the default, minor-version-specific ABI.
I don't quite follow that logic. Not many people are using the stable ABI, that's clear, but that should be seen as a chance, not as proof that APIs are not used and thus not necessary.
If you're saying: "frozen applications should not use the stable ABI" that's fine, but it should be documented as such.
If you need something that's not part of the stable ABI yet, then you can't use it. That goes for any part of the stable ABI.
Exactly, which is why it should include a good set of the APIs to make it useful.
Applications that embed or statically link Python (including frozen applications) gain nothing from using the stable ABI. Only extension modules benefit, since they can be built once and imported from multiple versions.
Py_FrozenMain isn't useful to modules (calling it won't do anything sensible, so about all they can do with it is take its address), so it doesn't really make sense to have it in the stable ABI.
My assumption had been that it was already in the stable ABI by mistake in 3.9 and earlier, so modules might already have been referencing it (however pointless that would be), so we needed to figure out a way to include it that let the new stable ABI checking test case pass on a statically linked Python build.
Petr now indicates that isn't the case - instead, it was added by mistake in Python 3.10. Given that, the simplest answer is to revert that change and continue leaving it out of the stable ABI.
If someone has an actual use case for it being in the stable ABI, then it could be reconsidered for addition, but so far no such use case has been presented.
Cheers, Nick.
On 19.05.2021 10:00, Nick Coghlan wrote:
On Wed, 19 May 2021, 2:23 am Marc-Andre Lemburg, <mal@egenix.com <mailto:mal@egenix.com>> wrote:
On 18.05.2021 17:27, Petr Viktorin wrote: >>> I'm definitely not proposing to remove Py_FrozenMain from the default, >>> minor-version-specific ABI. >> >> I don't quite follow that logic. Not many people are using the >> stable ABI, that's clear, but that should be seen as a chance, >> not as proof that APIs are not used and thus not necessary. >> >> If you're saying: "frozen applications should not use the stable >> ABI" that's fine, but it should be documented as such. > > If you need something that's not part of the stable ABI yet, then you can't use > it. That goes for any part of the stable ABI. Exactly, which is why it should include a good set of the APIs to make it useful.Applications that embed or statically link Python (including frozen applications) gain nothing from using the stable ABI. Only extension modules benefit, since they can be built once and imported from multiple versions.
That's one perspective, but not necessarily one which has much benefit these days, since it fairly easy to recompile modules against new Python versions.
The other, and I believe, much more important perspective is stability in the API. APIs not included in the stable ABI do not get the benefit of being stable for the rest of the major version editions of Python, so using them can potentially incur additional maintenance overhead and breakage going forward.
As such, the stable ABI is important for both extension modules and applications.
Py_FrozenMain() has been around for more then 2 decades by now and unchanged, so it's probably one of the more stable APIs we have :-)
Of course, if you want to frame the stable ABI in a different way that's reasonable as well.
What I'd like to learn is what the criteria of that framing would look like, because so far, the stable ABI is a list of APIs that Martin von Löwis started a while back, without a clear indication of why APIs should be in the stable ABI or why not.
As it stands right now, it's a fairly high level API definition, which goes into depth only in particular areas, but leaves out important other ones, with seemingly no justification.
E.g. why is the low level capsule API included, but not the capsule definitions of the datetime module ?
IMO, a lot more APIs would have to be added to make the stable API useful for more than the handful of extensions / applications using it at the moment.
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, May 19 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On 19. 05. 21 10:41, Marc-Andre Lemburg wrote:
On 19.05.2021 10:00, Nick Coghlan wrote:
On Wed, 19 May 2021, 2:23 am Marc-Andre Lemburg, <mal@egenix.com <mailto:mal@egenix.com>> wrote:
On 18.05.2021 17:27, Petr Viktorin wrote: >>> I'm definitely not proposing to remove Py_FrozenMain from the default, >>> minor-version-specific ABI. >> >> I don't quite follow that logic. Not many people are using the >> stable ABI, that's clear, but that should be seen as a chance, >> not as proof that APIs are not used and thus not necessary. >> >> If you're saying: "frozen applications should not use the stable >> ABI" that's fine, but it should be documented as such. > > If you need something that's not part of the stable ABI yet, then you can't use > it. That goes for any part of the stable ABI. Exactly, which is why it should include a good set of the APIs to make it useful.Applications that embed or statically link Python (including frozen applications) gain nothing from using the stable ABI. Only extension modules benefit, since they can be built once and imported from multiple versions.
That's one perspective, but not necessarily one which has much benefit these days, since it fairly easy to recompile modules against new Python versions.
The other, and I believe, much more important perspective is stability in the API. APIs not included in the stable ABI do not get the benefit of being stable for the rest of the major version editions of Python, so using them can potentially incur additional maintenance overhead and breakage going forward.
Python's normal C API is covered by Python's general backward compatibility policy (PEP 387): they can be changed/removed after two releases of deprecations, or an exception from the Steering Council.
(Of course, this being API and not ABI, you need to recompile for every version. Things can get replaced by macros, structs can grow, etc.)
And I think that's pretty reasonable. Of course, to change anything there still needs to be some need to change it -- if Py_FrozenMain is OK the way it is, it should be left alone.
As such, the stable ABI is important for both extension modules and applications.
Py_FrozenMain() has been around for more then 2 decades by now and unchanged, so it's probably one of the more stable APIs we have :-)
Of course, if you want to frame the stable ABI in a different way that's reasonable as well.
What I'd like to learn is what the criteria of that framing would look like, because so far, the stable ABI is a list of APIs that Martin von Löwis started a while back, without a clear indication of why APIs should be in the stable ABI or why not.
Ultimately, the question is: can we support this API for another few decades?
But yes, what we have now is just a somewhat arbitrary list. That's the base we can now build on.
As it stands right now, it's a fairly high level API definition, which goes into depth only in particular areas, but leaves out important other ones, with seemingly no justification.
E.g. why is the low level capsule API included, but not the capsule definitions of the datetime module ?
IMO, a lot more APIs would have to be added to make the stable API useful for more than the handful of extensions / applications using it at the moment.
I agree.
On 18. 05. 21 18:23, Marc-Andre Lemburg wrote:
On 18.05.2021 17:27, Petr Viktorin wrote:
I'm definitely not proposing to remove Py_FrozenMain from the default, minor-version-specific ABI.
I don't quite follow that logic. Not many people are using the stable ABI, that's clear, but that should be seen as a chance, not as proof that APIs are not used and thus not necessary.
If you're saying: "frozen applications should not use the stable ABI" that's fine, but it should be documented as such.
If you need something that's not part of the stable ABI yet, then you can't use it. That goes for any part of the stable ABI.
Exactly, which is why it should include a good set of the APIs to make it useful.
It would certainly be good to make the Stable ABI complete, but that'll take time. In 3.10 are we finally adding a rigid definition of what the stable ABI even is :)
The missing piece can be added later, but if we add it now, removing it would be much harder.
Didn't we just provide proof that this particular API is needed ? Why add it later ?
I never was a great fan of the stable ABI, but now do see a point since we're moving towards a new HPy based API.
The stable ABI could provide a path forward to have a basic set of APIs defined which can be made to work with HPy initially and then later extended further.
However, for this to work out, we'd have to have a better definition of what is supposed to go into the stable ABI. The PEPs don't provide much guidance on this and this discussion also doesn't clarify this.
Yes, the Py_FrozenMain() API is not used by many applications, but it is essential for a subset of applications. Is that good enough ?
Py_FrozenMain:
- is not documented
- was untested until yesterday (thanks to Victor!)
- is not present in all builds of Python
I'm all for fixing that and adding it to the stable ABI, but I think it's too late for Python 3.10. Let's add it in 3.11?
Should the stable ABI be driven by "I need this to work" or rather "anything which doesn't expose internals is fine" ?
"Anything which doesn't expose internals is fine" is a better goal; more comprehensive guidelines are here: https://devguide.python.org/c-api/#guidelines-for-changing-the-limited-api
To give you an example:
When I wrote mxODBC, the database interface for Python in 1997, I found that Python was missing native object types for handling date/time values.
So I created mxDateTime, which exposes a C API for creating and working with date/time values. This became a standard for many years, until Python grew its own datetime module -- also exposing a C API.
Now, the stable ABI does not include this datetime C API, so for my purposes, it's not usable.
Well, that nicely illustrates the general state of the stable ABI now. (Though I don't yet know much about how a capsule-based API will interact with the stable ABI.)
But for in 3.10 I'm want to define what the stable ABI *is*, so it can be meaningfully improved later.
On 19.05.2021 10:39, Petr Viktorin wrote:
To give you an example:
When I wrote mxODBC, the database interface for Python in 1997, I found that Python was missing native object types for handling date/time values.
So I created mxDateTime, which exposes a C API for creating and working with date/time values. This became a standard for many years, until Python grew its own datetime module -- also exposing a C API.
Now, the stable ABI does not include this datetime C API, so for my purposes, it's not usable.
Well, that nicely illustrates the general state of the stable ABI now. (Though I don't yet know much about how a capsule-based API will interact with the stable ABI.)
But for in 3.10 I'm want to define what the stable ABI *is*, so it can be meaningfully improved later. Fair enough.
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, May 19 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On 18. 05. 21 12:23, Marc-Andre Lemburg wrote:
On 18.05.2021 11:16, Petr Viktorin wrote:
On 14. 05. 21 20:07, Ronald Oussoren wrote:
On 14 May 2021, at 19:05, Victor Stinner <vstinner@python.org <mailto:vstinner@python.org>> wrote:
Hi Petr,
I reproduced the issue with Python is built without --enable-shared.
But when Python is built with --enable-shared, Py_FrozenMain() is exported as expected!
I createdhttps://bugs.python.org/issue44133 <https://bugs.python.org/issue44133>and https://github.com/python/cpython/pull/26130 <https://github.com/python/cpython/pull/26130>to fix the issue.
I guess exporting it for completeness sake is harmless, but I wonder how useful exporting the symbol from the python binary in a static build is.
AFAIK Py_FrozenMain is the main entry point for frozen binaries (linked to libpython). It does not seem to have other uses.
That's what I now think as well.
That's the case, yes.
It's OK if Py_FrozenMain is exported, but also OK if it's missing (unless you're compiling a special build with frozen modules, which won't work at all without Py_FrozenMain).
Skipping any tests for it based on something like
hasattr(ctypes.pythonapi, 'Py_FrozenMain')should work fine.I'll remove it from the Stable ABI list; there's still time for that in 3.10.
If you are embedding a Python with frozen modules, you will need to call the Py_FrozenMain() API from your application's main() (or other entry point within your application).
Removing the export will break such applications. Removing the API from the stable ABI makes it impossible to use the stable ABI from such applications.
Is that intended ? Perhaps I'm missing something.
Is there a supported way to make such binary other than Tools/freeze/freeze.py? The (lack of) docs suggests this is an implementation detail for the freeze script.
Tools/freeze does not se the Stable ABI.
On 18.05.2021 13:31, Petr Viktorin wrote:
On 18. 05. 21 12:23, Marc-Andre Lemburg wrote:
On 18.05.2021 11:16, Petr Viktorin wrote:
On 14. 05. 21 20:07, Ronald Oussoren wrote:
On 14 May 2021, at 19:05, Victor Stinner <vstinner@python.org <mailto:vstinner@python.org>> wrote:
Hi Petr,
I reproduced the issue with Python is built without --enable-shared.
But when Python is built with --enable-shared, Py_FrozenMain() is exported as expected!
I createdhttps://bugs.python.org/issue44133 <https://bugs.python.org/issue44133>and https://github.com/python/cpython/pull/26130 <https://github.com/python/cpython/pull/26130>to fix the issue.
I guess exporting it for completeness sake is harmless, but I wonder how useful exporting the symbol from the python binary in a static build is.
AFAIK Py_FrozenMain is the main entry point for frozen binaries (linked to libpython). It does not seem to have other uses.
That's what I now think as well.
That's the case, yes.
It's OK if Py_FrozenMain is exported, but also OK if it's missing (unless you're compiling a special build with frozen modules, which won't work at all without Py_FrozenMain).
Skipping any tests for it based on something like
hasattr(ctypes.pythonapi, 'Py_FrozenMain')should work fine.I'll remove it from the Stable ABI list; there's still time for that in 3.10.
If you are embedding a Python with frozen modules, you will need to call the Py_FrozenMain() API from your application's main() (or other entry point within your application).
Removing the export will break such applications. Removing the API from the stable ABI makes it impossible to use the stable ABI from such applications.
Is that intended ? Perhaps I'm missing something.
Is there a supported way to make such binary other than Tools/freeze/freeze.py? The (lack of) docs suggests this is an implementation detail for the freeze script.
Tools/freeze does not use the Stable ABI.
I don't know what you mean with "supported". It is certainly possible to use the same mechanism freeze uses, to build an embedded version of Python which you add to an application.
The code is really minimal:
int main(int argc, char **argv) { extern int Py_FrozenMain(int, char **);
PyImport_FrozenModules = _PyImport_FrozenModules;
return Py_FrozenMain(argc, argv);}
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, May 18 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On Fri, May 14, 2021 at 7:05 PM Victor Stinner <vstinner@python.org> wrote:
Hi Petr,
I reproduced the issue with Python is built without --enable-shared.
But when Python is built with --enable-shared, Py_FrozenMain() is exported as expected!
I created https://bugs.python.org/issue44133 and https://github.com/python/cpython/pull/26130 to fix the issue.
This issue is not specific to the "Py_FrozenMain" symbol. *Any* symbol can miss depending on how Python was built and where the function lives. If you move an exported function to a new C function and the function is not used directly by Python, the function is no longer exported.
This bug is a major issue, because C extensions are no longer linked to libpython since Python 3.8. Python must export all symbols that it pretends to export, especially if it built without libpython dynamic library (libpythonXY.so).
Victor
On 19 May 2021, at 12:15, Victor Stinner <vstinner@python.org> wrote:
On Fri, May 14, 2021 at 7:05 PM Victor Stinner <vstinner@python.org> wrote:
Hi Petr,
I reproduced the issue with Python is built without --enable-shared.
But when Python is built with --enable-shared, Py_FrozenMain() is exported as expected!
I created https://bugs.python.org/issue44133 and https://github.com/python/cpython/pull/26130 to fix the issue.
This issue is not specific to the "Py_FrozenMain" symbol. *Any* symbol can miss depending on how Python was built and where the function lives. If you move an exported function to a new C function and the function is not used directly by Python, the function is no longer exported.
Do you mean “new C file” instead of “new C function”? Because all C compilers I’ve used so far default to including whose object files when a program uses at least one symbol from that object file. That’s why Py_FrozenMain() is currently the only missing symbol, the rest of the core has sufficiently large source code files to ensure all of them will be included. The issue could crop up for other symbols if, for example, functions that are included only for backward compatibility were to be moved to separate files.
That said, newer version of at least the GCC stack strip unused symbols when using the right compiler invocation (“-ffunction-sections” and related options).
This bug is a major issue, because C extensions are no longer linked to libpython since Python 3.8. Python must export all symbols that it pretends to export, especially if it built without libpython dynamic library (libpythonXY.so).
It has been a problem longer than that, if I recall correctly framework builds on macOS have never linked extensions to libpython. That said, as I wrote above this is currently not that large of an issue due to the way the code is structured and the way linkers work.
Ronald
—
Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/
On 19.05.2021 12:45, Ronald Oussoren via capi-sig wrote:
This bug is a major issue, because C extensions are no longer linked to libpython since Python 3.8. Python must export all symbols that it pretends to export, especially if it built without libpython dynamic library (libpythonXY.so).
It has been a problem longer than that, if I recall correctly framework builds on macOS have never linked extensions to libpython. That said, as I wrote above this is currently not that large of an issue due to the way the code is structured and the way linkers work.
I believe it's the same on Linux.
I checked a couple packages against Python 3.7 on Linux: none of them links against libpythonXX.so. They all link against the symbols exposed by the Python executable.
However, the system built python executable does link against libpythonXX.so. If you build Python using standard settings, libpythonXX.so is not used or even created.
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, May 19 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On Wed, May 19, 2021 at 3:35 PM Marc-Andre Lemburg <mal@egenix.com> wrote:
On 19.05.2021 12:45, Ronald Oussoren via capi-sig wrote:
This bug is a major issue, because C extensions are no longer linked to libpython since Python 3.8. Python must export all symbols that it pretends to export, especially if it built without libpython dynamic library (libpythonXY.so).
It has been a problem longer than that, if I recall correctly framework builds on macOS have never linked extensions to libpython. That said, as I wrote above this is currently not that large of an issue due to the way the code is structured and the way linkers work.
I believe it's the same on Linux.
There is no such thing as "Linux". There are "Linux distributions" :-) It depends on how Python provided by your Linux distribution was built.
Fedora builds Python with --enable-shared: "python" is linked to "libpython". Debian/Ubuntu builds Python without --enable-shared. If I recall correctly: C extensions are no linked to libpython if Python is not linked to libpython.
Since Python 3.8, distutils no longer links C extensions to libpython (in any case). Fedora fixed packages which linked C extensions to libpython, extensions usually not built by distutils. We also had to modify some applications embedding Pyhon to use python-3.9-embed to link them to libpython. Example on Fedora 34:
$ pkg-config python-3.9 --libs # empty result $ pkg-config python-3.9-embed --libs # link to libpython -lpython3.9
Victor
Night gathers, and now my watch begins. It shall not end until my death.
On 20.05.2021 11:47, Victor Stinner wrote:
On Wed, May 19, 2021 at 3:35 PM Marc-Andre Lemburg <mal@egenix.com> wrote:
On 19.05.2021 12:45, Ronald Oussoren via capi-sig wrote:
This bug is a major issue, because C extensions are no longer linked to libpython since Python 3.8. Python must export all symbols that it pretends to export, especially if it built without libpython dynamic library (libpythonXY.so).
It has been a problem longer than that, if I recall correctly framework builds on macOS have never linked extensions to libpython. That said, as I wrote above this is currently not that large of an issue due to the way the code is structured and the way linkers work.
I believe it's the same on Linux.
There is no such thing as "Linux". There are "Linux distributions" :-) It depends on how Python provided by your Linux distribution was built.
Fedora builds Python with --enable-shared: "python" is linked to "libpython". Debian/Ubuntu builds Python without --enable-shared. If I recall correctly: C extensions are no linked to libpython if Python is not linked to libpython.
Since Python 3.8, distutils no longer links C extensions to libpython (in any case). Fedora fixed packages which linked C extensions to libpython, extensions usually not built by distutils. We also had to modify some applications embedding Pyhon to use python-3.9-embed to link them to libpython. Example on Fedora 34:
$ pkg-config python-3.9 --libs # empty result $ pkg-config python-3.9-embed --libs # link to libpython -lpython3.9
I had checked a larger number of pip downloaded wheel files and none of the .so files link to libpythonXX.so.
I now checked the OS Python 3.6 packages on my SUSE system and they do indeed also link against libpythonXX.so, just like the main Python 3.6 binary on the system.
So in conclusion, it really all depends on how you install Python to begin with.
The default configure settings of Python don't use a shared libpythonXX.so, so packages compiled against such installation will also never use libpythonXX.so, but instead get the symbols from the main executable.
Given that the libpythonXX.so is not the default, the problem is indeed a lot older than Python 3.8. How old depends a lot on the default optimization settings of the linker and compilers.
In any case, you are right: care has to be taken to make sure that we don't accidentally lose exports simply because Python itself no longer uses a function and the symbol gets optimized away from the main executable.
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, May 20 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On Wed, May 19, 2021 at 12:45 PM Ronald Oussoren <ronaldoussoren@mac.com> wrote:
Do you mean “new C file” instead of “new C function”? Because all C compilers I’ve used so far default to including whose object files when a program uses at least one symbol from that object file. That’s why Py_FrozenMain() is currently the only missing symbol, the rest of the core has sufficiently large source code files to ensure all of them will be included. The issue could crop up for other symbols if, for example, functions that are included only for backward compatibility were to be moved to separate files.
Sorry, you're right: "new C file".
If no symbol of an object file (.o) is used, GCC ignores all symbols of the object file. If at least one symbol is used, all symbols of this object file are exported. It's weird, but it's the behavior that I saw in my manual tests. It's even more surprising because the issue goes away if no static library (libpython.a) is used. It really depends on how exactly C files are linked altogether.
Victor
Night gathers, and now my watch begins. It shall not end until my death.
On 20 May 2021, at 11:49, Victor Stinner <vstinner@python.org> wrote:
On Wed, May 19, 2021 at 12:45 PM Ronald Oussoren <ronaldoussoren@mac.com> wrote:
Do you mean “new C file” instead of “new C function”? Because all C compilers I’ve used so far default to including whose object files when a program uses at least one symbol from that object file. That’s why Py_FrozenMain() is currently the only missing symbol, the rest of the core has sufficiently large source code files to ensure all of them will be included. The issue could crop up for other symbols if, for example, functions that are included only for backward compatibility were to be moved to separate files.
Sorry, you're right: "new C file".
If no symbol of an object file (.o) is used, GCC ignores all symbols of the object file. If at least one symbol is used, all symbols of this object file are exported. It's weird, but it's the behavior that I saw in my manual tests.
It is not weird, but documented behaviour ;-). And that’s probably due to historical accidents, such simplicity in early linkers.
It's even more surprising because the issue goes away if no static library (libpython.a) is used. It really depends on how exactly C files are linked altogether.
Linking a shared library will include everything that’s included in the link, or at least every object file containing visible symbols. I’ve not checked if visibility=hidden changes can result in object files being dropped, but that’s not interesting for our current discussion as symbols in the public API aren’t. hidden.
The linker will exclude unused object files in static libraries to avoid creating unnecessarily large output files, otherwise every binary that links staticly to libc would get huge. There are AFAIK 3 ways to avoid this (1) don’t link with libpython.a, but include all object files in the link command, (2) use (linker-specific) flags to include all files in libpython.a or (3) ensure that all object files contain a symbol that is referenced from the file containing the main function (directly or indirectly). IMHO that’s something we don’t have to look into right now, Py_FrozenMain is currently the only problematic symbol and is not useable from extensions.
Ronald
Victor
Night gathers, and now my watch begins. It shall not end until my death.
—
Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/
Hi,
On Fedora, the static library libpythonX.Y.a is *not* installed. I made our downstream change upstream: I added --without-static-libpython configure option to Python 3.10. https://docs.python.org/dev/using/configure.html#cmdoption-without-static-li...
Pablo wrote: "There are some build systems and downstream distributors that expect the .a to be there, so I propose to at least not changing the default." So the static library is still installed by default. https://bugs.python.org/issue43103
We can continue to install the static library, but build the Python binary by using directly the whole list of object files. IMO that's the simplest and most portable option. I'm not sure why a static library was created to link the Python binary in the first place.
Victor
On Thu, May 20, 2021 at 2:25 PM Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 20 May 2021, at 11:49, Victor Stinner <vstinner@python.org> wrote:
On Wed, May 19, 2021 at 12:45 PM Ronald Oussoren <ronaldoussoren@mac.com> wrote:
Do you mean “new C file” instead of “new C function”? Because all C compilers I’ve used so far default to including whose object files when a program uses at least one symbol from that object file. That’s why Py_FrozenMain() is currently the only missing symbol, the rest of the core has sufficiently large source code files to ensure all of them will be included. The issue could crop up for other symbols if, for example, functions that are included only for backward compatibility were to be moved to separate files.
Sorry, you're right: "new C file".
If no symbol of an object file (.o) is used, GCC ignores all symbols of the object file. If at least one symbol is used, all symbols of this object file are exported. It's weird, but it's the behavior that I saw in my manual tests.
It is not weird, but documented behaviour ;-). And that’s probably due to historical accidents, such simplicity in early linkers.
It's even more surprising because the issue goes away if no static library (libpython.a) is used. It really depends on how exactly C files are linked altogether.
Linking a shared library will include everything that’s included in the link, or at least every object file containing visible symbols. I’ve not checked if visibility=hidden changes can result in object files being dropped, but that’s not interesting for our current discussion as symbols in the public API aren’t. hidden.
The linker will exclude unused object files in static libraries to avoid creating unnecessarily large output files, otherwise every binary that links staticly to libc would get huge. There are AFAIK 3 ways to avoid this (1) don’t link with libpython.a, but include all object files in the link command, (2) use (linker-specific) flags to include all files in libpython.a or (3) ensure that all object files contain a symbol that is referenced from the file containing the main function (directly or indirectly). IMHO that’s something we don’t have to look into right now, Py_FrozenMain is currently the only problematic symbol and is not useable from extensions.
Ronald
Victor
Night gathers, and now my watch begins. It shall not end until my death.
—
Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/
-- Night gathers, and now my watch begins. It shall not end until my death.
On 21.05.2021 12:54, Victor Stinner wrote:
Hi,
On Fedora, the static library libpythonX.Y.a is *not* installed. I made our downstream change upstream: I added --without-static-libpython configure option to Python 3.10. https://docs.python.org/dev/using/configure.html#cmdoption-without-static-li...
Pablo wrote: "There are some build systems and downstream distributors that expect the .a to be there, so I propose to at least not changing the default." So the static library is still installed by default. https://bugs.python.org/issue43103
We can continue to install the static library, but build the Python binary by using directly the whole list of object files. IMO that's the simplest and most portable option. I'm not sure why a static library was created to link the Python binary in the first place.
If you have lots of object files to link, creating a library is best practice. You create a library with all your object files and then link the main() entry point object file of your choice against this.
The library also allows to statically link Python when embedding it or using it with different entry point implementations (e.g. when creating frozen apps), which is why it's installed.
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, May 21 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
participants (5)
-
Marc-Andre Lemburg -
Nick Coghlan -
Petr Viktorin -
Ronald Oussoren -
Victor Stinner