Making PY_SSIZE_T_CLEAN not mandatory.
Hi, folks, Since Python 3.8, PyArg_Parse*() APIs and Py_BuildValue() APIs emitted DeprecationWarning when '#' format is used without PY_SSIZE_T_CLEAN defined. In Python 3.10, they raise a RuntimeError, not a warning. Extension modules can not use '#' format with int. So how about making PY_SSIZE_T_CLEAN not mandatory in Python 3.11? Extension modules can use '#' format with ssize_t, without PY_SSIZE_T_CLEAN defined. Or should we wait one more version? Regards, -- Inada Naoki <songofacandy@gmail.com>
So how about making PY_SSIZE_T_CLEAN not mandatory in Python 3.11? Extension modules can use '#' format with ssize_t, without PY_SSIZE_T_CLEAN defined. Or should we wait one more version?
Hi, Inada, I suggest we should wait until at least Python 3.12 or Python 4.0. There have an another question. There have many C API defined under PY_SSIZE_T_CLEAN, for example: _PyArg_Parse_SizeT(). Could we remove or merge them after making PY_SSIZE_T_CLEAN not mandatory?
On Tue, Jun 8, 2021 at 12:53 AM Hai Shi <shihai1992@gmail.com> wrote:
So how about making PY_SSIZE_T_CLEAN not mandatory in Python 3.11? Extension modules can use '#' format with ssize_t, without PY_SSIZE_T_CLEAN defined. Or should we wait one more version?
Hi, Inada, I suggest we should wait until at least Python 3.12 or Python 4.0.
Serhiy and you suggest Python 3.12 and I agree with it. Thank you for your reply.
There have an another question. There have many C API defined under PY_SSIZE_T_CLEAN, for example: _PyArg_Parse_SizeT(). Could we remove or merge them after making PY_SSIZE_T_CLEAN not mandatory?
They are part of stable ABIs. So we can remove/merge them at Python 4.0. Regards, -- Inada Naoki <songofacandy@gmail.com>
07.06.21 08:41, Hai Shi пише:
There have an another question. There have many C API defined under PY_SSIZE_T_CLEAN, for example: _PyArg_Parse_SizeT(). Could we remove or merge them after making PY_SSIZE_T_CLEAN not mandatory?
We should support binary compatibility to some degree, so there should be several steps: * Make macro PyArg_Parse an alias of _PyArg_Parse_SizeT. Keep function PyArg_Parse. * Make function PyArg_Parse always raising an exception. * Remove function PyArg_Parse. * [Optionally] Now we can re-add function PyArg_Parse as an alias of _PyArg_Parse_SizeT and remove macro PyArg_Parse. * [Optionally in 4.0 or 5.0] Remove _PyArg_Parse_SizeT. But we can squish several last steps in 4.0 which do not need to support binary compatibility with 3.x.
On 08. 06. 21 10:05, Serhiy Storchaka wrote:
07.06.21 08:41, Hai Shi пише:
There have an another question. There have many C API defined under PY_SSIZE_T_CLEAN, for example: _PyArg_Parse_SizeT(). Could we remove or merge them after making PY_SSIZE_T_CLEAN not mandatory?
We should support binary compatibility to some degree, so there should be several steps:
* Make macro PyArg_Parse an alias of _PyArg_Parse_SizeT. Keep function PyArg_Parse.
One more thing about the stable ABI: in the future, I'd like to make it more useful in languages other than C. This usually means avoiding macros. Would it make sense to expose _PyArg_Parse_SizeT as a public function, like PyArgT_Parse? (The macro redirecting PyArg_Parse to this function could of course stay, to help C users.)
* Make function PyArg_Parse always raising an exception.
This would break extensions that use the stable ABI. (Yes, even starting to raise RuntimeError in 3.10 broke things. And yes, it's not strictly an ABI issue, but it has the same effect for users: they still need to recompile extensions to keep them working.)
* Remove function PyArg_Parse. * [Optionally] Now we can re-add function PyArg_Parse as an alias of _PyArg_Parse_SizeT and remove macro PyArg_Parse. * [Optionally in 4.0 or 5.0] Remove _PyArg_Parse_SizeT.
But we can squish several last steps in 4.0 which do not need to support binary compatibility with 3.x.
On Tue, Jun 8, 2021 at 6:02 PM Petr Viktorin <encukou@gmail.com> wrote:
* Make function PyArg_Parse always raising an exception.
This would break extensions that use the stable ABI. (Yes, even starting to raise RuntimeError in 3.10 broke things. And yes, it's not strictly an ABI issue, but it has the same effect for users: they still need to recompile extensions to keep them working.)
I think we can skip this step. Extension modules using # format is already broken since Python 3.10. Adding # format support with size_t won't break so much. We can do it in Python 3.12 or 3.13. Regards, -- Inada Naoki <songofacandy@gmail.com>
07.06.21 06:05, Inada Naoki пише:
Since Python 3.8, PyArg_Parse*() APIs and Py_BuildValue() APIs emitted DeprecationWarning when '#' format is used without PY_SSIZE_T_CLEAN defined. In Python 3.10, they raise a RuntimeError, not a warning. Extension modules can not use '#' format with int.
So how about making PY_SSIZE_T_CLEAN not mandatory in Python 3.11? Extension modules can use '#' format with ssize_t, without PY_SSIZE_T_CLEAN defined.
Or should we wait one more version?
Many users still use 3.6 or 3.7. Jumping from 3.7 to 3.11 could break extensions in bad way (crash, truncated data, leaked sensitive information, execution of arbitrary code). Also, deprecation warnings in 3.8 and 3.9 can be easily ignored. I propose to wait until both of conditions became true: * 3.7 no longer maintained * 3.10 reaches security-only mode.
On Mon, Jun 7, 2021 at 4:52 PM Serhiy Storchaka <storchaka@gmail.com> wrote:
Many users still use 3.6 or 3.7. Jumping from 3.7 to 3.11 could break extensions in bad way (crash, truncated data, leaked sensitive information, execution of arbitrary code). Also, deprecation warnings in 3.8 and 3.9 can be easily ignored.
I propose to wait until both of conditions became true:
* 3.7 no longer maintained * 3.10 reaches security-only mode.
Makes sense. Python 3.7 will get security fix until 2023-06. https://www.python.org/dev/peps/pep-0537/#and-beyond-schedule Python 3.12 will be released at 2023-10. So we can change PY_SSIZE_T_CLEAN by default from 3.12. Regards, -- Inada Naoki <songofacandy@gmail.com>
On 7 Jun 2021, at 05:05, Inada Naoki <songofacandy@gmail.com> wrote:
Hi, folks,
Since Python 3.8, PyArg_Parse*() APIs and Py_BuildValue() APIs emitted DeprecationWarning when '#' format is used without PY_SSIZE_T_CLEAN defined. In Python 3.10, they raise a RuntimeError, not a warning. Extension modules can not use '#' format with int.
So how about making PY_SSIZE_T_CLEAN not mandatory in Python 3.11? Extension modules can use '#' format with ssize_t, without PY_SSIZE_T_CLEAN defined.
Or should we wait one more version?
Its a bit late to complain (and I’m not affected by this myself), but those functions are part of the stable ABI. The change in 3.10 will break any extensions that use the stable ABI, use these functions and don’t use PY_SSIZE_T_CLEAN. I have no idea how many of those exist, especially given that the stable ABI doesn’t seem to be used a lot. I guess this depends a little on what promises the stable ABI makes, the functions are still there but behave differently than before. P.S. I’d be in favour of just dropping PY_SSIZE_T_CLEAN completely (that is use Py_ssize_t unconditionally) to simplify the code base, apart from being slightly worried about the impact on the stable ABI. AFAIK The define was meant as a temporary transition mechanism when Py_ssize_t was introduced in the, by now, ancient past. Ronald — Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/
On Wed, Jun 9, 2021 at 10:32 AM Ronald Oussoren via Python-Dev <python-dev@python.org> wrote:
Its a bit late to complain (and I’m not affected by this myself), but those functions are part of the stable ABI. The change in 3.10 will break any extensions that use the stable ABI, use these functions and don’t use PY_SSIZE_T_CLEAN. I have no idea how many of those exist, especially given that the stable ABI doesn’t seem to be used a lot.
Require PY_SSIZE_T_CLEAN macro to be defined in an incompatible *API* change. At the ABI level, what changed if that C extensions built (with Python 3.9 and older) without PY_SSIZE_T_CLEAN now raise an error on Python 3.10 (for a few specific argument formats using "#"). Ah you are right, it's an incompatible ABI change. It might be possible to keep the backward compatibility at the ABI level by adding a 3rd flavor of "parse" functions: * parse with size_t: no change * parse without size_t: stable ABI * parse without size_t which raises an exception on "#" formats: new Python 3.10 functions It's already painful to have 2 flavors of each functions. Adding a 3rd flavor would make the maintenance burden even worse, whereas Inada-san wants to opposite (remove the 2nd flavor to only have one!). A more general question is: do we still want to keep backward compatibility with Python 3.2 released 10 years ago, or is it ok to start with a new stable ABI which drops backward compatibility with Python 3.5 (for example)? It's maybe time to replace "abi3" with "abi4" in Python 3.10? Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On 9 Jun 2021, at 11:13, Victor Stinner <vstinner@python.org> wrote:
On Wed, Jun 9, 2021 at 10:32 AM Ronald Oussoren via Python-Dev <python-dev@python.org> wrote:
Its a bit late to complain (and I’m not affected by this myself), but those functions are part of the stable ABI. The change in 3.10 will break any extensions that use the stable ABI, use these functions and don’t use PY_SSIZE_T_CLEAN. I have no idea how many of those exist, especially given that the stable ABI doesn’t seem to be used a lot.
Require PY_SSIZE_T_CLEAN macro to be defined in an incompatible *API* change. At the ABI level, what changed if that C extensions built (with Python 3.9 and older) without PY_SSIZE_T_CLEAN now raise an error on Python 3.10 (for a few specific argument formats using "#"). Ah you are right, it's an incompatible ABI change.
:-(
It might be possible to keep the backward compatibility at the ABI level by adding a 3rd flavor of "parse" functions:
* parse with size_t: no change * parse without size_t: stable ABI * parse without size_t which raises an exception on "#" formats: new Python 3.10 functions
It's already painful to have 2 flavors of each functions. Adding a 3rd flavor would make the maintenance burden even worse, whereas Inada-san wants to opposite (remove the 2nd flavor to only have one!).
I don’t think it is necessary to introduce a 3th variant, for 3.11+ we could do something like this: * [3.11] Add deprecation warnings in the C headers to the few functions with a “PY_SSIZE_T_CLEAN” variant when “PY_SSIZE_T_CLEAN” is not defined * [3.12+] Change the headers to behave as if “PY_SSIZE_T_CLEAN” is defined, and only keep the non-PY_SSIZE_T_CLEAN variants for the stable ABI (which would include dropping non-PY_SSIZE_T_CLEAN variants for private functions). The PY_SSIZE_T_CLEAN variants would keep their "_PysizeT” suffix in the (stable or unstable) ABI. This wouldn’t allow dropping the non-PY_SSIZE_T_CLEAN variants entirely, at least not until we’re fine with breaking the stable ABI. Another disadvantage is that this might require changes in code that doesn’t even use “#” in format strings in 3.11.
A more general question is: do we still want to keep backward compatibility with Python 3.2 released 10 years ago, or is it ok to start with a new stable ABI which drops backward compatibility with Python 3.5 (for example)?
It's maybe time to replace "abi3" with "abi4" in Python 3.10?
Personally I’d say it is too soon for that, especially when the CPython speedup project (Guido, Mark, et.al.) is just started and HPy is far from finished. Either project might teach us what changes are needed for a long term stable ABI. Ronald — Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/
I think stable ABI keeps symbols, signatures, and memory layouts. I don't think stable ABI keeps all behaviors. For example, Py_CompileString() is stable ABI. When we add `async` keyword, Py_CompileString() starts raising an Error for source code using `async` name. Is it ABI change? I don't think so. I want to drop Py_UNICODE support in Python 3.12. It is another incompatible change in PyArg_Parse*() *API*. Users can not use "u" format after it. It is an incompatible *API* change, but not an *ABI* change. I suspect we had made many incompatible *API* changes in stable ABIs already. If I am wrong, can we stop keeping stable ABI at Python 3.12? Python 4.0 won't come in foreseeable future. Stable ABI blocks Python evolution. Regards, -- Inada Naoki <songofacandy@gmail.com>
On Wed, 9 Jun 2021 at 11:36, Inada Naoki <songofacandy@gmail.com> wrote:
If I am wrong, can we stop keeping stable ABI at Python 3.12? Python 4.0 won't come in foreseeable future. Stable ABI blocks Python evolution.
Conversely, the stable ABI allows projects to build cross-version binary wheels. Not many projects do that yet, but it's definitely something we'd like to see more of. Needing new binary builds every version blocks users from testing new versions of Python in advance of the release. Also, I often use the stable ABI when embedding, so that I can replace the Python interpreter without needing to recompile my application and redeploy new binaries everywhere (my use case is pretty niche, though, so I wouldn't like to claim I represent a typical user...). But I do agree that we should either start keeping to the commitments that we made around the stability of the stable ABI, or we should abandon it properly and declare it no longer supported. Having something that sort of works except when we accidentally broke it doesn't help anyone. Paul
On 09. 06. 21 13:09, Paul Moore wrote:
On Wed, 9 Jun 2021 at 11:36, Inada Naoki <songofacandy@gmail.com> wrote:
If I am wrong, can we stop keeping stable ABI at Python 3.12? Python 4.0 won't come in foreseeable future. Stable ABI blocks Python evolution.
Conversely, the stable ABI allows projects to build cross-version binary wheels. Not many projects do that yet, but it's definitely something we'd like to see more of. Needing new binary builds every version blocks users from testing new versions of Python in advance of the release. [...]
But I do agree that we should either start keeping to the commitments that we made around the stability of the stable ABI, or we should abandon it properly and declare it no longer supported. Having something that sort of works except when we accidentally broke it doesn't help anyone.
I don't think I made actual commitments regarding the API. The docs do say: "we recommend testing an extension with all minor Python versions it supports". Also, when the API breaks, you get a Python exception; if the ABI does, you get segfaults. So breaking the API is much less severe, but still -- please think about the effect on users who just want their compiled extensions to keep working. On 09. 06. 21 13:09, Paul Moore wrote:
Also, I often use the stable ABI when embedding, so that I can replace the Python interpreter without needing to recompile my application and redeploy new binaries everywhere (my use case is pretty niche, though, so I wouldn't like to claim I represent a typical user...).
I hope this use case becomes non-niche. I would love it if embedders tell people to just use any Python they have lying around, instead of vendoring it (or more realistically, embedding JS or Lua instead).
On 6/9/2021 2:20 PM, Petr Viktorin wrote:
On 09. 06. 21 13:09, Paul Moore wrote:
Also, I often use the stable ABI when embedding, so that I can replace the Python interpreter without needing to recompile my application and redeploy new binaries everywhere (my use case is pretty niche, though, so I wouldn't like to claim I represent a typical user...).
I hope this use case becomes non-niche. I would love it if embedders tell people to just use any Python they have lying around, instead of vendoring it (or more realistically, embedding JS or Lua instead).
I also hope it becomes non-niche, but I'd rather you started embedding/vendoring CPython rather than using anything that just happens to be laying around. The number one issue that *all* of my customers (and their customers) have is installation. For most of them, the best way to solve it is to not make them install Python themselves, which in many cases means vendoring. The more acceptable and easy we can make this process, the more Python will be a viable choice against JS or Lua (though with all the other C API, threading and initialization issues, it's unlikely that embedding CPython is going to become significantly more attractive than those two - even IronPython still lives on for embedding because it works so well). Cheers, Steve
I don’t see how the stable ABI works as a substitute for vendoring Python. A lot of other things can still vary even when the C API remains the same! (E.g. syntax, and stdlib behavior.) On Thu, Jun 17, 2021 at 11:49 Steve Dower <steve.dower@python.org> wrote:
On 6/9/2021 2:20 PM, Petr Viktorin wrote:
On 09. 06. 21 13:09, Paul Moore wrote:
Also, I often use the stable ABI when embedding, so that I can replace the Python interpreter without needing to recompile my application and redeploy new binaries everywhere (my use case is pretty niche, though, so I wouldn't like to claim I represent a typical user...).
I hope this use case becomes non-niche. I would love it if embedders tell people to just use any Python they have lying around, instead of vendoring it (or more realistically, embedding JS or Lua instead).
I also hope it becomes non-niche, but I'd rather you started embedding/vendoring CPython rather than using anything that just happens to be laying around.
The number one issue that *all* of my customers (and their customers) have is installation. For most of them, the best way to solve it is to not make them install Python themselves, which in many cases means vendoring. The more acceptable and easy we can make this process, the more Python will be a viable choice against JS or Lua (though with all the other C API, threading and initialization issues, it's unlikely that embedding CPython is going to become significantly more attractive than those two - even IronPython still lives on for embedding because it works so well).
Cheers, Steve _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/36MZXQC3... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido (mobile)
On Fri, 18 Jun 2021 at 01:57, Guido van Rossum <guido@python.org> wrote:
I don’t see how the stable ABI works as a substitute for vendoring Python. A lot of other things can still vary even when the C API remains the same! (E.g. syntax, and stdlib behavior.)
IMO it doesn't. However for certain applications (the sort of thing I was referring to) - where the user is writing their own scripts and the embedding API is used merely to expose an interface to the Python language, dynamically linking to whatever version of Python the user has installed can be precisely the right thing to do - the user gets access to the version of the language they expect, the installed packages they expect to see, etc. Consider something like the vim integration with Python (which *doesn't* use the stable ABI, so you have to use a distribution built for precisely the Python version you have installed - the stable ABI would be better from the POV of supporting fewer binary builds). Vendoring is a viable alternative here as well, but as I say, it just has different trade-offs. As I said, it's a niche usage, but it has its place. I don't personally have an opinion on promoting the option of linking to the system Python - it just suits my use cases. Paul
Le 18/06/2021 à 08:50, Paul Moore a écrit :
IMO it doesn't. However for certain applications (the sort of thing I was referring to) - where the user is writing their own scripts and the embedding API is used merely to expose an interface to the Python language, dynamically linking to whatever version of Python the user has installed can be precisely the right thing to do - the user gets access to the version of the language they expect, the installed packages they expect to see, etc.
As a user, I second this. When trying to drive applications from the outside (as opposed to extending them through plugins), it is annoying when two applications won't work together because each one insists on using its own vendored python. Of course, there are often real blockers, such as incompatible event loops. But not always… Cheers, Baptiste
Can you elaborate on that use case? Which two applications are you thinking of, and what was your goal in driving them? This sounds interesting but I haven’t encountered this myself. On Fri, Jun 18, 2021 at 09:44 Baptiste Carvello < devel2021@baptiste-carvello.net> wrote:
Le 18/06/2021 à 08:50, Paul Moore a écrit :
IMO it doesn't. However for certain applications (the sort of thing I was referring to) - where the user is writing their own scripts and the embedding API is used merely to expose an interface to the Python language, dynamically linking to whatever version of Python the user has installed can be precisely the right thing to do - the user gets access to the version of the language they expect, the installed packages they expect to see, etc.
As a user, I second this. When trying to drive applications from the outside (as opposed to extending them through plugins), it is annoying when two applications won't work together because each one insists on using its own vendored python.
Of course, there are often real blockers, such as incompatible event loops. But not always…
Cheers, Baptiste _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PPKL7466... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido (mobile)
Hi, Le 18/06/2021 à 21:00, Guido van Rossum a écrit :
Can you elaborate on that use case? Which two applications are you thinking of, and what was your goal in driving them? This sounds interesting but I haven’t encountered this myself.
Well, I'm not sure the case I was thinking of is still relevant to anything: that was plotting 3D crystal models using crystallography library CCTBX [1] and visualization application Mayavi [2], some 15-20 years ago. BTW, I misremembered a bit: only CCTBX insisted on using a vendored python ("libtbx.python"), Mayavi used the system python. Anyway, it was more pain to make Mayavi use libtbx.python, than to make CCTBX work with the system python. Also, I must admit that even applications embedding the system python can have some limitations. For example, GIMP and GDB can execute python scripts, but their API can't be "imported" from the outside. Which means no arguments passed to the script over the command line ("sys.argv"), no venvs, no REPL. But at least you can install additional packages (pip / distro package manager) and limitations can be more or less hacked around. For a sophisticated example, the debugger extension Voltron [3] provides REPL access to GDB objects over a client-server connexion. Cheers, Baptiste [1] https://cci.lbl.gov/docs/cctbx/ [2] https://docs.enthought.com/mayavi/mayavi/ [3] https://github.com/snare/voltron
On Fri, Jun 18, 2021 at 09:44 Baptiste Carvello <devel2021@baptiste-carvello.net <mailto:devel2021@baptiste-carvello.net>> wrote:
Le 18/06/2021 à 08:50, Paul Moore a écrit : > > IMO it doesn't. However for certain applications (the sort of thing I > was referring to) - where the user is writing their own scripts and > the embedding API is used merely to expose an interface to the Python > language, dynamically linking to whatever version of Python the user > has installed can be precisely the right thing to do - the user gets > access to the version of the language they expect, the installed > packages they expect to see, etc.
As a user, I second this. When trying to drive applications from the outside (as opposed to extending them through plugins), it is annoying when two applications won't work together because each one insists on using its own vendored python.
Of course, there are often real blockers, such as incompatible event loops. But not always…
Cheers, Baptiste _______________________________________________ Python-Dev mailing list -- python-dev@python.org <mailto:python-dev@python.org> To unsubscribe send an email to python-dev-leave@python.org <mailto:python-dev-leave@python.org> https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PPKL7466... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido (mobile)
Okay, I think your evidence can then be discounted. Really, any app that relies on the publicly installed Python runs a serious risk of breaking when that Python gets updated, regardless of whether the ABI changes or not. On Mon, Jun 21, 2021 at 2:46 AM Baptiste Carvello < devel2021@baptiste-carvello.net> wrote:
Hi,
Le 18/06/2021 à 21:00, Guido van Rossum a écrit :
Can you elaborate on that use case? Which two applications are you thinking of, and what was your goal in driving them? This sounds interesting but I haven’t encountered this myself.
Well, I'm not sure the case I was thinking of is still relevant to anything: that was plotting 3D crystal models using crystallography library CCTBX [1] and visualization application Mayavi [2], some 15-20 years ago. BTW, I misremembered a bit: only CCTBX insisted on using a vendored python ("libtbx.python"), Mayavi used the system python. Anyway, it was more pain to make Mayavi use libtbx.python, than to make CCTBX work with the system python.
Also, I must admit that even applications embedding the system python can have some limitations. For example, GIMP and GDB can execute python scripts, but their API can't be "imported" from the outside. Which means no arguments passed to the script over the command line ("sys.argv"), no venvs, no REPL. But at least you can install additional packages (pip / distro package manager) and limitations can be more or less hacked around. For a sophisticated example, the debugger extension Voltron [3] provides REPL access to GDB objects over a client-server connexion.
Cheers, Baptiste
[1] https://cci.lbl.gov/docs/cctbx/ [2] https://docs.enthought.com/mayavi/mayavi/ [3] https://github.com/snare/voltron
On Fri, Jun 18, 2021 at 09:44 Baptiste Carvello <devel2021@baptiste-carvello.net <mailto:devel2021@baptiste-carvello.net>> wrote:
Le 18/06/2021 à 08:50, Paul Moore a écrit : > > IMO it doesn't. However for certain applications (the sort of thing I > was referring to) - where the user is writing their own scripts and > the embedding API is used merely to expose an interface to the Python > language, dynamically linking to whatever version of Python the user > has installed can be precisely the right thing to do - the user gets > access to the version of the language they expect, the installed > packages they expect to see, etc.
As a user, I second this. When trying to drive applications from the outside (as opposed to extending them through plugins), it is annoying when two applications won't work together because each one insists on using its own vendored python.
Of course, there are often real blockers, such as incompatible event loops. But not always…
Cheers, Baptiste _______________________________________________ Python-Dev mailing list -- python-dev@python.org <mailto:python-dev@python.org> To unsubscribe send an email to python-dev-leave@python.org <mailto:python-dev-leave@python.org> https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/PPKL7466...
Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido (mobile)
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KP3SE6UW... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On 21. 06. 21 20:20, Guido van Rossum wrote:
Okay, I think your evidence can then be discounted. Really, any app that relies on the publicly installed Python runs a serious risk of breaking when that Python gets updated, regardless of whether the ABI changes or not.
Unfortunately, this includes scripts for any extensible software (Mayavi, GIMP, etc) -- even if Python is bundled with that software, upgrading the software risks breaking the scripts.
On Mon, Jun 21, 2021 at 2:46 AM Baptiste Carvello <devel2021@baptiste-carvello.net <mailto:devel2021@baptiste-carvello.net>> wrote:
Hi,
Le 18/06/2021 à 21:00, Guido van Rossum a écrit : > Can you elaborate on that use case? Which two applications are you > thinking of, and what was your goal in driving them? This sounds > interesting but I haven’t encountered this myself.
Well, I'm not sure the case I was thinking of is still relevant to anything: that was plotting 3D crystal models using crystallography library CCTBX [1] and visualization application Mayavi [2], some 15-20 years ago. BTW, I misremembered a bit: only CCTBX insisted on using a vendored python ("libtbx.python"), Mayavi used the system python. Anyway, it was more pain to make Mayavi use libtbx.python, than to make CCTBX work with the system python.
Also, I must admit that even applications embedding the system python can have some limitations. For example, GIMP and GDB can execute python scripts, but their API can't be "imported" from the outside. Which means no arguments passed to the script over the command line ("sys.argv"), no venvs, no REPL. But at least you can install additional packages (pip / distro package manager) and limitations can be more or less hacked around. For a sophisticated example, the debugger extension Voltron [3] provides REPL access to GDB objects over a client-server connexion.
Cheers, Baptiste
[1] https://cci.lbl.gov/docs/cctbx/ <https://cci.lbl.gov/docs/cctbx/> [2] https://docs.enthought.com/mayavi/mayavi/ <https://docs.enthought.com/mayavi/mayavi/> [3] https://github.com/snare/voltron <https://github.com/snare/voltron>
> On Fri, Jun 18, 2021 at 09:44 Baptiste Carvello > <devel2021@baptiste-carvello.net <mailto:devel2021@baptiste-carvello.net> > <mailto:devel2021@baptiste-carvello.net <mailto:devel2021@baptiste-carvello.net>>> wrote: > > Le 18/06/2021 à 08:50, Paul Moore a écrit : > > > > IMO it doesn't. However for certain applications (the sort of thing I > > was referring to) - where the user is writing their own scripts and > > the embedding API is used merely to expose an interface to the Python > > language, dynamically linking to whatever version of Python the user > > has installed can be precisely the right thing to do - the user gets > > access to the version of the language they expect, the installed > > packages they expect to see, etc. > > As a user, I second this. When trying to drive applications from the > outside (as opposed to extending them through plugins), it is annoying > when two applications won't work together because each one insists on > using its own vendored python. > > Of course, there are often real blockers, such as incompatible event > loops. But not always… > > Cheers, > Baptiste > _______________________________________________ > Python-Dev mailing list -- python-dev@python.org <mailto:python-dev@python.org> > <mailto:python-dev@python.org <mailto:python-dev@python.org>> > To unsubscribe send an email to python-dev-leave@python.org <mailto:python-dev-leave@python.org> > <mailto:python-dev-leave@python.org <mailto:python-dev-leave@python.org>> > https://mail.python.org/mailman3/lists/python-dev.python.org/ <https://mail.python.org/mailman3/lists/python-dev.python.org/> > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/PPKL7466... <https://mail.python.org/archives/list/python-dev@python.org/message/PPKL7466BIG6DPCUIJURLE5ZGFNHBNSM/> > Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/> > > -- > --Guido (mobile) >
_______________________________________________ Python-Dev mailing list -- python-dev@python.org <mailto:python-dev@python.org> To unsubscribe send an email to python-dev-leave@python.org <mailto:python-dev-leave@python.org> https://mail.python.org/mailman3/lists/python-dev.python.org/ <https://mail.python.org/mailman3/lists/python-dev.python.org/> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KP3SE6UW... <https://mail.python.org/archives/list/python-dev@python.org/message/KP3SE6UWSV3VDCJOWCXUZIBPDWFJHRLU/> Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/>
-- --Guido van Rossum (python.org/~guido <http://python.org/~guido>) /Pronouns: he/him //(why is my pronoun here?)/ <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GC6PMZP3... Code of Conduct: http://python.org/psf/codeofconduct/
On Tue, 22 Jun 2021 at 10:06, Petr Viktorin <encukou@gmail.com> wrote:
On 21. 06. 21 20:20, Guido van Rossum wrote:
Okay, I think your evidence can then be discounted. Really, any app that relies on the publicly installed Python runs a serious risk of breaking when that Python gets updated, regardless of whether the ABI changes or not.
Unfortunately, this includes scripts for any extensible software (Mayavi, GIMP, etc) -- even if Python is bundled with that software, upgrading the software risks breaking the scripts.
I'm confused by what you mean by this, or why it is a problem? If I upgrade GIMP (and it vendors some version/variant of Python), it is not unreasonable that this would break a script that I have written in GIMPython? (GIMP should probably mention that it has changed its Python and how in the changelog/release notes) If I upgrade my OS, and I use the system Python, scripts I have written might break too. (Of course, GIMP is a placeholder here, I do not actually know what it does in terms of Python (vendoring), if at all.
On June 22, 2021 11:18:46 AM GMT+02:00, Henk-Jaap Wagenaar <wagenaarhenkjaap@gmail.com> wrote:
On Tue, 22 Jun 2021 at 10:06, Petr Viktorin <encukou@gmail.com> wrote:
Okay, I think your evidence can then be discounted. Really, any app
On 21. 06. 21 20:20, Guido van Rossum wrote: that
relies on the publicly installed Python runs a serious risk of breaking when that Python gets updated, regardless of whether the ABI changes or not.
Unfortunately, this includes scripts for any extensible software (Mayavi, GIMP, etc) -- even if Python is bundled with that software, upgrading the software risks breaking the scripts.
I'm confused by what you mean by this, or why it is a problem?
Not necessarily a problem, I just want to point out that there are situations where you need to depend on Python managed by someone else.
If I upgrade GIMP (and it vendors some version/variant of Python), it is not unreasonable that this would break a script that I have written in GIMPython? (GIMP should probably mention that it has changed its Python and how in the changelog/release notes)
If I upgrade my OS, and I use the system Python, scripts I have written might break too.
(Of course, GIMP is a placeholder here, I do not actually know what it does in terms of Python (vendoring), if at all.
GIMP itself doesn't, but it's sometimes distributed in flatpak/appimage with its own bundled/pinned Python. (AFAIK it's usually Python 2.7, for a few reasons; one of them being that upgrading would break scripts) -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Hi, I'm afraid we are talking past each other. In my case, the applications are either part of my Python-based work setup, which needs to be rebuilt anyway in the very infrequent occurrence of a major Python upgrade (I could get away with using the system python for that on a "stable" distro). Or they are installed using my distro's package manager, which takes care of upgrading dependent packages together. This is the classical "DLL hell" problem, right? Or is there some Python specificity that I don't get? Cheers, Baptiste Le 21/06/2021 à 20:20, Guido van Rossum a écrit :
Okay, I think your evidence can then be discounted. Really, any app that relies on the publicly installed Python runs a serious risk of breaking when that Python gets updated, regardless of whether the ABI changes or not.
On Mon, Jun 21, 2021 at 2:46 AM Baptiste Carvello <devel2021@baptiste-carvello.net <mailto:devel2021@baptiste-carvello.net>> wrote:
Hi,
Le 18/06/2021 à 21:00, Guido van Rossum a écrit : > Can you elaborate on that use case? Which two applications are you > thinking of, and what was your goal in driving them? This sounds > interesting but I haven’t encountered this myself.
Well, I'm not sure the case I was thinking of is still relevant to anything: that was plotting 3D crystal models using crystallography library CCTBX [1] and visualization application Mayavi [2], some 15-20 years ago. BTW, I misremembered a bit: only CCTBX insisted on using a vendored python ("libtbx.python"), Mayavi used the system python. Anyway, it was more pain to make Mayavi use libtbx.python, than to make CCTBX work with the system python.
Also, I must admit that even applications embedding the system python can have some limitations. For example, GIMP and GDB can execute python scripts, but their API can't be "imported" from the outside. Which means no arguments passed to the script over the command line ("sys.argv"), no venvs, no REPL. But at least you can install additional packages (pip / distro package manager) and limitations can be more or less hacked around. For a sophisticated example, the debugger extension Voltron [3] provides REPL access to GDB objects over a client-server connexion.
Cheers, Baptiste
[1] https://cci.lbl.gov/docs/cctbx/ [2] https://docs.enthought.com/mayavi/mayavi/ [3] https://github.com/snare/voltron
> On Fri, Jun 18, 2021 at 09:44 Baptiste Carvello > <devel2021@baptiste-carvello.net <mailto:devel2021@baptiste-carvello.net> > <mailto:devel2021@baptiste-carvello.net <mailto:devel2021@baptiste-carvello.net>>> wrote: > > Le 18/06/2021 à 08:50, Paul Moore a écrit : > > > > IMO it doesn't. However for certain applications (the sort of thing I > > was referring to) - where the user is writing their own scripts and > > the embedding API is used merely to expose an interface to the Python > > language, dynamically linking to whatever version of Python the user > > has installed can be precisely the right thing to do - the user gets > > access to the version of the language they expect, the installed > > packages they expect to see, etc. > > As a user, I second this. When trying to drive applications from the > outside (as opposed to extending them through plugins), it is annoying > when two applications won't work together because each one insists on > using its own vendored python. > > Of course, there are often real blockers, such as incompatible event > loops. But not always… > > Cheers, > Baptiste > _______________________________________________ > Python-Dev mailing list -- python-dev@python.org <mailto:python-dev@python.org> > <mailto:python-dev@python.org <mailto:python-dev@python.org>> > To unsubscribe send an email to python-dev-leave@python.org <mailto:python-dev-leave@python.org> > <mailto:python-dev-leave@python.org <mailto:python-dev-leave@python.org>> > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/PPKL7466... > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > --Guido (mobile) >
_______________________________________________ Python-Dev mailing list -- python-dev@python.org <mailto:python-dev@python.org> To unsubscribe send an email to python-dev-leave@python.org <mailto:python-dev-leave@python.org> https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KP3SE6UW... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido <http://python.org/~guido>) /Pronouns: he/him //(why is my pronoun here?)/ <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On 9 Jun 2021, at 12:28, Inada Naoki <songofacandy@gmail.com> wrote:
I think stable ABI keeps symbols, signatures, and memory layouts. I don't think stable ABI keeps all behaviors.
As often “it depends”. Behaviour is IMHO part of the API/ABI contract. That said, that does not necessarily mean that we cannot change behaviour at all, but that the cost to users for such changes should be taken into account.
For example, Py_CompileString() is stable ABI. When we add `async` keyword, Py_CompileString() starts raising an Error for source code using `async` name. Is it ABI change? I don't think so.
I agree. But its not as easy as “it is not an ABI change because it only changes functionality of a function”. The interface contract of Py_CompileString is that it compiles Python code. If the rules for what valid Python code is change (such as the introduction of ‘async’ as a hard keyword) the function can start to reject input that was accepted earlier. That’s IMHO different from a change to Py_CompileString that starts raising an error unconditionally because we no longer want to expose the API.
I want to drop Py_UNICODE support in Python 3.12. It is another incompatible change in PyArg_Parse*() *API*. Users can not use "u" format after it. It is an incompatible *API* change, but not an *ABI* change.
It is an ABI change: an extensions targetting the stable ABI no longer works due to a change in implementation. That doesn’t necessarily mean the change cannot be made, especially when a deprecation warning is emitted before the feature is removed.
I suspect we had made many incompatible *API* changes in stable ABIs already.
If I am wrong, can we stop keeping stable ABI at Python 3.12? Python 4.0 won't come in foreseeable future. Stable ABI blocks Python evolution.
The annoying part of the stable ABI is that it still exposes some implementation details and behaviour that make it harder to write correct code (such as borrowed references, these can be very convenient but are also easy to misuse). That’s one reason why HPy is an interesting project, even when only targeting CPython. And to be clear: I’m not opposed to the change for the “#” format character and the removal of the “u” format you mention earlier. Ronald
Regards, -- Inada Naoki <songofacandy@gmail.com>
— Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/
participants (11)
-
Baptiste Carvello -
Guido van Rossum -
Hai Shi -
Henk-Jaap Wagenaar -
Inada Naoki -
Paul Moore -
Petr Viktorin -
Ronald Oussoren -
Serhiy Storchaka -
Steve Dower -
Victor Stinner