code.replace() and Python 3.11 exception table
("Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API") On Fri, Apr 1, 2022 at 11:01 AM Chris Angelico <rosuav@gmail.com> wrote:
On Fri, 1 Apr 2022 at 19:51, Victor Stinner <vstinner@python.org> wrote:
In Python, sadly the types.CodeType type also has a public constructor and many projects break at each Python release because the API changes. Hopefully, it seems like the new CodeType.replace() method added to Python 3.8 mitigated the issue. IMO CodeType.replace() is a better abstraction and closer to what developers need in practice.
It certainly has been for me. When I want to do bytecode hackery, I usually start by creating a function with def/lambda, then construct a modified function using f.__code__.replace(). It's the easiest way to ensure that all the little details are correct.
Python 3.11 added the concept of "exception table" (code.co_exceptiontable). You have to build this table, otherwise Python can no longer catch exceptions :-) I don't know how to build this exception table. It seems like currently there is no Python function in the stdlib to build this table. Example: --- def f(): try: print("raise") raise ValueError except ValueError: print("except") else: print("else") print("exit func") def g(): pass if 1: code = f.__code__ g.__code__ = g.__code__.replace( co_code=code.co_code, co_consts=code.co_consts, co_names=code.co_names, co_flags=code.co_flags, co_stacksize=code.co_stacksize) else: g.__code__ = f.__code__ # this code path works on Python 3.11 g() --- Output with Python 3.10 (ok): --- raise except exit func --- Output with Python 3.11 (oops): --- raise Traceback (most recent call last): ... ValueError --- By the way, this change is not documented at all: * https://docs.python.org/dev/library/types.html#types.CodeType * https://docs.python.org/dev/whatsnew/3.11.html I understand that these changes come from the "Zero cost exception handling" change: https://bugs.python.org/issue40222 Victor -- Night gathers, and now my watch begins. It shall not end until my death.
I created https://bugs.python.org/issue47185 to discuss this issue: either recompute automatically co_exceptiontable, or at least document the change. Victor On Fri, Apr 1, 2022 at 11:21 AM Victor Stinner <vstinner@python.org> wrote:
("Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API")
On Fri, Apr 1, 2022 at 11:01 AM Chris Angelico <rosuav@gmail.com> wrote:
On Fri, 1 Apr 2022 at 19:51, Victor Stinner <vstinner@python.org> wrote:
In Python, sadly the types.CodeType type also has a public constructor and many projects break at each Python release because the API changes. Hopefully, it seems like the new CodeType.replace() method added to Python 3.8 mitigated the issue. IMO CodeType.replace() is a better abstraction and closer to what developers need in practice.
It certainly has been for me. When I want to do bytecode hackery, I usually start by creating a function with def/lambda, then construct a modified function using f.__code__.replace(). It's the easiest way to ensure that all the little details are correct.
Python 3.11 added the concept of "exception table" (code.co_exceptiontable). You have to build this table, otherwise Python can no longer catch exceptions :-)
I don't know how to build this exception table. It seems like currently there is no Python function in the stdlib to build this table.
Example: --- def f(): try: print("raise") raise ValueError except ValueError: print("except") else: print("else") print("exit func")
def g(): pass
if 1: code = f.__code__ g.__code__ = g.__code__.replace( co_code=code.co_code, co_consts=code.co_consts, co_names=code.co_names, co_flags=code.co_flags, co_stacksize=code.co_stacksize) else: g.__code__ = f.__code__ # this code path works on Python 3.11
g() ---
Output with Python 3.10 (ok): --- raise except exit func ---
Output with Python 3.11 (oops): --- raise Traceback (most recent call last): ... ValueError ---
By the way, this change is not documented at all:
* https://docs.python.org/dev/library/types.html#types.CodeType * https://docs.python.org/dev/whatsnew/3.11.html
I understand that these changes come from the "Zero cost exception handling" change: https://bugs.python.org/issue40222
Victor -- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
Does this mean that this line in the bytecode library is likely to fail with 3.11, with no way to fix it? https://github.com/MatthieuDartiailh/bytecode/blob/7b0423234b0e999b45a4eb0c5... On Fri, 1 Apr 2022, 10:40 Victor Stinner, <vstinner@python.org> wrote:
I created https://bugs.python.org/issue47185 to discuss this issue: either recompute automatically co_exceptiontable, or at least document the change.
Victor
On Fri, Apr 1, 2022 at 11:21 AM Victor Stinner <vstinner@python.org> wrote:
("Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API")
On Fri, Apr 1, 2022 at 11:01 AM Chris Angelico <rosuav@gmail.com> wrote:
On Fri, 1 Apr 2022 at 19:51, Victor Stinner <vstinner@python.org>
wrote:
In Python, sadly the types.CodeType type also has a public constructor and many projects break at each Python release because the API changes. Hopefully, it seems like the new CodeType.replace() method added to Python 3.8 mitigated the issue. IMO CodeType.replace() is a better abstraction and closer to what developers need in practice.
It certainly has been for me. When I want to do bytecode hackery, I usually start by creating a function with def/lambda, then construct a modified function using f.__code__.replace(). It's the easiest way to ensure that all the little details are correct.
Python 3.11 added the concept of "exception table" (code.co_exceptiontable). You have to build this table, otherwise Python can no longer catch exceptions :-)
I don't know how to build this exception table. It seems like currently there is no Python function in the stdlib to build this table.
Example: --- def f(): try: print("raise") raise ValueError except ValueError: print("except") else: print("else") print("exit func")
def g(): pass
if 1: code = f.__code__ g.__code__ = g.__code__.replace( co_code=code.co_code, co_consts=code.co_consts, co_names=code.co_names, co_flags=code.co_flags, co_stacksize=code.co_stacksize) else: g.__code__ = f.__code__ # this code path works on Python 3.11
g() ---
Output with Python 3.10 (ok): --- raise except exit func ---
Output with Python 3.11 (oops): --- raise Traceback (most recent call last): ... ValueError ---
By the way, this change is not documented at all:
* https://docs.python.org/dev/library/types.html#types.CodeType * https://docs.python.org/dev/whatsnew/3.11.html
I understand that these changes come from the "Zero cost exception handling" change: https://bugs.python.org/issue40222
Victor -- Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6N6DX3JT... Code of Conduct: http://python.org/psf/codeofconduct/
On Fri, Apr 1, 2022 at 8:56 AM Gabriele <phoenix1987@gmail.com> wrote:
Does this mean that this line in the bytecode library is likely to fail with 3.11, with no way to fix it?
https://github.com/MatthieuDartiailh/bytecode/blob/7b0423234b0e999b45a4eb0c5...
Yes, that constructor is not considered stable. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
Hi Gabriele, On 01/04/2022 4:50 pm, Gabriele wrote:
Does this mean that this line in the bytecode library is likely to fail with 3.11, with no way to fix it?
You can pass the exception table the same way you pass all the other arguments. The exception table depends on the code, but that is nothing new. The bytecode library already recomputes the consts, names, etc. TBH, calling `types.CodeType` didn't work for earlier versions either. It just sort of worked, some of the time. Cheers, Mark.
https://github.com/MatthieuDartiailh/bytecode/blob/7b0423234b0e999b45a4eb0c5... <https://github.com/MatthieuDartiailh/bytecode/blob/7b0423234b0e999b45a4eb0c5...>
On Fri, 1 Apr 2022, 10:40 Victor Stinner, <vstinner@python.org <mailto:vstinner@python.org>> wrote:
I created https://bugs.python.org/issue47185 <https://bugs.python.org/issue47185> to discuss this issue: either recompute automatically co_exceptiontable, or at least document the change.
Victor
On Fri, Apr 1, 2022 at 11:21 AM Victor Stinner <vstinner@python.org <mailto:vstinner@python.org>> wrote: > > ("Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" > private C API to the internal C API") > > On Fri, Apr 1, 2022 at 11:01 AM Chris Angelico <rosuav@gmail.com <mailto:rosuav@gmail.com>> wrote: > > > > On Fri, 1 Apr 2022 at 19:51, Victor Stinner <vstinner@python.org <mailto:vstinner@python.org>> wrote: > > > In Python, sadly the types.CodeType type also has a public constructor > > > and many projects break at each Python release because the API > > > changes. Hopefully, it seems like the new CodeType.replace() method > > > added to Python 3.8 mitigated the issue. IMO CodeType.replace() is a > > > better abstraction and closer to what developers need in practice. > > > > It certainly has been for me. When I want to do bytecode hackery, I > > usually start by creating a function with def/lambda, then construct a > > modified function using f.__code__.replace(). It's the easiest way to > > ensure that all the little details are correct. > > Python 3.11 added the concept of "exception table" > (code.co_exceptiontable). You have to build this table, otherwise > Python can no longer catch exceptions :-) > > I don't know how to build this exception table. It seems like > currently there is no Python function in the stdlib to build this > table. > > Example: > --- > def f(): > try: > print("raise") > raise ValueError > except ValueError: > print("except") > else: > print("else") > print("exit func") > > def g(): pass > > if 1: > code = f.__code__ > g.__code__ = g.__code__.replace( > co_code=code.co_code, > co_consts=code.co_consts, > co_names=code.co_names, > co_flags=code.co_flags, > co_stacksize=code.co_stacksize) > else: > g.__code__ = f.__code__ # this code path works on Python 3.11 > > g() > --- > > Output with Python 3.10 (ok): > --- > raise > except > exit func > --- > > Output with Python 3.11 (oops): > --- > raise > Traceback (most recent call last): > ... > ValueError > --- > > By the way, this change is not documented at all: > > * https://docs.python.org/dev/library/types.html#types.CodeType <https://docs.python.org/dev/library/types.html#types.CodeType> > * https://docs.python.org/dev/whatsnew/3.11.html <https://docs.python.org/dev/whatsnew/3.11.html> > > I understand that these changes come from the "Zero cost exception > handling" change: > https://bugs.python.org/issue40222 <https://bugs.python.org/issue40222> > > Victor > -- > Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org <mailto:python-dev@python.org> To unsubscribe send an email to python-dev-leave@python.org <mailto:python-dev-leave@python.org> https://mail.python.org/mailman3/lists/python-dev.python.org/ <https://mail.python.org/mailman3/lists/python-dev.python.org/> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6N6DX3JT... <https://mail.python.org/archives/list/python-dev@python.org/message/6N6DX3JT...> Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/>
As the maintainer of bytecode (thanks to Victor), I expect that adding support for 3.11 will be challenging at least. However I hoped that by waiting for the first beta most changes would be at least documented. What would be the best channel to reach people that may clarify how things work starting with 3.11 ? Best Matthieu Dartiailh On Fri, Apr 1, 2022, 18:34 Mark Shannon <mark@hotpy.org> wrote:
Hi Gabriele,
On 01/04/2022 4:50 pm, Gabriele wrote:
Does this mean that this line in the bytecode library is likely to fail with 3.11, with no way to fix it?
You can pass the exception table the same way you pass all the other arguments. The exception table depends on the code, but that is nothing new. The bytecode library already recomputes the consts, names, etc.
TBH, calling `types.CodeType` didn't work for earlier versions either. It just sort of worked, some of the time.
Cheers, Mark.
https://github.com/MatthieuDartiailh/bytecode/blob/7b0423234b0e999b45a4eb0c5... < https://github.com/MatthieuDartiailh/bytecode/blob/7b0423234b0e999b45a4eb0c5...
On Fri, 1 Apr 2022, 10:40 Victor Stinner, <vstinner@python.org <mailto:
vstinner@python.org>> wrote:
I created https://bugs.python.org/issue47185 <
https://bugs.python.org/issue47185> to discuss this issue:
either recompute automatically co_exceptiontable, or at least
document
the change.
Victor
On Fri, Apr 1, 2022 at 11:21 AM Victor Stinner <vstinner@python.org
<mailto:vstinner@python.org>> wrote:
> > ("Re: C API: Move PEP 523 "Adding a frame evaluation API to
CPython"
> private C API to the internal C API") > > On Fri, Apr 1, 2022 at 11:01 AM Chris Angelico <rosuav@gmail.com
<mailto:rosuav@gmail.com>> wrote:
> > > > On Fri, 1 Apr 2022 at 19:51, Victor Stinner <
vstinner@python.org <mailto:vstinner@python.org>> wrote:
> > > In Python, sadly the types.CodeType type also has a public
constructor
> > > and many projects break at each Python release because the API > > > changes. Hopefully, it seems like the new CodeType.replace()
method
> > > added to Python 3.8 mitigated the issue. IMO
CodeType.replace() is a
> > > better abstraction and closer to what developers need in
practice.
> > > > It certainly has been for me. When I want to do bytecode
hackery, I
> > usually start by creating a function with def/lambda, then
construct a
> > modified function using f.__code__.replace(). It's the easiest
way to
> > ensure that all the little details are correct. > > Python 3.11 added the concept of "exception table" > (code.co_exceptiontable). You have to build this table, otherwise > Python can no longer catch exceptions :-) > > I don't know how to build this exception table. It seems like > currently there is no Python function in the stdlib to build this > table. > > Example: > --- > def f(): > try: > print("raise") > raise ValueError > except ValueError: > print("except") > else: > print("else") > print("exit func") > > def g(): pass > > if 1: > code = f.__code__ > g.__code__ = g.__code__.replace( > co_code=code.co_code, > co_consts=code.co_consts, > co_names=code.co_names, > co_flags=code.co_flags, > co_stacksize=code.co_stacksize) > else: > g.__code__ = f.__code__ # this code path works on Python 3.11 > > g() > --- > > Output with Python 3.10 (ok): > --- > raise > except > exit func > --- > > Output with Python 3.11 (oops): > --- > raise > Traceback (most recent call last): > ... > ValueError > --- > > By the way, this change is not documented at all: > > * https://docs.python.org/dev/library/types.html#types.CodeType <
https://docs.python.org/dev/library/types.html#types.CodeType>
https://docs.python.org/dev/whatsnew/3.11.html>
> > I understand that these changes come from the "Zero cost exception > handling" change: > https://bugs.python.org/issue40222 <
https://bugs.python.org/issue40222>
> > Victor > -- > Night gathers, and now my watch begins. It shall not end until my
death.
-- Night gathers, and now my watch begins. It shall not end until my
death.
_______________________________________________ Python-Dev mailing list -- python-dev@python.org <mailto:
python-dev@python.org>
To unsubscribe send an email to python-dev-leave@python.org <mailto:
python-dev-leave@python.org>
https://mail.python.org/mailman3/lists/python-dev.python.org/ <
https://mail.python.org/mailman3/lists/python-dev.python.org/>
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/6N6DX3JT... < https://mail.python.org/archives/list/python-dev@python.org/message/6N6DX3JT...
Code of Conduct: http://python.org/psf/codeofconduct/ <
http://python.org/psf/codeofconduct/>
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6ODPJHN7... Code of Conduct: http://python.org/psf/codeofconduct/
By beta 1 things should be stable (modulo bug fixes). But documentation may lag. If you can’t figure something out by reading the code by all means ask! On Fri, Apr 1, 2022 at 11:40 Matthieu Dartiailh <m.dartiailh@gmail.com> wrote:
As the maintainer of bytecode (thanks to Victor), I expect that adding support for 3.11 will be challenging at least. However I hoped that by waiting for the first beta most changes would be at least documented. What would be the best channel to reach people that may clarify how things work starting with 3.11 ?
Best
Matthieu Dartiailh
On Fri, Apr 1, 2022, 18:34 Mark Shannon <mark@hotpy.org> wrote:
Hi Gabriele,
On 01/04/2022 4:50 pm, Gabriele wrote:
Does this mean that this line in the bytecode library is likely to fail with 3.11, with no way to fix it?
You can pass the exception table the same way you pass all the other arguments. The exception table depends on the code, but that is nothing new. The bytecode library already recomputes the consts, names, etc.
TBH, calling `types.CodeType` didn't work for earlier versions either. It just sort of worked, some of the time.
Cheers, Mark.
https://github.com/MatthieuDartiailh/bytecode/blob/7b0423234b0e999b45a4eb0c5... < https://github.com/MatthieuDartiailh/bytecode/blob/7b0423234b0e999b45a4eb0c5...
On Fri, 1 Apr 2022, 10:40 Victor Stinner, <vstinner@python.org <mailto:
vstinner@python.org>> wrote:
I created https://bugs.python.org/issue47185 <
https://bugs.python.org/issue47185> to discuss this issue:
either recompute automatically co_exceptiontable, or at least
document
the change.
Victor
On Fri, Apr 1, 2022 at 11:21 AM Victor Stinner <vstinner@python.org
<mailto:vstinner@python.org>> wrote:
> > ("Re: C API: Move PEP 523 "Adding a frame evaluation API to
CPython"
> private C API to the internal C API") > > On Fri, Apr 1, 2022 at 11:01 AM Chris Angelico <rosuav@gmail.com
<mailto:rosuav@gmail.com>> wrote:
> > > > On Fri, 1 Apr 2022 at 19:51, Victor Stinner <
vstinner@python.org <mailto:vstinner@python.org>> wrote:
> > > In Python, sadly the types.CodeType type also has a public
constructor
> > > and many projects break at each Python release because the
API
> > > changes. Hopefully, it seems like the new CodeType.replace()
method
> > > added to Python 3.8 mitigated the issue. IMO
CodeType.replace() is a
> > > better abstraction and closer to what developers need in
practice.
> > > > It certainly has been for me. When I want to do bytecode
hackery, I
> > usually start by creating a function with def/lambda, then
construct a
> > modified function using f.__code__.replace(). It's the easiest
way to
> > ensure that all the little details are correct. > > Python 3.11 added the concept of "exception table" > (code.co_exceptiontable). You have to build this table, otherwise > Python can no longer catch exceptions :-) > > I don't know how to build this exception table. It seems like > currently there is no Python function in the stdlib to build this > table. > > Example: > --- > def f(): > try: > print("raise") > raise ValueError > except ValueError: > print("except") > else: > print("else") > print("exit func") > > def g(): pass > > if 1: > code = f.__code__ > g.__code__ = g.__code__.replace( > co_code=code.co_code, > co_consts=code.co_consts, > co_names=code.co_names, > co_flags=code.co_flags, > co_stacksize=code.co_stacksize) > else: > g.__code__ = f.__code__ # this code path works on Python
3.11
> > g() > --- > > Output with Python 3.10 (ok): > --- > raise > except > exit func > --- > > Output with Python 3.11 (oops): > --- > raise > Traceback (most recent call last): > ... > ValueError > --- > > By the way, this change is not documented at all: > > * https://docs.python.org/dev/library/types.html#types.CodeType
<https://docs.python.org/dev/library/types.html#types.CodeType>
https://docs.python.org/dev/whatsnew/3.11.html>
> > I understand that these changes come from the "Zero cost
exception
> handling" change: > https://bugs.python.org/issue40222 <
https://bugs.python.org/issue40222>
> > Victor > -- > Night gathers, and now my watch begins. It shall not end until
my death.
-- Night gathers, and now my watch begins. It shall not end until my
death.
_______________________________________________ Python-Dev mailing list -- python-dev@python.org <mailto:
python-dev@python.org>
To unsubscribe send an email to python-dev-leave@python.org
<mailto:python-dev-leave@python.org>
https://mail.python.org/mailman3/lists/python-dev.python.org/ <
https://mail.python.org/mailman3/lists/python-dev.python.org/>
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/6N6DX3JT... < https://mail.python.org/archives/list/python-dev@python.org/message/6N6DX3JT...
Code of Conduct: http://python.org/psf/codeofconduct/ <
http://python.org/psf/codeofconduct/>
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6ODPJHN7... Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JIMRLSC3... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido (mobile)
participants (5)
-
Gabriele -
Guido van Rossum -
Mark Shannon -
Matthieu Dartiailh -
Victor Stinner