pypy specific code flags DONT_IMPLY_DEDENT and SOURCE_IS_UTF8
Hi - I have a cross-version Python bytecode disassembler xdis ( https://pypi.org/project/xdis/) and I notice that flags 0x0100 and 0x0200 (DONT_IMPLY_DEDENT and SOURCE_IS_UTF8 respectively) conflict in Pypy 3.6 with Python 3.6's ITERABLE_COROUTINE and ASYNC_GENERATOR. However in pypy3.6-7.2.0, inspect.CO_ITERABLE_COROUTINE and inspect.CO_ASYNC_GENERATOR are defined to their Python 3.6 values. So what should a disassembler do which it sees 0x0100 as a flag in a code object? Does that reflect the PyPy meaning or the 3.6 meaning? And how can one tell if the code has the other property? Thanks.
Hi, On Mon, 16 Dec 2019 at 11:24, Rocky Bernstein <rb@dustyfeet.com> wrote:
I have a cross-version Python bytecode disassembler xdis (https://pypi.org/project/xdis/) and I notice that flags 0x0100 and 0x0200 (DONT_IMPLY_DEDENT and SOURCE_IS_UTF8 respectively) conflict in Pypy 3.6 with Python 3.6's ITERABLE_COROUTINE and ASYNC_GENERATOR.
Looking in the CPython 3.6 source code, I see that PyCF_DONT_IMPLY_DEDENT = 0x0200 as well there, even though ``inspect.CO_ASYNC_GENERATOR`` is also 0x0200. The same with PyCF_SOURCE_IS_UTF8 = 0x0100 and ``inspect.CO_ITERABLE_COROUTINE``. So the conflict you're talking about seems to exist in CPython 3.6 too. Can you give us a concrete example of code where CPython 3.6 differs from PyPy 3.6? A bientôt, Armin.
I did a little test, and for this program: # from 3.7 test_asyncgen.py def test_async_gen_iteration_01(self): async def gen(): await awaitable() a = yield 123 the 3.6 ASYNC_GENERATOR flag is added in the code object created for "gen()" as 3.6 does. So I infer that flag 0x200 in a code object doesn't imply the PYPY_DONT_IMPLY_DEDENT property. Does that mean that in 3.6 PYPY_DON'T_IMPLY_DEDENT flag will *never *appear in a code object? Or just for 3.6 and above or for all Pypy versions? I don't want to try guessing what the principle being used is looking at all versions of Pypy across all the pypy-specific flags listed in pypy/interpreter/astcompiler/consts.py if there is a principle being followed and someone can just tell me what it is or point to a document that describes what's up here. On Mon, Dec 16, 2019 at 5:33 AM Armin Rigo <armin.rigo@gmail.com> wrote:
CPython 3.6 bytecode differs from PyPy 3.6 bytecode and so far as I know it CPython bytecode version x.y differs from Pypy version x.y Otherwise I wouldn't have to go through the extra effort to try to add code to make Pypy disassembly and decompilation possible in a cross-version way. I have for example that there are opcodes FORMAT_VALUE, BUILD_STRING and JUMP_IF_NOT_DEBUG for example in Pypy3.6 that are not in CPython (among I think other opcodes). But again I'd prefer not to guess this kind of thing if it is documented somewhere or someone who knows this already can just let me know.
A bientôt,
Armin.
Hi Rocky, On Mon, 16 Dec 2019 at 11:57, Rocky Bernstein <rb@dustyfeet.com> wrote:
I could probably dig the answer, but the general rule is that PyPy should do the same thing as CPython. These two PyCF_XXX flags are passed as arguments to compile(), and should never end up inside the co_flags object; the CO_XXX flags are what ends up there.
I meant, an example where the flags that are produced are different. It's entirely possible that there is a problem somewhere, e.g. we might stick the wrong kind of flags inside co_flags. If you're asking this question here in the first place, then maybe you already have such an example? If not, then you can assume that there is no difference as far as the flags are concerned; the only difference is that PyPy has got a very small number of extra opcodes. (Note that this includes JUMP_IF_NOT_DEBUG, but not FORMAT_VALUE and BUILD_STRING, which are from CPython.) A bientôt, Armin.
Thanks - got it. Many thanks for the clarification. I reread and missed the part where you mentioned that those two flags are conflicts in CPython as well. I am sorry for my confusion there. (I'll ask CPython folks what's up if there is need). On Mon, Dec 16, 2019 at 6:05 AM Armin Rigo <armin.rigo@gmail.com> wrote:
Hi, On Mon, 16 Dec 2019 at 11:24, Rocky Bernstein <rb@dustyfeet.com> wrote:
I have a cross-version Python bytecode disassembler xdis (https://pypi.org/project/xdis/) and I notice that flags 0x0100 and 0x0200 (DONT_IMPLY_DEDENT and SOURCE_IS_UTF8 respectively) conflict in Pypy 3.6 with Python 3.6's ITERABLE_COROUTINE and ASYNC_GENERATOR.
Looking in the CPython 3.6 source code, I see that PyCF_DONT_IMPLY_DEDENT = 0x0200 as well there, even though ``inspect.CO_ASYNC_GENERATOR`` is also 0x0200. The same with PyCF_SOURCE_IS_UTF8 = 0x0100 and ``inspect.CO_ITERABLE_COROUTINE``. So the conflict you're talking about seems to exist in CPython 3.6 too. Can you give us a concrete example of code where CPython 3.6 differs from PyPy 3.6? A bientôt, Armin.
I did a little test, and for this program: # from 3.7 test_asyncgen.py def test_async_gen_iteration_01(self): async def gen(): await awaitable() a = yield 123 the 3.6 ASYNC_GENERATOR flag is added in the code object created for "gen()" as 3.6 does. So I infer that flag 0x200 in a code object doesn't imply the PYPY_DONT_IMPLY_DEDENT property. Does that mean that in 3.6 PYPY_DON'T_IMPLY_DEDENT flag will *never *appear in a code object? Or just for 3.6 and above or for all Pypy versions? I don't want to try guessing what the principle being used is looking at all versions of Pypy across all the pypy-specific flags listed in pypy/interpreter/astcompiler/consts.py if there is a principle being followed and someone can just tell me what it is or point to a document that describes what's up here. On Mon, Dec 16, 2019 at 5:33 AM Armin Rigo <armin.rigo@gmail.com> wrote:
CPython 3.6 bytecode differs from PyPy 3.6 bytecode and so far as I know it CPython bytecode version x.y differs from Pypy version x.y Otherwise I wouldn't have to go through the extra effort to try to add code to make Pypy disassembly and decompilation possible in a cross-version way. I have for example that there are opcodes FORMAT_VALUE, BUILD_STRING and JUMP_IF_NOT_DEBUG for example in Pypy3.6 that are not in CPython (among I think other opcodes). But again I'd prefer not to guess this kind of thing if it is documented somewhere or someone who knows this already can just let me know.
A bientôt,
Armin.
Hi Rocky, On Mon, 16 Dec 2019 at 11:57, Rocky Bernstein <rb@dustyfeet.com> wrote:
I could probably dig the answer, but the general rule is that PyPy should do the same thing as CPython. These two PyCF_XXX flags are passed as arguments to compile(), and should never end up inside the co_flags object; the CO_XXX flags are what ends up there.
I meant, an example where the flags that are produced are different. It's entirely possible that there is a problem somewhere, e.g. we might stick the wrong kind of flags inside co_flags. If you're asking this question here in the first place, then maybe you already have such an example? If not, then you can assume that there is no difference as far as the flags are concerned; the only difference is that PyPy has got a very small number of extra opcodes. (Note that this includes JUMP_IF_NOT_DEBUG, but not FORMAT_VALUE and BUILD_STRING, which are from CPython.) A bientôt, Armin.
Thanks - got it. Many thanks for the clarification. I reread and missed the part where you mentioned that those two flags are conflicts in CPython as well. I am sorry for my confusion there. (I'll ask CPython folks what's up if there is need). On Mon, Dec 16, 2019 at 6:05 AM Armin Rigo <armin.rigo@gmail.com> wrote:
participants (2)
-
Armin Rigo -
Rocky Bernstein