Hi Joe, On 26/12/15 22:36, Joe Jevnik wrote:

The number and meaning of the arguments are documented in the dis module: https://docs.python.org/3.6/library/dis.html

OK - I *did* find that, but perhaps didn't immediately understand what it was telling me. So, something documented as "OP_CODE" is a 1-byte op, something documented as "OP_CODE(foo)" is a 2-byte op - and unless I missed one, there are no 3-byte ops? Thanks, E.

On 26/12/15 23:10, Joe Jevnik wrote:

All arguments are 2 bytes, if there needs to be more, EXTENDED_ARG is used

OK, got it - many thanks. E.

Ned also neglected to mention his byterun project which is a pure Python implementation of the CPython eval loop: https://github.com/nedbat/byterun On Sat, 26 Dec 2015, 16:38 Ned Batchelder wrote:

...

On 26/12/15 23:10, Joe Jevnik wrote:

...

All arguments are 2 bytes, if there needs to be more, EXTENDED_ARG is used

OK, got it - many thanks. One thing to understand that may not be immediately apparent: the byte code can (and does) change between versions, so Python 2.7 doesn't have

On 12/26/15 6:13 PM, Erik wrote: the exact same byte code as 3.4, which is also different from 3.5.

--Ned.

...

E. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

https://mail.python.org/mailman/options/python-dev/ned%40nedbatchelder.com

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org

On 27 December 2015 at 12:23, Guido van Rossum wrote:

On Sat, Dec 26, 2015 at 6:06 PM, Brett Cannon wrote:

...

Ned also neglected to mention his byterun project which is a pure Python implementation of the CPython eval loop: https://github.com/nedbat/byterun

From the commit log it looks like it's a co-production between Ned and Allison Kaptur (who gave the talk I mentioned).

It occurred to me that "byterun" would make a good see-also link from the dis module docs, and looking into that idea brought me to this article Allison wrote about it for the "500 lines" project: http://aosabook.org/en/500L/a-python-interpreter-written-in-python.html For a detailed semantic reference, byterun's eval loop is likely one of the most readable sources of information: https://github.com/nedbat/byterun/blob/master/byterun/pyvm2.py In terms of formal documentation, the main problem with providing reference bytecode tables is keeping them up to date as the eval loop changes. However, it would theoretically be possible to create a custom Sphinx directive that uses the dis module to generate the tables automatically during the docs build process, rather than maintaining them by hand - something like that could be experimented with outside CPython, and potentially incorporated into the dis module docs if folks are able to figure out something that works well. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Thanks for your help so far (I'm experimenting with the peephole optimizer - hence my question before as I was trying to work out how to know what the small integer hard-coded offsets should be when looking ahead in the bytecode). I've successfully added a new opcode (generated by the optimizer and understood by the interpreter loop) but when adding a second I unexpectedly got the following error. I'm not doing anything different to what I did with the first opcode as far as I can tell (I have a TARGET(FOO) in ceval.c and have obviously defined the new opcode's value in opcode.h). """ ./python -E -S -m sysconfig --generate-posix-vars ;\ if test $? -ne 0 ; then \ echo "generate-posix-vars failed" ; \ rm -f ./pybuilddir.txt ; \ exit 1 ; \ fi XXX lineno: 241, opcode: 1 Fatal Python error: Py_Initialize: can't import _frozen_importlib Traceback (most recent call last): File "<frozen importlib._bootstrap>", line 698, in <module> File "<frozen importlib._bootstrap>", line 751, in BuiltinImporter File "<frozen importlib._bootstrap>", line 241, in _requires_builtin SystemError: unknown opcode Aborted (core dumped) generate-posix-vars failed make: *** [pybuilddir.txt] Error 1 """ If I #ifdef out the code in peephole.c which generates my new (2nd) opcode, then the error does not occur. I tried a "make clean" first, but that didn't help (I realise that does not necessarily rule out a makefile dependency issue). Does anyone know if this is a well-known symptom of forgetting to add something somewhere when adding a new opcode, or do I need to track it down some more myself? I did not have this problem when introducing my first new opcode. Thanks, E.

You can look at https://docs.python.org/devguide/compiler.html to see if you missed something. As for the _frozen_importlib problem, that typically manifests itself when you have invalid bytecode (that module is frozen bytecode that gets compiled into the interpreter and is the first bit of Python code that gets run). On Sun, 27 Dec 2015, 16:41 Guido van Rossum wrote:

Can you show the diffs you have so far? Somebody's got to look at your code.

--Guido (mobile) On Dec 27, 2015 16:51, "Erik" wrote:

...

Thanks for your help so far (I'm experimenting with the peephole optimizer - hence my question before as I was trying to work out how to know what the small integer hard-coded offsets should be when looking ahead in the bytecode).

I've successfully added a new opcode (generated by the optimizer and understood by the interpreter loop) but when adding a second I unexpectedly got the following error. I'm not doing anything different to what I did with the first opcode as far as I can tell (I have a TARGET(FOO) in ceval.c and have obviously defined the new opcode's value in opcode.h).

""" ./python -E -S -m sysconfig --generate-posix-vars ;\ if test $? -ne 0 ; then \ echo "generate-posix-vars failed" ; \ rm -f ./pybuilddir.txt ; \ exit 1 ; \ fi XXX lineno: 241, opcode: 1 Fatal Python error: Py_Initialize: can't import _frozen_importlib Traceback (most recent call last): File "<frozen importlib._bootstrap>", line 698, in <module> File "<frozen importlib._bootstrap>", line 751, in BuiltinImporter File "<frozen importlib._bootstrap>", line 241, in _requires_builtin SystemError: unknown opcode Aborted (core dumped) generate-posix-vars failed make: *** [pybuilddir.txt] Error 1 """

If I #ifdef out the code in peephole.c which generates my new (2nd) opcode, then the error does not occur. I tried a "make clean" first, but that didn't help (I realise that does not necessarily rule out a makefile dependency issue).

Does anyone know if this is a well-known symptom of forgetting to add something somewhere when adding a new opcode, or do I need to track it down some more myself? I did not have this problem when introducing my first new opcode.

Thanks, E. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe:

...

https://mail.python.org/mailman/options/python-dev/guido%40python.org

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org

On 28 December 2015 at 11:00, Erik wrote:

On 28/12/15 00:41, Guido van Rossum wrote:

...

Can you show the diffs you have so far? Somebody's got to look at your code.

Sounds like it's not a well-known symptom then.

The symptom is well known (at least to folks that have worked on the compiler and eval loop since the switch to importlib as the import system implementation), but the circumstances where it can arise are *very* limited. Specifically, being unable to load the import system while working on CPython is usually a sign that: 1. The interpreter's bytecode generation is inconsistent with the implementation of the eval loop 2. importlib._bootstrap includes code that triggers the inconsistent bytecode processing path 3. Freezing importlib._bootstrap to create _frozen_importlib thus produces a frozen module that won't load with the given eval loop implementation If you're not hacking on bytecode generation or the eval loop (1), or your changes to the bytecode generator and/or eval loop don't impact the code in importlib._bootstrap (2), then you won't see this kind of bug (3).

I agree, but that Somebody should be me (initially, at least) - I don't want to waste other people's time if I made a silly mistake.

In this particular case, it's hard to help debug the error without being able to see both the new code generation changes and the corresponding eval loop changes. It's also the case that to rule out the bootstrapping cycle as a potential source of problems, you can try the following manual dance: 1. Revert to a clean checkout and rebuild 2. Apply the eval loop changes, and rebuild 3. Apply the code generation changes, and rebuild That generally *shouldn't* be necessary (it's why there's a separate build step to freeze the import system), but it can be a useful exercise to help figure out the source of the "unknown opcode" problem. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Brett Cannon wrote:

Ned also neglected to mention his byterun project which is a pure Python implementation of the CPython eval loop: <a href="https://github.com/nedbat/byterun">https://github.com/nedbat/byterun</a>

I would also encourage you to take a look at Numba. It is an LLVM based JIT compiler for Python bytecode, written for hardcore numerical algorithms in Python. It can often achieve the same performance as -O2 in C after a short burn-in while inferring the types of the arguments and variables. Using it is mostly as easy as adding an @numba.jit decorator to the function we want to accelerate. Numba is rapidly becoming what Google's long dead swallow should have been. :-) Sturla