[Python-ideas] PEP 511: API for code transformers

[Victor Stinner]

I have a fully working implementation of the PEP 509, 510 and 511 (all
together). You can install it to play with it if you want ;-)

Get and compile patched (FAT) Python with:
--------------
hg clone http://hg.python.org/sandbox/fatpython
cd fatpython
./configure && make
--------------


Enjoy slow and non optimized bytecode :-)
-------------
$ ./python -o noopt -c 'import dis; dis.dis(compile("1+1", "test", "exec"))'
  1           0 LOAD_CONST               0 (1)
              3 LOAD_CONST               0 (1)
              6 BINARY_ADD
              7 POP_TOP
              8 LOAD_CONST               1 (None)
             11 RETURN_VALUE
-------------


Ok, now if you want to play with fat & fatoptimizer modules (FAT Python):
--------------
./python -m venv ENV
cd ENV
git clone https://github.com/haypo/fat.git
git clone https://github.com/haypo/fatoptimizer.git
(cd fat; ../bin/python setup.py install)
(cd fatoptimizer; ../bin/python setup.py install)
cd ..
--------------

I'm not using virtual environment for my development, I prefer to copy
manually fatoptimizer/fatoptimizer/ directory and the build .so file
of the fat module into the Lib/ directory of the standard library. If
you installed the patched Python into /opt/fatpython (./confgure
--prefix=/opt/fatpython && make && sudo make install), you can also
use "python setup.py install" in fat/ and fatoptimizer/ to install
them easily.

The drawback of the virtualenv is that it's easy to use the wrong
python (./python vs ENV/bin/python) and don't have FAT Python enabled
because of  http://bugs.python.org/issue26099 which ignores silently
import errors in sitecustomize...

Ensure that FAT Python is enabled with:
--------
$ ./python -X fat -c 'import sys; print(sys.implementation.optim_tag)'
fat-opt
--------
You must get "fat-opt" (and not "opt").

Note: The optimizer tag is "fat-opt" and not "fat" because
fatoptimizer keeps the peephole optimizer.


Enable FAT Python using the "-X fat" command line option:
--------------
$ ENV/bin/python -X fat
>>> def func(): return len("abc")
...

>>> import dis
>>> dis.dis(func)
  1           0 LOAD_GLOBAL              0 (len)
              3 LOAD_CONST               1 ('abc')
              6 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
              9 RETURN_VALUE

>>> import fat
>>> fat.get_specialized(func)
[(<code object func at 0x7f9d3155b1e0, file "<stdin>", line 1>,
[<fat.GuardBuiltins object at 0x7f9d39191198>])]

>>> dis.dis(fat.get_specialized(func)[0][0])
  1           0 LOAD_CONST               1 (3)
              3 RETURN_VALUE
--------------


Play with microbenchmarks:
---------------
$ ENV/bin/python -m timeit -s 'def f(): return len("abc")' 'f()'
10000000 loops, best of 3: 0.122 usec per loop

$ ENV/bin/python -X fat -m timeit -s 'def f(): return len("abc")' 'f()'
10000000 loops, best of 3: 0.0932 usec per loop
---------------
Oh look! It's faster without having to touch the code ;-)


I'm using Lib/sitecustomize.py to register the optimizer if -X fat is used:
-------
import sys
if sys._xoptions.get('fat'):
    import fatoptimizer; fatoptimizer._register()
-------


If you want to run optimized code without registering the optimizer,
it doesn't work because .pyc are missing:
---
$ ENV/bin/python -o fat-opt
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ImportError: missing AST transformers for
'.../Lib/encodings/__init__.py': optim_tag='fat', transformers
tag='noopt'
---


You have to compile optimized .pyc files:
---
# the optimizer is slow, so add -v to enable fatoptimizer logs for more fun
ENV/bin/python -X fat -v -m compileall
# why does compileall not compile encodings/*.py?
ENV/bin/python -X fat -m py_compile
/home/haypo/prog/python/fatpython/Lib/encodings/{__init__,aliases,latin_1,utf_8}.py
---


Finally, enjoy optimized code with no registered optimized:
---
# hum, use maybe ENV/bin/activate instead of my magic tricks
$ export PYTHONPATH=ENV/lib/python3.6/site-packages/

$ ENV/bin/python -o fat-opt -c 'import sys;
print(sys.implementation.optim_tag, sys.get_code_transformers())'
fat-opt []
---


Remember that you cannot import .py files in this case, only .pyc:
---
$ touch x.py
$ ENV/bin/python -o fat-opt -c 'import x'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: missing AST transformers for '.../x.py':
optim_tag='fat-opt', transformers tag='noopt'
---

Victor