On Wed, 27 Jan 2016 at 20:57 Kevin Conway <kevinjacobconway@gmail.com> wrote:

I'm willing to take this conversation offline as it seems this thread has cooled down quite a bit.

I would still like to hear more, though, about how adding this as a facility in the language improves over the current, external implementations of Python code optimizers. Python already has tools for reading in source files, parsing them into AST, modifying that AST, and writing the final bytecode to files as part of the standard library. I don't see anything in PEP0511 that improves upon that.

Out of curiosity, do you consider this PEP as adding something to Python that didn't previously exist or do you consider this PEP to more aligned with PEP0249 (DB2API) and PEP0484 (Type Hints) which are primarily designed to marshal the community in a common direction? I understand that you have other PEPs in flight that are designed to make certain optimizations easier (or possible). Looking at this PEP in isolation, however, leaves me wanting more explanation as to its value.

The PEP is about empowering people to write AST transformers without having to use third-party tools to integrate it into their workflow. As you pointed out, there is very little here that isn't possible today with some toolchain that reads Python source code, translates it into an AST, optimizes it, and then writes out the .pyc file. But that all does require going to PyPI or writing your own solution.

But if Victor's PEP gets in, then there will be a standard hook point that all Python code will go through which will make adding AST transformers much easier. Whether this ease of use is beneficial is part of the discussion around this PEP.

You mention the need for monkey-patching or hooking into the import process as a part of the rational. The PyCC project, while it may not be the best example for optimizer design, does not need to patch or hook into any thing to function. Instead, it acts as an alternative bytecode compiler that drops .pyc just like the standard compiler would. Other than the trade-off of using a 3rd party library versus adding a -o flag, what significant advantage does a sys.add_optimizer() call provide?

The -o addition is probably the biggest thing the PEP is proposing. The overwriting of .pyc files with optimizations that are not necessarily expected is not the best, so -o would allow for stopping the abuse of .pyc file naming. The AST registration parts is all just to make this stuff easier.

-Brett

Again, I'm very much behind your motivation and hope you are incredibly successful in making Python a faster place to live. I'm only trying to get in your head and see what you see.

On Wed, Jan 27, 2016 at 10:45 AM Victor Stinner <victor.stinner@gmail.com> wrote:
Hi,

2016-01-16 17:56 GMT+01:00 Kevin Conway <kevinjacobconway@gmail.com>:
> I'm a big fan of your motivation to build an optimizer for cPython code.
> What I'm struggling with is understanding why this requires a PEP and
> language modification. There are already several projects that manipulate
> the AST for performance gains such as [1] or even my own ham fisted attempt
> [2].

Oh cool, I didn't know PyCC [2]! I added it to the PEP 511 in the AST
optimizers section of Prior Art.

I wrote astoptimizer [1] and this project uses monkey-patching of the
compile() function, I mentioned this monkey-patching hack in the
rationale of the PEP:
https://www.python.org/dev/peps/pep-0511/#rationale

I would like to avoid monkey-patching because it causes various issues.

The PEP 511 also makes transformations more visible: transformers are
explicitly registered in sys.set_code_transformers() and the .pyc
filename is modified when the code is transformed.

It also adds a new feature: it becomes possible to run transformed
code without having to register the tranformer at runtime. This is
made possible with the addition of the -o command line option.

Victor

_______________________________________________
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/