[Python-ideas] Incorporating something like byteplay into the stdlib

[Andrew Barnert]

On Feb 12, 2016, at 02:57, Petr Viktorin <encukou at gmail.com> wrote:
> 
> Are you sure the API you come up with will be good enough to be
> immortalized in CPython itself?

That's exactly why I'm sticking as close as possible to the existing dis module--which is already immortalized in CPython--and leaning on the byteplay module--which has clearly won the competition in the wild--for things that dis doesn't cover.

> Why not put it on PyPI, and only look into immortalizing it after it
> survives some real use (and competition)?

You want me to put a module on PyPI that patches the compiler and the peephole optimizer? I doubt that's even possible, and it would be a very bad idea if it were.

If you weren't reading the proposal, just skimming for individual things to disagree with, you might have missed that the main idea is to provide a useful interface for the compiler to pass to PyCode_Optimize, and exposing functions to convert back and forth to that format a la byteplay is a secondary idea that adds the benefits that import hooks and decorators could then use the same interface as PEP 511 optimizers. You can't do the secondary thing without the main thing.

>> Anyway, the advantage here would be that import hooks and decorators can use the exact same API as PEP 511 transformers, except they have to call to_code at the end instead of just returning an object and letting the compiler call it for them. (They'll also probably have to recurse manually on (dis.Bytecode(const) for const in co_consts if isinstance(const, types.CodeType)), while PEP 511 transformers won't.)
> 
> Why can't Byteplay be fixed? (Or modularized, or rewritten?)

If the problem is the interface to the peephole optimizer and PEP 511 optimizers, fixing byteplay doesn't help that.

If we did fix the PyCode_Optimize interface, then sure, instead of exposing the same code, we could keep it hidden and separately change byteplay to duplicate the same interface we could have exposed. But why?

Also, byteplay is actually a very small module, and the only hard part of it is the part that keeps in sync with the dis module across each new Python version and does the same things backward-compatibly for pre-3.4 versions. There's really nothing to "modularize" there. If 3.6 had an extended dis module and C API as I've suggested, in 3.6+ it could just become a shim around what's already in the stdlib, which would then be useful for backward compat for anyone who wants to write bytecode processing import hooks or decorators that work with 3.5 and 3.6, or maybe even with 2.7 and 3.6. I think that probably _is_ worth doing, but that's not a proposal for stdlib, it's a proposal for byteplay (and only if this proposal--and PEP 511, of course--goes through).

>> So, what's wrong with that? The biggest problem is that, after each new Python release, anyone using a bytecode transformer will have to wait until byteplay is updated before they can update Python.
> 
> That's a problem common to any library. You always need to wait until
> your dependencies are ported to a new version of a language.

In theory, yes. In practice, when Python 3.6 comes out, every library in my site-packages from 3.5 will continue to work, most of them without even needing a recompile, except for byteplay, and that's been the same for almost every release in Python's history. So that does make it special in practice.

> Byteplay looks special because it explicitly depends on an interface
> with no backwards compatibility guarantees. But once everyone starts
> writing bytecode transformers, it won't be special any more: *any*
> bytecode utility library will have the same problem. Do we add them all
> to force them to follow Python's release cycle?

Since byteplay is the only one that's still alive and being updated, and "all of them" is just one, yes, that's exactly what we do.

And "forcing it to follow Python's release cycle" is not a problem. The only time there are significant updates to the library is when there's a new version of Python to deal with, or when someone finds a bug in the way it deals with the latest version of Python that wasn't discovered immediately.

At any rate, if everyone begins writing bytecode transformers, that makes the existing problem worse, it doesn't solve it.

>> Also, some people are going to try to do it from scratch instead of adding a dependency, and get it wrong, and publish processors that work on most code but in weird edge cases cause a SystemError or even a segfault which will be a nightmare to debug.
> 
> Shame on them. But is this really a reason to include one blessed
> convenience interface in the language itself?

Yes. Expecting people to update the lnotab, etc., as currently designed is a bad idea, especially given the way things go wrong if they get it wrong. If we're providing an interface that exposes this stuff to normal developers and expects them to deal with it, we should have tools to make it tractable. 

>> And it's annoying that anyone who wants to hack on the bytecode representation pretty much has to disable the peephole optimizer.
> 
> I'm sure this individual problem can be solved in a less invasive way.

I'm not. Short of rewriting the peephole optimizer to a new interface, how would you solve it?

I think this is a pretty minor problem in the first place (how often do people want to hack on the bytecode representation? and how often do people do so who aren't pretty experienced with Python and CPython?), which is why I tossed it in at the end, after the real problems.

>> Or we could just remove bytecode transformers from PEP 511. PEP 511 still seems worth doing to me, even if it only has AST transformers, especially since all or nearly all of the examples anyone's come up with for it are implementable (and easier to implement) at the AST level.
> 
> This looks like an OK solution. It can always be added back.

Well, I assume Victor has an argument for why it's not OK, but I think it would make things simpler.