[Python-Dev] Inconsistent behaviour in import/zipimport hooks

[Brett Cannon]

On 11/9/05, Guido van Rossum <guido at python.org> wrote:
> On 11/9/05, Brett Cannon <bcannon at gmail.com> wrote:
> > Plus I wouldn't be
> > surprised if we started to move away from bytecode optimization and
> > instead tried to do more AST transformations which would remove
> > possible post-load optimizations.
> >
> > I would have  no issue with removing .pyo files and have .pyc files
> > just be as optimized as they  the current settings are and leave it at
> > that.  Could have some metadata listing what optimizations occurred,
> > but do we really need to have a specific way to denote if bytecode has
> > been optimized?  Binary files compiled from C don't note what -O
> > optimization they were compiled with.  If someone distributes
> > optimized .pyc files chances are they are going to have a specific
> > compile step with py_compile and they will know what optimizations
> > they are using.
>
> Currently, .pyo files have some important semantic differences with
> .pyc files; -O doesn't remove docstrings (that's -OO) but it does
> remove asserts. I wouldn't want to accidentally use a .pyc file
> without asserts compiled in unless the .py file wasn't around.
>
> For application distribution, the following probably would work:
>
> - instead of .pyo files, we use .pyc files
> - the .pyc file records whether optimizations were applied, whether
> asserts are compiled, and whether docstrings are retained
> - if the compiler finds a .pyc that is inconsistent with the current
> command line, it ignores it and rewrites it (if it is writable) just
> as if the .py file were newer
>
> However, this would be a major pain for the standard library and other
> shared code -- there it's really nice to have a cache for each of the
> optimization levels since usually regular users can't write the
> .py[co] files there, meaning very slow always-recompilation if the
> standard .pyc files aren't of the right level, causing unacceptable
> start-up times.
>

What if PEP 304 came into being?  Then people would have a place to
have the shared code's recompiled version stored and thus avoid the
overhead from repeated use.

> The only solutions I can think of that use a single file actually
> *increase* the file size by having unoptimized and optimized code
> side-by-side, or some way to quickly skip the assertions -- the -OO
> option is a special case that probably needs to be done differently
> anyway and only for final distribution.
>

One option would be to introduce an ASSERTION bytecode that has an
argument specifying the amount of bytecode for the assertion.  The
eval loop can then just igonore the bytecode if assertions are being
evaluated and fall through to the bytecode for the assertions (and
thus be the equivalent of NOP) or use the argument to jump forward
that number of bytes in the bytecode and completely skip over the
assertion (and thus be just like a JUMP_FORWARD).  Either way
assertions becomes slightly more costly but it should be very minimal.

-Brett