[Python-Dev] Inconsistent behaviour in import/zipimport hooks

[Guido van Rossum]

On 11/9/05, Brett Cannon <bcannon at gmail.com> wrote:
> On 11/9/05, Guido van Rossum <guido at python.org> wrote:
> > Maybe it makes more sense to deprecate .pyo altogether and instead
> > have a post-load optimizer optimize .pyc files according to the
> > current optimization settings?
>
> But I thought part of the point of .pyo files was that they left out
> docstrings and thus had a smaller footprint?

Very few people care about the smaller footprint (although one piped up here).

> Plus I wouldn't be
> surprised if we started to move away from bytecode optimization and
> instead tried to do more AST transformations which would remove
> possible post-load optimizations.
>
> I would have  no issue with removing .pyo files and have .pyc files
> just be as optimized as they  the current settings are and leave it at
> that.  Could have some metadata listing what optimizations occurred,
> but do we really need to have a specific way to denote if bytecode has
> been optimized?  Binary files compiled from C don't note what -O
> optimization they were compiled with.  If someone distributes
> optimized .pyc files chances are they are going to have a specific
> compile step with py_compile and they will know what optimizations
> they are using.

Currently, .pyo files have some important semantic differences with
.pyc files; -O doesn't remove docstrings (that's -OO) but it does
remove asserts. I wouldn't want to accidentally use a .pyc file
without asserts compiled in unless the .py file wasn't around.

For application distribution, the following probably would work:

- instead of .pyo files, we use .pyc files
- the .pyc file records whether optimizations were applied, whether
asserts are compiled, and whether docstrings are retained
- if the compiler finds a .pyc that is inconsistent with the current
command line, it ignores it and rewrites it (if it is writable) just
as if the .py file were newer

However, this would be a major pain for the standard library and other
shared code -- there it's really nice to have a cache for each of the
optimization levels since usually regular users can't write the
.py[co] files there, meaning very slow always-recompilation if the
standard .pyc files aren't of the right level, causing unacceptable
start-up times.

The only solutions I can think of that use a single file actually
*increase* the file size by having unoptimized and optimized code
side-by-side, or some way to quickly skip the assertions -- the -OO
option is a special case that probably needs to be done differently
anyway and only for final distribution.

--
--Guido van Rossum (home page: http://www.python.org/~guido/)