[Python-Dev] Proto-PEP regarding writing bytecode files

[Moore, Paul]

From: Skip Montanaro [mailto:skip@pobox.com]
> Here's a first stab at a PEP about controlling generation
> of bytecode files.  Feedback appreciated.

As someone else (Martin?) mentioned, this proposal needs to be a bit
more explicit about the way *reading* of bytecode files is affected,
as well as writing.

This is of particular interest to me, as it will potentially affect
how people write import hooks. For example, I can't immediately see
how this proposal will interact with the new zipimport module. And
that module doesn't attempt to handle writing of PYC modules,
something I can imagine an import hook wanting to handle (although
I'd have to say that this is probably unusual).

In particular, the PEP needs to say something about what, if any,
requirements it places on the writers of import hooks ("If an import
hook attempts to cache compiled bytecode, in a similar way to the
builtin filesystem support, then it needs to check the environment
variable, and...")

> - Interpretation of a module's __file__ attribute.  I believe the
>   __file__ attribute of a module should reflect the true location
>   of the bytecode file.  If people want to locate a module's source
>   code, they should use imp.find_module(module).

Again, this is going to be something import hook writers need to be
aware of, and to cater for. Offhand, I can't recall what the import
hook PEP considers the fate of imp.find_module to be [... click,
click, browse, read...]

OK, if you have a module loaded from an import hook, you don't have
imp.find_module. You have a new function, imp.get_loader(), but that
returns only a loader, not a file name.

So you need to expand on this: "If people want to locate a module's
source code, they will not be able to except by using
imp.find_module(module), which does not take account of import hooks.
Whether this is good enough will depend upon the application".

[But in any case, PEP 302 points out about __file__ that "This must
be a string, but it may be a dummy value, for example "<frozen>""
so anyone using __file__ already needs to be aware of border cases
involving non-filesystem modules]

Personally, I have no real need for, nor a great interest in, this
feature. But others may care a lot about it, and have little need for
import hooks. I therefore can't put a good judgement on whether these
issues are significant - but I do think the PEP needs to address them,
if only to say that the two features don't work well together, and
that is understood and not considered to be a significant issue.

Paul.

PS It's also possible that this issue should be integrated into the
   import hook protocol somehow - have the loader.load_module()
   method take a "save_bytecode" parameter, or something. For a
   simple yes/no flag, that makes sense, but for a more general
   bytecode cache location, that wouldn't be sufficient.