[Python-Dev] Proposal: .pyc file format change

[Peter Funk]

[Guido van Rossum]:
> Given Christian Tismer's testimonial and inspection of marshal.c, I
> think Peter's small patch is acceptable.
> 
> A bigger question is whether we should freeze the magic number and add
> a version number.  In theory I'm all for that, but it means more
> changes; there are several tools (e.c. Lib/py_compile.py,
> Tools/freeze/modulefinder.py and Tools/scripts/checkpyc.py) that have
> intimate knowledge of the .pyc file format that would have to be
> modified to match.
> 
> The current format of a .pyc file is as follows:
> 
> bytes 0-3   magic number
> bytes 4-7   timestamp (mtime of .py file)
> bytes 8-*   marshalled code object

Proposal:
The future format (Python 1.6 and newer) of a .pyc file should be as follows:

bytes 0-3   a new magic number, which should be definitely frozen in 1.6.
bytes 4-7   a version number (which should be == 1 in Python 1.6)
bytes 8-11  timestamp (mtime of .py file) (same as earlier)
bytes 12-*  marshalled code object (same as earlier)

> The magic number itself is used to convey various bits of information,
> all implicit:
[...]
This mechanism to construct the magic number should not be changed.

But now once again a new value must be choosen to prevent havoc 
with .pyc files floating around, where people already played with the 
Python 1.6 alpha releases.  But this change should be definitely the 
last one, which will ever happen during the future life time of Python.

The unmarshaller should do the following with the magic number read:
If the read magic is the old magic number from 1.5.2, skip reading a
version number and assume 0 as the version number.  

If the read magic is this new value instead, it should also read the 
version number and raise a new 'ByteCodeToNew' exception, if the read 
version number is greater than a #defind version number of this 
Python interpreter.  

If future incompatible extensions to the byte code format will happen, 
then this number should be incremented to 2, 3 and so on.

For safety, 'imp.get_magic()' should return the old 1.5.2 magic
number and only 'imp.get_magic(imp.PYC_FINAL)' should return the new 
final magic number.  A new function 'imp.get_version()' should be 
introduced, which will return the current compiled in version number
of this Python interpreter.

Of course all Python modules reading .pyc files must be changed 
ccordingly, so that are able to deal with new .pyc files.  
This shouldn't be too hard.

This proposed change of the .pyc file format must be described in the final
Python 1.6 annoucement, if there are people out there, who borrowed
code from 'Tools/scripts/checkpyc.py' or some such.

Regards, Peter