Fred, Thank you for your quick response. Fred L. Drake:

Peter, In theory, perhaps it could; I don't know if the extra work is worth it, however. [...] Do you have reason to suspect that either case is sufficiently common to complicate the .pyc loader, or is there another reason that I've missed (very possible, I admit)?

Well, currently we (our company) deliver no source code to our customers. I don't want to discuss this policy and the reasoning behind here. But this situation may also apply to other commercial software vendors using Python. During late 2000 there may be several customers out there running Python 1.6 and others still running Python 1.5.2. So we will have several choices to deal with this situation: 1. Supply two different binary distribution packages: one containing 1.5.2 .pyc files and one containing 1.6 .pyc files. This will introduce some new logistic problems. 2. Upgrade to Python 1.6 at each customer site at once. This will be difficult. 3. Patch the 1.6 unmarshaller to support loading 1.5.2 .pyc files and supply our own patched Python distribution. (and this would also be "carrying owls to Athen" for Linux systems) [I don't know whether this ^^^^^^^^^^^^^^^^^^^^^^ careless translated german idiom makes any sense in english ;-) ] I personally don't like this. 4. Change our policy and distribute also the .py sources. Beside the difficulty to convince the management about this one, this also introduces new technical "challenges". The Unix text files have to be converted from LF lineends to CR lineends or MacPython wouldn't be able to parse the files. So the mac source distributions must be build from a different directory tree. No choice looks very attractive. Adding a '|| (magic == 0x994e)' or some such somewhere in the 1.6 unmarshaller should do the trick. But I don't want to submit a patch, if God^H^HGuido thinks, this isn't worth the effort. <wink> Regards, Peter -- Peter Funk, Oldenburger Str.86, D-27777 Ganderkesee, Germany, Fax:+49 4222950260 office: +49 421 20419-0 (ArtCom GmbH, Grazer Str.8, D-28359 Bremen)

On Tue, 23 May 2000, Eric S. Raymond wrote:

Peter Funk <pf@artcom-gmbh.de>:

...

(and this would also be "carrying owls to Athen" for Linux systems) [I don't know whether this ^^^^^^^^^^^^^^^^^^^^^^ careless translated german idiom makes any sense in english ;-) ]

There is a precise equivalent: "carrying coals to Newcastle".

That's interesting... I've never heard either, but I think I can guess the meaning now. I agree; it looks like there's some work to do in getting the .pyc loader to be a little more concerned about importing compatible marshal formats. I have an idea about how I'd like to see in done which may be a little less magical. I'll work up a patch later this week. I won't check in any changes for this until we've heard from Guido on the matter, and he'll probably be unavailable for the next couple of days. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>

Guido van Rossum wrote:

...

3. Patch the 1.6 unmarshaller to support loading 1.5.2 .pyc files

I agree that this is the correct solution.

...

No choice looks very attractive. Adding a '|| (magic == 0x994e)' or some such somewhere in the 1.6 unmarshaller should do the trick. But I don't want to submit a patch, if God^H^HGuido thinks, this isn't worth the effort. <wink>

That's BDFL for you, thank you. ;-)

Before accepting the trivial patch, I would like to see some analysis that shows that in fact all 1.5.2 .pyc files work correctly with 1.6. This means you have to prove that (a) the 1.5.2 marshal format is a subset of the 1.6 marshal format (easy enough probably) and (b) the 1.5.2 bytecode opcodes are a subset of the 1.6 bytecode opcodes. That one seems a little trickier; I don't remember if we moved opcodes or changed existing opcodes' semantics. You may be lucky, but it will cause an extra constraint on the evolution of the bytecode, so I'm somewhat reluctant.

Be assured, I know the opcodes by heart. We only appended to the end of opcode space, there are no changes. But I can't tell about marshal. ciao - chris -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaunstr. 26 : *Starship* http://starship.python.net 14163 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF where do you want to jump today? http://www.stackless.com

Peter Funk <pf@artcom-gmbh.de>:

BTW: import.c contains the following comment: /* XXX Perhaps the magic number should be frozen and a version field added to the .pyc file header? */

Judging from my decade long experience with exotic image and CAD data formats I think this is always the way to go for binary data files. Using this method newer versions of a program can always recognize the file format version and convert files generated by older versions in an appropriate way.

I have similar experience, notably with hacking graphics file formats. I concur with this recommendation. -- <a href="http://www.tuxedo.org/~esr">Eric S. Raymond</a> The end move in politics is always to pick up a gun. -- R. Buckminster Fuller

Given Christian Tismer's testimonial and inspection of marshal.c, I think Peter's small patch is acceptable. A bigger question is whether we should freeze the magic number and add a version number. In theory I'm all for that, but it means more changes; there are several tools (e.c. Lib/py_compile.py, Tools/freeze/modulefinder.py and Tools/scripts/checkpyc.py) that have intimate knowledge of the .pyc file format that would have to be modified to match. The current format of a .pyc file is as follows: bytes 0-3 magic number bytes 4-7 timestamp (mtime of .py file) bytes 8-* marshalled code object The magic number itself is used to convey various bits of information, all implicit: - the Python version - whether \r and \n are swapped (some old Mac compilers did this) - whether all string literals are Unicode (experimental -U flag) The current (1.6) value of the magic number (as a string -- the .pyc file format is byte order independent) is '\374\304\015\012' on most platforms; it's '\374\304\012\015' for the old Mac compilers mentioned; and it's '\375\304\015\012' with -U. Can anyone come up with a proposal? I'm swamped! --Guido van Rossum (home page: http://www.python.org/~guido/)

Peter Funk wrote:

[Guido van Rossum]:

...

Given Christian Tismer's testimonial and inspection of marshal.c, I think Peter's small patch is acceptable.

A bigger question is whether we should freeze the magic number and add a version number. In theory I'm all for that, but it means more changes; there are several tools (e.c. Lib/py_compile.py, Tools/freeze/modulefinder.py and Tools/scripts/checkpyc.py) that have intimate knowledge of the .pyc file format that would have to be modified to match.

The current format of a .pyc file is as follows:

bytes 0-3 magic number bytes 4-7 timestamp (mtime of .py file) bytes 8-* marshalled code object

Proposal: The future format (Python 1.6 and newer) of a .pyc file should be as follows:

bytes 0-3 a new magic number, which should be definitely frozen in 1.6. bytes 4-7 a version number (which should be == 1 in Python 1.6) bytes 8-11 timestamp (mtime of .py file) (same as earlier) bytes 12-* marshalled code object (same as earlier)

This will break all tools relying on having the code object available in bytes[8:] and believe me: there are lots of those around ;-) You cannot really change the file header, only add things to the end of the PYC file... Hmm, or perhaps we should move the version number to the code object itself... after all, the changes we want to refer to using the version number are located in the code object and not the PYC file layout. Unmarshalling it would then raise the error. -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

Peter Funk wrote:

[M.-A. Lemburg]:

...

...

Proposal: The future format (Python 1.6 and newer) of a .pyc file should be as follows:

bytes 0-3 a new magic number, which should be definitely frozen in 1.6. bytes 4-7 a version number (which should be == 1 in Python 1.6) bytes 8-11 timestamp (mtime of .py file) (same as earlier) bytes 12-* marshalled code object (same as earlier)

This will break all tools relying on having the code object available in bytes[8:] and believe me: there are lots of those around ;-)

In some way, this is intentional: If these tools (are there are really that many out there, that munge with .pyc byte code files?) simply use 'imp.get_magic()' and then silently assume a specific content of the marshalled code object, they probably need changes anyway, since the code needed to deal with the new unicode object is missing from them.

That's why I proposed to change the marshalled code object and not the PYC file: the problem is not only related to PYC files, it touches all areas where marshal is used. If you try to load a code object using Unicode in Python 1.5 you'll get all sorts of errors, e.g. EOFError, SystemError. Since marshal uses a specific format, that format should receive the version number. Ideally that version would be prepended to the format (not sure whether this is possible), so that the PYC file layout would then look like this: word 0: magic word 1: timestamp word 2: version in the marshalled code object word 3-*: rest of the marshalled code object Please make sure that options such as the -U option are also respected... -- A different approach to all this would be fixing only the first two bytes of the magic word, e.g. byte 0: 'P' byte 1: 'Y' byte 2: version number (counting from 1) byte 3: option byte (8 bits: one for each option; bit 0: -U cmd switch) This would be b/w compatible and still provide file(1) with enough information to be able to tell the file type. -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

[...]

For simplicity, comapibility and ease of change, I vote with +1 for adopting the solution of

byte 0: 'P' byte 1: 'Y' byte 2: version number (counting from 1) byte 3: option byte (8 bits: one for each option; bit 0: -U cmd switch)

If that turns out to be insufficient in some future, do a complete redesign.

What about the CR/LF issue with some Mac Compilers (see Guido's mail for details)? Can we simply drop this? Regards, Peter -- Peter Funk, Oldenburger Str.86, D-27777 Ganderkesee, Germany, Fax:+49 4222950260 office: +49 421 20419-0 (ArtCom GmbH, Grazer Str.8, D-28359 Bremen)

I don't think we should have a two-byte magic value. Especially where those two bytes are printable, 7-bit ASCII. "But it is four bytes," you say. Nope. It is two plus a couple parameters that can now change over time. To ensure uniqueness, I think a four-byte magic should stay. I would recommend the approach of adding opcodes into the marshal format. Specifically, 'V' followed by a single byte. That can only occur at the beginning. If it is not present, then you know that you have an old marshal value. Cheers, -g On Sun, 28 May 2000, Christian Tismer wrote:

Peter Funk wrote:

...

[...]

...

For simplicity, comapibility and ease of change, I vote with +1 for adopting the solution of

byte 0: 'P' byte 1: 'Y' byte 2: version number (counting from 1) byte 3: option byte (8 bits: one for each option; bit 0: -U cmd switch)

If that turns out to be insufficient in some future, do a complete redesign.

What about the CR/LF issue with some Mac Compilers (see Guido's mail for details)? Can we simply drop this?

Well, forgot about that. How about swapping bytes 0 and 1?

-- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaunstr. 26 : *Starship* http://starship.python.net 14163 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF where do you want to jump today? http://www.stackless.com

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://www.python.org/mailman/listinfo/python-dev

-- Greg Stein, http://www.lyra.org/

On Tue, May 30, 2000 at 09:08:15AM +0200, Peter Funk wrote:

...

I would recommend the approach of adding opcodes into the marshal format. Specifically, 'V' followed by a single byte. That can only occur at the beginning. If it is not present, then you know that you have an old marshal value.

But this would not solve the problem with 8 byte versus 4 byte timestamps in the header on 64-bit OSes. Trent Mick pointed this out.

I kind of intimated but did not make it clear: I wouldn't worry about the limitations of a 4 byte timestamp too much. That value is not going to overflow for another 38 years. Presumably the .pyc header (if such a thing even still exists then) will change by then. [peter summarizes .pyc header format options]

If we decide to move the version number into the marshal, if we can also move the .py-timestamp there. This way the timestamp will be handled in the same way as large integer literals. Quoting from the docs:

"""Caveat: On machines where C's long int type has more than 32 bits (such as the DEC Alpha), it is possible to create plain Python integers that are longer than 32 bits. Since the current marshal module uses 32 bits to transfer plain Python integers, such values are silently truncated. This particularly affects the use of very long integer literals in Python modules -- these will be accepted by the parser on such machines, but will be silently be truncated when the module is read from the .pyc instead. [...] A solution would be to refuse such literals in the parser, since they are inherently non-portable. Another solution would be to let the marshal module raise an exception when an integer value would be truncated. At least one of these solutions will be implemented in a future version."""

Should this be 1.6? Changing the format of .pyc files over and over again in the 1.x series doesn't look very attractive.

I *hope* it gets into 1.6, because I have implemented the latter suggestion (raise an exception is truncation of a PyInt to 32-bits will cause data loss) in the docs that you quoted and will be submitting a patch for it on Wed or Thurs. Ciao, Trent -- Trent Mick trentm@activestate.com

Peter Funk wrote:

Greg Stein:

...

I don't think we should have a two-byte magic value. Especially where those two bytes are printable, 7-bit ASCII. [...] To ensure uniqueness, I think a four-byte magic should stay.

Looking at /etc/magic I see many 16-bit magic numbers kept around from the good old days. But you are right: Choosing a four-byte magic value would make the chance of a clash with some other file format much less likely.

Just for quotes: the current /etc/magic I have on my Linux machine doesn't know anything about PYC or PYO files, so I don't really see much of a problem here -- noone seems to be interested in finding out the file type for these files anyway ;-) Also, I don't really get the 16-bit magic argument: we still have a 32-bit magic number -- one with a 16-bit fixed value and predefined ranges for the remaining 16 bits. This already is much better than what we have now w/r to making file(1) work on PYC files.

...

I would recommend the approach of adding opcodes into the marshal format. Specifically, 'V' followed by a single byte. That can only occur at the beginning. If it is not present, then you know that you have an old marshal value.

But this would not solve the problem with 8 byte versus 4 byte timestamps in the header on 64-bit OSes. Trent Mick pointed this out.

The switch to 8 byte timestamps is only needed when the current 4 bytes can no longer hold the timestamp value. That will happen in 2038... Note that import.c writes the timestamp in 4 bytes until it reaches an overflow situation.

I think, the situation we have now, is very unsatisfactory: I don't see a reasonable solution, which allows us to keep the length of the header before the marshal-block at a fixed length of 8 bytes together with a frozen 4 byte magic number.

Adding a version to the marshal format is a Good Thing -- independent of this discussion.

Moving the version number into the marshal doesn't help to resolve this conflict. So either you have to accept a new magic on 64 bit systems or you have to enlarge the header.

No you don't... please read the code: marshal only writes 8 bytes in case 4 bytes aren't enough to hold the value.

To come up with a new proposal, the following questions should be answered: 1. Is there really too much code out there, which depends on the hardcoded assumption, that the marshal part of a .pyc file starts at byte 8? I see no further evidence for or against this. MAL pointed this out in <http://www.python.org/pipermail/python-dev/2000-May/005756.html>

I have several references in my tool collection, the import stuff uses it, old import hooks (remember ihooks ?) also do, etc.

2. If we decide to enlarge the header, do we really need a new header field defining the length of the header ? This was proposed by Christian Tismer in <http://www.python.org/pipermail/python-dev/2000-May/005792.html>

In Py3K we can do this right (breaking things is allowed)... and I agree with Christian that a proper file format needs a header length field too. Basically, these values have to be present, IMHO: 1. Magic 2. Version 3. Length of Header 4. (Header Attribute)*n -- Start of Data --- Header Attribute can be pretty much anything -- timestamps, names of files or other entities, bit sizes, architecture flags, optimization settings, etc.

3. The 'imp' module exposes somewhat the structure of an .pyc file through the function 'get_magic()'. I proposed changing the signature of 'imp.get_magic()' in an upward compatible way. I also proposed adding a new function 'imp.get_version()'. What do you think about this idea?

imp.get_magic() would have to return the proposed 32-bit value ('PY' + version byte + option byte). I'd suggest adding additional functions which can read and write the header given a PYCHeader object which would hold the values version and options.

4. Greg proposed prepending the version number to the marshal format. If we do this, we definitely need a frozen way to find out, where the marshalled code object actually starts. This has also the disadvantage of making the task to come up with a /etc/magic definition whichs displays the version number of a .pyc file slightly harder.

If we decide to move the version number into the marshal, if we can also move the .py-timestamp there. This way the timestamp will be handled in the same way as large integer literals. Quoting from the docs:

"""Caveat: On machines where C's long int type has more than 32 bits (such as the DEC Alpha), it is possible to create plain Python integers that are longer than 32 bits. Since the current marshal module uses 32 bits to transfer plain Python integers, such values are silently truncated. This particularly affects the use of very long integer literals in Python modules -- these will be accepted by the parser on such machines, but will be silently be truncated when the module is read from the .pyc instead. [...] A solution would be to refuse such literals in the parser, since they are inherently non-portable. Another solution would be to let the marshal module raise an exception when an integer value would be truncated. At least one of these solutions will be implemented in a future version."""

Should this be 1.6? Changing the format of .pyc files over and over again in the 1.x series doesn't look very attractive.

-- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/