PEP 3147 - new .pyc format

[Steven D'Aprano]

On Sat, 30 Jan 2010 14:14:54 -0800, John Roth wrote:

> PEP 3147 has just been posted, proposing that, beginning in release 3.2
> (and possibly 2.7) compiled .pyc and .pyo files be placed in a directory
> with a .pyr extension. The reason is so that compiled versions of a
> program can coexist, which isn't possible now.


http://www.python.org/dev/peps/pep-3147/


Reading through the PEP, I went from an instinctive "oh no, that sounds 
horrible" reaction to "hmmm, well, that doesn't sound too bad". I don't 
think I need this, but I could live with it.

Firstly, it does sound like there is a genuine need for a solution to the 
problem of multiple Python versions. This is not the first PEP trying to 
solve it, so even if you personally don't see the need, we can assume 
that others do.

Secondly, the current behaviour will remain unchanged. Python will 
compile spam.py to spam.pyc (or spam.pyo with the -O switch) by default. 
If you don't need to support multiple versions, you don't need to care 
about this PEP. I like this aspect of the PEP very much. I would argue 
that any solution MUST support the status quo for those who don't care 
about multiple versions.

To get the new behaviour, you have to explicitly ask for it. You ask for 
it by calling python with the -R switch, by setting an environment 
variable, or explicitly providing the extra spam/<magic>.pyc files.

Thirdly, the magic file names aren't quite as magic as they appear at 
first glance. They represent the hexified magic number of the version of 
Python. More about the magic number here:

http://nedbatchelder.com/blog/200804/the_structure_of_pyc_files.html

Unfortunately the magic number doesn't seem to be documented anywhere I 
can find other than in the source code (import.c). The PEP gives some 
examples:

    f2b30a0d.pyc # Python 2.5
    f2d10a0d.pyc # Python 2.6
    f2d10a0d.pyo # Python 2.6 -O
    f2d20a0d.pyc # Python 2.6 -U
    0c4f0a0d.pyc # Python 3.1

but how can one map magic numbers to versions, short of reading import.c? 
I propose that sys grow an object sys.magic which is the hexlified magic 
number.


> 2. I think the proposed logic is too complex. If this is installed in
> 3.2, then that release should simply store its .pyc file in the .pyr
> directory, without the need for either a command line switch or an
> environment variable (both are mentioned in the PEP.)

I disagree. Making the new behaviour optional is an advantage, even if it 
leads to extra complexity. It is pointless forcing .pyc files to be in a 
subdirectory if you don't need multiple versions.


> 3. Tool support. There are tools that look for the .pyc files; these
> need to be upgraded somehow. The ones that ship with Python should, of
> course, be fixed with the PEP, but there are others.

Third party tools will be the responsibility of the third parties.


> 4. I'm in favor of putting the source in the .pyr directory as well, but
> that's got a couple more issues. One is tool support, which is likely to
> be worse for source, and the other is some kind of algorithm for
> identifying which source goes with which object.

It certain does.

What's the advantage of forcing .py files to live inside a directory with 
the same name?

Modules:
    mymodule.py  =>  mymodule/mymodule.py

Packages:
    mypackage/__init__.py  =>  mypackage/__init__/__init__.py
    mypackage/spam.py  =>  mypackage/spam/spam.py


Seems like a pointless and annoying extra layer to me.



-- 
Steven