[New-bugs-announce] [issue15333] import on Windows will recompile a pyc file created on Unix

[Andrew MacKeith]

New submission from Andrew MacKeith <andrew at mackeith.net>:

In certain cases, a compiled Python file (.pyc) created on Unix will be recompiled when imported on Windows, despite the original code file (.py) being the same.
The cause is the use of the c fstat function in import.c.

This behavior is contrary to the stated Python behavior that a precompiled Python file is portable between platforms.

The fix to this bug would be to use the posixmodule stat implementation in place of fstat in import.c.

An example is the following, in Python 2.6.2:

This was tested on July 11th, 2012, USA EDT 
(i.e. DST is in effect, and local time is GMT - 6 hours). 
Lib/re.py has a mtime of 01/01/2009  11:46 AM according to Windows DIR.

Windows fstat st_mtime returns 1230828370 (0x495cf352) 16:46 GMT
Linux   fstat st_mtime returns 1230824770 (0x495ce542) 15:46 GMT

The effect of this is that when import opens the .pyc file and reads the mtime (the 2nd 4-byte field of the .pyc file) it sees it as "bad" and therefore recompiles the file.

The Python os.stat function works correctly on both Windows and Unix. There is a note in posixmodule.c that warns of a problem with Windows fstat:
"""
#ifdef MS_WINDOWS
/* The CRT of Windows has a number of flaws wrt. its stat() implementation:
   - time stamps are restricted to second resolution
   - file modification times suffer from forth-and-back conversions between
     UTC and local time
   Therefore, we implement our own stat, based on the Win32 API directly.
*/
"""

A Python file can also be precompiled using py_compile.compile; this uses os.stat to get the mtime of the .py file, and writes that to the .pyc file. 
This has the Unix behavior, and is "correct".

Output from time functions on Linux:
(The results are the same on Windows)

>>> print time.gmtime(0x495ce542)
time.struct_time(tm_year=2009, tm_mon=1, tm_mday=1, tm_hour=15, tm_min=46, tm_sec=10, tm_wday=3, tm_yday=1, tm_isdst=0)

>>> print time.localtime(0x495ce542)
time.struct_time(tm_year=2009, tm_mon=1, tm_mday=1, tm_hour=10, tm_min=46, tm_sec=10, tm_wday=3, tm_yday=1, tm_isdst=0)

>>> print time.gmtime(0x495cf352)
time.struct_time(tm_year=2009, tm_mon=1, tm_mday=1, tm_hour=16, tm_min=46, tm_sec=10, tm_wday=3, tm_yday=1, tm_isdst=0)

>>> print time.localtime(0x495cf352)
time.struct_time(tm_year=2009, tm_mon=1, tm_mday=1, tm_hour=11, tm_min=46, tm_sec=10, tm_wday=3, tm_yday=1, tm_isdst=0)

>>>

There has been no change in the relevant import.c code between Python-2.6 and Python-3.3.

X:\work\testimportmtime>c:\python26\python
Python 2.6.5 (r265:79096, Mar 19 2010, 18:02:59) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> print platform.platform()
Windows-post2008Server-6.1.7601-SP1
>>> print platform.architecture()
('64bit', 'WindowsPE')
>>> print platform.python_compiler()
MSC v.1500 64 bit (AMD64)

----------
components: Interpreter Core
files: testImportRe.py
messages: 165309
nosy: mackeith
priority: normal
severity: normal
status: open
title: import on Windows will recompile a pyc file created on Unix
type: behavior
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3
Added file: http://bugs.python.org/file26367/testImportRe.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue15333>
_______________________________________