[New-bugs-announce] [issue13863] Using py_compile does not prevent recompilation due to 'bad mtime'.

[Mark Dickinson]

New submission from Mark Dickinson <dickinsm at gmail.com>:

(Marking this as 'Interpreter Core' because the issue really seems to come from Python/import.c rather than from the py_compile module.)

On a Windows 7 VM (64-bit, NTFS) running Python 2.7 (also reproduced on a non-VM Windows 7 installation), I'm seeing the following surprising behaviour:


C:\Python27\Lib>dir decimal.py*
 Volume in drive C is Local
 Volume Serial Number is 5083-D43D

 Directory of C:\Python27\Lib

06/10/2011  08:46 PM           219,383 decimal.py
01/25/2012  07:05 PM           165,322 decimal.pyc
01/25/2012  04:54 PM           169,386 decimal.pyo
               3 File(s)        554,091 bytes
               0 Dir(s)     966,266,880 bytes free

C:\Python27\Lib>python
Enthought Python Distribution -- www.enthought.com
Version: 7.2-2 (64-bit)

Python 2.7.2 |EPD 7.2-2 (64-bit)| (default, Sep 14 2011, 11:25:00) [MSC v.1500 64 bit (AMD64)] on win32
Type "packages", "demo" or "enthought" for more information.
>>> import py_compile; py_compile.compile('decimal.py')
>>> import os; os.stat('decimal.pyc').st_mtime
1327518430.306176
>>> import decimal
>>> os.stat('decimal.pyc').st_mtime
1327518443.9094632

Notice that in spite of calling py_compile.compile on decimal.py, the .pyc file is still regenerated on import.
Relevant details: note that the decimal.py timestamp is in the summer, and that it's currently winter time.  Also, my Windows 7 timezone setting is UTC, with 'automatically adjust for DST' set to true.

This bit me today when working with a 'post-install' script for an MSI installer for a large Python application, where the post-install script did a compileall.compile_path to make sure that all the .pyc files were up to date, and thereby avoid a *long* (> 90 second) application first startup time.  It turned out that on application startup some of the .pyc files got regenerated while others didn't, and it took me far too long to notice that it was the .py files with summer timestamps that lead to .pyc regeneration, and that those with winter timestamps were okay.

When I set my timezone to plain UTC with no DST adjustments, the above issue disappears.  Also, when I reset my timezone to that of Saudi Arabia (no DST), the issue also disappears.


The cause of the issue seems to be that:

(1) import.c uses an 'fstat' call to get mtime for a .py file.
(2) on my Windows installation, the result of fstat on a file varies (a) with time of year, and (b) with computer DST and timezone settings.  (No such problems on OS X.)
(3) in contrast, py_compile uses os.stat(...).st_mtime, which appears to be invariant under time changes.


The difference between the 'fstat' result and the os.stat result can be seen directly:

Python 2.7.2 |EPD 7.2-2 (64-bit)| (default, Sep 14 2011, 11:25:00) [MSC v.1500 64 bit (AMD64)] on win32
Type "packages", "demo" or "enthought" for more information.
>>> import decimal  # make sure .pyc file is up to date
>>> import os; os.stat('decimal.py').st_mtime  # actual mtime of .py file
1307738784.0
>>> import struct; struct.unpack('LL', open('decimal.pyc', 'rb').read(8))[1]  # stored mtime from .pyc; uses fstat
1307735184



This presumably also means that in this part of the world, .pyc files will be regenerated on import after any DST change, since the reported 'fstat' result used by import.c will no longer match the stored mtime timestamp in the .pyc file.

----------
components: Interpreter Core
messages: 151969
nosy: mark.dickinson
priority: normal
severity: normal
status: open
title: Using py_compile does not prevent recompilation due to 'bad mtime'.
type: behavior
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13863>
_______________________________________