Python 2.5.1 broken os.stat module

Joe Salmeri JoeSalmeri at hotmail.com
Sun Jun 3 03:45:46 CEST 2007


> It's not clear whether it's an error, however, localtime() does
> something different from what dir does.

Hi Martin,

First off thank you for spending your time to investigate this bug with me.

Thanks for pointing out the issue with Windows XP caching the access 
timestamps,
I was not aware of that.

The reality is that I am most interested in the last write time because that 
is
the timestamp that Windows displays when you issue a dir command (WITHOUT a 
/t option),
what Windows Explorer displays in detail view, and what every other program 
I have
found so far references.

I re-reviewed all differences found by the program I posted for the 
Python\lib directory
date comparison and if you eliminate access times all differences for 
created and last
write that were reported were a difference of 1 hour.

Based on a bunch of testing that I did and some references listed below I 
believe
that you are CORRECT, Python 2.5.1 is reporting the correct times and 
Windows is
reporting the times incorrectly.

The short explaination of this issue is that the timestamp shown when you do 
a dir
on a file that is on an NTFS volume changes by an hour when DST starts and 
also when
DST ends, even though the file has NOT been modified!!!  Note that this only 
happens
if you have the setting turned on to automatically adjust for DST (this is 
the default
in Windows).

Even though Python 2.5.1 is correct this presents a NEW problem because it 
differs
from what XP SP 2 (wrongly) reports.  I am sure this will cause a lot of 
confusion
for people as it already appears that others have stumbled on this 
difference.

Here are some tests I did which show the problem that Windows has.

Setup Test Environment Virtual machine
    XP SP2
    Set TimeZone to GMT with automatically adjust for DST turned on
    Set Date and Time to 01/02/2007 12:00:00 AM
    Create file with notepad named joe.txt

    This establishes our baseline:

    joe.txt last written 01/02/2007 12:00 AM GMT

Check what Windows, Python 2.4.2 and Python 2.5.1 Report

    dir /tc

        01/02/2007  12:00 AM                14 joe.txt

    dir /tw

        01/02/2007  12:00 AM                14 joe.txt

    Python 2.4.2

        Creation    Time: 01/02/2007 00:00:04
        Last Write  Time: 01/02/2007 00:00:08

    Python 2.5.1

        Creation    Time: 01/02/2007 00:00:04
        Last Write  Time: 01/02/2007 00:00:08

    Great so far everybody is in agreement

Change Environment
    Set TimeZone to GMT-5:00 Eastern Time (US & Canada) (where I am located)
    DST still set to automatically adjust
    No changes to date and time at this point

    With these changes joe.txt should now report 01/01/2007 07:00 PM since I
    am 5 hours behind GMT

Check what Windows, Python 2.4.2 and Python 2.5.1 Report

    dir /tc

        01/01/2007  07:00 PM                14 joe.txt

    dir /tw

        01/01/2007  07:00 PM                14 joe.txt

    Python 2.4.2

        Creation    Time: 01/01/2007 19:00:04
        Last Write  Time: 01/01/2007 19:00:08

    Python 2.5.1

        Creation    Time: 01/01/2007 19:00:04
        Last Write  Time: 01/01/2007 19:00:08

    Great so far everyone is still in agreement

Change Environment
    Leave TimeZone set to GMT-5:00 Eastern Time (US & Canada) (where I am 
located)
    DST still set to automatically adjust
    Change date to todays date 06/02/2007 and time 02:00:00 PM

    This test simulates my machine moving in time forward from 01/1/2007 to 
06/02/2007
    with no changes being made on the computer.  Imagine I turned it off on 
01/02/2007
    and went on a long vacation and just came home on 06/02/2007 and turned 
it back on.

Check what Windows, Python 2.4.2 and Python 2.5.1 Report

    dir /tc

        01/01/2007  08:00 PM                14 joe.txt

    dir /tw

        01/01/2007  08:00 PM                14 joe.txt

    Python 2.4.2

        Creation    Time: 01/01/2007 20:00:04
        Last Write  Time: 01/01/2007 20:00:08

    Python 2.5.1

        Creation    Time: 01/01/2007 19:00:04
        Last Write  Time: 01/01/2007 19:00:08


    This is where the problem occurs.
    This file was originally created        01/02/2007 12:00 AM GMT
    which is the same locally as            01/01/2007 07:00 PM EST
    but now everything EXCEPT Python 2.51
    is reporting that the file changed on   01/01/2007 08:00 PM EST

    Well that couldn't have happened since no changes were made (remember I
    was on vacation :-)).

    Everything was fine until our current date time enters DST.
    At that point the file now shows that it was last written an hour later.

    The last write timestamp for the file should not change or be affected
    because of the start/end of DST has occurred on my computer.

    To the end user it appears the file was modifed when in fact it was not.

    Python 2.5.1 is CORRECT
    Windows is WRONG (dir /t, Explorer)
    Python 2.4.2 is WRONG

One Possible Work Around for the problem:
    Turn OFF the automatically adjust for DST setting
    NOTE:  Turning this off causes issues with time sync. See details at the 
end.

With it turned off the timestamps are all correct
    dir /tc

        01/01/2007  07:00 PM                14 joe.txt

    dir /tw

        01/01/2007  07:00 PM                14 joe.txt


    Python 2.4.2

        Creation    Time: 01/01/2007 19:00:04
        Last Write  Time: 01/01/2007 19:00:08

    Python 2.5.1

        Creation    Time: 01/01/2007 19:00:04
        Last Write  Time: 01/01/2007 19:00:08

I was concered that since I had the automatically adjust DST setting turned 
on when
I originally started with the time set to GMT 12:00 AM that it might have 
affected
my results so I reran the tests.

Setup Test Environment Virtual machine
    XP SP2
    Set TimeZone to GMT WITHOUT automatically adjust for DST
    Set Date and Time to 01/02/2007 12:00:00 AM
    Create file with notepad named joe.txt

    This establishes our baseline:

    joe.txt last written 01/02/2007 12:00 AM

Check what Windows, Python 2.4.2 and Python 2.5.1 Report

    dir /tc

        01/02/2007  12:00 AM                16 joe.txt

    dir /tw

        01/02/2007  12:00 AM                16 joe.txt

    Python 2.4.2

        Creation    Time: 01/02/2007 00:00:04
        Last Write  Time: 01/02/2007 00:00:09

    Python 2.5.1

        Creation    Time: 01/02/2007 00:00:04
        Last Write  Time: 01/02/2007 00:00:09

    Just like the first time everone is in agreement

Change Environment
    Set TimeZone to GMT-5:00 Eastern Time (US & Canada) (where I am located)
    DST still NOT set to automatically adjust
    No changes to date and time at this point

    With these changes joe.txt should now report 01/01/2007 07:00 PM since I
    am 5 hours behind GMT

Check what Windows, Python 2.4.2 and Python 2.5.1 Report

    dir /tc

        01/01/2007  07:00 PM                16 joe.txt

    dir /tw

        01/01/2007  07:00 PM                16 joe.txt

    Python 2.4.2

        Creation    Time: 01/01/2007 19:00:04
        Last Write  Time: 01/01/2007 19:00:09

    Python 2.5.1

        Creation    Time: 01/01/2007 19:00:04
        Last Write  Time: 01/01/2007 19:00:09

    Just like the first time everyone still in agreement

Change Environment
    Leave TimeZone set to GMT-5:00 Eastern Time (US & Canada) (where I am 
located)
    DST still NOT set to automatically adjust
    Change date to todays date 06/02/2007 and time 02:00:00 PM

Check what Windows, Python 2.4.2 and Python 2.5.1 Report

    dir /tc

        01/01/2007  07:00 PM                16 joe.txt

    dir /tw

        01/01/2007  07:00 PM                16 joe.txt

    Python 2.4.2

        Creation    Time: 01/01/2007 19:00:04
        Last Write  Time: 01/01/2007 19:00:09

    Python 2.5.1

        Creation    Time: 01/01/2007 19:00:04
        Last Write  Time: 01/01/2007 19:00:09

    Everyone still in agreement.

    This confirms that the bug is in Windows when the automatically adjust 
for DST
    setting is turned on.  Windows should NOT be using the DST setting to 
adjust
    the file times.  It makes it look like the files were modified when in 
fact
    they have not.

    If you turn it on after the last test then you get the following results 
which
    again prove that Python 2.5.1 is correct and that Windows and Python 
2.4.2 are
    wrong.

    dir /tc

        01/01/2007  08:00 PM                16 joe.txt

    dir /tw

        01/01/2007  08:00 PM                16 joe.txt

    Python 2.4.2

        Creation    Time: 01/01/2007 20:00:04
        Last Write  Time: 01/01/2007 20:00:09

    Python 2.5.1

        Creation    Time: 01/01/2007 19:00:04
        Last Write  Time: 01/01/2007 19:00:09

The real issue now is that even though Python 2.5.1 is correct it is 
confusing to
someone that is trying to compare their results to what Windows reports.

If you turn of automatically adjust DST and run the program I provided to 
compare
the dates between windows and python in the other email message against the
Python\Lib directory then all create dates and all write dates between 
Python and
Windows compare.  Access dates do not match but you have already explained 
why that
can occur (caching).

As shown above, one way to fix this is to turn OFF the automatically adjust 
for
DST setting in Windows.

The problem with turning it OFF is that if you have automatic time sync 
turned ON
when the time sync occurs and automatically adjust for DST is turned OFF and 
it is DST
your time will be an hour off after the sync :-(.

Also any computers that are syncing their time with this computer that have
automatically adjust for DST turned ON will have their sync'd times one hour
off since this computer had automatically adjust for DST turned OFF.

The following link goes into even greater detail as to the source of the 
problem
in Windows:

http://search.cpan.org/~shay/Win32-UTCFileTime-1.45/lib/Win32/UTCFileTime.pm

Based on the above article it does not seem like Microsoft will be fixing 
this
which leaves us stuck because Python and Windows are not reporting the same
values when DST starts/ends and you have automatically adjust for DST turned 
ON.

Having this difference between Python and Windows makes it difficult to 
write
code that depends on those values being in sync.

It makes me wonder if there should be an option in localtime to "fudge" the 
dates
so that they match what Windows reports....

Thanks to all that assisted in investigating this.






More information about the Python-list mailing list