Interesting! Indeed, please create an issue and post a link here.

From a quick look at the code, I can't see any obvious bugs here, the info seems to be coming directly from FindNextFileW. This will likely require some more digging.

On Sun, Oct 18, 2020 at 7:37 AM Gregory P. Smith greg@krypto.org wrote:

Could you please file this as an issue on bugs.python.org?

Thanks! -Greg

On Sat, Oct 17, 2020 at 7:25 PM Rob Cliffe via Python-Dev < python-dev@python.org> wrote:

...

TLDR: In os.scandir directory entries, atime is always a copy of mtime rather than the actual access time.

Demo program: Windows 10, Python 3.8.3:

# osscandirtest.py import time, os with open('Test', 'w') as f: f.write('Anything\n') # Write to a file time.sleep(10) with open('Test', 'r') as f: f.readline() # Read the file print(os.stat('Test')) for DirEntry in os.scandir('.'): if DirEntry.name == 'Test': stat = DirEntry.stat() print(f'scandir DirEntry {stat.st_ctime=} {stat.st_mtime=} {stat.st_atime=}')

Sample output:

os.stat_result(st_mode=33206, st_ino=8162774324687317, st_dev=2230120362, st_nlink=1, st_uid=0, st_gid=0, st_size=10, st_atime=1600631381, st_mtime=1600631371, st_ctime=1600631262) scandir DirEntry stat.st_ctime=1600631262.951019 stat.st_mtime=1600631371.7062848 stat.st_atime=1600631371.7062848

For os.stat, atime is 10 seconds more than mtime, as would be expected. But for os.scandir, atime is a copy of mtime. ISTM that this is a bug, and in fact recently it stopped me from using os.scandir in a program where I needed the access timestamp. No big deal, but ... If it is a feature for some reason, presumably it should be documented.

Best wishes Rob Cliffe _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RIKQAXZV... Code of Conduct: http://python.org/psf/codeofconduct/

---

Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/INJBNXRK... Code of Conduct: http://python.org/psf/codeofconduct/

On 10/18/2020 12:25 PM, Rob Cliffe via Python-Dev wrote:

How do I do that, please?  I can't see an obvious create option on that web page.  Do I need to log in?

Yes, you need to log in before you can open an issue. You might need to create an account first if you don't have one: it's called "Register" on bpo. After you've logged in, there's a Create New button.

Eric

Thanks Rob Cliffe

On 18/10/2020 05:31, Gregory P. Smith wrote:

...

Could you please file this as an issue on bugs.python.org http://bugs.python.org?

Thanks! -Greg

On Sat, Oct 17, 2020 at 7:25 PM Rob Cliffe via Python-Dev <python-dev@python.org mailto:python-dev@python.org> wrote:

TLDR: In os.scandir directory entries, atime is always a copy of
mtime
rather than the actual access time.

Demo program: Windows 10, Python 3.8.3:

# osscandirtest.py
import time, os
with open('Test', 'w') as f: f.write('Anything\n') # Write to a file
time.sleep(10)
with open('Test', 'r') as f: f.readline() # Read the file
print(os.stat('Test'))
for DirEntry in os.scandir('.'):
     if DirEntry.name == 'Test':
         stat = DirEntry.stat()
         print(f'scandir DirEntry {stat.st_ctime=} {stat.st_mtime=}
{stat.st_atime=}')

Sample output:

os.stat_result(st_mode=33206, st_ino=8162774324687317,
st_dev=2230120362, st_nlink=1, st_uid=0,
st_gid=0, st_size=10, st_atime=1600631381, st_mtime=1600631371,
st_ctime=1600631262)
scandir DirEntry stat.st_ctime=1600631262.951019
stat.st_mtime=1600631371.7062848 stat.st_atime=1600631371.7062848

For os.stat, atime is 10 seconds more than mtime, as would be
expected.
But for os.scandir, atime is a copy of mtime.
ISTM that this is a bug, and in fact recently it stopped me from
using
os.scandir in a program where I needed the access timestamp. No big
deal, but ...
If it is a feature for some reason, presumably it should be
documented.

Best wishes
Rob Cliffe
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
<mailto:python-dev@python.org>
To unsubscribe send an email to python-dev-leave@python.org
<mailto:python-dev-leave@python.org>
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/RIKQAXZVUAQBLECFMNN2PUOH322B2BYD/
Code of Conduct: http://python.org/psf/codeofconduct/

---

Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/377JYZMK... Code of Conduct: http://python.org/psf/codeofconduct/

On 19.10.2020 14:47, Steve Dower wrote:

On 19Oct2020 1242, Steve Dower wrote:

...

On 15Oct2020 2239, Rob Cliffe via Python-Dev wrote:

...

TLDR: In os.scandir directory entries, atime is always a copy of mtime rather than the actual access time.

Correction - os.stat() updates the access time to _now_, while os.scandir() returns the last access time without updating it.

Let me correct myself first :)

*Windows* has decided not to update file access time metadata *in directory entries* on reads. os.stat() always[1] looks at the file entry metadata, while os.scandir() always looks at the directory entry metadata.

Is this behavior documented somewhere?

Such weirdness certaintly something that needs to be documented but I really don't like describing such quirks that are out of our control and may be subject to change in Python documentation. So we should only consider doing so if there are no other options.

My suggested approach still applies, other than the bit where we might fix os.stat(). The best we can do is regress os.scandir() to have similarly poor performance, but the best *you* can do is use os.stat() for accurate timings when files might be being modified while your program is running, and don't do it when you just need names/kinds (and I'm okay adding that note to the docs).

Cheers, Steve

[1]: With some fallback to directory entries in exceptional cases that don't apply here. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QHHJFYED... Code of Conduct: http://python.org/psf/codeofconduct/ -- Regards, Ivan

On 10/19/20 9:52 AM, Gregory P. Smith wrote:

On Mon, Oct 19, 2020 at 6:28 AM Ivan Pozdeev via Python-Dev <python-dev@python.org mailto:python-dev@python.org> wrote:

On 19.10.2020 14:47, Steve Dower wrote:
> On 19Oct2020 1242, Steve Dower wrote:
>> On 15Oct2020 2239, Rob Cliffe via Python-Dev wrote:
>>> TLDR: In os.scandir directory entries, atime is always a copy of
mtime rather than the actual access time.
>>
>> Correction - os.stat() updates the access time to _now_, while
os.scandir() returns the last access time without updating it.
>
> Let me correct myself first :)
>
> *Windows* has decided not to update file access time metadata *in
directory entries* on reads. os.stat() always[1] looks at the file
entry
> metadata, while os.scandir() always looks at the directory entry
metadata.

Is this behavior documented somewhere?

Such weirdness certaintly something that needs to be documented but
I really don't like describing such quirks that are out of our control
and may be subject to change in Python documentation. So we should
only consider doing so if there are no other options.

I'm sure this is covered in MSDN.  Linking to that if it has it in a concise explanation would make sense from a note in our docs.

If I'm understanding Steve correctly this is due to Windows/NTFS storing the access time potentially redundantly in two different places. One within the directory entry itself and one with the file's own metadata.  Those of us with a traditional posix filesystem background may raise eyeballs at this duplication, seeing a directory as a place that merely maps names to inodes with the inode structure (equiv: file entry metadata) being the sole source of truth.  Which ones get updated when and by what actions is up to the OS.

So yes, just document the "quirk" as an intended OS behavior.  This is one reason scandir() can return additional information on windows vs what it can return on posix.  The entire point of scandir() is to return as much as possible from the directory without triggering reads of the inodes/file-entry-metadata. :)

-gps

depending on atimes isn't a consistently reliable mechanism anyway, since filesystems on Linux et. al. are allowed to be mounted so as to not independently update access times.

On 20/10/20 4:52 am, Gregory P. Smith wrote:

Those of us with a traditional posix filesystem background may raise eyeballs at this duplication, seeing a directory as a place that merely maps names to inodes

This is probably a holdover from MS-DOS, where there was no separate inode-like structure -- it was all in the directory entry.

On 10/19/20, Steve Dower steve.dower@python.org wrote:

On 19Oct2020 1242, Steve Dower wrote:

...

On 15Oct2020 2239, Rob Cliffe via Python-Dev wrote:

...

TLDR: In os.scandir directory entries, atime is always a copy of mtime rather than the actual access time.

Correction - os.stat() updates the access time to _now_, while os.scandir() returns the last access time without updating it.

Let me correct myself first :)

*Windows* has decided not to update file access time metadata *in directory entries* on reads. os.stat() always[1] looks at the file entry metadata, while os.scandir() always looks at the directory entry metadata.

My suggested approach still applies, other than the bit where we might fix os.stat(). The best we can do is regress os.scandir() to have similarly poor performance, but the best *you* can do is use os.stat() for accurate timings when files might be being modified while your program is running, and don't do it when you just need names/kinds (and I'm okay adding that note to the docs).

If this is the correction to which you're referring in the previous message, I assumed you stood by the claim that os.stat() may update st_atime. That shouldn't be the case, so there shouldn't be anything that needs to be fixed there, unless I'm missing what you think needs to be fixed. If it's actually a problem, then I'd really, really like a test case that reproduces it. If it was just a misinterpreted test case or mis-remembered fact, then that's good news for me. ;-)

Regarding updating the access time in the directory entry, in my previous reply I explained that NTFS should update it with a one-hour granularity. With FAT, it's daily.

Regarding the view that this is only about "accurate timings when files might be being modified while your program is running", in my previous messages I stressed that the directory entry for a hard link may have the wrong size, change time, write time, and access time if it wasn't the last link used to update the file. That has nothing to do with the file being modified while the program is running. It's a stale directory entry. If you call os.stat() on the stale link, NTFS will update it with the correct metadata.

On 10/19/20, Steve Dower steve.dower@python.org wrote:

On 15Oct2020 2239, Rob Cliffe via Python-Dev wrote:

...

TLDR: In os.scandir directory entries, atime is always a copy of mtime rather than the actual access time.

Correction - os.stat() updates the access time to _now_, while os.scandir() returns the last access time without updating it.

os.stat() shouldn't affect st_atime because it doesn't access the file data. That has me curious if it can be reproduced.

With NTFS in Windows 10, I'd expect the os.stat() st_atime to change immediately when the file data is read or modified. With other filesystems, it may not be updated until the kernel file object that was used to access the file's data is closed.

Note that updating the access time in NTFS can be disabled by the "NtfsDisableLastAccessUpdate" value in "HKLM\System\CurrentControlSet\Control\FileSystem". The default value in Windows 10 should be 0x80000002, which means the value is system managed and updating the access time is enabled.

If it's only the access time that changes, the directory entry may be updated with a significant granularity such as hourly or daily. For NTFS, it's hourly. To confirm this, wait an hour from the current access time in the directory entry; open the file; read some data; and close the file. The access time in the directory entry should be updated.

For details, download the [MS-FSA] PDF [1] and look for all references to the following sections:

* 2.1.4.17 Algorithm for Noting That a File Has Been Modified * 2.1.4.19 Algorithm for Noting That a File Has Been Accessed * 2.1.4.18 Algorithm for Updating Duplicated Information

Also check the tables in Appendix A, which provide the update granularity of file time stamps (presumably for directory entries) for common Windows filesystems.

[1] https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-fsa/860b1516...

Going back to my initial message, I can't stress enough that this problem is at its worst when a file has multiple hardlinks. If a particular link in a directory wasn't the last link used to access the file, its duplicated metadata may have the wrong file size, access time, modify time, and change time (the latter is not reported by Python). As is, for the current implementation, I'd only rely on the basic attributes such as whether it's a directory or reparse point (symlink, mountpoint, etc) when using scandir() to quickly process a directory. For reliable stat information, call os.stat().

I do think, however, that os.scandir() can be improved in Windows without significant performance loss if it calls GetFileAttributesExW to get st_file_attributes, st_size, st_ctime (create time), st_mtime, and st_atime. This API call is relatively fast because it doesn't require opening the file via CreateFileW, which is one of the more expensive operations in os.stat(). But I haven't tried modifying scandir() to benchmark it.

Ultimately, I'm waiting for Windows 10 to provide a WinAPI function that calls the relatively new NTAPI function NtQueryInformationByName [2] (by name, not by handle!) to get the FileStatInformation, as well as for this information to be made available by handle via GetFileInformationByHandleEx. Compared to GetFileAttributesExW, the FileStatInformation additionally provides the file ID (if implemented by the filesystem), change time, reparse tag, number of links, and the effective access of the security context of the caller (i.e. process or thread access token). The latter is something that we've never impemented with os.stat(). It's not the same as POSIX owner-group-other permissions. It would need a new attribute such as st_effective_access. It could be used to provide a real implementation of os.access() in Windows.

https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/nf-ntifs...

On 10/19/20, Steve Dower steve.dower@python.org wrote:

Resolving the path is the most expensive part, even if the file is not opened (I've been working with the NTFS team on this area, and we've been benchmarking/analysing all of it).

If you say it's been extensively benchmarked and there's no direct way around the speed bottleneck, then I take your word for it. To clarify what I had in mind, I was hoping that because NTFS implements the fast I/O function FastIoQueryOpen [1] (via NtfsNetworkOpenCreate, as given by its FastIoDispatch table) that IRP_MJ_CREATE would be bypassed and that the filesystem would not incur a significant cost to parse the remaining path. I figured that most of the work would be in the ObObjectObjectByName and IopParseDevice executive calls that lead up to querying the filesystem.

Anyway, it's unfortunate that the Windows API doesn't support NT handle-relative names, except in the registry API. If we could call NTAPI NtQueryAttributesFile [2] directly, then the ObjectAttributes argument could be relative to a directory handle set in the RootDirectory field. That would eliminate the vast majority of the path-resolution cost. A handle-relative open or query goes straight to the filesystem device, which goes straight to the directory that contains the file.

To eliminate the cost of opening the directory handle, scandir() could be rewritten to use CreateFileW and GetFileInformationByHandleEx: FileIdBothDirectoryInfo [3] instead of FindFirstFileW / FindNextFileW. Just cache the directory handle in place of caching the find handle. scandir() would gain fd support in Windows. Opening a directory via os.open requires the flag _O_OBTAIN_DIR (0x2000), defined in fcntl.h.

FileIdBothDirectoryInfo provides the file ID, so the implementation would support the inode() method without calling stat(). It would still directly support is_dir() and is_file() based on the file attributes, and is_symlink() based on the file attributes and the EaSize field. The Windows Protocols document that the latter contains the reparse tag for a reparse point. The field is reused because a reparse point can't have extended attributes.

All that said, I don't prefer to call NtQueryAttributesFile or any other NTAPI function in Windows Python. I'd rather do the best possible with just the Windows API. I wish there were a new GetFileAttributesExExW function that supported handle-relative names. Even better would be a new function that calls NtQueryInformationByName -- something like GetFileInformationByName -- for FileStatInfo (and FileCaseSensitiveInfo as well, which is becoming more of an issue), also with support for handle-relative names.

[1] https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/ns-wdm-_fa... [2] https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/nf-wdm-zwq... [3] https://docs.microsoft.com/en-us/windows/win32/api/winbase/ns-winbase-file_i...

On 20Oct2020 0520, Rob Cliffe wrote:

On 19/10/2020 12:42, Steve Dower wrote:

...

On 15Oct2020 2239, Rob Cliffe via Python-Dev wrote:

...

TLDR: In os.scandir directory entries, atime is always a copy of mtime rather than the actual access time.

Correction - os.stat() updates the access time to _now_, while os.scandir() returns the last access time without updating it.

Eryk replied with a deeper explanation of the cause, but fundamentally this is what you are seeing.

Feel free to file a bug, but we'll likely only add a vague note to the docs about how Windows works here rather than changing anything. If anything, we should probably fix os.stat() to avoid updating the access time so that both functions behave the same, but that might be too complicated.

Cheers, Steve

Sorry - what you say does not match the behaviour I observe, which is that

Yes, I posted a correction already (immediately after sending the first email).

What you are seeing is what Windows decided was the best approach. If you want to avoid that, os.stat() will get the latest available information. But I don't want to penalise people who don't need it by slowing down their scandir calls unnecessarily.

A documentation patch to make this difference between os.stat() and DirEntry even clearer would be fine.

Cheers, Steve

On Fri, Oct 23, 2020, at 02:14, Random832 wrote:

What correction, exactly, do you mean? The post I saw with the word "Correction" on it is the one that *makes* the claim people are taking issue with.

okay, sorry, I see the other correction post now...

My issue I guess was the same as Eryk Sun, it wasn't clear which parts of the previous post you were correcting and which (if any) you stood by, since they were about the behavior of different parts of the system, so it didn't register as a correction to that part when I originally read it.

On 10/26/20, Victor Stinner vstinner@python.org wrote:

Le lun. 19 oct. 2020 à 13:50, Steve Dower steve.dower@python.org a écrit :

...

Feel free to file a bug, but we'll likely only add a vague note to the docs about how Windows works here rather than changing anything.

I agree that this surprising behavior can be documented. Attempting to provide accurate access time in os.scandir() is likely to slow-down the function which would defeat its whole purpose.

I don't think the access time (st_atime) is a significant concern. I'm concerned with the reliability of the file size (st_size) and last-write time (st_mtime) in stat() results. Developers are used to various filesystem policies on various platforms that limit when the access time gets updated, if at all. FAT32 filesystems only have an access date, and the driver in Windows fixes the access time at midnight. Updating the access time in NTFS and ReFS can be completely disabled at the system level; otherwise it's updated with a granularity of one hour if it's only the access time that would be updated.

The biggest concern for me is NTFS hardlinks, for which the st_size and st_mtime in the directory entry is unreliable. When a file with multiple hardlinks is modified, the filesystem only updates the duplicated information in the directory entry of the opened link. Because the entry in the directory doesn't include the link count or even a boolean value to indicate that a file has multiple hardlinks, if you don't know whether or not there's a possibility of hardlinks, then os.stat() is required in order to reliably determine st_size and st_mtime, to the extent that reliably knowing st_mtime is possible.

A general problem that affects even os.stat() is that a modified file may only be noted by setting a flag (FO_FILE_MODIFIED) in the kernel file object of the particular open. Whether it's immediately noted in the last-write time of the shared FCB (file control block) is up to filesystem policy.

Starting with Windows 10 1809 (as noted in [MS-FSA]), NTFS immediately notes the modification time, so the st_mtime value from os.stat() is current. In prior versions of NTFS, and with other Microsoft filesystems such as FAT32, the last-write time is only noted when the file is flushed to disk via FlushFileBuffers (i.e. os.fsync) or when the open is closed.

This means that st_size may change without also changing st_mtime. I'm using Windows 10 2004 currently, so I can't show an NTFS example, but the following shows the behavior with FAT32:

f = open('spam.txt', 'w') st1 = os.stat('spam.txt') time.sleep(10) f.write('spam') f.flush() st2 = os.stat('spam.txt')

The above write was noted only by setting the FO_FILE_MODIFIED flag on the kernel file object. (The file object can be inspected with a local kernel debugger.) The write time wasn't noted in the FCB, i.e. st_mtime hasn't changed in st2:

>>> st2.st_size - st1.st_size 4 >>> st2.st_mtime - st1.st_mtime 0.0

The last-write time is noted when FlushFileBuffers (os.fsync) is called on the open:

>>> os.fsync(f.fileno()) >>> st3 = os.stat('spam.txt') >>> st3.st_mtime - st1.st_mtime 10.0

Note also that, with NTFS, to the extent that the FCB metadata is current, calling os.stat() on a link updates the duplicated information in the directory entry. So calling os.stat() on a NTFS file may update the entry that's returned by a subsequent os.scandir() call.

On Tue, Oct 27, 2020 at 10:00 AM Greg Ewing greg.ewing@canterbury.ac.nz wrote:

On 27/10/20 8:24 am, Victor Stinner wrote:

...

I would rather want to kill the whole concept of "access" time in operating systems (or just configure the OS to not update it anymore). I guess that it's really hard to make it efficient and accurate at the same time...

Also it's kind of weird that just looking at data on the disk can change something about it. Sometimes it's an advantage to *not* have quantum computing!

And yet, it's of incredible value to be able to ask "now, where was that file... the one that I was looking at last week, called something about calendars, and it had a cat picture in it". Being able to answer that kinda depends on recording accesses one way or another, so the weirdnesses are bound to happen.

ChrisA

Greg Ewing writes:

Also it's kind of weird that just looking at data on the disk can change something about it.

The "something about it" *did* change. The world is a dynamic entity, it does change. What you think is weird is that the metadata change is recorded.

Note: you can "fix" directory updates by mounting the filesystem r/o.

Sometimes it's an advantage to *not* have quantum computing!

I think effective encryption is a bigger one, myself. ;-)