Interesting! Indeed, please create an issue and post a link here.

From a quick look at the code, I can't see any obvious bugs here, the info seems to be coming directly from FindNextFileW. This will likely require some more digging.

On Sun, Oct 18, 2020 at 7:37 AM Gregory P. Smith greg@krypto.org wrote:

On 10/18/2020 12:25 PM, Rob Cliffe via Python-Dev wrote:

Yes, you need to log in before you can open an issue. You might need to create an account first if you don't have one: it's called "Register" on bpo. After you've logged in, there's a Create New button.

Eric

TLDR: In os.scandir directory entries, atime is always a copy of
mtime
rather than the actual access time.

Demo program: Windows 10, Python 3.8.3:

# osscandirtest.py
import time, os
with open('Test', 'w') as f: f.write('Anything\n') # Write to a file
time.sleep(10)
with open('Test', 'r') as f: f.readline() # Read the file
print(os.stat('Test'))
for DirEntry in os.scandir('.'):
     if DirEntry.name == 'Test':
         stat = DirEntry.stat()
         print(f'scandir DirEntry {stat.st_ctime=} {stat.st_mtime=}
{stat.st_atime=}')

Sample output:

os.stat_result(st_mode=33206, st_ino=8162774324687317,
st_dev=2230120362, st_nlink=1, st_uid=0,
st_gid=0, st_size=10, st_atime=1600631381, st_mtime=1600631371,
st_ctime=1600631262)
scandir DirEntry stat.st_ctime=1600631262.951019
stat.st_mtime=1600631371.7062848 stat.st_atime=1600631371.7062848

For os.stat, atime is 10 seconds more than mtime, as would be
expected.
But for os.scandir, atime is a copy of mtime.
ISTM that this is a bug, and in fact recently it stopped me from
using
os.scandir in a program where I needed the access timestamp. No big
deal, but ...
If it is a feature for some reason, presumably it should be
documented.

Best wishes
Rob Cliffe
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
<mailto:python-dev@python.org>
To unsubscribe send an email to python-dev-leave@python.org
<mailto:python-dev-leave@python.org>
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/RIKQAXZVUAQBLECFMNN2PUOH322B2BYD/
Code of Conduct: http://python.org/psf/codeofconduct/

On 19.10.2020 14:47, Steve Dower wrote:

Is this behavior documented somewhere?

Such weirdness certaintly something that needs to be documented but I really don't like describing such quirks that are out of our control and may be subject to change in Python documentation. So we should only consider doing so if there are no other options.

>

On 10/19/20 9:52 AM, Gregory P. Smith wrote:

On 19.10.2020 14:47, Steve Dower wrote:
> On 19Oct2020 1242, Steve Dower wrote:
>> On 15Oct2020 2239, Rob Cliffe via Python-Dev wrote:
>>> TLDR: In os.scandir directory entries, atime is always a copy of
mtime rather than the actual access time.
>>
>> Correction - os.stat() updates the access time to _now_, while
os.scandir() returns the last access time without updating it.
>
> Let me correct myself first :)
>
> *Windows* has decided not to update file access time metadata *in
directory entries* on reads. os.stat() always[1] looks at the file
entry
> metadata, while os.scandir() always looks at the directory entry
metadata.

Is this behavior documented somewhere?

Such weirdness certaintly something that needs to be documented but
I really don't like describing such quirks that are out of our control
and may be subject to change in Python documentation. So we should
only consider doing so if there are no other options.

depending on atimes isn't a consistently reliable mechanism anyway, since filesystems on Linux et. al. are allowed to be mounted so as to not independently update access times.

On 20/10/20 4:52 am, Gregory P. Smith wrote:

This is probably a holdover from MS-DOS, where there was no separate inode-like structure -- it was all in the directory entry.

-- Greg

On 10/19/20, Steve Dower steve.dower@python.org wrote:

If this is the correction to which you're referring in the previous message, I assumed you stood by the claim that os.stat() may update st_atime. That shouldn't be the case, so there shouldn't be anything that needs to be fixed there, unless I'm missing what you think needs to be fixed. If it's actually a problem, then I'd really, really like a test case that reproduces it. If it was just a misinterpreted test case or mis-remembered fact, then that's good news for me. ;-)

Regarding updating the access time in the directory entry, in my previous reply I explained that NTFS should update it with a one-hour granularity. With FAT, it's daily.

Regarding the view that this is only about "accurate timings when files might be being modified while your program is running", in my previous messages I stressed that the directory entry for a hard link may have the wrong size, change time, write time, and access time if it wasn't the last link used to update the file. That has nothing to do with the file being modified while the program is running. It's a stale directory entry. If you call os.stat() on the stale link, NTFS will update it with the correct metadata.

On 10/19/20, Steve Dower steve.dower@python.org wrote:

os.stat() shouldn't affect st_atime because it doesn't access the file data. That has me curious if it can be reproduced.

With NTFS in Windows 10, I'd expect the os.stat() st_atime to change immediately when the file data is read or modified. With other filesystems, it may not be updated until the kernel file object that was used to access the file's data is closed.

Note that updating the access time in NTFS can be disabled by the "NtfsDisableLastAccessUpdate" value in "HKLM\System\CurrentControlSet\Control\FileSystem". The default value in Windows 10 should be 0x80000002, which means the value is system managed and updating the access time is enabled.

If it's only the access time that changes, the directory entry may be updated with a significant granularity such as hourly or daily. For NTFS, it's hourly. To confirm this, wait an hour from the current access time in the directory entry; open the file; read some data; and close the file. The access time in the directory entry should be updated.

For details, download the [MS-FSA] PDF [1] and look for all references to the following sections:

* 2.1.4.17 Algorithm for Noting That a File Has Been Modified
* 2.1.4.19 Algorithm for Noting That a File Has Been Accessed
* 2.1.4.18 Algorithm for Updating Duplicated Information

Also check the tables in Appendix A, which provide the update granularity of file time stamps (presumably for directory entries) for common Windows filesystems.

[1] https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-fsa/860b1516...

Going back to my initial message, I can't stress enough that this problem is at its worst when a file has multiple hardlinks. If a particular link in a directory wasn't the last link used to access the file, its duplicated metadata may have the wrong file size, access time, modify time, and change time (the latter is not reported by Python). As is, for the current implementation, I'd only rely on the basic attributes such as whether it's a directory or reparse point (symlink, mountpoint, etc) when using scandir() to quickly process a directory. For reliable stat information, call os.stat().

I do think, however, that os.scandir() can be improved in Windows without significant performance loss if it calls GetFileAttributesExW to get st_file_attributes, st_size, st_ctime (create time), st_mtime, and st_atime. This API call is relatively fast because it doesn't require opening the file via CreateFileW, which is one of the more expensive operations in os.stat(). But I haven't tried modifying scandir() to benchmark it.

Ultimately, I'm waiting for Windows 10 to provide a WinAPI function that calls the relatively new NTAPI function NtQueryInformationByName [2] (by name, not by handle!) to get the FileStatInformation, as well as for this information to be made available by handle via GetFileInformationByHandleEx. Compared to GetFileAttributesExW, the FileStatInformation additionally provides the file ID (if implemented by the filesystem), change time, reparse tag, number of links, and the effective access of the security context of the caller (i.e. process or thread access token). The latter is something that we've never impemented with os.stat(). It's not the same as POSIX owner-group-other permissions. It would need a new attribute such as st_effective_access. It could be used to provide a real implementation of os.access() in Windows.

https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/nf-ntifs...

On 10/19/20, Steve Dower steve.dower@python.org wrote: >

If you say it's been extensively benchmarked and there's no direct way around the speed bottleneck, then I take your word for it. To clarify what I had in mind, I was hoping that because NTFS implements the fast I/O function FastIoQueryOpen [1] (via NtfsNetworkOpenCreate, as given by its FastIoDispatch table) that IRP_MJ_CREATE would be bypassed and that the filesystem would not incur a significant cost to parse the remaining path. I figured that most of the work would be in the ObObjectObjectByName and IopParseDevice executive calls that lead up to querying the filesystem.

Anyway, it's unfortunate that the Windows API doesn't support NT handle-relative names, except in the registry API. If we could call NTAPI NtQueryAttributesFile [2] directly, then the ObjectAttributes argument could be relative to a directory handle set in the RootDirectory field. That would eliminate the vast majority of the path-resolution cost. A handle-relative open or query goes straight to the filesystem device, which goes straight to the directory that contains the file.

To eliminate the cost of opening the directory handle, scandir() could be rewritten to use CreateFileW and GetFileInformationByHandleEx: FileIdBothDirectoryInfo [3] instead of FindFirstFileW / FindNextFileW. Just cache the directory handle in place of caching the find handle. scandir() would gain fd support in Windows. Opening a directory via os.open requires the flag _O_OBTAIN_DIR (0x2000), defined in fcntl.h.

FileIdBothDirectoryInfo provides the file ID, so the implementation would support the inode() method without calling stat(). It would still directly support is_dir() and is_file() based on the file attributes, and is_symlink() based on the file attributes and the EaSize field. The Windows Protocols document that the latter contains the reparse tag for a reparse point. The field is reused because a reparse point can't have extended attributes.

All that said, I don't prefer to call NtQueryAttributesFile or any other NTAPI function in Windows Python. I'd rather do the best possible with just the Windows API. I wish there were a new GetFileAttributesExExW function that supported handle-relative names. Even better would be a new function that calls NtQueryInformationByName -- something like GetFileInformationByName -- for FileStatInfo (and FileCaseSensitiveInfo as well, which is becoming more of an issue), also with support for handle-relative names.

[1] https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/ns-wdm-_fa... [2] https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/nf-wdm-zwq... [3] https://docs.microsoft.com/en-us/windows/win32/api/winbase/ns-winbase-file_i...

On 20Oct2020 0520, Rob Cliffe wrote:

Yes, I posted a correction already (immediately after sending the first email).

What you are seeing is what Windows decided was the best approach. If you want to avoid that, os.stat() will get the latest available information. But I don't want to penalise people who don't need it by slowing down their scandir calls unnecessarily.

A documentation patch to make this difference between os.stat() and DirEntry even clearer would be fine.

Cheers, Steve

On Fri, Oct 23, 2020, at 02:14, Random832 wrote:

okay, sorry, I see the other correction post now...

My issue I guess was the same as Eryk Sun, it wasn't clear which parts of the previous post you were correcting and which (if any) you stood by, since they were about the behavior of different parts of the system, so it didn't register as a correction to that part when I originally read it.

On 10/26/20, Victor Stinner vstinner@python.org wrote:

I don't think the access time (st_atime) is a significant concern. I'm concerned with the reliability of the file size (st_size) and last-write time (st_mtime) in stat() results. Developers are used to various filesystem policies on various platforms that limit when the access time gets updated, if at all. FAT32 filesystems only have an access date, and the driver in Windows fixes the access time at midnight. Updating the access time in NTFS and ReFS can be completely disabled at the system level; otherwise it's updated with a granularity of one hour if it's only the access time that would be updated.

The biggest concern for me is NTFS hardlinks, for which the st_size and st_mtime in the directory entry is unreliable. When a file with multiple hardlinks is modified, the filesystem only updates the duplicated information in the directory entry of the opened link. Because the entry in the directory doesn't include the link count or even a boolean value to indicate that a file has multiple hardlinks, if you don't know whether or not there's a possibility of hardlinks, then os.stat() is required in order to reliably determine st_size and st_mtime, to the extent that reliably knowing st_mtime is possible.

A general problem that affects even os.stat() is that a modified file may only be noted by setting a flag (FO_FILE_MODIFIED) in the kernel file object of the particular open. Whether it's immediately noted in the last-write time of the shared FCB (file control block) is up to filesystem policy.

Starting with Windows 10 1809 (as noted in [MS-FSA]), NTFS immediately notes the modification time, so the st_mtime value from os.stat() is current. In prior versions of NTFS, and with other Microsoft filesystems such as FAT32, the last-write time is only noted when the file is flushed to disk via FlushFileBuffers (i.e. os.fsync) or when the open is closed.

This means that st_size may change without also changing st_mtime. I'm using Windows 10 2004 currently, so I can't show an NTFS example, but the following shows the behavior with FAT32:

f = open('spam.txt', 'w')
st1 = os.stat('spam.txt')
time.sleep(10)
f.write('spam')
f.flush()
st2 = os.stat('spam.txt')

The above write was noted only by setting the FO_FILE_MODIFIED flag on the kernel file object. (The file object can be inspected with a local kernel debugger.) The write time wasn't noted in the FCB, i.e. st_mtime hasn't changed in st2:

>>> st2.st_size - st1.st_size
4
>>> st2.st_mtime - st1.st_mtime
0.0

The last-write time is noted when FlushFileBuffers (os.fsync) is called on the open:

>>> os.fsync(f.fileno())
>>> st3 = os.stat('spam.txt')
>>> st3.st_mtime - st1.st_mtime
10.0

Note also that, with NTFS, to the extent that the FCB metadata is current, calling os.stat() on a link updates the duplicated information in the directory entry. So calling os.stat() on a NTFS file may update the entry that's returned by a subsequent os.scandir() call.

On Tue, Oct 27, 2020 at 10:00 AM Greg Ewing greg.ewing@canterbury.ac.nz wrote: >

And yet, it's of incredible value to be able to ask "now, where was that file... the one that I was looking at last week, called something about calendars, and it had a cat picture in it". Being able to answer that kinda depends on recording accesses one way or another, so the weirdnesses are bound to happen.

ChrisA

Greg Ewing writes:

The "something about it" did change. The world is a dynamic entity, it does change. What you think is weird is that the metadata change is recorded.

Note: you can "fix" directory updates by mounting the filesystem r/o.

I think effective encryption is a bigger one, myself. ;-)