[Python-Dev] Re: What to do about invalid escape sequences

Aug. 10, 2019

      On 8/10/19, Rob Cliffe via Python-Dev <python-dev@python.org> wrote:
...
On 10/08/2019 11:50:35, eryk sun wrote:
...
On 8/9/19, Steven D'Aprano <steve@pearwood.info> wrote:
...
I'm also curious why the string needs to *end* with a backslash. Both of
these are the same path:
C:\foo\bar\baz\
     C:\foo\bar\baz
Also, the former is simply more *informative* - it tells the reader that
baz is expected to be a directory, not a file.
This is an important point that I overlooked. The trailing backslash
is more than just a redundant character to inform human readers. Refer
to [MS-FSA] 2.1.5.1 "Server Requests an Open of a File" [1]. A
create/open fails with STATUS_OBJECT_NAME_INVALID if either of the
following is true:

    * PathName contains a trailing backslash and
      CreateOptions.FILE_NON_DIRECTORY_FILE is
      TRUE.

    * PathName contains a trailing backslash and
      StreamTypeToOpen is DataStream

For NtCreateFile or NtOpenFile (in the NT API), the
FILE_NON_DIRECTORY_FILE option restricts the call to a regular file,
and FILE_DIRECTORY_FILE restricts it to a directory. With neither
option, the call can target either a file or directory. A trailing
backslash is another information channel. It tells the filesystem that
the target has to be a directory. If we specify
FILE_NON_DIRECTORY_FILE with a trailing backslash on the name, this is
an immediate failure as an invalid name without even checking the
entry. If we specify neither option and use a trailing backslash, it's
an invalid name if the filesystem finds a regular file or data stream.
Had the call specified the FILE_DIRECTORY_FILE option, it would
instead fail with STATUS_NOT_A_DIRECTORY.

We can see this in practice in the published source for the fastfat
filesystem driver. FatCommonCreate [2] (for a create or open) has the
following code to handle the second case (in this code, an FCB is a
file control block for a regular file, and a DCB is a directory
control block):

    if (NodeType(Fcb) == FAT_NTC_FCB) {
        //
        //  Check if we were only to open a directory
        //
        if (OpenDirectory) {
            DebugTrace(0, Dbg, "Cannot open file as directory\n", 0);
            try_return( Iosb.Status = STATUS_NOT_A_DIRECTORY );
        }
        DebugTrace(0, Dbg, "Open existing fcb, Fcb = %p\n", Fcb);
        if ( TrailingBackslash ) {
            try_return( Iosb.Status = STATUS_OBJECT_NAME_INVALID );
        }

We observe the first case with a typical CreateFileW call, which uses
the option FILE_NON_DIRECTORY_FILE. In the following example "baz" is
a regular file:

    >>> f = open(r'foo\bar\baz') # success
    >>> try: open('foo\\bar\\baz\\')
    ... except OSError as e: print(e)
    ...
    [Errno 22] Invalid argument: 'foo\\bar\\baz\\'

C EINVAL (22) is mapped from Windows ERROR_INVALID_NAME (123), which
is mapped from NT STATUS_OBJECT_NAME_INVALID (0xC0000033).

We can observe the second case with os.stat(), which calls CreateFileW
with backup semantics, which omits the FILE_NON_DIRECTORY_FILE option
in order to allow the call to open either a file or directory. In this
case the filesystem has to actually check that "baz" is a data file
before it can fail the call, as was shown in the fasfat code snippet
above:

    >>> try: os.stat('foo\\bar\\baz\\')
    ... except OSError as e: print(e)
    ...
    [WinError 123] The filename, directory name, or
    volume label syntax is incorrect: 'foo\\bar\\baz\\'

[1] https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-fsa/8ada5fbe...
[2] https://github.com/microsoft/Windows-driver-samples/blob/74200/filesys/fastf...