What does sanitizepart do with newlines \n \r \r\n in filenames? Are these control characters? What does sanitizepart do with a leading slash? assert os.path.join("a", "/b") == "/b" A new safejoin() or joinsafe() or join(safe='True') could call sanitizepart() such that: assert joinsafe("a\n", "/b") == "a\\n/b" On Mon, May 11, 2020, 1:11 PM Andrew Barnert via Python-ideas < python-ideas@python.org> wrote:

On May 11, 2020, at 00:40, Steve Jorgensen wrote:

...

Proposal:

Add a new function (possibly `os.path.sanitizepart`) to sanitize a value

for use as a single component of a path. In the default case, the value must also not be a reference to the current or parent directory ("." or "..") and must not contain control characters.

“Also” in addition to what? Are there other requirements enforced besides these two that aren’t specified anywhere?

If not: the result can contain the path separator, illegal characters that aren’t control characters, nonprinting characters that aren’t control characters, and characters whose bytes (in the filesystem’s encoding) are ASCII control characters?

And it can be a reserved name, or even something like C:; as long as it’s not the Unix . or ..?

What’s the use case where you need to sanitize these things but nothing else? As I said on the previous proposal, I have had a variety of times where I needed to sanitize filenames, but I don’t think this would have been what I wanted for _any_ of them, much less for most.

Are there existing tools, libraries, recommendations, etc. that this is based on, or is it just an educated guess at what’s important? For something that’s meant to go into the stdlib with a name that strongly implies “if you use this, you’re safe from stupid or malicious filenames”, it would be misleading, and possibly dangerous, if it didn’t actually make you safe because it didn’t catch common mistakes/exploits that everyone else considers important to catch. And without any cites to what people everyone else considers important, why should anyone trust that this proposal isn’t missing, or getting wrong, anything critical?

Why isn’t this also available in pathlib? Is it the kind of thing you don’t envision high-level pathlib-style code ever needing to do, only low-level os-style code?

...

When `replace` is supplied, it is used as a replacement for any invalid characters or for the first character of an invalid name. When `prefix` is not also supplied, this is also used as the replacement for the first character of the name if it is invalid, not simply due to containing invalid characters.

What’s the use case for separate prefix and replace? Or just for prefix in the first place?

...

When `escape` is supplied (typically "%") it is used as the escape character in the same way that "%" is used in URL encoding.

Why allow other escape strings? Has anyone ever wanted URL-encoding but with some other string in place or %, in this or any other context?

The escape character is not itself escaped?

More generally, what’s the use case for %-encoding filenames like this? Are people expecting it to interact transparently with URLs, so if I save a file “spam\0eggs” in a Python script and then try to browse to file:///spam\0eggs” in a browser, the browser will convert the \0 character to %00 the same way my Python script did and therefore find the file? If so, doesn’t it need to escape all the same characters that URLs do, not a different set? If not, isn’t using something similar to URL-encoding but not identical just going to confuse people rather than help then?

What happens if you supply a string longer than one character as escape? Or replace or prefix, for that matter?

Overall, it seems like there is a problem to be solved, but I don’t see any reason to be confident that this is the solution for anyone, and if it’s not the solution for _most_ people, adding it to the stdlib will just mean people don’t search for and find the right one, all the while misleading themselves into thinking they’re safe when they’re not, which will make the overall problem worse, not better.

_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/ZBBMQ3... Code of Conduct: http://python.org/psf/codeofconduct/

On 11 May 2020, at 18:09, Andrew Barnert via Python-ideas wrote:

More generally, what’s the use case for %-encoding filenames like this? Are people expecting it to interact transparently with URLs, so if I save a file “spam\0eggs” in a Python script and then try to browse to file:///spam\0eggs file:///spam/0eggs” in a browser, the browser will convert the \0 character to %00 the same way my Python script did and therefore find the file?

No. The \0 can never be part of a valid file in Unix, macOS or Windows. Barry

On 11 May 2020, at 22:38, Andrew Barnert wrote:

On May 11, 2020, at 12:59, Barry Scott wrote:

...

...

On 11 May 2020, at 18:09, Andrew Barnert via Python-ideas mailto:python-ideas@python.org> wrote:

More generally, what’s the use case for %-encoding filenames like this? Are people expecting it to interact transparently with URLs, so if I save a file “spam\0eggs” in a Python script and then try to browse to file:///spam\0eggs file:///spam/0eggs” in a browser, the browser will convert the \0 character to %00 the same way my Python script did and therefore find the file?

No.

The \0 can never be part of a valid file in Unix, macOS or Windows.

Of course. Which is exactly the kind of thing this sanitize function is meant for.

Hence my question: if my Python script is sanitizing all filenames with this function with escape='%', is the expectation that it’ll actually give me something that can be used if I paste the same thing into a browser and let it url-escape a file URL? If so, will that actually work? If not, what _is_ the intended use for this option?

I misunderstood I thought you where saying that using escaping allowed bad chars to work. Barry

On Wed, 13 May 2020 20:52:38 +1000 Steven D'Aprano wrote:

I don't know how complete HFS+ support is on Linux or Windows, but in principle any OS that supports HFS+ or (maybe) Joliet could have files with NULs.

Remember that NULs may be legal next time you are stress testing your file IO code *wink*

NULs may be theoretically legal on your filesystem of choice, but standard POSIX and Windows APIs don't let you pass filenames with NULs in them correctly. So the point is moot. If you know of a system function which accepts filenames with embedded NULs (which probably means it also takes the filename length as a separate parameter), I'd be curious to know about it. Regards Antoine.

On 5/13/20, Antoine Pitrou wrote:

If you know of a system function which accepts filenames with embedded NULs (which probably means it also takes the filename length as a separate parameter), I'd be curious to know about it.

Windows is layered over the base NT system, which uses counted strings and a root object namespace that reserves only the path separator, backslash. Null characters are allowed, at least as far as the object manager cares, but using them is a bad idea, if only because such names aren't generally accessible in Windows. But let's look at an example just for kicks. When the object manager parses a path up to a Device object (e.g. "\Device\NamedPipe"), the I/O manager takes over parsing the remaining path, which calls the device driver's IRP_MJ_CREATE routine with the remaining path. Whether or not a name with nulls is allowed depends on the device driver -- or a filesystem driver if the device is mounted. Almost all filesystem drivers reject a component name that contains nulls as invalid. One exception is the named-pipe filesystem (NPFS). NPFS doesn't disallow any characters. It even allows backslash in pipe names since it doesn't support subdirectories, and if you check via os.listdir('//./pipe'), you should see several Winsock pipes with backslash in their name. Creating a pipe with nulls in its name is impossible via WINAPI CreateNamedPipeW. It requires native NtCreateNamedPipeFile, with the name passed in an OBJECT_ATTRIBUTES record [1]. This system function is undocumented, but just to show that it's possible in principle, I created a pipe named "spam\x00eggs". We can query the name via GetFileInformationByHandleEx: FileNameInfo [2], which returns a counted string: >>> GetFileInformationByHandleEx(h, FileNameInfo) '\\spam\x00eggs' The name is in the root path of the device, but we don't get the fully-qualified name "\\Device\\NamedPipe\\spam\x00eggs". WINAPI GetFinalPathNameByHandleW [3] can figure this out, at least for the native NT path (from NtQueryObject). However, it works with null-terminated strings, so the pipe name gets truncated as "spam": >>> flags = VOLUME_NAME_NT | FILE_NAME_OPENED >>> GetFinalPathNameByHandle(h, flags) '\\Device\\NamedPipe\\spam' [1]: https://docs.microsoft.com/en-us/windows/win32/api/ntdef/ns-ntdef-_object_at... [2]: https://docs.microsoft.com/en-us/windows/win32/api/winbase/ns-winbase-file_n... [3]: https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getfin...

Andrew Barnert wrote:

On May 11, 2020, at 00:40, Steve Jorgensen stevej@stevej.name wrote:

...

Proposal: Add a new function (possibly os.path.sanitizepart) to sanitize a value for use as a single component of a path. In the default case, the value must also not be a reference to the current or parent directory ("." or "..") and must not contain control characters. <snip> If not: the result can contain the path separator, illegal characters that aren’t control characters, nonprinting characters that aren’t control characters, and characters whose bytes (in the filesystem’s encoding) are ASCII control characters? And it can be a reserved name, or even something like C:; as long as it’s not the Unix . or ..?

Are there non-printing characters outside of those in the Unicode general category of "C" that make sense to omit? There are combining characters and such that do not have glyphs but are visible in the sense that they modify the glyphs displayed for the characters that they combine with. Regarding names like "C:", you are absolutely right to point that out. When the platform is Windows, certainly, "<letter>:" should not be allowed, and perhaps colon should not be allowed at all. I'll need to research that a bit. This matters because if the path part is used without explicit "./" prefixed to it, then it will refer to a root path, so same problem as allowing a name starting with "/" in *NIX. That should be unconditionally disallowed in the case of WIN or GENERAL systems.

On Mon, May 11, 2020 at 09:12:52PM -0000, Steve Jorgensen wrote:

When the platform is Windows, certainly, "<letter>:" should not be allowed, and perhaps colon should not be allowed at all.

https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file Forbidden characters: chr(0) < > : " / \ | ? * characters in range from chr(1) through chr(31), a space or a period at the end of file/directory name. Forbidden file names (with any extensions): CON, PRN, AUX, NUL, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, and LPT9. Oleg. -- Oleg Broytman https://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.