Mailman 3 Sanitize filename (path part) - Python-ideas

newer
Sanitize filename (path part) 2nd...

Sanitize filename (path part)

Steve Jorgensen

10 May 2020 10 May '20

6:04 a.m.

I believe the Python standard library should include a means of sanitizing a filesystem entry, and this should not be something requiring a 3rd party package. One of reasons I think this should be in the standard lib is because that provides a common, simple means for code reviewers and static analysis services such as Veracode to recognize that a value is sanitized in an accepted manner. What I am envisioning is a function (presumably in `os.path` with a signature roughly like {{{ sanitizepart(name, permissive=False, mode=ESCAPE, system=None) }}} When `permissive` is `False`, characters that are generally unsafe are rejected. When `permissive` is `True`, only path separator characters are rejected. Generally unsafe characters besides path separators would include things like a leading ".", any non-printing character, any wildcard, piping and redirection characters, etc. The `mode` argument indicates what to do with unacceptable characters. Escape them (`ESCAPE`), omit them (`OMIT`) or raise an exception (`RAISE`). This could also double as an escape character argument when a string is given. The default escape character should probably be "%" (same as URL encoding). The `system` argument accepts a combination of bit flags indicating what operating system's rules to apply, or `None` meaning to use rules for the current platform. Systems would probably include `SYS_POSIX`, `SYS_WIN`, and `SYS_MISC` where miscellaneous means to enforce rules for all commonly used systems. One example of a distinction is that on a POSIX system, backslash characters are not path separators, but on Windows, both forward and backward slashes are path separators. {{{ from os import path from os.path import sanitizepart print(repr( os.path.sanitizepart('/ABC\\QRS%', system=path.SYS_WIN)) # => '%2fABC%5cQRS%%' os.path.sanitizepart('/ABC\\QRS%', True, mode=path.STRIP, system=path.SYS_POSIX)) # => 'ABC\\QRS%' os.path.sanitizepart('../AB&CD*\x01\n', system=path.SYS_POSIX)) # => '%2e.%2fABC%26CD%2a%01%10' os.path.sanitizepart('../AB&CD*\x01\n', True, system=path.SYS_POSIX)) # => '..%2eAB&CD*\x01\n' }}}

Show replies by date

Steve Jorgensen

10 May 10 May

6:08 a.m.

Steve Jorgensen wrote:

...

I believe the Python standard library should include a means of sanitizing a filesystem entry, and this should not be something requiring a 3rd party package. One of reasons I think this should be in the standard lib is because that provides a common, simple means for code reviewers and static analysis services such as Veracode to recognize that a value is sanitized in an accepted manner. What I am envisioning is a function (presumably in os.path with a signature roughly like {{{ sanitizepart(name, permissive=False, mode=ESCAPE, system=None) }}} When permissive is False, characters that are generally unsafe are rejected. When permissive is True, only path separator characters are rejected. Generally unsafe characters besides path separators would include things like a leading ".", any non-printing character, any wildcard, piping and redirection characters, etc. The mode argument indicates what to do with unacceptable characters. Escape them (ESCAPE), omit them (OMIT) or raise an exception (RAISE). This could also double as an escape character argument when a string is given. The default escape character should probably be "%" (same as URL encoding). The system argument accepts a combination of bit flags indicating what operating system's rules to apply, or None meaning to use rules for the current platform. Systems would probably include SYS_POSIX, SYS_WIN, and SYS_MISC where miscellaneous means to enforce rules for all commonly used systems. One example of a distinction is that on a POSIX system, backslash characters are not path separators, but on Windows, both forward and backward slashes are path separators. {{{ from os import path from os.path import sanitizepart print(repr( os.path.sanitizepart('/ABC\QRS%', system=path.SYS_WIN)) # => '%2fABC%5cQRS%%' os.path.sanitizepart('/ABC\\QRS%', True, mode=path.STRIP, system=path.SYS_POSIX))

# => 'ABC\QRS%' os.path.sanitizepart('../AB&CD*\x01\n', system=path.SYS_POSIX))

# => '%2e.%2fABC%26CD%2a%01%10' os.path.sanitizepart('../AB&CD*\x01\n', True, system=path.SYS_POSIX))

# => '..%2eAB&CD*\x01\n' }}}

Existing work: https://pypi.org/project/pathvalidate/#sanitize-a-filename

Steve Jorgensen

3:19 p.m.

Steve Jorgensen wrote:

...

Steve Jorgensen wrote:

...
I believe the Python standard library should include a means of sanitizing a filesystem entry, and this should not be something requiring a 3rd party package. One of reasons I think this should be in the standard lib is because that provides a common, simple means for code reviewers and static analysis services such as Veracode to recognize that a value is sanitized in an accepted manner. What I am envisioning is a function (presumably in os.path with a signature roughly like {{{ sanitizepart(name, permissive=False, mode=ESCAPE, system=None) }}} When permissive is False, characters that are generally unsafe are rejected. When permissive is True, only path separator characters are rejected. Generally unsafe characters besides path separators would include things like a leading ".", any non-printing character, any wildcard, piping and redirection characters, etc. The mode argument indicates what to do with unacceptable characters. Escape them (ESCAPE), omit them (OMIT) or raise an exception (RAISE). This could also double as an escape character argument when a string is given. The default escape character should probably be "%" (same as URL encoding). The system argument accepts a combination of bit flags indicating what operating system's rules to apply, or None meaning to use rules for the current platform. Systems would probably include SYS_POSIX, SYS_WIN, and SYS_MISC where miscellaneous means to enforce rules for all commonly used systems. One example of a distinction is that on a POSIX system, backslash characters are not path separators, but on Windows, both forward and backward slashes are path separators. {{{ from os import path from os.path import sanitizepart print(repr( os.path.sanitizepart('/ABC\QRS%', system=path.SYS_WIN)) # => '%2fABC%5cQRS%%' os.path.sanitizepart('/ABC\QRS%', True, mode=path.STRIP, system=path.SYS_POSIX)) # => 'ABC\QRS%' os.path.sanitizepart('../AB&CD*\x01\n', system=path.SYS_POSIX)) # => '%2e.%2fABC%26CD%2a%01%10' os.path.sanitizepart('../AB&CD*\x01\n', True, system=path.SYS_POSIX)) # => '..%2eAB&CD*\x01\n' }}} Existing work: https://pypi.org/project/pathvalidate/#sanitize-a-filename

More existing work: * https://pypi.org/project/sanitize-filename/ * http://detox.sourceforge.net/ * https://sourceforge.net/p/glindra/news/2005/08/glindra-rename--lower--portab...

Andrew Barnert

11:03 a.m.

On May 9, 2020, at 17:35, Steve Jorgensen wrote:

...

I believe the Python standard library should include a means of sanitizing a filesystem entry, and this should not be something requiring a 3rd party package.

One of reasons I think this should be in the standard lib is because that provides a common, simple means for code reviewers and static analysis services such as Veracode to recognize that a value is sanitized in an accepted manner.

This does seem like a good idea. People who do this themselves get it wrong all the time, occasionally with disastrous consequences, so if Python can solve that, that would be great. But, at least historically, this has been more complicated than what you’re suggesting here. For example, don’t you have to catch things like directories named “Con” or files whose 8.3 representation has “CON” as the 8 part? I don’t think you can hang an entire Windows system by abusing those anymore, but you can still produce filenames that some APIs, and some tools (possibly including Explorer, cmd, powershell, Cygwin, mingw/native shells, Python itself…) can’t access (or can only access if the user manually specified a \\.\ absolute path, or whatever). Is there an established algorithm/rule that lots of people in the industry trust that Python can just reference, instead of having to research or invent it? Because otherwise, we run the risk of making things worse instead of better.

...

What I am envisioning is a function (presumably in `os.path` with a signature roughly like {{{ sanitizepart(name, permissive=False, mode=ESCAPE, system=None) }}}

Maybe it would make more sense to put this in pathlib. Then you construct a PurePath of the appropriate type, and call sanitize() on it (maybe with a flag that ensures that it’s a single path component if you expected it to be one). I think some, but not all, of this logic already exists in pathlib.

...

When `permissive` is `False`, characters that are generally unsafe are rejected. When `permissive` is `True`, only path separator characters are rejected. Generally unsafe characters besides path separators would include things like a leading ".", any non-printing character, any wildcard, piping and redirection characters, etc.

I think neither of these is what I’d usually want. I never want to sanitize just pathsep characters without sanitizing all illegal characters. I do often want to sanitize all illegal characters (just \0 and the path sep on POSIX, a larger set that I don’t know by heart on Windows). I don’t think I’ve ever wanted to sanitize the set of potentially-unsafe characters you’re proposing here. I have wanted to sanitize (or pop up an “are you sure?” dialog, etc.) a wider range of potentially confusing characters. For example, newlines or Unicode separators can be very confusing in filenames. I’ve used one of those “potentially misleading URL” libs for this even though files and URLs aren’t quite the same and it was definitely overzealous, but if I’m not really confident that someone has thought through the details and widely vetted them, I’d rather have overzealous than underzealous for something like this. Meanwhile, on POSIX, it’s actually bytes rather than characters that are illegal. Any character that, in the filesystem’s encoding, would have a \0 or \x2f is therefore illegal. Of course in UTF-8, the only such characters are NUL and /, so in scripts I write for my own use on my own systems where I know all the filesystems are UTF-8 I don’t worry about this But mething meant for hardening/verification tools seems like it needs to meet a higher standard and work on more varied systems. And I don’t know how you could even apply the right rule without knowing what the file system encoding is (which means you need the full path, not just the component to be checked) or requiring bytes rather than str (but then it doesn’t work for Windows, and resolving that whole mess gets extra fun, and even on POSIX it’s a lot less common to use). Speaking of encodings and Windows, isn’t any character not in the user’s OEM code page likely to be confusing? Sure, it’ll work with other Python 3.8 scripts, but it’ll crash or do the wrong thing or display mojibake when used with lots of other tools.

...

The `mode` argument indicates what to do with unacceptable characters. Escape them (`ESCAPE`), omit them (`OMIT`) or raise an exception (`RAISE`).

What’s the exception, and what attributes does it have? Usually I don’t care too much as long as the traceback/log entry/whatever is good enough for debugging, but for this function, I think I’d often want to be able to programmatically access the character(s) that triggered the error so I can tell the user. Especially if the rule isn’t a fixed, well-known one that you can describe the way Windows Explorer does when you try to use an illegal character.

...

This could also double as an escape character argument when a string is given. The default escape character should probably be "%" (same as URL encoding).

But % only makes sense with a specific encoding of the escaped character, which is a totally different encoding than the one used by other escape mechanisms, so how can just an escape string select between them? If I give it \U expecting to get JSON escapes but instead get % escapes with \U in place of %, that won’t be at all useful. In fact, passing any string at all besides % won’t be at all useful, because I don’t think there’s any other escape mechanism with the same rules as %-encoding but a different escape character. Not only that, but %-encoding doesn’t make sense with a different list of characters to be encoded than the one actually used by URLs, most obviously because % itself is not an unsafe filename character but you’d better be escaping it anyway, or the system is ridiculously easy to break/exploit. More importantly, what would %-encoding be good for? No other program—including Finder/Explorer, native GUI apps, native shell tools, etc.—will know how to generate the same name from the same user input, much less how to convert it back to something human-readable, etc. Even browsers following file: URLs won’t be able to use these names, and in fact it’ll be pretty confusing that (a) it’s misleadingly close to URL escaping but not the same, and (b) you have to %-escape the %-escaped filename to actually get a usable file URL out of it.

Steve Jorgensen

2:22 p.m.

Responding to points individually to avoid confusing multi-topic threads. :) Andrew Barnert wrote: < snip >

...

...
When permissive is False, characters that are generally unsafe are rejected. When permissive is True, only path separator characters are rejected. Generally unsafe characters besides path separators would include things like a leading ".", any non-printing character, any wildcard, piping and redirection characters, etc. I think neither of these is what I’d usually want. I never want to sanitize just pathsep characters without sanitizing all illegal characters. I do often want to sanitize all illegal characters (just \0 and the path sep on POSIX, a larger set that I don’t know by heart on Windows).

Sanitization and validation are not the same thing though. \0 is invalid and will result in an error when passed to a function that attempts to use it to reference a file, so allowing that character to pass through sanitization doesn't constitute an exploitable vulnerability. Having said that, it's usually friendlier to fail sooner rather than later, so it maybe it actually does make sense for sanitization to fail for illegal characters as well as for valid, unsafe characters. Hmm. I just realized that "..." and (to a lesser extent) "." are valid path parts but are nevertheless usually not safe to allow.

Steve Jorgensen

2:45 p.m.

Andrew Barnert wrote:

...

...
I believe the Python standard library should include a means of sanitizing a filesystem entry, and this should not be something requiring a 3rd party package. One of reasons I think this should be in the standard lib is because that provides a common, simple means for code reviewers and static analysis services such as Veracode to recognize that a value is sanitized in an accepted manner. This does seem like a good idea. People who do this themselves get it wrong all

On May 9, 2020, at 17:35, Steve Jorgensen stevej@stevej.name wrote: the time, occasionally with disastrous consequences, so if Python can solve that, that would be great. But, at least historically, this has been more complicated than what you’re suggesting here. For example, don’t you have to catch things like directories named “Con” or files whose 8.3 representation has “CON” as the 8 part? I don’t think you can hang an entire Windows system by abusing those anymore, but you can still produce filenames that some APIs, and some tools (possibly including Explorer, cmd, powershell, Cygwin, mingw/native shells, Python itself…) can’t access (or can only access if the user manually specified a \.\ absolute path, or whatever).

Yes. I am aware of some of the unsafe names in DOS and older Windows. As I mentioned in my other reply, there is a distinction between the ones that are merely invalid and those that are actually unsafe. In researching existing Linux tools just now, I was reminded that a leading dash is frequently unsafe because many tools will treat an argument starting with dash as an option argument.

...

Is there an established algorithm/rule that lots of people in the industry trust that Python can just reference, instead of having to research or invent it? Because otherwise, we run the risk of making things worse instead of better.

An excellent point! I just started digging into that and found references to detox and Glindra. Neither of those seems to be well maintained though. The documentation pages for Glindra no longer exist and detox is not in standard package repositories for CentOS later than 6 (and only in EPEL for that. Still digging.

Steve Jorgensen

2:59 p.m.

Steve Jorgensen wrote:

...

...
On May 9, 2020, at 17:35, Steve Jorgensen stevej@stevej.name wrote: I believe the Python standard library should include a means of sanitizing a filesystem entry, and this should not be something requiring a 3rd party package. One of reasons I think this should be in the standard lib is because that provides a common, simple means for code reviewers and static analysis services such as Veracode to recognize that a value is sanitized in an accepted manner. This does seem like a good idea. People who do this themselves get it wrong all the time, occasionally with disastrous consequences, so if Python can solve that, that would be great. But, at least historically, this has been more complicated than what you’re suggesting here. For example, don’t you have to catch things like directories named “Con” or files whose 8.3 representation has “CON” as the 8 part? I don’t think you can hang an entire Windows system by abusing those anymore, but you can still produce filenames that some APIs, and some tools (possibly including Explorer, cmd, powershell, Cygwin, mingw/native shells, Python itself…) can’t access (or can only access if the user manually specified a .\ absolute path, or whatever). Yes. I am aware of some of the unsafe names in DOS and older Windows. As I mentioned in my other reply, there is a distinction between the ones that are merely invalid and those that are actually unsafe. In researching existing Linux tools just now, I was reminded that a leading dash is frequently unsafe because many tools will treat an argument starting with dash as an option argument. Is there an established algorithm/rule that lots of people in the industry trust that Python can just reference, instead of having to research or invent it? Because otherwise, we run the risk of making things worse instead of better. An excellent point! I just started digging into that and found references to detox and Glindra. Neither of those seems to be well maintained though. The documentation

Andrew Barnert wrote: pages for Glindra no longer exist and detox is not in standard package repositories for CentOS later than 6 (and only in EPEL for that. Still digging.

Extremely apropos to the question of what charters might be problematic and/or unsafe: https://dwheeler.com/essays/fixing-unix-linux-filenames.html

Steve Jorgensen

3:04 p.m.

Steve Jorgensen wrote:

...

Steve Jorgensen wrote:

...
Andrew Barnert wrote: On May 9, 2020, at 17:35, Steve Jorgensen stevej@stevej.name wrote: I believe the Python standard library should include a means of sanitizing a filesystem entry, and this should not be something requiring a 3rd party package. One of reasons I think this should be in the standard lib is because that provides a common, simple means for code reviewers and static analysis services such as Veracode to recognize that a value is sanitized in an accepted manner. This does seem like a good idea. People who do this themselves get it wrong all the time, occasionally with disastrous consequences, so if Python can solve that, that would be great. But, at least historically, this has been more complicated than what you’re suggesting here. For example, don’t you have to catch things like directories named “Con” or files whose 8.3 representation has “CON” as the 8 part? I don’t think you can hang an entire Windows system by abusing those anymore, but you can still produce filenames that some APIs, and some tools (possibly including Explorer, cmd, powershell, Cygwin, mingw/native shells, Python itself…) can’t access (or can only access if the user manually specified a .\ absolute path, or whatever). Yes. I am aware of some of the unsafe names in DOS and older Windows. As I mentioned in my other reply, there is a distinction between the ones that are merely invalid and those that are actually unsafe. In researching existing Linux tools just now, I was reminded that a leading dash is frequently unsafe because many tools will treat an argument starting with dash as an option argument. Is there an established algorithm/rule that lots of people in the industry trust that Python can just reference, instead of having to research or invent it? Because otherwise, we run the risk of making things worse instead of better. An excellent point! I just started digging into that and found references to detox and Glindra. Neither of those seems to be well maintained though. The documentation pages for Glindra no longer exist and detox is not in standard package repositories for CentOS later than 6 (and only in EPEL for that. Still digging. Extremely apropos to the question of what charters might be problematic and/or unsafe: https://dwheeler.com/essays/fixing-unix-linux-filenames.html

That article links to another by the same author that is specific to vulnerabilities caused by file names. https://dwheeler.com/secure-programs/Secure-Programs-HOWTO/file-names.html

Wes Turner

12 May 12 May

2:18 a.m.

...

[...] If a component is an absolute path, all previous components are

FWIW, here are some of the CWE codes for related vulnerabilities/weaknesses in implementations: CWE-73: External Control of File Name or Path https://cwe.mitre.org/data/definitions/73.html CWE-707: Improper Neutralization https://cwe.mitre.org/data/definitions/707.html CWE-22: Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal') https://cwe.mitre.org/data/definitions/22.html Because this behavior of os.path.join is documented, it's not a vuln in Python, it's a vuln in every downstream component that (1) uses os.path.join with user supplied input; and that (2) doesn't strip a leading '/' from path parts before joining them with os.path.join. https://docs.python.org/3/library/os.path.html#os.path.join thrown away and joining continues from the absolute path component. [quoting from "part 2"] What does sanitizepart do with a leading slash? assert os.path.join("a", "/b") == "/b" A new safejoin() or joinsafe() or join(safe='True') could call sanitizepart() such that: assert joinsafe("a\n", "/b") == "a\\n/b" On Sun, May 10, 2020 at 5:36 AM Steve Jorgensen wrote:

...

...
Steve Jorgensen wrote:

...
Andrew Barnert wrote: On May 9, 2020, at 17:35, Steve Jorgensen stevej@stevej.name wrote: I believe the Python standard library should include a means of sanitizing a filesystem entry, and this should not be something requiring a 3rd party package. One of reasons I think this should be in the standard lib is because

...
...
common, simple means for code reviewers and static analysis services such as Veracode to recognize that a value is sanitized in an accepted manner. This does seem like a good idea. People who do this themselves get it wrong all the time, occasionally with disastrous consequences, so if Python can solve that, that would be great. But, at least historically, this has been more complicated than what you’re suggesting here. For example, don’t you have to catch things like directories named “Con” or files whose 8.3 representation has “CON” as the 8 part? I don’t think you can hang an entire Windows system by abusing those anymore, but you can still produce filenames that some APIs, and some tools (possibly including Explorer, cmd, powershell, Cygwin, mingw/native shells, Python itself…) can’t access (or can only access if the user manually specified a .\ absolute path, or whatever). Yes. I am aware of some of the unsafe names in DOS and older Windows. As I mentioned in my other reply, there is a distinction between the ones

...
...
invalid and those that are actually unsafe. In researching existing Linux tools just now, I was reminded that a leading dash is frequently unsafe because many tools will treat an argument starting with dash as an option argument. Is there an established algorithm/rule that lots of people in the industry trust that Python can just reference, instead of having to research or invent it? Because otherwise, we run the risk of making things worse instead of better. An excellent point! I just started digging into that and found references to detox and Glindra. Neither of those seems to be well maintained

Steve Jorgensen wrote: that provides a that are merely though. The documentation

...
...
pages for Glindra no longer exist and detox is not in standard package repositories for CentOS later than 6 (and only in EPEL for that. Still digging. Extremely apropos to the question of what charters might be problematic and/or unsafe: https://dwheeler.com/essays/fixing-unix-linux-filenames.html

That article links to another by the same author that is specific to vulnerabilities caused by file names. https://dwheeler.com/secure-programs/Secure-Programs-HOWTO/file-names.html _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/FDZOXS... Code of Conduct: http://python.org/psf/codeofconduct/

Wes Turner

2:19 a.m.

(Is it almost always better to just use a hash of the provided filename (maybe in a p/a/ir/tree234 implementation to avoid the max files in a directory limit of whichever filesystem) instead of the user-supplied filename string?) On Mon, May 11, 2020 at 4:48 PM Wes Turner wrote:

...

FWIW, here are some of the CWE codes for related vulnerabilities/weaknesses in implementations:

CWE-73: External Control of File Name or Path https://cwe.mitre.org/data/definitions/73.html

CWE-707: Improper Neutralization https://cwe.mitre.org/data/definitions/707.html

CWE-22: Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal') https://cwe.mitre.org/data/definitions/22.html

Because this behavior of os.path.join is documented, it's not a vuln in Python, it's a vuln in every downstream component that (1) uses os.path.join with user supplied input; and that (2) doesn't strip a leading '/' from path parts before joining them with os.path.join.

...
[...] If a component is an absolute path, all previous components are

https://docs.python.org/3/library/os.path.html#os.path.join thrown away and joining continues from the absolute path component.

[quoting from "part 2"] What does sanitizepart do with a leading slash?

assert os.path.join("a", "/b") == "/b"

A new safejoin() or joinsafe() or join(safe='True') could call sanitizepart() such that:

assert joinsafe("a\n", "/b") == "a\\n/b"

On Sun, May 10, 2020 at 5:36 AM Steve Jorgensen wrote:

...
...
Steve Jorgensen wrote:

...
Andrew Barnert wrote: On May 9, 2020, at 17:35, Steve Jorgensen stevej@stevej.name wrote: I believe the Python standard library should include a means of sanitizing a filesystem entry, and this should not be something requiring a 3rd party package. One of reasons I think this should be in the standard lib is because

...
...
common, simple means for code reviewers and static analysis services such as Veracode to recognize that a value is sanitized in an accepted manner. This does seem like a good idea. People who do this themselves get it wrong all the time, occasionally with disastrous consequences, so if Python can solve that, that would be great. But, at least historically, this has been more complicated than what you’re suggesting here. For example, don’t you have to catch things like directories named “Con” or files whose 8.3 representation has “CON” as the 8 part? I don’t think you can hang an entire Windows system by abusing those anymore, but you can still produce filenames that some APIs, and some tools (possibly including Explorer, cmd, powershell, Cygwin, mingw/native shells, Python itself…) can’t access (or can only access if the user manually specified a .\ absolute path, or whatever). Yes. I am aware of some of the unsafe names in DOS and older Windows. As I mentioned in my other reply, there is a distinction between the ones

...
...
invalid and those that are actually unsafe. In researching existing Linux tools just now, I was reminded that a leading dash is frequently unsafe because many tools will treat an argument starting with dash as an option argument. Is there an established algorithm/rule that lots of people in the industry trust that Python can just reference, instead of having to research or invent it? Because otherwise, we run the risk of making things worse instead of better. An excellent point! I just started digging into that and found references to detox and Glindra. Neither of those seems to be well maintained

...
...
pages for Glindra no longer exist and detox is not in standard

...
...
CentOS later than 6 (and only in EPEL for that. Still digging. Extremely apropos to the question of what charters might be

Steve Jorgensen wrote: that provides a that are merely though. The documentation package repositories for problematic

...
and/or unsafe: https://dwheeler.com/essays/fixing-unix-linux-filenames.html

That article links to another by the same author that is specific to vulnerabilities caused by file names. https://dwheeler.com/secure-programs/Secure-Programs-HOWTO/file-names.html _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/FDZOXS... Code of Conduct: http://python.org/psf/codeofconduct/

Dan Sommers

10 May 10 May

10:10 p.m.

On Sun, 10 May 2020 00:34:43 -0000 "Steve Jorgensen" wrote:

...

I believe the Python standard library should include a means of sanitizing a filesystem entry, and this should not be something requiring a 3rd party package.

I'm not disagreeing.

...

What I am envisioning is a function (presumably in `os.path` with a signature roughly like {{{ sanitizepart(name, permissive=False, mode=ESCAPE, system=None) }}}

...

When `permissive` is `False`, characters that are generally unsafe are rejected. When `permissive` is `True`, only path separator characters are rejected. Generally unsafe characters besides path separators would include things like a leading ".", any non-printing character, any wildcard, piping and redirection characters, etc.

Okay, now I'm disagreeing. ;-) I know what sanitize means (in English and in the technical sense I believe you intend here), but can you provide some context and actual use cases? Sanitize on input so that your application code doesn't "accidentally" spit out the contents of /etc/shadow? Sanitize on output so that your code doesn't produce syntactically broken links in an HTML document or weird results in an xterm? Sanitize in both directions for safe round tripping to a database server? All of those use cases potentially require separate handling, especially in terms of quoting and escaping. For another example, suppose I'm writing a command line utility on a POSIX system to compute a hash of the contents of a file. There's nothing wrong with ".profile" as a file name. Why are you rejecting leading "." characters? What about leading "-"s, or embedded "|"s? Yes, certain shells and shell commands can make them "difficult" to deal with in one way or another, but they're not "generally unsafe." A very, very, very long time ago, we wrote some software for a customer who liked to "editing" our data files to make minor corrections instead of using our software. Our solution was to use "illegal" filenames that the shell rejected, but that an application could access directly anyway. I guess the point is that "sanitize" can mean different things to different parts of a system. Dan -- “Atoms are not things.” – Werner Heisenberg Dan Sommers, http://www.tombstonezero.net/dan

Steve Jorgensen

11 May 11 May

1:34 a.m.

Dan Sommers wrote:

...

...
I believe the Python standard library should include a means of sanitizing a filesystem entry, and this should not be something requiring a 3rd party package. I'm not disagreeing. What I am envisioning is a function (presumably in os.path with a signature roughly like {{{ sanitizepart(name, permissive=False, mode=ESCAPE, system=None) }}} When permissive is False, characters that are generally unsafe are rejected. When permissive is True, only path separator characters are rejected. Generally unsafe characters besides path separators would include things like a leading ".", any non-printing character, any wildcard, piping and redirection characters, etc. Okay, now I'm disagreeing. ;-) I know what sanitize means (in English and in the technical sense I believe you intend here), but can you provide some context and actual use cases? Sanitize on input so that your application code doesn't "accidentally" spit out the contents of /etc/shadow? Sanitize on output so that your code doesn't produce syntactically broken links in an HTML document or weird results in an xterm? Sanitize in both directions for safe round

On Sun, 10 May 2020 00:34:43 -0000 "Steve Jorgensen" stevej@stevej.name wrote: tripping to a database server? All of those use cases potentially require separate handling, especially in terms of quoting and escaping. For another example, suppose I'm writing a command line utility on a POSIX system to compute a hash of the contents of a file. There's nothing wrong with ".profile" as a file name. Why are you rejecting leading "." characters? What about leading "-"s, or embedded "|"s? Yes, certain shells and shell commands can make them "difficult" to deal with in one way or another, but they're not "generally unsafe." A very, very, very long time ago, we wrote some software for a customer who liked to "editing" our data files to make minor corrections instead of using our software. Our solution was to use "illegal" filenames that the shell rejected, but that an application could access directly anyway. I guess the point is that "sanitize" can mean different things to different parts of a system. Dan

I totally get what you're saying. For the sake of simplicity, I thought that the 2 permissiveness options should be one that only prevents path traversal and one that is extremely conservative, omitting characters that are often safe and appropriate but may be unsafe in some cases. In regard to dot files, those can be safe in some cases, but unsafe in others — writing to configuration files that will be read by shell helpers or editors, for instance.

Eric V. Smith

2:12 a.m.

On 5/10/2020 4:04 PM, Steve Jorgensen wrote:

...

I totally get what you're saying. For the sake of simplicity, I thought that the 2 permissiveness options should be one that only prevents path traversal and one that is extremely conservative, omitting characters that are often safe and appropriate but may be unsafe in some cases.

In regard to dot files, those can be safe in some cases, but unsafe in others — writing to configuration files that will be read by shell helpers or editors, for instance.

I don't see how it's realistic to come up with a version that would fit in the stdlib, especially when the stdlib itself has no need for it. It seems like this would be best on PyPI, and I understand there's already at least a few examples of that. Eric

Steve Jorgensen

1:49 a.m.

Dan Sommers wrote: <snip>

...

I know what sanitize means (in English and in the technical sense I believe you intend here), but can you provide some context and actual use cases? Sanitize on input so that your application code doesn't "accidentally" spit out the contents of /etc/shadow? Sanitize on output so that your code doesn't produce syntactically broken links in an HTML document or weird results in an xterm? Sanitize in both directions for safe round tripping to a database server?

I'm thinking of this specifically in terms of sanitizing input, assuming that later usage of the value might or might not properly protect against potential vulnerabilities. This is also limited to the case where the value is supposed to be a single path referring to an entry within a single directory context.

Stephen J. Turnbull

11:08 a.m.

Steve Jorgensen writes:

...

I'm thinking of this specifically in terms of sanitizing input, assuming that later usage of the value might or might not properly protect against potential vulnerabilities. This is also limited to the case where the value is supposed to be a single path referring to an entry within a single directory context.

This sounds extremely specialized to me. For example, presumably you're not referring to dotted module specifications in Python, but those usually do map to filesystem paths in implementations, and I can imagine vulnerabilities (the one on top of my head requires a fair amount of Python ignorance and environmental serendipity, which sort of proves my point about situation-specificity) using Python module paths as mapped to filesystem paths. ISTM that it might be useful to provide a toolbox for scanning paths with various validation operations, but that it's really up to applications to decide which operations to use and what parameters (eg, evil code point set, bytes vs code points vs code units vs characters), and so on. PyPI seems ideal for that, until it matures more than a discussion on the mailing lists can provide. Steve (T)

Steve Jorgensen

11:43 a.m.

Stephen J. Turnbull wrote:

...

...
I'm thinking of this specifically in terms of sanitizing input, assuming that later usage of the value might or might not properly protect against potential vulnerabilities. This is also limited to the case where the value is supposed to be a single path referring to an entry within a single directory context. This sounds extremely specialized to me. For example, presumably you're not referring to dotted module specifications in Python, but

Steve Jorgensen writes: those usually do map to filesystem paths in implementations, and I can imagine vulnerabilities (the one on top of my head requires a fair amount of Python ignorance and environmental serendipity, which sort of proves my point about situation-specificity) using Python module paths as mapped to filesystem paths. ISTM that it might be useful to provide a toolbox for scanning paths with various validation operations, but that it's really up to applications to decide which operations to use and what parameters (eg, evil code point set, bytes vs code points vs code units vs characters), and so on. PyPI seems ideal for that, until it matures more than a discussion on the mailing lists can provide. Steve (T)

…so maybe it makes sense to have only the more specific sanitization in the standard library, then. In the POSIX case, I think that means just blocking "/" characters and "." or ".." values.

Barry Scott

12 May 12 May

1:22 a.m.

...

On 10 May 2020, at 01:34, Steve Jorgensen wrote:

I believe the Python standard library should include a means of sanitizing a filesystem entry, and this should not be something requiring a 3rd party package.

snip I found that I needed to have code that could tell me if a filename was valid for the OS I'm on. I'm not sure where sanitising would be useful, if in valid I ask the user to fix with suitable feedback in my UI. There is more than one problem to address. 1. Is the string valid as the path to a filename on this OS and a particular file system? 2. Does this valid path refer to a device and not a file? 3. Is this path meeting the security requirements of the application? (1) is possible to code against specs for Windows, macOS and posix for the default file system. Knowing the exact file system allows further constraints to be checked for. (2) on posix is usually a check using stat() for the type of the file. On Windows this check is complicated by needing to know the names of all the devices and check for them. There are API calls that allow this list to be determined at runtime. And the parsing rules mean that "COM1" is an RS232 port as is "c:\windows\com1" and "com1.txt" (3) needs a threat-model to determine that paths that are considered a security risk. Implementing (1) and (2) is doable. (3) might be possible as a API that takes a list of black-listed locations to check for. I have code for (1) and a weak version of (2) in SCM Workbench. For (3) I have relied on file system permissions to prevent harm. Windows version (MSDN documents the char set to that is allowed): __filename_bad_chars_set = set( '\\:/\000?<>*|"' ) __filename_reserved_names = set( ['nul', 'con', 'aux', 'prn', 'com1', 'com2', 'com3', 'com4', 'com5', 'com6', 'com7', 'com8', 'com9', 'lpt1', 'lpt2', 'lpt3', 'lpt4', 'lpt5', 'lpt6', 'lpt7', 'lpt8', 'lpt9', ] ) def isInvalidFilename( filename ): name_set = set( filename ) if len( name_set.intersection( __filename_bad_chars_set ) ) != 0: return True name = filename.split( '.' )[0] if name.lower() in __filename_reserved_names: return True return False macOS and Unix version (I only use Unicode input so avoid the random bytes problems): __filename_bad_chars_set = set( '/\000' ) def isInvalidFilename( filename ): name_set = set( filename ) if len( name_set.intersection( __filename_bad_chars_set ) ) != 0: return True return False Barry

Andrew Barnert

3:54 a.m.

On May 11, 2020, at 13:31, Barry Scott wrote:

...

macOS and Unix version (I only use Unicode input so avoid the random bytes problems):

But that doesn’t avoid the problem. If someone gives you a character whose encoding on the target filesystem includes a null or pathsep byte, your sanitizer will pass it as safe, when it shouldn’t. This isn’t possible on macOS because the OS won’t let you mount any filesystem whose encoding isn’t UTF-8, but it is possible on most other *nixes, and it has been used as an attack in the past. Is it still a realistic problem today? I don’t know. I’m pretty sure the modern versions of Shift-JIS, EUC-*, Big5, and GB can never have continuation bytes below 0x30, but even if I’m right, are these (and UTF-8, of course) the only multi-byte encodings anyone ever uses on Unix filesystems?

Barry Scott

2:02 p.m.

...

On 11 May 2020, at 23:24, Andrew Barnert wrote:

On May 11, 2020, at 13:31, Barry Scott wrote:

...
macOS and Unix version (I only use Unicode input so avoid the random bytes problems):

But that doesn’t avoid the problem. If someone gives you a character whose encoding on the target filesystem includes a null or pathsep byte, your sanitizer will pass it as safe, when it shouldn’t.

Do you have a example that shows an encoding that produces a NUL or pathsep? I'm not aware of any.

...

This isn’t possible on macOS because the OS won’t let you mount any filesystem whose encoding isn’t UTF-8, but it is possible on most other *nixes, and it has been used as an attack in the past.

Indeed the case of mounting an NTFS filesystem on Linux now requires the use of the NTFS rules to validate the filename,

...

Is it still a realistic problem today? I don’t know. I’m pretty sure the modern versions of Shift-JIS, EUC-*, Big5, and GB can never have continuation bytes below 0x30, but even if I’m right, are these (and UTF-8, of course) the only multi-byte encodings anyone ever uses on Unix filesystems?

I suspect that legacy encoding are used in organisations with old data, but do have direct experience of this. Barry

Andrew Barnert

13 May 13 May

12:32 a.m.

On May 12, 2020, at 01:32, Barry Scott wrote:

...

...
On 11 May 2020, at 23:24, Andrew Barnert wrote:

...
On May 11, 2020, at 13:31, Barry Scott wrote:

macOS and Unix version (I only use Unicode input so avoid the random bytes problems):

But that doesn’t avoid the problem. If someone gives you a character whose encoding on the target filesystem includes a null or pathsep byte, your sanitizer will pass it as safe, when it shouldn’t.

Do you have a example that shows an encoding that produces a NUL or pathsep? I'm not aware of any.

UTF-1 encodes U+D7FF to the bytes F7 2F C3. BOCU has similar examples. In the other direction, MUTF-8 decodes the bytes CO 80 to U+0000. There were a number of cross-site scripting and misleading-link attacks abusing (mostly) BOCU in this way, which is part of the reason WHATWG banned them as charsets. Although there were other reasons (they banned stuff like SCSU and CESU-8 and UTF-7 at the same time, and I don’t think any of them have the same problem). And if there were widespread legitimate uses of these codecs, they probably wouldn’t have been banned (see UTF-16LE, which is even easier to exploit this way, but unfortunately way too common). I don’t think Python comes with codecs for any of these encodings. And I don’t know of anyone who ever used them for filenames. (SCSU was the default fs encoding on Symbian flash memory drives, but again, I don’t think it has this problem.) So this may well not be a practical problem.

...

...
Is it still a realistic problem today? I don’t know. I’m pretty sure the modern versions of Shift-JIS, EUC-*, Big5, and GB can never have continuation bytes below 0x30, but even if I’m right, are these (and UTF-8, of course) the only multi-byte encodings anyone ever uses on Unix filesystems?

I suspect that legacy encoding are used in organisations with old data, but do have direct experience of this.

I have direct experience of some of those East Asian codecs, albeit 15 or so years ago. I’m pretty sure the only ones they used were all safe. I also have experience even further back of mounting drives from Ataris and classic Macs and IBM mainframes and all kinds of other crazy things under Unix, but the filesystem drivers recoded filenames on the fly, along with providing a Unix-style hierarchical filesystem, so user-level code didn’t have to worry about MacKorean or EBCDIC or whatever any more than it had to worry about : as a pathsep and absolute paths being the ones that _don’t_ start with a pathsep and so on. So, based on my experience, it doesn’t seem likely to come up even in shops full of old data. But that experience isn’t worth much…

1439

Age (days ago)

1441

Last active (days ago)

List overview

Download

19 comments

7 participants

participants (7)

Andrew Barnert
Barry Scott
Dan Sommers
Eric V. Smith
Stephen J. Turnbull
Steve Jorgensen
Wes Turner

Sanitize filename (path part)

tags

participants (7)