Missing expandvars equivalent in pathlib
All, I know that /os.path/ includes a function /expandvars(..)/ which expands any environment variables in a given path, but from looking at the /pathlib/ documentation It seems there is no equivalent to /os.path.expandvars(..) on any class/ in /pathlib/, and the recommendation seems to be to use /pathlib/ to do any and all path manipulations, with the exception of expanding environment variables. It appears that the recommended /pathlib/ compatible code to fully convert any file path (which may have environment variables and or ~ or ~user constructs) to a fully resolved absolute path is : import os.path import pathlib given_path = input('Provide the full path') abs_path = pathlib.Path(os.path.expandvars(pathlib.Path(givenpath).expanduser())).resolve() It seems to me that /pathlib.Path/ would benefit massively from an /os.path.expandvars(..)/ equivalent - so that the equivalent code could be : import os.path import pathlib given_path = input('Provide the full path') abs_path = pathlib.Path(givenpath).expandvars().expanduser().resolve() I know the need to do this is likely to be a corner case, but even still it continues dependency on /os.path/ for those programs that need to resolve environment variables in paths, and increases the cognitive load for this operation, , and it seems like a missed feature in /pathlib/ A change such as this shouldn't affect backwards compatibility. What do people think - have I missed something or is pathlib missing something ? /Pathlib documentation / /https://docs.python.org/3/library/pathlib.html#module-pathlib/ -- Anthony Flury *Moble*: +44 07743 282707 *Home*: +44 (0)1206 391294 *email*: anthony.flury@btinternet.com
+1 -- I would really like pathlib to be able to used for everything one would need to do with paths. -CHB On Thu, Feb 10, 2022 at 3:05 AM anthony.flury via Python-ideas < python-ideas@python.org> wrote:
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
FYI there was a patch for this in the past and it was rejected: https://bugs.python.org/issue21301 Damian (he/him) On Thu, Feb 10, 2022 at 12:04 PM Christopher Barker <pythonchb@gmail.com> wrote:
10.02.22 12:59, anthony.flury via Python-ideas пише:
expandvars() does not operate on paths, it operates on strings and bytestrings. There is nothing path-specific here. Expanding environment variables consists of three distinct steps: 1. Represent a path as a string. 2. Expand environment variables in the string. 3. Create a new path from a new string. Note that there are two implementations of expandvars(): in posixpath and ntpath. You may want to apply Posix substitution on Windows and Windows substitution on Linux or macOS, so it cannot be tied to PosixPath or WindowsPath. Perhaps the shlex module would more appropriate place for expandvars() than os.path, but what done is done.
On Fri, Feb 11, 2022 at 12:28 AM Serhiy Storchaka <storchaka@gmail.com> wrote:
sure -- but it does live in os.paths now, the docs talk about paths, and it is useful for paths -- so it seems a fine idea to have that functionality in pathlib. Note that there are two implementations of expandvars(): in posixpath
I'm a bit confused here: in os.path it's tied to posixpath or ntpath, so what is it any different to tie it to PosixPath or WindowsPath? Also, IIUC, it expands environment variable from the current environment -- so expanding an environment variable from a non-native system seems pretty esoteric. Perhaps the shlex module would more appropriate place for expandvars()
than os.path, but what done is done.
How about we add it to shlex for the more general (and esoteric) cases, and also add it to pathlib for the path cases? Yes, having essentially the same thing in three places is not-good, but pathlib already duplicated much of os.path, that's exactly the point. ANd while we are at it, it could take a flag indicating what style of envvar expnasion to use: % or $ or both. "The easy things should be easy" If a user wants to expand an environment variable in a path, there should be an easy and obvious way to do that. NOTE: I don't think anyone is suggesting that os.path be deprecated -- but I do think it's a worthy goal to be able to do be able to do everything you need to do with pathlib without having to reach back into os.path -- needing both is kind the worst of both worlds. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Fri, 11 Feb 2022 at 16:37, Christopher Barker <pythonchb@gmail.com> wrote:
One way that tying it to paths is bad is that it ignores the path structure. If environment variable a is "d1/f1" and b is "d2/f2" then "${a}x${b}" is "d1/f1xd2/f2", which could be very confusing to someone who sees the value of b and expects the result to be a file in a directory called d2. Yes, I know that the documentation could make this clear (the docs in os.path don't, BTW) and I know that it's no different than the os.path version, but adding an expandvars method to pathlib feels like throwing good money after bad... Let's leave it where it is and keep the status quo, or fix it *properly* and put it somewhere more logical like shlex. Personally I don't use it that often, so I'm fine with just leaving it alone. Paul
Just had a thought kernel: what if there were an f-string mini-language directive to grab environment variables and expand user paths? That seems to me like it could be even more useful beyond just working with paths. Maybe something like: f"{my_var:$}" This would return the same as os.path.expandvars("$my_var") I am not sure about user name expansion though. Maybe something like: f"{user:~}" ...would return the user directory, and this: f"{:~}" ...would give the same result as os.path.expanduser("~") --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler On Fri, Feb 11, 2022 at 2:04 PM Paul Moore <p.f.moore@gmail.com> wrote:
On Fri, Feb 11, 2022 at 4:58 PM MRAB <python@mrabarnett.plus.com> wrote:
well right now $ is not a valid format string for f-strings. what i'm suggesting is to add $ to the format spec mini-languagem and have the result be an expanded environment variable value. but you're right, i guess it would apply this to the *contents *of the variable my_var, not the STRING my_var. so maybe the spelling would need to use a nested quotation, like this: f"{'my_var':$}" this would be the same as os.path.expandvars("$my_var") --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
On Sat, 12 Feb 2022 at 09:34, Ricky Teachey <ricky@teachey.org> wrote:
Why shoehorn it into f-strings? Even worse: Why should f-strings fetch external information in such a hidden way? One thing I'm not fully aware of here is what precisely is wrong with the status quo. Are you looking for a method instead of a function? Do you need something that takes a Path and returns a Path? The existing os.path.expandvars will happily accept a Path and return a str:
I'm confused, since anything involving f-strings clearly has nothing to do with pathlib (other than in the trivial sense that you could expand vars before constructing a Path), but the OP was very definitely talking about Path objects. ChrisA
On Fri, Feb 11, 2022 at 5:44 PM Chris Angelico <rosuav@gmail.com> wrote:
yes but path objects are usually constructed from a string, and you could support path objects easily in the f-string using my suggested ~ format specifier: Path(f"{my_path_obj:~}") granted this is a bit weird since you're going from a path obj, to a string, and back to a path obj again. i thought of the idea because Serhiy Storchaka brought up the point that the two expand methods that currently live in os.path have more to do with strings than they do paths. because of this f-strings seemed to me like a potentially convenient place to put this functionality. doesn't feel like a shoehorn at all. but i suppose in practice, producing an expanded path like above doesn't seem all that useful. so it's an idea in search of a use case, which means is probably isn't a very good idea. --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
you can already put an arbitrary expression in a f-string -- that's the point. So what's wrong with: In [12]: eget = os.environ.get In [13]: f"{eget('HOME')}/bin" Out[13]: '/Users/chris/bin' This seems a rare enough need that special built-in support is not worth it. -CHB On Fri, Feb 11, 2022 at 6:01 PM Eric V. Smith <eric@trueblade.com> wrote:
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On 2/11/2022 2:01 PM, Paul Moore wrote:
I'd like to see a Path.expandvars(), something like: -------------------------------------- import os def _expand_part(part, allow_sep_in_env_var): # Expand environment variables in 'part'. If allow_sep_in_env_var is # False, the resulting string cannot contain a path separator. s = os.path.expandvars(part) if s == part: return part if not allow_sep_in_env_var and os.path.sep in s: raise ValueError(f"expanded variable {part!r} may not contain a path separator") return s def expandvars(path, allow_sep_in_env_var=False): return type(path)( *list(_expand_part(part, allow_sep_in_env_var) for part in path.parts) ) from pathlib import Path os.environ["USER"] = "foo" os.environ["a"] = "d1/f1" os.environ["b"] = "d1/f1" print(f'{expandvars(Path("/usr/local/bin")) = }') print(f'{expandvars(Path("/home/${USER}")) = }') print(f'{expandvars(Path("${a}x${b}"), True) = }') print(f'{expandvars(Path("${a}x${b}")) = }') -------------------------------------- Output: expandvars(Path("/usr/local/bin")) = PosixPath('/usr/local/bin') expandvars(Path("/home/${USER}")) = PosixPath('/home/foo') expandvars(Path("${a}x${b}"), True) = PosixPath('d1/f1xd1/f1') Traceback (most recent call last): File "expandpaths.py", line 26, in <module> print(f'{expandvars(Path("${a}x${b}")) = }') File "expandpaths.py", line 14, in expandvars return type(path)(*list(_expand_part(part, allow_sep_in_env_var) for part in path.parts)) File "expandpaths.py", line 14, in <genexpr> return type(path)(*list(_expand_part(part, allow_sep_in_env_var) for part in path.parts)) File "expandpaths.py", line 10, in _expand_part raise ValueError(f'expanded variable {part!r} may not contain a path separator') ValueError: expanded variable '${a}x${b}' may not contain a path separator It would no doubt have to do something trickier with forward and back slash detection after the expansion. Eric
On Sun, 13 Feb 2022 at 02:45, Eric V. Smith <eric@trueblade.com> wrote:
Why would you ever set it to False?!? In realistic use, interpolating environment variables into paths is *expected* to be able to include path separators. $HOME might be "/home/foo" or "/root", temporary directories could be anywhere (and, on Windows, on any drive), etc, etc, etc. What is the use-case for a feature designed for paths that can't handle paths? ChrisA
On Sun, 13 Feb 2022 at 02:57, Eric V. Smith <eric@trueblade.com> wrote:
See Paul’s example, copied above. Maybe the code isn’t expecting it.
There wasn't an actual example, just a hypothetical one. I have never once seen something in real usage where you interpolate environment variables into a path, without expecting the environment variables to be paths. It's different with URLs, but pathlib doesn't handle URLs, it handles paths. ChrisA
On Sat, 12 Feb 2022 at 16:02, Chris Angelico <rosuav@gmail.com> wrote:
My example was not intended to illustrate that having slashes within variables wasn't reasonable, but rather that concatenating a string to a variable without an intervening slash *could* be confusing. But equally it might not be: backup = os.path.expandvars("${ORIGFILE}.bak") The point I was trying to make is that thinking of expandvars as acting on strings is straightforward, whereas thinking of the operands as paths can lead to expectations that might be wrong. I'm arguing that we should not link expandvars with paths, even though historically it's part of os.path. Paul
On Sun, 13 Feb 2022 at 03:15, Paul Moore <p.f.moore@gmail.com> wrote:
That's fair, but I don't know of any situation where ORIGFILE would come from an environment variable but be required to be a file name without a path. For instance, if you have an env var saying "put the backup over here", then it most certainly could be used the way you describe, but then the normal expectation would be for it to allow a full path. Much more common, for the expansion you're describing, would be for ORIGFILE to come from elsewhere in the code, so expandvars is irrelevant. ChrisA
On Fri, Feb 11, 2022 at 07:01:12PM +0000, Paul Moore wrote:
"Very confusing". Huh. Maybe they would be less confused if they would look at the entire path expression, instead of just the ${b} envar in isolation. Real environment variables are not usually called "${a}", they have names that reflect their meaning. And they typically have values which are more meaningful than "d1/f1". Real environment variables can be file names, or suffixes or prefixes, or directiory components, but your example where ${a} combines a directory component with a prefix, and ${b} combines a suffix with a file name, is surely a terrible design. Who would do that? No wonder you gave them single letter names, it would be awful to have to come up with a descriptive name for doing this. Or a use-case. But that's okay, people use terrible designs all the time. It is not our job to protect them from themselves. If it was, Python wouldn't have metaclasses, or classes, or functions, or variables, or code. The conclusion you would like us to draw from this invented example is that we should forego supporting envars in pathlib because of the miniscule risk that somebody using a really shitty design with badly thought-out envars might be confused by one of the envars in isolation. I'm sorry Paul, but this sort of made-up, utterly artificial toy example adds no benefit to the discussion, it just adds fear, uncertainty and doubt to an extremely straight-forward feature request. -- Steve
On Sun, 13 Feb 2022 at 01:09, Steven D'Aprano <steve@pearwood.info> wrote:
OK, if that's your view then fine. I agree the example was made up, although I dispute "utterly artificial", as it reflects a genuine concern I have, it's just not one that I can find or construct a non-artificial example for right now. But yes, the example wasn't good. Let's keep it simple. I'm -1 on the request, I don't think expandvars should be added to pathlib. I think os.path isn't a very good place for it currently, but it's what we have. I'm not massively enthusiastic about it being in shlex, but I can't think of a better place for it. So I'm inclined to stick with the status quo, or if it's *really* worth doing anything, move it to shlex. Paul
On Sun, 13 Feb 2022 at 21:47, Paul Moore <p.f.moore@gmail.com> wrote:
Finding a non-artificial example would be important here, given that there are plenty of non-artificial examples to the contrary (cases where env vars are used for arbitrary parts of paths, not being restricted to path components). If there is some incredibly rare situation where that functionality is needed, it can be coded directly in the application.
It could also go into os (alongside environ itself). I'm +0.5 on paths getting a method for variable expansion, +0.5 for os.path.expandvars learning to return a Path if given a Path (it already returns bytes or str based on the input). Either would work. ChrisA
Paul Moore writes:
Let's keep it simple. I'm -1 on the request, I don't think expandvars should be added to pathlib.
Would you be happier with a getenv_as_path, so that one could do p = Path.getenv_as_path('HOME') / Path('.python_history') and on my Mac p == Path('/Users/steve/.python_history')? Steve
On Sun, 13 Feb 2022 at 14:26, Stephen J. Turnbull <stephenjturnbull@gmail.com> wrote:
I don't think *any* of this is much better than the status quo: p = Path(os.getenv("HOME")) / ".python_history" The main point I see of expandvars is to have a way of letting the *user* pass a string that might reference environment variables, and expand them. There's some problematic aspects of that already, namely the platform-dependent handling of % symbols. I assume that the intended use case for expandvars, based on the fact that it's in os.path, is to allow expansion of a variable containing a directory name, such as $HOME, as in the example you give. And that's mostly fine. Windows has some weirdness in that some environment variables can (I think) contain trailing backslashes (it definitely happens with batch file substitution variables, I haven't been able to find an example with environment variables, but I wouldn't bet either way, given how the extremely limited capabilities of batch files can get used to (clumsily) set environment variables at time (look at some of the wrapper scripts for Java apps, for examples). Again, that's probably fine, as sequences of multiple slashes and backslashes should get normalised. But again, I wouldn't be 100% sure. file_next_to_bat_wrapper = os.path.expandvars("%BATFILE_DIR%the_file.txt") is a Python translation of a *very* common batfile idiom, where %~dp0 is "the directory containing the bat file", which has a trailing slash, and you need to *not* add an extra slash if you don't want later filename parsing (in a bat file) to get confused. I can easily imagine someone who's new to Python using this sort of idiom by analogy, and then getting bitten by a variable with no trailing slash. *shrug* As I say, my arguments are mostly "from experience" and "vague concerns" - which people, notably the other Steven, have dismissed (with some justification). Generally, I think that expandvars is mildly risky, but perfectly fine for simple use. People wanting to write high-reliability production code should probably think a bit harder about what they actually want, and whether the limitations of expandvars are acceptable for their needs. I'd be surprised if after doing so, they didn't write their own custom parser (we did for pip, I'm pretty sure). I think that expending a lot of effort trying to "improve" expandvars is probably not going to make it much more useful for that latter group. It might make it more discoverable, and just possibly more convenient, for people who won't have a problem with its limitations, but it might *also* make some people who should use a custom solution, think it's fine and use it when they shouldn't. To be honest though, I'm not sure of the point of trying to persuade me. I'm fairly fundamentally opposed to changing anything here, but I can't express my reservations in a way that will persuade anyone else. Which is fine. Conversely, if you manage to change my mind, it's not likely to make much difference - no-one is going to write or merge a PR purely on the basis that I stopped objecting to the idea ;-) Paul
On Mon, 14 Feb 2022 at 02:34, Paul Moore <p.f.moore@gmail.com> wrote:
Which is why the default should be to follow platform expectations. If I pass a parameter to a script saying "~/foo/bar", that script should be able to expanduser/expandvars to make that behave the way I expect it to. If I'm on Windows and I tell something to write to a file in "%TEMP%\spam.csv", then I expect it to understand what that means. Cross-platform support is nice, but the most common need is for the current platform's normal behaviour.
Not sure, but if there are issues with the way that Python on Windows expands variables, they can be raised as bugs.
And perfectly fine for strings provided by the user. In fact, it's extremely frustrating to work with a program that DOESN'T do proper var and user expansion - I expect "~git/reponame/.git" to expand to the home directory of the git user, not my home directory with the word "git" after it. As long as Python's expanduser/expandvars behave as per platform expectations, the risk is on the user, not on the script.
I want my users to be able to treat my script the way they treat everything else in the system. Not sure what you mean by "limitations of expandvars". It does precisely what it is supposed to do. The only problem is that it always returns a str or a bytes, not a Path, so you'd have to then construct a Path. On the other hand, if you're writing something where platform is irrelevant, then expandvars isn't just limited, it's actually wrong, and you should be doing expansion differently. ChrisA
On Sun, 13 Feb 2022 at 15:43, Chris Angelico <rosuav@gmail.com> wrote:
For better or worse, though, Windows (as an OS) doesn't have a "normal behaviour". %-expansion is a feature of CMD and .bat files, which varies depending on the Windows version and the version of CMD. A significant proportion of command line programs on Windows are ported from Unix and add (some form of) Unix-like behaviour, either as well as, or instead of, % expansion. Remember, I'm supporting the existing behaviour as "good enough" here. I'm not the one advocating change. If you believe that "the current platform's normal behaviour" is anything other than what os.path.expandvars does, then the onus is on you to define what you mean. But I thought the debate here was about "putting a version of expandvars into pathlib". Maybe someone ought to define what that means, in practice. Oh, and just for fun, I just discovered that expandvars does (arguably broken) quoting, too:
Note that it *keeps* the quotes. That's *definitely* not normal platform behaviour on Windows... But anyway, I thought the debate here was about "putting a version of expandvars into pathlib". Maybe someone ought to define what that means, in practice. Remember to make sure that the documentation of Path.expandvars() makes it clear why (Path("Dev team's docs") / Path("${USER}'s files") / Path("somefile.txt")).expandvars() returns Path("Dev team's docs/${USER}'s files/somefile.txt") and not Path("Dev team's docs/Gustav's files/somefile.txt"), so that people don't report it as a bug... Yes, they might do the same with os.path.expandvars, but that's easily responded to - "documentation bug, the docs should mention quoting and should emphasise that the operation treats the argument as a string, not as a path". I'm not sure the same response works when it's a method of a Path object ;-) And I repeat, *please* remember that I'm not the one arguing for change here, I'm arguing that the proposed change is potentially more complex than people are suggesting, and that actually, the benefit isn't sufficient to justify the work. Although as it's not clear yet who is offering to do the work, it may be premature of me to make that argument... Paul
On Sun, 13 Feb 2022 at 15:43, Chris Angelico <rosuav@gmail.com> wrote:
That may or may not work as Windows has inconsistent treatment of multiple separators depending on where they appear in a path. If TEMP is a drive spec, say "t:\", then it expands to "t:\\spam.csv", which is an invalid windows path. If TEMP is a directory spec, "c:\temp\", then it expands to "c:\temp\\spam.csv", which works fine. C:\> dir c:\\temp\junk The filename, directory name, or volume label syntax is incorrect. C:\> dir c:\temp\\junk Volume in drive C has no label. Volume Serial Number is FC52-C692 Directory of c:\temp 2022-02-13 10:09 0 junk
On Mon, 14 Feb 2022 at 05:15, Eric Fahlgren <ericfahlgren@gmail.com> wrote:
Ugh, what a mess. Thanks for the reminder that I don't understand Windows very well. In my defense, I haven't used it much in years, so I'm not sure what the best practice would be here; but I do know that, if there is such a thing as best practice, Path("%TEMP%\\spam.csv").expandvars() should do it. (Or os.path.expandvars(Path("%TEMP%\\spam.csv")) or however it's spelled.) At least Unix has it written into the standard that multiple slashes are equivalent to one. ChrisA
On 2/13/22, Eric Fahlgren <ericfahlgren@gmail.com> wrote:
"c:\\temp\junk" isn't always invalid in CMD, and definitely not in the Windows API. The problem occurs because the DIR command in the CMD shell has legacy support to ignore the drive (e.g. "C:") when the root of the path is exactly two backslashes -- because DOS in the 1980s (i.e. they went out of their to add this behavior in CMD to make it compatible with DOS). To see this, check the "C$" administrative share on "localhost": C:\>dir /b C:\\localhost\C$\Temp\spam.txt File Not Found C:\>echo spam >C:\\Temp\spam.txt C:\>dir /b C:\\localhost\C$\Temp\spam.txt spam.txt Even though using two backslashes for the root of a drive path is allowed in the Windows API itself, it's sill problematic. The path part r"\\path\to\file" can't be used as relative to the current drive of the process because it's always a UNC absolute path. So it should be normalized to r"\path\to\file" as soon as possible, e.g. via GetFullPathNameW(): >>> print(nt._getfullpathname(r'C:\\path\to\file')) C:\path\to\file
On 2/13/22, Paul Moore <p.f.moore@gmail.com> wrote:
For better or worse, though, Windows (as an OS) doesn't have a "normal behaviour". %-expansion is a feature of CMD and .bat files, which
You're overlooking ExpandEnvironmentStringsW() [1], ExpandEnvironmentStringsForUserW(), and PathUnExpandEnvStringsW() [2], which provide basic support for `%` based environment variables in strings. Python's standard library supports winreg.ExpandEnvironmentStrings(). It is critical that the system supports this functionality in order to evaluate REG_EXPAND_SZ values in the registry. [1] https://docs.microsoft.com/en-us/windows/win32/api/processenv/nf-processenv-... [2] https://docs.microsoft.com/en-us/windows/win32/api/shlwapi/nf-shlwapi-pathun...
On Fri, Feb 11, 2022 at 10:26:03AM +0200, Serhiy Storchaka wrote:
expandvars() does not operate on paths, it operates on strings and bytestrings. There is nothing path-specific here.
Paths can be strings and byte-strings too. When I using my OS shell, and I type: ls -l ~/code/ I am passing a path to ls as argument. And when I do either of these in Python: os.listdir(os.path.expanduser('~/code/')) os.listdir(os.path.expandvars('${HOME}/code/')) I am still passing a path to listdir. Its just not a pathlib.Path instance, but it is still a path, just as "123 Main Road" is a street address and "Jane Doe" is a name. Of course environment variables are not restricted to being path components, but neither are Paths: >>> snake = Path('Chordata/Reptilia/Squamata/Serpentes') >>> python = snake/'Pythonidae/Python' >>> rock_python = python/'sebae' >>> burmese_python = python/'bivittatus' Is this an abuse of Path? Hell yes. What are we, the Path Police? :-) The point is, regardless of whatever odd non-path-like things people might happen to have in environment variables, they also have paths in environment variables, and it is a strange lack that pathlib doesn't make it easy to construct paths from environment variables. If people want to shoot themselves in the foot: myfile = Path('${HOME}/${LS_COLORS}/file.txt').expandvars() that's up to them, and who knows, maybe somebody does actually have a file with that path, no matter how odd it looks.
That's odd. I would expect PosixPath.expandvars to use Posix-style substitution, and WindowsPath.expandvars to use Windows-style substitution. -- Steve
+1 -- I would really like pathlib to be able to used for everything one would need to do with paths. -CHB On Thu, Feb 10, 2022 at 3:05 AM anthony.flury via Python-ideas < python-ideas@python.org> wrote:
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
FYI there was a patch for this in the past and it was rejected: https://bugs.python.org/issue21301 Damian (he/him) On Thu, Feb 10, 2022 at 12:04 PM Christopher Barker <pythonchb@gmail.com> wrote:
10.02.22 12:59, anthony.flury via Python-ideas пише:
expandvars() does not operate on paths, it operates on strings and bytestrings. There is nothing path-specific here. Expanding environment variables consists of three distinct steps: 1. Represent a path as a string. 2. Expand environment variables in the string. 3. Create a new path from a new string. Note that there are two implementations of expandvars(): in posixpath and ntpath. You may want to apply Posix substitution on Windows and Windows substitution on Linux or macOS, so it cannot be tied to PosixPath or WindowsPath. Perhaps the shlex module would more appropriate place for expandvars() than os.path, but what done is done.
On Fri, Feb 11, 2022 at 12:28 AM Serhiy Storchaka <storchaka@gmail.com> wrote:
sure -- but it does live in os.paths now, the docs talk about paths, and it is useful for paths -- so it seems a fine idea to have that functionality in pathlib. Note that there are two implementations of expandvars(): in posixpath
I'm a bit confused here: in os.path it's tied to posixpath or ntpath, so what is it any different to tie it to PosixPath or WindowsPath? Also, IIUC, it expands environment variable from the current environment -- so expanding an environment variable from a non-native system seems pretty esoteric. Perhaps the shlex module would more appropriate place for expandvars()
than os.path, but what done is done.
How about we add it to shlex for the more general (and esoteric) cases, and also add it to pathlib for the path cases? Yes, having essentially the same thing in three places is not-good, but pathlib already duplicated much of os.path, that's exactly the point. ANd while we are at it, it could take a flag indicating what style of envvar expnasion to use: % or $ or both. "The easy things should be easy" If a user wants to expand an environment variable in a path, there should be an easy and obvious way to do that. NOTE: I don't think anyone is suggesting that os.path be deprecated -- but I do think it's a worthy goal to be able to do be able to do everything you need to do with pathlib without having to reach back into os.path -- needing both is kind the worst of both worlds. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Fri, 11 Feb 2022 at 16:37, Christopher Barker <pythonchb@gmail.com> wrote:
One way that tying it to paths is bad is that it ignores the path structure. If environment variable a is "d1/f1" and b is "d2/f2" then "${a}x${b}" is "d1/f1xd2/f2", which could be very confusing to someone who sees the value of b and expects the result to be a file in a directory called d2. Yes, I know that the documentation could make this clear (the docs in os.path don't, BTW) and I know that it's no different than the os.path version, but adding an expandvars method to pathlib feels like throwing good money after bad... Let's leave it where it is and keep the status quo, or fix it *properly* and put it somewhere more logical like shlex. Personally I don't use it that often, so I'm fine with just leaving it alone. Paul
Just had a thought kernel: what if there were an f-string mini-language directive to grab environment variables and expand user paths? That seems to me like it could be even more useful beyond just working with paths. Maybe something like: f"{my_var:$}" This would return the same as os.path.expandvars("$my_var") I am not sure about user name expansion though. Maybe something like: f"{user:~}" ...would return the user directory, and this: f"{:~}" ...would give the same result as os.path.expanduser("~") --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler On Fri, Feb 11, 2022 at 2:04 PM Paul Moore <p.f.moore@gmail.com> wrote:
On Fri, Feb 11, 2022 at 4:58 PM MRAB <python@mrabarnett.plus.com> wrote:
well right now $ is not a valid format string for f-strings. what i'm suggesting is to add $ to the format spec mini-languagem and have the result be an expanded environment variable value. but you're right, i guess it would apply this to the *contents *of the variable my_var, not the STRING my_var. so maybe the spelling would need to use a nested quotation, like this: f"{'my_var':$}" this would be the same as os.path.expandvars("$my_var") --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
On Sat, 12 Feb 2022 at 09:34, Ricky Teachey <ricky@teachey.org> wrote:
Why shoehorn it into f-strings? Even worse: Why should f-strings fetch external information in such a hidden way? One thing I'm not fully aware of here is what precisely is wrong with the status quo. Are you looking for a method instead of a function? Do you need something that takes a Path and returns a Path? The existing os.path.expandvars will happily accept a Path and return a str:
I'm confused, since anything involving f-strings clearly has nothing to do with pathlib (other than in the trivial sense that you could expand vars before constructing a Path), but the OP was very definitely talking about Path objects. ChrisA
On Fri, Feb 11, 2022 at 5:44 PM Chris Angelico <rosuav@gmail.com> wrote:
yes but path objects are usually constructed from a string, and you could support path objects easily in the f-string using my suggested ~ format specifier: Path(f"{my_path_obj:~}") granted this is a bit weird since you're going from a path obj, to a string, and back to a path obj again. i thought of the idea because Serhiy Storchaka brought up the point that the two expand methods that currently live in os.path have more to do with strings than they do paths. because of this f-strings seemed to me like a potentially convenient place to put this functionality. doesn't feel like a shoehorn at all. but i suppose in practice, producing an expanded path like above doesn't seem all that useful. so it's an idea in search of a use case, which means is probably isn't a very good idea. --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
you can already put an arbitrary expression in a f-string -- that's the point. So what's wrong with: In [12]: eget = os.environ.get In [13]: f"{eget('HOME')}/bin" Out[13]: '/Users/chris/bin' This seems a rare enough need that special built-in support is not worth it. -CHB On Fri, Feb 11, 2022 at 6:01 PM Eric V. Smith <eric@trueblade.com> wrote:
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On 2/11/2022 2:01 PM, Paul Moore wrote:
I'd like to see a Path.expandvars(), something like: -------------------------------------- import os def _expand_part(part, allow_sep_in_env_var): # Expand environment variables in 'part'. If allow_sep_in_env_var is # False, the resulting string cannot contain a path separator. s = os.path.expandvars(part) if s == part: return part if not allow_sep_in_env_var and os.path.sep in s: raise ValueError(f"expanded variable {part!r} may not contain a path separator") return s def expandvars(path, allow_sep_in_env_var=False): return type(path)( *list(_expand_part(part, allow_sep_in_env_var) for part in path.parts) ) from pathlib import Path os.environ["USER"] = "foo" os.environ["a"] = "d1/f1" os.environ["b"] = "d1/f1" print(f'{expandvars(Path("/usr/local/bin")) = }') print(f'{expandvars(Path("/home/${USER}")) = }') print(f'{expandvars(Path("${a}x${b}"), True) = }') print(f'{expandvars(Path("${a}x${b}")) = }') -------------------------------------- Output: expandvars(Path("/usr/local/bin")) = PosixPath('/usr/local/bin') expandvars(Path("/home/${USER}")) = PosixPath('/home/foo') expandvars(Path("${a}x${b}"), True) = PosixPath('d1/f1xd1/f1') Traceback (most recent call last): File "expandpaths.py", line 26, in <module> print(f'{expandvars(Path("${a}x${b}")) = }') File "expandpaths.py", line 14, in expandvars return type(path)(*list(_expand_part(part, allow_sep_in_env_var) for part in path.parts)) File "expandpaths.py", line 14, in <genexpr> return type(path)(*list(_expand_part(part, allow_sep_in_env_var) for part in path.parts)) File "expandpaths.py", line 10, in _expand_part raise ValueError(f'expanded variable {part!r} may not contain a path separator') ValueError: expanded variable '${a}x${b}' may not contain a path separator It would no doubt have to do something trickier with forward and back slash detection after the expansion. Eric
On Sun, 13 Feb 2022 at 02:45, Eric V. Smith <eric@trueblade.com> wrote:
Why would you ever set it to False?!? In realistic use, interpolating environment variables into paths is *expected* to be able to include path separators. $HOME might be "/home/foo" or "/root", temporary directories could be anywhere (and, on Windows, on any drive), etc, etc, etc. What is the use-case for a feature designed for paths that can't handle paths? ChrisA
On Sun, 13 Feb 2022 at 02:57, Eric V. Smith <eric@trueblade.com> wrote:
See Paul’s example, copied above. Maybe the code isn’t expecting it.
There wasn't an actual example, just a hypothetical one. I have never once seen something in real usage where you interpolate environment variables into a path, without expecting the environment variables to be paths. It's different with URLs, but pathlib doesn't handle URLs, it handles paths. ChrisA
On Sat, 12 Feb 2022 at 16:02, Chris Angelico <rosuav@gmail.com> wrote:
My example was not intended to illustrate that having slashes within variables wasn't reasonable, but rather that concatenating a string to a variable without an intervening slash *could* be confusing. But equally it might not be: backup = os.path.expandvars("${ORIGFILE}.bak") The point I was trying to make is that thinking of expandvars as acting on strings is straightforward, whereas thinking of the operands as paths can lead to expectations that might be wrong. I'm arguing that we should not link expandvars with paths, even though historically it's part of os.path. Paul
On Sun, 13 Feb 2022 at 03:15, Paul Moore <p.f.moore@gmail.com> wrote:
That's fair, but I don't know of any situation where ORIGFILE would come from an environment variable but be required to be a file name without a path. For instance, if you have an env var saying "put the backup over here", then it most certainly could be used the way you describe, but then the normal expectation would be for it to allow a full path. Much more common, for the expansion you're describing, would be for ORIGFILE to come from elsewhere in the code, so expandvars is irrelevant. ChrisA
On Fri, Feb 11, 2022 at 07:01:12PM +0000, Paul Moore wrote:
"Very confusing". Huh. Maybe they would be less confused if they would look at the entire path expression, instead of just the ${b} envar in isolation. Real environment variables are not usually called "${a}", they have names that reflect their meaning. And they typically have values which are more meaningful than "d1/f1". Real environment variables can be file names, or suffixes or prefixes, or directiory components, but your example where ${a} combines a directory component with a prefix, and ${b} combines a suffix with a file name, is surely a terrible design. Who would do that? No wonder you gave them single letter names, it would be awful to have to come up with a descriptive name for doing this. Or a use-case. But that's okay, people use terrible designs all the time. It is not our job to protect them from themselves. If it was, Python wouldn't have metaclasses, or classes, or functions, or variables, or code. The conclusion you would like us to draw from this invented example is that we should forego supporting envars in pathlib because of the miniscule risk that somebody using a really shitty design with badly thought-out envars might be confused by one of the envars in isolation. I'm sorry Paul, but this sort of made-up, utterly artificial toy example adds no benefit to the discussion, it just adds fear, uncertainty and doubt to an extremely straight-forward feature request. -- Steve
On Sun, 13 Feb 2022 at 01:09, Steven D'Aprano <steve@pearwood.info> wrote:
OK, if that's your view then fine. I agree the example was made up, although I dispute "utterly artificial", as it reflects a genuine concern I have, it's just not one that I can find or construct a non-artificial example for right now. But yes, the example wasn't good. Let's keep it simple. I'm -1 on the request, I don't think expandvars should be added to pathlib. I think os.path isn't a very good place for it currently, but it's what we have. I'm not massively enthusiastic about it being in shlex, but I can't think of a better place for it. So I'm inclined to stick with the status quo, or if it's *really* worth doing anything, move it to shlex. Paul
On Sun, 13 Feb 2022 at 21:47, Paul Moore <p.f.moore@gmail.com> wrote:
Finding a non-artificial example would be important here, given that there are plenty of non-artificial examples to the contrary (cases where env vars are used for arbitrary parts of paths, not being restricted to path components). If there is some incredibly rare situation where that functionality is needed, it can be coded directly in the application.
It could also go into os (alongside environ itself). I'm +0.5 on paths getting a method for variable expansion, +0.5 for os.path.expandvars learning to return a Path if given a Path (it already returns bytes or str based on the input). Either would work. ChrisA
Paul Moore writes:
Let's keep it simple. I'm -1 on the request, I don't think expandvars should be added to pathlib.
Would you be happier with a getenv_as_path, so that one could do p = Path.getenv_as_path('HOME') / Path('.python_history') and on my Mac p == Path('/Users/steve/.python_history')? Steve
On Sun, 13 Feb 2022 at 14:26, Stephen J. Turnbull <stephenjturnbull@gmail.com> wrote:
I don't think *any* of this is much better than the status quo: p = Path(os.getenv("HOME")) / ".python_history" The main point I see of expandvars is to have a way of letting the *user* pass a string that might reference environment variables, and expand them. There's some problematic aspects of that already, namely the platform-dependent handling of % symbols. I assume that the intended use case for expandvars, based on the fact that it's in os.path, is to allow expansion of a variable containing a directory name, such as $HOME, as in the example you give. And that's mostly fine. Windows has some weirdness in that some environment variables can (I think) contain trailing backslashes (it definitely happens with batch file substitution variables, I haven't been able to find an example with environment variables, but I wouldn't bet either way, given how the extremely limited capabilities of batch files can get used to (clumsily) set environment variables at time (look at some of the wrapper scripts for Java apps, for examples). Again, that's probably fine, as sequences of multiple slashes and backslashes should get normalised. But again, I wouldn't be 100% sure. file_next_to_bat_wrapper = os.path.expandvars("%BATFILE_DIR%the_file.txt") is a Python translation of a *very* common batfile idiom, where %~dp0 is "the directory containing the bat file", which has a trailing slash, and you need to *not* add an extra slash if you don't want later filename parsing (in a bat file) to get confused. I can easily imagine someone who's new to Python using this sort of idiom by analogy, and then getting bitten by a variable with no trailing slash. *shrug* As I say, my arguments are mostly "from experience" and "vague concerns" - which people, notably the other Steven, have dismissed (with some justification). Generally, I think that expandvars is mildly risky, but perfectly fine for simple use. People wanting to write high-reliability production code should probably think a bit harder about what they actually want, and whether the limitations of expandvars are acceptable for their needs. I'd be surprised if after doing so, they didn't write their own custom parser (we did for pip, I'm pretty sure). I think that expending a lot of effort trying to "improve" expandvars is probably not going to make it much more useful for that latter group. It might make it more discoverable, and just possibly more convenient, for people who won't have a problem with its limitations, but it might *also* make some people who should use a custom solution, think it's fine and use it when they shouldn't. To be honest though, I'm not sure of the point of trying to persuade me. I'm fairly fundamentally opposed to changing anything here, but I can't express my reservations in a way that will persuade anyone else. Which is fine. Conversely, if you manage to change my mind, it's not likely to make much difference - no-one is going to write or merge a PR purely on the basis that I stopped objecting to the idea ;-) Paul
On Mon, 14 Feb 2022 at 02:34, Paul Moore <p.f.moore@gmail.com> wrote:
Which is why the default should be to follow platform expectations. If I pass a parameter to a script saying "~/foo/bar", that script should be able to expanduser/expandvars to make that behave the way I expect it to. If I'm on Windows and I tell something to write to a file in "%TEMP%\spam.csv", then I expect it to understand what that means. Cross-platform support is nice, but the most common need is for the current platform's normal behaviour.
Not sure, but if there are issues with the way that Python on Windows expands variables, they can be raised as bugs.
And perfectly fine for strings provided by the user. In fact, it's extremely frustrating to work with a program that DOESN'T do proper var and user expansion - I expect "~git/reponame/.git" to expand to the home directory of the git user, not my home directory with the word "git" after it. As long as Python's expanduser/expandvars behave as per platform expectations, the risk is on the user, not on the script.
I want my users to be able to treat my script the way they treat everything else in the system. Not sure what you mean by "limitations of expandvars". It does precisely what it is supposed to do. The only problem is that it always returns a str or a bytes, not a Path, so you'd have to then construct a Path. On the other hand, if you're writing something where platform is irrelevant, then expandvars isn't just limited, it's actually wrong, and you should be doing expansion differently. ChrisA
On Sun, 13 Feb 2022 at 15:43, Chris Angelico <rosuav@gmail.com> wrote:
For better or worse, though, Windows (as an OS) doesn't have a "normal behaviour". %-expansion is a feature of CMD and .bat files, which varies depending on the Windows version and the version of CMD. A significant proportion of command line programs on Windows are ported from Unix and add (some form of) Unix-like behaviour, either as well as, or instead of, % expansion. Remember, I'm supporting the existing behaviour as "good enough" here. I'm not the one advocating change. If you believe that "the current platform's normal behaviour" is anything other than what os.path.expandvars does, then the onus is on you to define what you mean. But I thought the debate here was about "putting a version of expandvars into pathlib". Maybe someone ought to define what that means, in practice. Oh, and just for fun, I just discovered that expandvars does (arguably broken) quoting, too:
Note that it *keeps* the quotes. That's *definitely* not normal platform behaviour on Windows... But anyway, I thought the debate here was about "putting a version of expandvars into pathlib". Maybe someone ought to define what that means, in practice. Remember to make sure that the documentation of Path.expandvars() makes it clear why (Path("Dev team's docs") / Path("${USER}'s files") / Path("somefile.txt")).expandvars() returns Path("Dev team's docs/${USER}'s files/somefile.txt") and not Path("Dev team's docs/Gustav's files/somefile.txt"), so that people don't report it as a bug... Yes, they might do the same with os.path.expandvars, but that's easily responded to - "documentation bug, the docs should mention quoting and should emphasise that the operation treats the argument as a string, not as a path". I'm not sure the same response works when it's a method of a Path object ;-) And I repeat, *please* remember that I'm not the one arguing for change here, I'm arguing that the proposed change is potentially more complex than people are suggesting, and that actually, the benefit isn't sufficient to justify the work. Although as it's not clear yet who is offering to do the work, it may be premature of me to make that argument... Paul
On Sun, 13 Feb 2022 at 15:43, Chris Angelico <rosuav@gmail.com> wrote:
That may or may not work as Windows has inconsistent treatment of multiple separators depending on where they appear in a path. If TEMP is a drive spec, say "t:\", then it expands to "t:\\spam.csv", which is an invalid windows path. If TEMP is a directory spec, "c:\temp\", then it expands to "c:\temp\\spam.csv", which works fine. C:\> dir c:\\temp\junk The filename, directory name, or volume label syntax is incorrect. C:\> dir c:\temp\\junk Volume in drive C has no label. Volume Serial Number is FC52-C692 Directory of c:\temp 2022-02-13 10:09 0 junk
On Mon, 14 Feb 2022 at 05:15, Eric Fahlgren <ericfahlgren@gmail.com> wrote:
Ugh, what a mess. Thanks for the reminder that I don't understand Windows very well. In my defense, I haven't used it much in years, so I'm not sure what the best practice would be here; but I do know that, if there is such a thing as best practice, Path("%TEMP%\\spam.csv").expandvars() should do it. (Or os.path.expandvars(Path("%TEMP%\\spam.csv")) or however it's spelled.) At least Unix has it written into the standard that multiple slashes are equivalent to one. ChrisA
On 2/13/22, Eric Fahlgren <ericfahlgren@gmail.com> wrote:
"c:\\temp\junk" isn't always invalid in CMD, and definitely not in the Windows API. The problem occurs because the DIR command in the CMD shell has legacy support to ignore the drive (e.g. "C:") when the root of the path is exactly two backslashes -- because DOS in the 1980s (i.e. they went out of their to add this behavior in CMD to make it compatible with DOS). To see this, check the "C$" administrative share on "localhost": C:\>dir /b C:\\localhost\C$\Temp\spam.txt File Not Found C:\>echo spam >C:\\Temp\spam.txt C:\>dir /b C:\\localhost\C$\Temp\spam.txt spam.txt Even though using two backslashes for the root of a drive path is allowed in the Windows API itself, it's sill problematic. The path part r"\\path\to\file" can't be used as relative to the current drive of the process because it's always a UNC absolute path. So it should be normalized to r"\path\to\file" as soon as possible, e.g. via GetFullPathNameW(): >>> print(nt._getfullpathname(r'C:\\path\to\file')) C:\path\to\file
On 2/13/22, Paul Moore <p.f.moore@gmail.com> wrote:
For better or worse, though, Windows (as an OS) doesn't have a "normal behaviour". %-expansion is a feature of CMD and .bat files, which
You're overlooking ExpandEnvironmentStringsW() [1], ExpandEnvironmentStringsForUserW(), and PathUnExpandEnvStringsW() [2], which provide basic support for `%` based environment variables in strings. Python's standard library supports winreg.ExpandEnvironmentStrings(). It is critical that the system supports this functionality in order to evaluate REG_EXPAND_SZ values in the registry. [1] https://docs.microsoft.com/en-us/windows/win32/api/processenv/nf-processenv-... [2] https://docs.microsoft.com/en-us/windows/win32/api/shlwapi/nf-shlwapi-pathun...
On Fri, Feb 11, 2022 at 10:26:03AM +0200, Serhiy Storchaka wrote:
expandvars() does not operate on paths, it operates on strings and bytestrings. There is nothing path-specific here.
Paths can be strings and byte-strings too. When I using my OS shell, and I type: ls -l ~/code/ I am passing a path to ls as argument. And when I do either of these in Python: os.listdir(os.path.expanduser('~/code/')) os.listdir(os.path.expandvars('${HOME}/code/')) I am still passing a path to listdir. Its just not a pathlib.Path instance, but it is still a path, just as "123 Main Road" is a street address and "Jane Doe" is a name. Of course environment variables are not restricted to being path components, but neither are Paths: >>> snake = Path('Chordata/Reptilia/Squamata/Serpentes') >>> python = snake/'Pythonidae/Python' >>> rock_python = python/'sebae' >>> burmese_python = python/'bivittatus' Is this an abuse of Path? Hell yes. What are we, the Path Police? :-) The point is, regardless of whatever odd non-path-like things people might happen to have in environment variables, they also have paths in environment variables, and it is a strange lack that pathlib doesn't make it easy to construct paths from environment variables. If people want to shoot themselves in the foot: myfile = Path('${HOME}/${LS_COLORS}/file.txt').expandvars() that's up to them, and who knows, maybe somebody does actually have a file with that path, no matter how odd it looks.
That's odd. I would expect PosixPath.expandvars to use Posix-style substitution, and WindowsPath.expandvars to use Windows-style substitution. -- Steve
participants (13)
-
anthony.flury -
Chris Angelico -
Christopher Barker -
Damian Shaw -
Eric Fahlgren -
Eric V. Smith -
Eryk Sun -
MRAB -
Paul Moore -
Ricky Teachey -
Serhiy Storchaka -
Stephen J. Turnbull -
Steven D'Aprano