...

...

On 24.03.2016 22:06, Koos Zevenhoven wrote:

Or even better, that you could do p"filename.txt", which would give you a Path string object.

That would tie Path objects deeply into the parser and compiler, which I'm not sure is a good idea. Also, it would be stretching the string-prefix concept considerably. Currently, the prefixes just represent different ways of specifying a string -- the end result is still always an instance of str. In this proposal, it would be a different type of object with greatly different behaviour. Not sure whether I feel positively or negatively about this, so +0j. -- Greg

On Mar 25, 2016, at 5:59 PM, Greg Ewing wrote:

Currently, the prefixes just represent different ways of specifying a string -- the end result is still always an instance of str.

While I’m not sure if the p”” string is a good idea, this isn’t exactly true. b”” creates a bytes object, not a str object. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 03/25/2016 03:03 PM, Greg Ewing wrote:

Koos Zevenhoven wrote:

...

...

...

Or even better, that you could do p"filename.txt", which would give you a Path string object. Has this been discussed?

Also, this would give a privileged place to one particular path library. How do we choose which one?

Um, the one in the stdlib? ;) -- ~Ethan~

On Mar 25, 2016, at 13:20, Koos Zevenhoven wrote:

So, let's start a new thread about how to deal with pathlib.Path objects, involving how/whether to convert between str and Path.

On Fri, Mar 25, 2016 at 7:14 PM, Sven R. Kunze wrote:

...

...

On 24.03.2016 22:06, Koos Zevenhoven wrote:

Or even better, that you could do p"filename.txt", which would give you a Path string object. Has this been discussed?

Interesting. I had this thought half a year ago. Quite recently I mentioned this to Andrew in a private conversation.

Yeah, I first thought about this at some point last summer when I had written the little np package for creating numpy arrays as np[3,4,5] (which I am planning to polish and maybe advertise), probably also inspired by the f-string discussions on this list. And now that you asked me about dealing with Paths...

Anyway, as you say, if different people have thought about this independently, maybe its worth discussing.

...

p'/etc/hosts' would make a perfect path which subclassed from str.

Having Path types subclass str would solve 90% of the problems with pathlib. Adding p-strings raises that to 91%; I don't think it's enough extra win to be worth it. The main reason not to use pathlib today is that almost nothing takes Path objects. A handful of functions (open, stat, access) have methods on the Path objects. But if you want to zlib.open a Path, or add one to a ZipFile, or pass one to json.load or etree.parse, or use it with any third-party library, you have to convert to str manually. And in the opposite direction, paths you get from sys.argv or WSGI or anywhere else are str, not Path, so you have to construct Path objects manually too. So, attempting to use Path all over the place means converting back and forth to str all over the place. And the benefit of having Path objects is so tiny that it's not worth the cost. Path seems to have been designed for a statically-typed language with implicit conversions. In fact, it's very similar to the Boost/C++ design. But that design works for C++ because every time you pass a path to a function that wants a string, or try to store a string in a path-typed variable, the compiler sees what you're doing, sees that the types are unrelated, and looks for a conversion operator or constructor so it can insert an implicit call. That doesn't, and can't, happen in Python. As mentioned in one of the other replies, pathlib's main PyPI predecessor does subclass str, and that makes it a lot more usable.

...

The p-string idea has not been discussed in the PEP I think. The subclassing thing however was and I think its resolution was a mistake. The explanation, at least from my point of view, is a bit weird.

I assume there are good arguments against making Path inherit from str. Unfortunately, those arguments don't seem to appear in the PEP. They really should. But, given that they don't, anyone proposing a change like this really should go back and look at the archived previous discussions, summarize and link the old arguments, and offer counter-arguments.

Anyway, I think the question of whether Path should subclass str is partly separate from the p-string concept. After all, b'byte strings' are not subclassed from str either.

Well, adding p-strings but _not_ subclassing str seems to add almost nothing. It makes it easier to construct literal Path objects that you still can't use anywhere, so you end up writing "spam(str(p'eggs.txt'))" instead of just "spam('eggs.txt')"? But I agree that they can be separated, for the opposite reason: a str-subclassing Path is useful with or without p-strings.

Le 26/03/2016 11:44, Andrew Barnert via Python-ideas a écrit :

On Mar 25, 2016, at 13:20, Koos Zevenhoven wrote:

...

So, let's start a new thread about how to deal with pathlib.Path objects, involving how/whether to convert between str and Path.

On Fri, Mar 25, 2016 at 7:14 PM, Sven R. Kunze wrote:

...

...

On 24.03.2016 22:06, Koos Zevenhoven wrote:

Or even better, that you could do p"filename.txt", which would give you a Path string object. Has this been discussed?

Interesting. I had this thought half a year ago. Quite recently I mentioned this to Andrew in a private conversation.

Yeah, I first thought about this at some point last summer when I had written the little np package for creating numpy arrays as np[3,4,5] (which I am planning to polish and maybe advertise), probably also inspired by the f-string discussions on this list. And now that you asked me about dealing with Paths...

Anyway, as you say, if different people have thought about this independently, maybe its worth discussing.

...

p'/etc/hosts' would make a perfect path which subclassed from str.

Having Path types subclass str would solve 90% of the problems with pathlib. Adding p-strings raises that to 91%; I don't think it's enough extra win to be worth it.

The main reason not to use pathlib today is that almost nothing takes Path objects. A handful of functions (open, stat, access) have methods on the Path objects. But if you want to zlib.open a Path, or add one to a ZipFile, or pass one to json.load or etree.parse, or use it with any third-party library, you have to convert to str manually. And in the opposite direction, paths you get from sys.argv or WSGI or anywhere else are str, not Path, so you have to construct Path objects manually too. So, attempting to use Path all over the place means converting back and forth to str all over the place. And the benefit of having Path objects is so tiny that it's not worth the cost.

Path seems to have been designed for a statically-typed language with implicit conversions. In fact, it's very similar to the Boost/C++ design. But that design works for C++ because every time you pass a path to a function that wants a string, or try to store a string in a path-typed variable, the compiler sees what you're doing, sees that the types are unrelated, and looks for a conversion operator or constructor so it can insert an implicit call. That doesn't, and can't, happen in Python.

As mentioned in one of the other replies, pathlib's main PyPI predecessor does subclass str, and that makes it a lot more usable.

...

...

The p-string idea has not been discussed in the PEP I think. The subclassing thing however was and I think its resolution was a mistake. The explanation, at least from my point of view, is a bit weird.

I assume there are good arguments against making Path inherit from str. Unfortunately, those arguments don't seem to appear in the PEP. They really should. But, given that they don't, anyone proposing a change like this really should go back and look at the archived previous discussions, summarize and link the old arguments, and offer counter-arguments.

...

Anyway, I think the question of whether Path should subclass str is partly separate from the p-string concept. After all, b'byte strings' are not subclassed from str either.

Well, adding p-strings but _not_ subclassing str seems to add almost nothing. It makes it easier to construct literal Path objects that you still can't use anywhere, so you end up writing "spam(str(p'eggs.txt'))" instead of just "spam('eggs.txt')"?

+1 to everything. Remember that because of this, all projects I work on still use path.py. We prefer to add an additional dependency than to deal with the casting, because it spreads everywhere. And we are not a conservative crew, some projects use Python 3.5 and asyncio. Although honestly, p'/foo/bar'.write_text(stuff) would be awesome at some many levels. Can you imagine how scripting becomes easier because of this ? Sys admin ? Shell sessions ?

But I agree that they can be separated, for the opposite reason: a str-subclassing Path is useful with or without p-strings.

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

I love argparse. I really do. But it's one of those lib forcing me to RTFM everytime I need it, for the last 10 years. We had pathlib added to the stdlib to help with open/os/shutil API and we could do the same with argparse. There are many libs helping with argparsing (clize, click, argh, docopt, etc), and I'd like to talk about the API of one of them, begins. You define the entry point of your script as a regular function you decorate:

...

...

import begin @begin.start ... def run(name='Arther', quest='Holy Grail', colour='blue', *knights): ... "tis but a scratch!"

@begin.start will do "if __name__ == '__main__'" for you (alhough you can opt out), and parse the function arguments then generate a argparse parser. When you call the script, it parses the arguments, and call your function with them passed as parameters. It also generates the --help: usage: example.py [-h] [-n NAME] [-q QUEST] [-c COLOUR] [knights [knights ...]] tis but a scratch! positional arguments: knights optional arguments: -h, --help show this help message and exit -n NAME, --name NAME (default: Arther) -q QUEST, --quest QUEST (default: Holy Grail) -c COLOUR, --colour COLOUR (default: blue) Of course, you can define more behavior using other decorators such as repetition, type, checks and sub commands. It's not a huge lib, but's it makes writting script out of the box a breeze. The only problem is : you usually don't want to do a pip install just to run a script. That's why I things it would be great to have something similar in the stdlib. What's more, it's really not going to evolve much (you can't get more old school than cmd), so having it not be updated for years after including it (or something like it) in the stdlib is not an issue. And we can even add some features that would be useful to benefit from the last Python goodness, such as an option to auto-spawn an event loop. This way you can create a quick script with just an async def function. Indeed, the boiler plate of getting the event loop, adding a coroutine and starting the loop is getting annoying very fast when you do a lot of quick things with any aio libs.

I have been watching this thread, and read the docs and the PEP, and... What is pathlib even for? Yes, its there to be an object oriented abstraction of a file system. Why? Why do I want this? Am I alone in wondering why this exists? Is it even worth improving? On 3/26/2016 06:59, Andrew Barnert via Python-ideas wrote:

On Mar 25, 2016, at 13:20, Koos Zevenhoven wrote:

...

So, let's start a new thread about how to deal with pathlib.Path objects, involving how/whether to convert between str and Path As a different point:

If we _don't_ either subclass str or find some way to make things magically work, is there any other way to start getting more uptake on pathlib? This may only be anecdotal, but from what I can tell, nobody is using it, because everyone who tries is put off by the need to convert back and forth everywhere.

Last year, everyone agreed that it would be good if at least the stdlib accepted paths everywhere, which might prompt third party libs to start doing the same. But nobody's started writing patches to do that. And I'm not sure we'd want it to happen in an uncoordinated way anyway, since there are at least four different ways to do it, and if we pick among them arbitrarily for different parts of the stdlib, we'll have a huge mess, and possibly behavioral inconsistencies.

The four ways I can think of are (in every function that currently takes a "path: str" argument, and should now take a "path: str | Path"):

* path = str(path) * path = Path(path) * if isinstance(path, Path): ... else: ... * try: f = path.open('w') except AttributeError: open(path, 'w')

It's also worth noting that all but the first require every module to depend on Path. Including C modules. And modules in the bootstrap. But the first version makes a bad guide for third-party code, because a lot of third-party code is dual-version/single-source libs, and you definitely don't want to unconditionally call str on a path argument that may be Unicode in 2.7 (or to unconditionally call a six-style Unicode function on a path argument that may be str in 2.7 on Linux).

So, assuming we all want a future in which pathlib is actually used by people, and assuming str subclassing is out and there's no other magic bullet, how do we get there from here? _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

I had to read the traditional way to figure out what you were even trying with the other two, so I don't know how 'natural' it is. As long as the traditional way continues to be supported, and this is what pathlib is intended to do, I guess I don't care. I know I'm not going to use it. On 3/26/2016 10:03, Michel Desmoulin wrote:

Because it makes a very common task (FS manipulation) much more natural.

This is the same reason people like request over urllib or like @decorator over func = decorator(func) or unpacking vs manual item extract.

In a directory of music files, you ave mp3 files you converted to other formats. Now you want to remove those conversion. Your task is to find all files having the same name as the ones with the mp3 ones, but with a different extension, remove them, then list the mp3 absolument path in a text file.

This example has baked in a lot of the tasks you do when you use Python as a scripting language at the core of you job such as if it's your glue language, if you are a tester or for sysadmin tasks. But it helps also everybody that once in a file does a one off script.

The traditional approach:

import os import glob import sys

root = os.path.abspath(sys.argv[1])

playlist = os.path.join(root, 'playlist.m3u8')

with open(playlist, 'w') as f:

for path in glob.glob(os.path.join(root, '*.mp3')):

name, ext = os.path.splitext(os.path.basename(path))

for to_remove in glob.glob(os.path.join(root, name + '.*')): if not to_remove.endswith('mp3'): os.remove(to_remove)

f.write(os.path.join(root, path) + "\n")

Now with pathlib you don't have to wonder about whether the feature you are looking for is on "os", "os.path" or "glob" or "open". You don't have to deal the file opening for such a small script. You don't have to switch between functions and methods all the time and have to choose between nested function calls or intermediary variables.

The pathlib version is way easier to figure out without knowing the stdlib by heart, it is one line shorter and one level of indent less. And you can discover it all from iPython with ".<tab>":

import sys import pathlib

root = pathlib.Path(sys.argv[1])

files = [] for path in root.glob('*.mp3'):

name = str(path).replace(path.suffix, '.*') files.append(str(path.absolute()))

for to_remove in root.glob(name): if to_remove.suffix != ".mp3": to_remove.unlink()

(root / 'playlist.m3u8').write_text('\n'.join(files))

And this true while pathlib is a half backed written lib, since the competition (that existed before pathlib and that pathlib failed to inspire from), can do even shorter, easier and cleaner:

import sys import path

root = path.Path(sys.argv[1]).realpath()

files = [] for p in root.glob('*.mp3'):

name = p.namebase + '.*' files.append(p)

for to_remove in root.glob(name): if to_remove.ext != ".mp3": to_remove.remove()

(root / 'playlist.m3u8').write_lines(files)

Because path.py:

- inherit from str - has all the methods from os, not just a few cherry picked - has logical names for attributes and methods - have more utilities than pathlib or os

So yes, if you do a lot of scripting, this is a must. It's also way easier for beginers to grasp.

Koos Zevenhoven writes:

One downside to p-strings like this is that they don't help you when you have the path in a plain str variable (i.e. not a literal),

IMO that kills this simple form of the idea, since hard-coding paths is a bad idea in many many programs, and paths are naturally derived from user input in a host of others. It becomes an attractive nuisance. How about generalizing f-strings, or perhaps giving p-strings (some of) the power of f-strings?

On Fri, 25 Mar 2016 at 16:50 Greg Ewing wrote:

...

...

...

On 24.03.2016 22:06, Koos Zevenhoven wrote:

Or even better, that you could do p"filename.txt", which would give you a Path string object.

That would tie Path objects deeply into the parser and compiler, which I'm not sure is a good idea.

I'll go one step further than Greg and say that it's not a good idea. To make something like this work you have two options. One is you do what importlib does and very carefully construct it to only rely on built-in modules (and if you look at https://hg.python.org/cpython/file/default/Lib/pathlib.py you will notice that is no where near true ATM). Two, is Python gets a two-tier syntax system for before and after pathlib is imported and available. That would mean that all the dependencies of pathlib would need to not use this syntax until it's bootstrapped in. If you notice, both solutions are extremely painful to implement and radiate well past the code in pathilb.py and would potentially become hard to maintain.

Also, it would be stretching the string-prefix concept considerably. Currently, the prefixes just represent different ways of specifying a string -- the end result is still always an instance of str. In this proposal, it would be a different type of object with greatly different behaviour.

Yep, that too.

Not sure whether I feel positively or negatively about this, so +0j.

I'm sure how I feel :) -1. I also want to mention two things. One, pathlib.path is a thing now and something most people are probably not aware of as an alternative to doing `str(path)`: https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.path . And two, wrapping path inputs in `pathlib.Path(path)` is a small price to pay for a nicer API. As time passes it will be less and less of a thing as APIs will start to come up that explicitly accept either, and then eventually they will only accept pathlib. Plus if you do it at the edges of your own API then you can do the conversion early and only get out the string as necessary later on as needed (much like how you handle text/binary data boundaries). I get the desire of succinctness as Python spoils us for that, but much like __future__ statements it's really a small price to pay in the grand scheme of things in order to migrate to a better API.

On Fri, Mar 25, 2016, at 17:59, Greg Ewing wrote:

...

...

...

On 24.03.2016 22:06, Koos Zevenhoven wrote:

Or even better, that you could do p"filename.txt", which would give you a Path string object.

That would tie Path objects deeply into the parser and compiler, which I'm not sure is a good idea.

It's not obviously _not_ a good idea. Are float and complex more "worthy" of being tied into the parser and compiler than Path?

On Sat, Mar 26, 2016 at 4:03 PM, Michel Desmoulin

wrote:

[...] Now with pathlib you don't have to wonder about whether the feature you are looking for is on "os", "os.path" or "glob" or "open". You don't have to deal the file opening for such a small script. You don't have to switch between functions and methods all the time and have to choose between nested function calls or intermediary variables.

Here we completely agree. I used to do scripting of filesystem-intensive tasks with bash, because it felt more natural, partly because I was doing that before I started using Python for anything. Due to pathlib I now do this stuff in Python, which I of course like better for other reasons. Pretty much everything is there, and nicely auto-completed in editors. The fact that you can do / for joining paths is not necessarily better than a method (which exists too as .joinpath()), but it works and may be easy to remember.

The pathlib version is way easier to figure out without knowing the stdlib by heart, it is one line shorter and one level of indent less. And you can discover it all from iPython with ".<tab>":

import sys import pathlib

root = pathlib.Path(sys.argv[1])

files = [] for path in root.glob('*.mp3'):

name = str(path).replace(path.suffix, '.*') files.append(str(path.absolute()))

for to_remove in root.glob(name): if to_remove.suffix != ".mp3": to_remove.unlink()

(root / 'playlist.m3u8').write_text('\n'.join(files))

Or even cleaner, and hopefully easier to understand (not tested, sorry): import sys import pathlib root = pathlib.Path(sys.argv[1]) files = [f.absolute() for f in root.glob("*.mp3")] for mp3file in files: for f in root.glob(mp3file.stem + ".*"): if f.suffix != ".mp3": f.unlink() # remove non-mp3 version (root / "playlist.m3u8").write_text("\n".join(files)) Yes, some of the properties could have more logical names like .remove() instead of .unlink(), and there could be even more functionality. Subclassing from str would have obvious benefits, but I'm just not sure whether it's a good idea to inherit things like str.find, str.splitlines, str.title, ... So maybe the best thing is not to inherit from str but to make Path quack as much like a str-duck as possible (and vice versa?) without introducing confusing things? BTW, here's a link to issue #22570 (Better stdlib support for Path objects) http://www.psf.upfronthosting.co.za/issue22570 - Koos

Chris Angelico wrote:

There's b"..." vs u"...", which do represent entirely different objects,

That's true, but they're not entirely unrelated -- bytes objects are the py3 version of what str was in py2. Also, they're both pretty fundamental to the language, and they're essentially pure representations of data, with not much semantics attached. Path objects, on the other hand, come with a pile of semantics relating to a particular application area -- manipulation of the file system. Moreover, those semantics vary depending on what operating system you're running on. So it's hard so see them as being part of the language the way str and bytes are. This feels like a slippery slope to me. If we include special syntax for pathnames, why shouldn't we have it for dates and times? Regular expressions? URLs? JSON data? SQL queries? XML data? Where do we draw the line?

plus f"..." which isn't even a literal at all, but more like a special syntax for an expression

It still evaluates to a str, though. -- Greg

I'm wondering how often literal pathames are really used in actual programs. In my experience, most pathnames are received as arguments, read from config files, etc. Most literals are just fragments that get concatenated with existing pathnames, and that's already covered by operations between path objects and strings. -- Greg

On 03/26/2016 03:55 PM, Greg Ewing wrote:

Ethan Furman wrote:

...

...

Also, this would give a privileged place to one particular path library. How do we choose which one?

Um, the one in the stdlib? ;)

Not everyone likes that one though. As things stand, you can easily substitute one of your own that's more to your liking. That won't be true if path objects get baked into the language syntax.

Yeah, I'm one of those (primarily because it's a pain to work with). My point was just that baked-in syntax would naturally go with stdlib objects (and it would still not preclude using your preferred library). -- ~Ethan~

On Sat, Mar 26, 2016 at 6:51 PM, Koos Zevenhoven wrote:

On Sun, Mar 27, 2016 at 1:13 AM, Greg Ewing wrote:

...

[...] This feels like a slippery slope to me. If we include special syntax for pathnames, why shouldn't we have it for dates and times? Regular expressions? URLs? JSON data? SQL queries? XML data? Where do we draw the line?

OT:

To be honest, I do think it feels like URL:s are becoming (or have become) just as important as paths, and that pathlib.Path should in the future work with URLs just like it now works with windows and posix paths. The difference between "http://domain.xyz/" and "C:\\" is not huge.

Here's some preliminary research (w/ links) that may be of use here: https://www.reddit.com/r/Python/comments/1r7h1t/python_objects_for_working_w... * URLObject (+1 from me) * rdflib.term.URIRef * fs.path Differences between fs paths and URIs: * urlencode IDK why I seem to remember having concluded that it would make sense to store paths as tuples / lists and join by os.path.sep

I also think there should be a Python type (stdlib or builtin), which handles JSON objects nicer than dicts do and has its own literal syntax.

collections.OrderedDict almost works here, except that .keys() and .values() must be casted to e.g. lists (because otherwise the comparison is between KeysView / ValuesView ~iterators) to_json = _repr_json_ # "IPython repr method examples" # https://gist.github.com/westurner/be22dba8110be099a35e#file-ordereddefaultdi... Then, if Path is not a subclass of str (or, py2 __builtin__.unicode, as in https://github.com/jaraco/path.py/blob/master/path.py ), comparisons between JSON-ified path objects will fail.

- Koos

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Sun, Mar 27, 2016 at 1:23 AM, Koos Zevenhoven wrote:

On Sat, Mar 26, 2016 at 9:10 PM, Brett Cannon wrote:

...

I also want to mention two things. One, pathlib.path is a thing now and something most people are probably not aware of as an alternative to doing `str(path)`: https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.path .

I assume you meant to type pathlib.Path.path, so that Path("...").path == str(Path("...")). That's a good start, and I'm looking forward to Serhiy's patch for making the stdlib accept Paths. But if Path will not subclass str, we also need new stdlib functions that *return* Paths.

Actually, now that .path is not out yet, would it make sense to call it Path.str or Path.strpath instead, and introduce the same thing on DirEntry and guarantee a str (instead of str or bytes as DirEntry.path now does)? Maybe that would lead to fewer broken implementations in third-party libraries too? - Koos

On Sat, 26 Mar 2016 at 16:23 Koos Zevenhoven wrote:

On Sat, Mar 26, 2016 at 9:10 PM, Brett Cannon wrote:

...

On Fri, 25 Mar 2016 at 16:50 Greg Ewing wrote:

...

...

...

...

On 24.03.2016 22:06, Koos Zevenhoven wrote:

Or even better, that you could do p"filename.txt", which would give you a Path string object.

That would tie Path objects deeply into the parser and compiler, which I'm not sure is a good idea.

I'll go one step further than Greg and say that it's not a good idea. To make something like this work you have two options. One is you do what importlib does and very carefully construct it to only rely on built-in modules (and if you look at https://hg.python.org/cpython/file/default/Lib/pathlib.py you will notice that is no where near true ATM). Two, is Python gets a two-tier syntax system for before and after pathlib is imported and available. That would mean that all the dependencies of pathlib would need to not use this syntax until it's bootstrapped in. If you notice, both solutions are extremely painful to implement and radiate well past the code in pathilb.py and would potentially become hard to maintain.

As I understand it, this could be implemented by making the compiler essentially turn p"/path/to/whatever" into something like _make_path("/path/to/whatever"), where _make_path would be builtin and do something like this.

def _make_path(str_path): import pathlib return pathlib.Path(str_path)

Unless of course Path could be modified to import its dependencies on demand and put in builtins. Am I missing something crucial?

The trick is whether that import statement will work when you call it. Import is currently the only bit of syntax that relies on Python code in CPython to work and making that happen took a lot of work. If you add in this concept of p-strings then suddenly we have two pieces of syntax that require Python code to work and on top of it p-strings would depend on import but also that we would then have to make sure to not have import depend on p-strings. And on top of it we would have to freeze pathlib and all of its dependencies which may or may not be preferable. My point is that it's not straight-forward and this proposal will have to consider some technical difficulties involved with it if it happens to go forward. -Brett

...

I also want to mention two things. One, pathlib.path is a thing now and something most people are probably not aware of as an alternative to doing `str(path)`: https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.path .

I assume you meant to type pathlib.Path.path, so that Path("...").path == str(Path("...")). That's a good start, and I'm looking forward to Serhiy's patch for making the stdlib accept Paths. But if Path will not subclass str, we also need new stdlib functions that *return* Paths.

- Koos

On 26.03.2016 19:17, Koos Zevenhoven wrote:

On Sat, Mar 26, 2016 at 7:32 PM, Stephen J. Turnbull mailto:stephen@xemacs.org> wrote:

Koos Zevenhoven writes:

How about generalizing f-strings, or perhaps giving p-strings (some of) the power of f-strings?

That'll be a good idea.

Just their combination: pf"{ path_as_string }" ? That would at least be very consistent.

Could be make implicit: p'{my_variable}'

On 27.03.2016 00:38, Greg Ewing wrote:

I'm wondering how often literal pathames are really used in actual programs.

Don't underestimate what made Python great. Exploration is very important.

In my experience, most pathnames are received as arguments, read from config files, etc. Most literals are just fragments that get concatenated with existing pathnames, and that's already covered by operations between path objects and strings.

True, matured programs usually work this way. But as we all know, agile is better than waterfall, so we might start with literals and replacing them with config variables later when needed. Best, Sven

On 26.03.2016 20:53, Koos Zevenhoven wrote:

Yes, some of the properties could have more logical names like .remove() instead of .unlink(), and there could be even more functionality. Subclassing from str would have obvious benefits, but I'm just not sure whether it's a good idea to inherit things like str.find, str.splitlines, str.title, ...

So maybe the best thing is not to inherit from str but to make Path quack as much like a str-duck as possible (and vice versa?) without introducing confusing things?

I would not like this limitation. Paths are strings (even for the os). It's just that Python devs are terribly lazy so, they want a string deluxe instead of a plain string in case a string represents a path. A lot of companies encode data right into the path of some files. So, when dealing with such situation, I would very like to have all capabilities of str available. Everything else would be frustrating IMHO. Best, Sven

On 29 March 2016 at 08:22, Sven R. Kunze wrote:

Thinking more about it, that would even come in nicely with path separators:

p'/{whereever}/{youwant}/{togo}'

That'll finally an easy-to-write and readable path generation. Great idea, Stephen.

P(wherever)/youwant/togo is as good, if not better - just "from pathlib import Path as P" (or use Path(wherever) if you don't like unnecessary 1-letter aliases...) Paul

On 29 March 2016 at 08:31, Sven R. Kunze wrote:

Paths are strings (even for the os).

On Unix, paths are bytestrings, not strings (in Python 3, string = Unicode string), but on Windows they are Unicode strings. Having an abstraction over this OS difference is useful. Paul

On 26.03.2016 23:55, Greg Ewing wrote:

Ethan Furman wrote:

...

...

Also, this would give a privileged place to one particular path library. How do we choose which one?

Um, the one in the stdlib? ;)

Not everyone likes that one though. As things stand, you can easily substitute one of your own that's more to your liking. That won't be true if path objects get baked into the language syntax.

I'd rather submit bug reports and help improving a commonly used library than re-inventing the wheel. If the help is not wanted, that's another thing. But I consider working together important. Furthermore, file paths are an essential part of programming even of todays operating systems. As are strings, numbers btw. Best, Sven

Forget about it. My thunderbird again screwed up my threading view. On 29.03.2016 10:14, Sven R. Kunze wrote:

On 26.03.2016 12:02, Michel Desmoulin wrote:

...

I love argparse. I really do. But it's one of those lib forcing me to RTFM everytime I need it, for the last 10 years.

You may want to look at: https://pypi.python.org/pypi/argh#examples

It's dead simple. Each function can become a cmdline command since it just re-uses the functions' signatures. :)

Best, Sven _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On 27.03.2016 19:36, Stephen J. Turnbull wrote:

Koos Zevenhoven writes:

...

To be honest, I do think it feels like URL:s are becoming (or have become) just as important as paths, and that pathlib.Path should in the future work with URLs just like it now works with windows and posix paths.

+1

... to the concept, but: Is there a semantic difference between a RFC 3986 path component (Section 3.3), and a pathlib path? If there is, this could be a difficult project. (May as well start now, though!)

I would interpret the path of pathlib as a subset of functionality of path of a URI. """ The path component contains data, usually organized in hierarchical form [...] serves to identify a resource within the scope of the URI's scheme and naming authority (if any). """ URI's scheme for path of pathlib could implicitly be: "file://" Relative paths of pathlib are a subset of relative references (Section 4.1). The only practical issue, I can think of is, how to distinguish (in the sense of avoiding hidden bugs) file paths and url paths. And name clashes when communicating with your fellow programmers: "I got that url path working" "the url path or the url path path?" Best, Sven

On 26.03.2016 20:21, Guido van Rossum wrote:

I'm not going to say how I feel. But I noticed one thing in this thread. These seems to be a growing difference between people who use Python for "serious" programming and those who consider themselves testers, QA integrators and the like (more power to them!). Maybe both groups can occasionally try to walk in the others' shoes.

+1

On 27.03.2016 19:50, Brett Cannon wrote:

The trick is whether that import statement will work when you call it. Import is currently the only bit of syntax that relies on Python code in CPython to work and making that happen took a lot of work. If you add in this concept of p-strings then suddenly we have two pieces of syntax that require Python code to work and on top of it p-strings would depend on import but also that we would then have to make sure to not have import depend on p-strings. And on top of it we would have to freeze pathlib and all of its dependencies which may or may not be preferable.

My point is that it's not straight-forward and this proposal will have to consider some technical difficulties involved with it if it happens to go forward.

What solution can you propose to solve this? Can this procedure be generalized and smoothed in order to reduce that "lot of work" and to some extend enable a simpler language evolution? Or would it be better to leave the bootstrapping infrastructure alone? Best, Sven

Le 29/03/2016 09:50, Sven R. Kunze a écrit :

On 25.03.2016 22:56, Michel Desmoulin wrote:

...

Although path.py, which I have been using for years now (and still prefer to pathlib) subclass str and never caused any problem whatsoever. So really, we should pinpoint where it could be an issue and see if this is a frequent problem, because right now, it seems more a decision based on purity than practicality.

I agree. The PEP says:

""" No confusion with builtins ---------------------------------- In this proposal, the path classes do not derive from a builtin type. This contrasts with some other Path class proposals which were derived from str . They also do not pretend to implement the sequence protocol: if you want a path to act as a sequence, you have to lookup a dedicated attribute (the parts attribute).

Not behaving like one of the basic builtin types also minimizes the potential for confusion if a path is combined by accident with genuine builtin types. """"

I have to admit I cannot follow these statements but they should have appeared to be necessary back then. As experience shows the PyPI module fared far better here.

I am great a fan of theory over practice as what has been proven in theory cannot be proven wrong in practice. However, this only holds if we talk about hard proof. For soft things like "human interaction", "mutual understanding" or "potential for confusion", only the practice of many many people can "prove" what's useful, what's "practical".

+1. Can somebody defend, with practical examples, the imperative to refuse to inherit from str ? And weight it against the benefits of it ? We have been doing it for years, and so far it's been really, really nice. I can't recall problem I had because of it ever. I can recall numerous editions of my code because I forgot str() on pathlib.Path.

Best, Sven

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, Mar 29, 2016 at 10:49 AM, Paul Moore wrote:

On 29 March 2016 at 08:22, Sven R. Kunze wrote:

...

Thinking more about it, that would even come in nicely with path separators:

p'/{whereever}/{youwant}/{togo}'

That'll finally an easy-to-write and readable path generation. Great idea, Stephen.

P(wherever)/youwant/togo

is as good, if not better - just "from pathlib import Path as P" (or use Path(wherever) if you don't like unnecessary 1-letter aliases...)

Yes. I assume you meant P('/') /wherever/youwant/togo. Or with p-strings, p'/' /wherever/youwant/togo I'm not necessarily against Sven's version either, but my first thoughts about the power of combined pf strings would be something like this (assuming Sven's implicit f): all_data = [p'data{i}.txt'.read_text() for i in range(1, 307)] which would read the contents of data1.txt, data2.txt, ..., and data306.txt into a list. And below, to the security issue raised by Sjoerd. On Tue, Mar 29, 2016 at 10:46 AM, Sjoerd Job Postmus wrote:

On Tue, Mar 29, 2016 at 06:27:09PM +1100, Chris Angelico wrote:

...

On Tue, Mar 29, 2016 at 6:22 PM, Sven R. Kunze wrote:

...

Thinking more about it, that would even come in nicely with path separators:

p'/{whereever}/{youwant}/{togo}'

That'll finally an easy-to-write and readable path generation. Great idea, Stephen.

Agreed, although this introduces a new edge case: what if 'togo' contains a slash? Should this raise an exception, or should it be interpreted as a multi-part path component? Arguments can be put for both sides.

As we all know, togo is user input containing ../../../../../../../../../../../../../../../etc/passwd

How about p'/' /wherever/youwant//togo That is, the floordiv operator could be used to prevent 'togo' from going up the directory tree with "../../" or "/etc/passwd". The // would thus restrict the user (who provides `togo`) into /wherever/youwant/ and it its subdirectories. -Koos

On 29 March 2016 at 11:24, Koos Zevenhoven wrote:

...

As we all know, togo is user input containing ../../../../../../../../../../../../../../../etc/passwd

How about

p'/' /wherever/youwant//togo

That is, the floordiv operator could be used to prevent 'togo' from going up the directory tree with "../../" or "/etc/passwd". The // would thus restrict the user (who provides `togo`) into /wherever/youwant/ and it its subdirectories.

There have been a number of previous threads about the security of path objects (and for that matter of os.path) and "sandboxing" paths to disallow traversing "up" out of a particular directory. I've no idea whether the floordiv operator would work for this purpose (I'm not a security expert, and most of my personal applications can afford to take a very lax view of such things, luckily) but it's an interesting idea. Potentially, it's a bit easy to miss, which may be a problem. Paul

On 29.03.2016 12:36, Paul Moore wrote:

I don't intend to get sucked into the role of "defender of not inheriting", but one point that should probably be considered is that on Unix, it is possible to have paths which are *not* valid Python strings.

I don't know how other system developers are working in this field but that might be something serious I encounter from time to time. As a developer and API designer, one has two options (off the top of my head): 1) design and code with Unicode/UTF-8 as file names 2) deal with arbitrary bytes in file names I like 1) more because it gives me the ability to extract useful data/text from paths, which as far as I can tell is quite some usecase. 2) is more like a black box variant to me, but then operations like "/" or "+" or "suffix" etc. have far less meaning (at least to me).

How would those work in an inherit-from-str environment? I suspect that using surrogateescape is the answer here, but I don't know.

Could you explain what this means?

Also, has anyone *converted* code from os.path manipulation of strings to a str-subclass path object like path.py? If you write your code from scratch intending to use path.py it's probably easier to avoid objects getting silently downgraded from path to str than if you're retrofitting a path library to legacy code. That's a consideration that is more applicable to the standard library than to a 3rd party module.

Good point. It's quite problematic to convert a large system at once, so a transition is necessary. Maybe, that is a reason why more people want path->str.

But it may be that now is the right time to revisit the choice to not inherit from str. I don't know, personally. I'm not even sure if a pathlib.Path that inherited from str is something I'd use more often than I currently do.

It's only one of its warts. In order to get a greater mass to make the transition, most of these issues needs to be removed. Quite recently, there has been a discussion to add something to pathlib (I can't remember now) but it failed to get through because of "purity over practicability". A wasted chance IIRC. That thing was exactly what the Zen warns about.

For open source work, the need to support Python 2 means I can't use pathlib (without adding the backport as a dependency). For personal code, it's just a case of mental inertia...

Adding the dependency works for me. Best, Sven

offtopic On 29.03.2016 12:57, Michel Desmoulin wrote:

Now I was sad when I learned request would never be part of the stdlib,

Looking at http://docs.python-requests.org/en/master/dev/philosophy/#standard-library Seems to be true. I wonder if it would be possible/necessary for the stdlib to include a set of libraries marked as "featured". Not like "provisional" but more like "You should use this one and be aware that it may upgrade faster than normal stdlib modules". Basically like a "pull module". Just a third-party module pulled from the source and upgraded with each Python version. That would maintain the maintainers flexibility and the increase the availability of these "defacto stdlib" modules. Just wondering. Best, Sven

On 29 March 2016 at 12:24, Sven R. Kunze wrote:

On 29.03.2016 13:21, Steven D'Aprano wrote:

...

So if I have a Linux file b'\xD8\x01', I can't create a path object to work with it.

I actually like this limitation. It forces a cleaner API and cleaner code as it is the one way.

However, it does mean that certain use cases are not supported by pathlib. One of the constant issues with core/stdlib development is having to consider how the design affects people with needs you don't personally have. This one's not such a big problem "pathlib doesn't support non-Unicode filenames" is a simple enough to express limitation. But that doesn't help the poor guy debugging an issue with his filesystem-scanning program where a user has a mangled non-Unicode filename somewhere in the bowels of his profile - does he declare that user's situation is unsupported, or does he abandon pathlib in favour of the old os APIs? (And if the , what's the likelihood

On 3/26/2016 7:13 PM, Greg Ewing wrote:

Path objects, on the other hand, come with a pile of semantics relating to a particular application area -- manipulation of the file system. Moreover, those semantics vary depending on what operating system you're running on. So it's hard so see them as being part of the language the way str and bytes are.

This feels like a slippery slope to me. If we include special syntax for pathnames, why shouldn't we have it for dates and times? Regular expressions? URLs? JSON data? SQL queries? XML data? Where do we draw the line?

PEP 501 tries to address these issues, with i-strings. There were other proposals on python-ideas relating to arbitrary string prefixes when PEPs 498 and 501 were being discussed. Eric.

On Tue, Mar 29, 2016 at 10:21 PM, Steven D'Aprano wrote:

Unless I'm missing something, pathlib doesn't support bytes filenames. So the question doesn't come up. Any path you can use with pathlib is a valid Python string, because you can only create paths using Python strings.

So if I have a Linux file b'\xD8\x01', I can't create a path object to work with it.

Actually you can. Here's an annotated interactive session illustrating byte smuggling:

...

...

# Create a file using a bytes name ... with open(b'\xD8\x01',"w") as f: f.write("Demo text\n") ... 10 import os, pathlib # By default, you get text strings. ... os.listdir() ['\udcd8\x01'] os.listdir(".") ['\udcd8\x01'] # But you can request byte strings. ... os.listdir(b".") [b'\xd8\x01'] # pathlib works with text strings the same way. ... list(pathlib.Path(".").glob("*")) [PosixPath('\udcd8\x01')] # And you can cast that to str and open it. ... open(str(pathlib.Path("\uDCD8\x01"))).read() 'Demo text\n'

So it looks like the only missing piece of the puzzle is a way to construct a Path from a bytes, which would do the same transformation of \xD8 to \uDCD8. ChrisA

Le 29/03/2016 14:31, Sven R. Kunze a écrit :

Patching your mails:

On 29.03.2016 14:10, Paul Moore wrote:

...

However, it does mean that certain use cases are not supported by pathlib. One of the constant issues with core/stdlib development is having to consider how the design affects people with needs you don't personally have. This one's not such a big problem "pathlib doesn't support non-Unicode filenames" is a simple enough to express limitation. But that doesn't help the poor guy debugging an issue with his filesystem-scanning program where a user has a mangled non-Unicode filename somewhere in the bowels of his profile - does he declare that user's situation is unsupported, or does he abandon pathlib in favour of the old os APIs?

Perfect example.

As I said, I've encountered this situation and most likely will encounter it again. We have mostly control over the file names. So, we might be able to fix a problematic file name. The issue you are bringing up (which is perfectly valid) is when you as a developer don't have control over the file name (aka fixing them).

In this regard, I might be pragmatic and say: "let's see what happens". If this is a frequent issue, then this is one of those warts pathlib needs to loose in order to become mainstream.

yes. It's a very rare use case. If you create a program that needs to be handling this, you still got the old API. For the 99% of the rest of the world, you have the modern one. Let's not try to make pathlib handle everything, it's a convenience wrapper, not a silver bullet.

But why not open another thread with a solution to this issue as has Koos done with p-strings?

...

(And either way, what's the likelihood of him ever wanting to use pathlib again?)

You mean like myself? As I wanted to use pathlib but found myself writing awkward code with it? (btw. that was not the path->str issue)

That as well needs to be fixed but it's no argument for delaying other improvements, right?

Exactly. pathlib needs improvements, and there is plenty of prior art to make suggestions about this, but if it's a central API, it will be on the spotlight to do so.

Best, Sven _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, Mar 29, 2016, at 05:35, Sven R. Kunze wrote:

Because you are asking specifically for regexes, I might be able to give an explanation.

Regular expressions have, let's say, the tendency to grow. It's more or less code, so they grow naturally. Readability, however, diminishes extremely fast with regexes when compared to regular code. ... And hey, I wouldn't mind a builtin type for regular expressions either if this makes things considerably easier. :)

Is it time again to bring up my regex AST idea?

On 29.03.2016 15:08, Paul Moore wrote:

On 29 March 2016 at 13:31, Sven R. Kunze wrote:

...

That as well needs to be fixed but it's no argument for delaying other improvements, right? Certainly. However, I don't think p-strings are an improvement - I'm not suggesting delaying them because other things need to be fixed, I'm suggesting that the whole proposal shouldn't be implemented because it's not a good idea.

Alright. Got it. :) p-strings are not the most pressing issue of pathlib I assume. Wrapping an (f-)string into a Path might work for now although I would love see the convenient p-string in some future Python version.

If you want custom string-like syntaxes, PEP 501 is where you should be getting involved. Adding a single-purpose custom string prefix is a proposal that *should* be delayed until the fate of a more general proposal of which it's a subset is decided. Because once syntax is added it's nigh-on impossible to get rid of.

Agreed. First things first. Note on PEP 501. Not sure if I like it (sorry Nick) but to me it feels a bit artificial/impracticable. Its examples seem very awkward to me. This additional pair of parentheses kills it for me. runquery & runcommand should do the wrapping themselves. If done, i-strings look like f-strings. runquery(sql(i"SELECT {column} FROM {table};")) runcommand(sh(i"cat {filename}")) This means for p-string: Path(f'/whereever/{youwant}/{togo}') == p'/whereever/{youwant}/{togo}' So, if Path is powerful enough, I think most people can live with that for now (but those parentheses .... don't we already have the quotation marks ... anyway) ;-) Best, Sven

On Tue, Mar 29, 2016 at 1:41 PM, Paul Moore wrote:

On 29 March 2016 at 11:24, Koos Zevenhoven wrote:

...

...

As we all know, togo is user input containing ../../../../../../../../../../../../../../../etc/passwd

How about

p'/' /wherever/youwant//togo

That is, the floordiv operator could be used to prevent 'togo' from going up the directory tree with "../../" or "/etc/passwd". The // would thus restrict the user (who provides `togo`) into /wherever/youwant/ and it its subdirectories.

There have been a number of previous threads about the security of path objects (and for that matter of os.path) and "sandboxing" paths to disallow traversing "up" out of a particular directory.

I've no idea whether the floordiv operator would work for this purpose (I'm not a security expert, and most of my personal applications can afford to take a very lax view of such things, luckily) but it's an interesting idea. Potentially, it's a bit easy to miss, which may be a problem.

Actually, after having looked into RFC 3986s, I'd say using | instead of // for the path sandboxing operator may be better for at least two reasons: 1a. Because // is used in relative references for URIs to mean essentially 'go all the way up to scheme://', which is quite opposite to the concept of sandboxing. 1b. Because | kind of looks like a wall. 2. Because | has lower precedence than /, meaning that dontmesswiththis | relative / paths == dontmesswiththis | (relative / paths) while dontmesswiththis // relative / paths == (dontmesswiththis // relative) / paths So unless (dontmesswiththis // relative) somehow remembers the sandboxing barrier, the | would do the right thing and // the wrong thing. Then of course you might want it to remember the barrier anyway. I won't open the pandora's box in front of myself now regarding the old threads you mention. But if someone else wants to do that and/or there is something to discuss regarding this, I suggest to move the sandboxing discussion to a new thread. -Koos

On 29 March 2016 at 16:48, Koos Zevenhoven wrote:

On Tue, Mar 29, 2016 at 2:22 PM, Sven R. Kunze wrote:

...

offtopic

I wonder if it would be possible/necessary for the stdlib to include a set of libraries marked as "featured". Not like "provisional" but more like "You should use this one and be aware that it may upgrade faster than normal stdlib modules".

Basically like a "pull module". Just a third-party module pulled from the source and upgraded with each Python version. That would maintain the maintainers flexibility and the increase the availability of these "defacto stdlib" modules.

+1.

It is a different thing to find a random module on PyPI and depend on it, than to depend on something that is ~officially recommended. Or maybe some kind of package reputation system on PyPI? More threads? :)

Been discussed a few times previously (look for things like "sumo distribution", "breaking out the stdlib", "recommended packages"). So yes, again, should be a new thread and should also start with a bit of research into previous discussions. Paul

On Tue, 29 Mar 2016 at 02:45 Michel Desmoulin wrote:

Le 29/03/2016 09:50, Sven R. Kunze a écrit :

...

On 25.03.2016 22:56, Michel Desmoulin wrote:

...

Although path.py, which I have been using for years now (and still prefer to pathlib) subclass str and never caused any problem whatsoever. So really, we should pinpoint where it could be an issue and see if this is a frequent problem, because right now, it seems more a decision based on purity than practicality.

I agree. The PEP says:

""" No confusion with builtins ---------------------------------- In this proposal, the path classes do not derive from a builtin type. This contrasts with some other Path class proposals which were derived from str . They also do not pretend to implement the sequence protocol: if you want a path to act as a sequence, you have to lookup a dedicated attribute (the parts attribute).

Not behaving like one of the basic builtin types also minimizes the potential for confusion if a path is combined by accident with genuine builtin types. """"

I have to admit I cannot follow these statements but they should have appeared to be necessary back then. As experience shows the PyPI module fared far better here.

I am great a fan of theory over practice as what has been proven in theory cannot be proven wrong in practice. However, this only holds if we talk about hard proof. For soft things like "human interaction", "mutual understanding" or "potential for confusion", only the practice of many many people can "prove" what's useful, what's "practical".

+1. Can somebody defend, with practical examples, the imperative to refuse to inherit from str ? And weight it against the benefits of it ?

We have been doing it for years, and so far it's been really, really nice. I can't recall problem I had because of it ever. I can recall numerous editions of my code because I forgot str() on pathlib.Path.

I may regret this, but I will give this a shot. :) I wrote a blog post at http://www.snarky.ca/why-pathlib-path-doesn-t-inherit-from-str because my response was getting a bit long. I have pasted it below in CommonMark. ----- Over on [python-ideas](https://mail.python.org/mailman/listinfo/python-ideas) a discussion has broken out about somehow trying to make `p'/some/path/to/a/file` return an [instance of `pathlib.Path`]( https://docs.python.org/3/library/pathlib.html#pathlib.Path). This led to a splinter discussion as to why `pathlib.Path` doesn't inherit from `str`? I figured instead of burying my response to this question in the thread I'd blog about it to try and explain one approach to API design. I think the key question in all of this is whether paths are semantically equivalent to a sequence of characters? Obviously the answer is "no" since paths have structure, directly represent something like files and directories, etc. Now paths do have a serialized representation as strings -- at least most of the time, but I'm ignoring Linux and the crazy situation of binary paths -- which is why C APIs take in `char *` as the representation of paths (and because C tends to make one try to stick with the types that are part of the C standard). So not all strings represent paths, but all paths can be represented by strings. I think we can all agree with that. OK, so if all paths can be represented as strings, why don't we just make `pathlib.Path` subclass `str`? Well, as I said earlier, not all strings are paths. You can't concatenate a string representing a path with some other random string and expect to get back a string that still represents a valid string (and I'm not talking about "valid" as in "the file doesn't exist", I'm talking about "syntactically not possible"). This is what [PEP 428]( https://www.python.org/dev/peps/pep-0428/) is talking about when it says:

Not behaving like one of the basic builtin types [list str] also minimizes the potential for confusion if a path is combined by accident with genuine builtin types [like str].

Now at this point someone someone will start saying, "but Brett, '[practicality beats purity](https://www.python.org/dev/peps/pep-0020/)' and all of those pre-existing, old OS APIs want a string as an argument!" And you're right, in terms of practicality it would be easier for `pathlib.Path` to inherit from `str` ... for the short/medium term. One thing people are forgetting is that "[explicit is better than implicit]( https://www.python.org/dev/peps/pep-0020/)" as well and as with most design decisions there's not one clear answer. Implicitly treating a path as a string currently works, but that's just because we have inherited a suboptimal representation for paths from C. Now if you use an explicit representation of paths like `pathlib.Path` then you gain semantic separation and understanding of what you are working with. You also avoid any issues that come with implicit `str` compatibility as I pointed out earlier. And before anyone says, "so what?", think about this: Python 2/3. While I'm sure some will say I'm overblowing the comparison, but Python 3 came about because the implicit compatibility of binary and textual data in Python 2 caused major headaches to the point that we made a backwards-incompatible change that has caused widespread ramifications (but which we seem to be coming out on the other side of). By not inheriting from `str`, `pathlib.Path` has avoided a similar potential issue from the get-go. Or another way of looking at it is to ask why doesn't `dict` inherit from `str` so that it's easier to make it easier to stick in the body of an HTTP response so it can be implicitly treated as JSON? If you going, "eww, no because they are different types" then you at least understand the argument I'm trying to make (if you're going "I like that idea", then I think [Perl]( https://www.perl.org/) might be more to your liking than Python and that's fine since the two languages just have different designs). Now back to that "practicality beats purity" side of this. Obviously there are lots of APIs out there that take a string as an argument for a file path. And I understand people don't like using `str(path)` or the upcoming/new [`getattr(path, 'path', path)` idiom]( https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.path) to work around the limitations forced upon them by other APIs that only take a `str` because they occasionally forget it. For this point, I have two answers. One is still "explicit is better than implicit" and you should have tests to begin with to catch when you forget to convert your `pathlib.Path` objects to `str` as needed. I'm just one of those programmers who's willing to type a bit more for easy-to-read code that's less error-prone within reason (and I obviously view this as reasonable). But more importantly, why don't you work with the code and/or projects that are forcing you to convert to `str` to start accepting `pathlib` objects as well for those same APIs? If projects start to work on making `pathlib` objects acceptable anywhere a path is accepted then once Python 3.4 is the oldest version of Python that is supported by the project then they can start considering dropping support for `str` as paths in new releases (obviously this also includes dropping support for Python 2 unless people use [pathlib2](https://pypi.python.org/pypi/pathlib2/)). And I fully admit the stdlib is not exempt from a place that needs updating, so for `importlib` I have [opened an issue](http://bugs.python.org/issue26667) to update it to show this isn't just talk on my end (unfortunately there's some unique bootstrapping problems to import where stuff like `sys.path` probably can't be updated since that has to exist before you can import `pathlib` and that sort of thing ripples out, but I'm hoping I can update at least some things and this is a very unique case of potentially needing to stick with `str`). Hopefully if people start asking for `pathlib` support from projects they will add it, eventually leading to an alleviation of the desire to have `pathlib.Path` inherit from `str`.

On 03/29/2016 11:51 AM, Brett Cannon wrote:

I wrote a blog post at http://www.snarky.ca/why-pathlib-path-doesn-t-inherit-from-str because my response was getting a bit long. I have pasted it below in CommonMark.

Thanks for that. For myself at least, I also find the complexity of pathlib completely unnecessary: I see the structure of paths as loose at best -- at least in regards to the kinds of things we do with paths: - changing file names either in part or in whole, - changing directory names or making new directories from portions of the names of previous directories, - looking for directory or file names that have some substring as part of their names - etc. And yes, it is possible to incorrectly combine two path pieces (or a path and a string) together and get something that is semantically invalid -- but you can do that with any thing more complex than a random jumble of characters: - sentences - names - phrases - paragraphs - etc. But oh well -- what's done is done and we're probably stuck with it now. -- ~Ethan~

On 29.03.2016 20:51, Brett Cannon wrote:

I may regret this, but I will give this a shot. :)

Never, Brett. I think everybody here appreciate a decent discussion. Who knows maybe, somebody advocating for path->str misses an important piece which in turn misses the PEP to describe clearly.

I wrote a blog post at http://www.snarky.ca/why-pathlib-path-doesn-t-inherit-from-str because my response was getting a bit long. I have pasted it below in CommonMark.

Thanks +1 So, let's see:

Over on [python-ideas](https://mail.python.org/mailman/listinfo/python-ideas) a discussion has broken out about somehow trying to make `p'/some/path/to/a/file` return an [instance of `pathlib.Path`](https://docs.python.org/3/library/pathlib.html#pathlib.Path). This led to a splinter discussion as to why `pathlib.Path` doesn't inherit from `str`? I figured instead of burying my response to this question in the thread I'd blog about it to try and explain one approach to API design.

I think the key question in all of this is whether paths are semantically equivalent to a sequence of characters? Obviously the answer is "no" since paths have structure, directly represent something like files and directories, etc. Now paths do have a serialized representation as strings -- at least most of the time, but I'm ignoring Linux and the crazy situation of binary paths -- which is why C APIs take in `char *` as the representation of paths (and because C tends to make one try to stick with the types that are part of the C standard). So not all strings represent paths, but all paths can be represented by strings. I think we can all agree with that.

OK, so if all paths can be represented as strings, why don't we just make `pathlib.Path` subclass `str`? Well, as I said earlier, not all strings are paths. You can't concatenate a string representing a path with some other random string and expect to get back a string that still represents a valid string (and I'm not talking about "valid" as in "the file doesn't exist", I'm talking about "syntactically not possible"). This is what [PEP 428](https://www.python.org/dev/peps/pep-0428/) is talking about when it says:

...

Not behaving like one of the basic builtin types [list str] also minimizes the potential for confusion if a path is combined by accident with genuine builtin types [like str].

Valid string? I assume you mean path, right?

Now at this point someone someone will start saying, "but Brett, '[practicality beats purity](https://www.python.org/dev/peps/pep-0020/)' and all of those pre-existing, old OS APIs want a string as an argument!" And you're right, in terms of practicality it would be easier for `pathlib.Path` to inherit from `str` ... for the short/medium term. One thing people are forgetting is that "[explicit is better than implicit](https://www.python.org/dev/peps/pep-0020/)" as well and as with most design decisions there's not one clear answer. Implicitly treating a path as a string currently works, but that's just because we have inherited a suboptimal representation for paths from C. Now if you use an explicit representation of paths like `pathlib.Path` then you gain semantic separation and understanding of what you are working with. You also avoid any issues that come with implicit `str` compatibility as I pointed out earlier.

I agree with the concatenation issue when using plain strings for paths. However, something that I cannot leave uncommented is "suboptimal representation for paths". What would your optimal representation for paths look like? I cannot believe that the current representation is so bad and has been for so long and nobody, really nobody, has anything done about it.

And before anyone says, "so what?", think about this: Python 2/3. While I'm sure some will say I'm overblowing the comparison, but Python 3 came about because the implicit compatibility of binary and textual data in Python 2 caused major headaches to the point that we made a backwards-incompatible change that has caused widespread ramifications (but which we seem to be coming out on the other side of). By not inheriting from `str`, `pathlib.Path` has avoided a similar potential issue from the get-go. Or another way of looking at it is to ask why doesn't `dict` inherit from `str` so that it's easier to make it easier to stick in the body of an HTTP response so it can be implicitly treated as JSON? If you going, "eww, no because they are different types" then you at least understand the argument I'm trying to make (if you're going "I like that idea", then I think [Perl](https://www.perl.org/) might be more to your liking than Python and that's fine since the two languages just have different designs).

I think most "practicality beats purity" folks don't want that either. They are just bloody lazy. They actually want the benefits of both, the pure datastructure with its convenience methods and the dirty str-like thing with its convenience methods. People don't like it when somebody takes one bag of convenience away from them if they easily can have both. ;-) Better sell it differently. So, let's read on. ;)

Now back to that "practicality beats purity" side of this. Obviously there are lots of APIs out there that take a string as an argument for a file path. And I understand people don't like using `str(path)` or the upcoming/new [`getattr(path, 'path', path)` idiom](https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.path) to work around the limitations forced upon them by other APIs that only take a `str` because they occasionally forget it. For this point, I have two answers.

Sorry to interrupt your here but you missed one important piece (didn't we agree on selling this better?): people do nasty little string manipulations/analysis/regex with their paths for whatever reasons. Right now, it's easier than ever because, well, because paths are strings. :) Selling pathlib with the "getattr(path, 'path', path)" idiom is not going to help those fellows. I mean, are you serious? It looks ridiculous. :-/ Especially the point that a "path" has a "path" is not really getting into my head. Either a "path" is a "path" or it better be named "file/directory" which has a "path". path.path vs file.path/dir.path You need to work on your selling skills, Brett. ;-)

One is still "explicit is better than implicit" and you should have tests to begin with to catch when you forget to convert your `pathlib.Path` objects to `str` as needed. I'm just one of those programmers who's willing to type a bit more for easy-to-read code that's less error-prone within reason (and I obviously view this as reasonable).

But more importantly, why don't you work with the code and/or projects that are forcing you to convert to `str` to start accepting `pathlib` objects as well for those same APIs? If projects start to work on making `pathlib` objects acceptable anywhere a path is accepted then once Python 3.4 is the oldest version of Python that is supported by the project then they can start considering dropping support for `str` as paths in new releases (obviously this also includes dropping support for Python 2 unless people use [pathlib2](https://pypi.python.org/pypi/pathlib2/) https://pypi.python.org/pypi/pathlib2/%29). And I fully admit the stdlib is not exempt from a place that needs updating, so for `importlib` I have [opened an issue](http://bugs.python.org/issue26667) to update it to show this isn't just talk on my end (unfortunately there's some unique bootstrapping problems to import where stuff like `sys.path` probably can't be updated since that has to exist before you can import `pathlib` and that sort of thing ripples out, but I'm hoping I can update at least some things and this is a very unique case of potentially needing to stick with `str`). Hopefully if people start asking for `pathlib` support from projects they will add it, eventually leading to an alleviation of the desire to have `pathlib.Path` inherit from `str`.

I think updating the stdlib looks sufficiently simple to start hacking CPython. So, I would be willing to help here once the github repo is up and running. :) Nice post, Brett. Seems you covered the most important point quite well and also gave an explanation of why people request path->str. Best, Sven

On Tue, 29 Mar 2016 at 11:59 Random832 wrote:

On Tue, Mar 29, 2016, at 14:51, Brett Cannon wrote:

...

OK, so if all paths can be represented as strings, why don't we just make `pathlib.Path` subclass `str`? Well, as I said earlier, not all strings are paths. You can't concatenate a string representing a path with some other random string and expect to get back a string that still represents a valid string

I assume you mean a valid path.

Yep, it's a typo.

You can't add an int to a float and expect to get an integer back, either.

Sure, but that's because int and float are both part of the same type hierarchy (https://www.python.org/dev/peps/pep-3141/).

If you're adding it to a random string you're clearly not using it in a path context, and the type of the result won't be Path anyway.

You're assuming it's always clear. If that was always true then no one would ever find a bug in their code when porting to Python 3. We can all assume we won't screw up, but the fact bugs exist shows that's not true.

On Mar 30, 2016 00:39, "Ethan Furman" wrote:

On 03/29/2016 02:30 PM, Greg Ewing wrote:

...

Paul Moore wrote:

...

P(wherever)/youwant/togo

If P were a suitable seed object, this could be

P/wherever/youwant/togo

An interesting idea, but how would you differentiate between relative and

absolute paths?

Indeed interesting. It would currently work with P = Path(''), kinda like my earlier p'' / ... example. It works even with relative paths, although they would not look like relative ones because of the first /. How short can a name be to be acceptable in builtins?-) K (mobile)

On Tue, 29 Mar 2016 at 14:00 Sven R. Kunze wrote:

On 29.03.2016 20:51, Brett Cannon wrote:

...

I may regret this, but I will give this a shot. :)

Never, Brett. I think everybody here appreciate a decent discussion. Who knows maybe, somebody advocating for path->str misses an important piece which in turn misses the PEP to describe clearly.

...

I wrote a blog post at http://www.snarky.ca/why-pathlib-path-doesn-t-inherit-from-str because my response was getting a bit long. I have pasted it below in CommonMark.

Thanks +1

So, let's see:

...

Over on [python-ideas](https://mail.python.org/mailman/listinfo/python-ideas) a discussion has broken out about somehow trying to make `p'/some/path/to/a/file` return an [instance of `pathlib.Path`]( https://docs.python.org/3/library/pathlib.html#pathlib.Path). This led to a splinter discussion as to why `pathlib.Path` doesn't inherit from `str`? I figured instead of burying my response to this question in the thread I'd blog about it to try and explain one approach to API design.

I think the key question in all of this is whether paths are semantically equivalent to a sequence of characters? Obviously the answer is "no" since paths have structure, directly represent something like files and directories, etc. Now paths do have a serialized representation as strings -- at least most of the time, but I'm ignoring Linux and the crazy situation of binary paths -- which is why C APIs take in `char *` as the representation of paths (and because C tends to make one try to stick with the types that are part of the C standard). So not all strings represent paths, but all paths can be represented by strings. I think we can all agree with that.

OK, so if all paths can be represented as strings, why don't we just make `pathlib.Path` subclass `str`? Well, as I said earlier, not all strings are paths. You can't concatenate a string representing a path with some other random string and expect to get back a string that still represents a valid string (and I'm not talking about "valid" as in "the file doesn't exist", I'm talking about "syntactically not possible"). This is what [PEP 428](https://www.python.org/dev/peps/pep-0428/) is talking about when it says:

...

Not behaving like one of the basic builtin types [list str] also minimizes the potential for confusion if a path is combined by accident with genuine builtin types [like str].

Valid string? I assume you mean path, right?

Yes.

...

Now at this point someone someone will start saying, "but Brett, '[practicality beats purity](https://www.python.org/dev/peps/pep-0020/)' and all of those pre-existing, old OS APIs want a string as an argument!" And you're right, in terms of practicality it would be easier for `pathlib.Path` to inherit from `str` ... for the short/medium term. One thing people are forgetting is that "[explicit is better than implicit](https://www.python.org/dev/peps/pep-0020/)" as well and as with most design decisions there's not one clear answer. Implicitly treating a path as a string currently works, but that's just because we have inherited a suboptimal representation for paths from C. Now if you use an explicit representation of paths like `pathlib.Path` then you gain semantic separation and understanding of what you are working with. You also avoid any issues that come with implicit `str` compatibility as I pointed out earlier.

I agree with the concatenation issue when using plain strings for paths.

However, something that I cannot leave uncommented is "suboptimal representation for paths". What would your optimal representation for paths look like?

I don't know, but I do like pathlib.Path more than a plain string.

I cannot believe that the current representation is so bad and has been for so long and nobody, really nobody, has anything done about it.

I never said it was bad, just suboptimal. If you want you can say using strings as paths is mediocre. I really don't want to argue about the label, just that pathlib.Path is a better representation of a path than a plain string and that strings on their own are nothing that special in terms of representing a string that demands we defend it on its own grounds as such.

...

And before anyone says, "so what?", think about this: Python 2/3. While I'm sure some will say I'm overblowing the comparison, but Python 3 came about because the implicit compatibility of binary and textual data in Python 2 caused major headaches to the point that we made a backwards-incompatible change that has caused widespread ramifications (but which we seem to be coming out on the other side of). By not inheriting from `str`, `pathlib.Path` has avoided a similar potential issue from the get-go. Or another way of looking at it is to ask why doesn't `dict` inherit from `str` so that it's easier to make it easier to stick in the body of an HTTP response so it can be implicitly treated as JSON? If you going, "eww, no because they are different types" then you at least understand the argument I'm trying to make (if you're going "I like that idea", then I think [Perl](https://www.perl.org/) might be more to your liking than Python and that's fine since the two languages just have different designs).

I think most "practicality beats purity" folks don't want that either. They are just bloody lazy. They actually want the benefits of both, the pure datastructure with its convenience methods and the dirty str-like thing with its convenience methods.

People don't like it when somebody takes one bag of convenience away from them if they easily can have both. ;-)

That's fine, but I also don't want to have the worry of latent bugs lurking in code either due to implicit type compatibility and so that clashes in this instance.

Better sell it differently. So, let's read on. ;)

...

Now back to that "practicality beats purity" side of this. Obviously there are lots of APIs out there that take a string as an argument for a file path. And I understand people don't like using `str(path)` or the upcoming/new [`getattr(path, 'path', path)` idiom]( https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.path) to work around the limitations forced upon them by other APIs that only take a `str` because they occasionally forget it. For this point, I have two answers.

Sorry to interrupt your here but you missed one important piece (didn't we agree on selling this better?): people do nasty little string manipulations/analysis/regex with their paths for whatever reasons. Right now, it's easier than ever because, well, because paths are strings. :)

Sure, but that doesn't mean the situation can't be improved. You can also get the string out of Path objects to hack on, but that doesn't mean you should be shipping paths around in your code as strings either. I'm not saying you *can't* use strings to represent a path, just that it isn't the best option available from a higher-level perspective.

Selling pathlib with the "getattr(path, 'path', path)" idiom is not going to help those fellows. I mean, are you serious? It looks ridiculous. :-/

I disagree.

Especially the point that a "path" has a "path" is not really getting into my head. Either a "path" is a "path" or it better be named "file/directory" which has a "path".

path.path vs file.path/dir.path

You need to work on your selling skills, Brett. ;-)

...

One is still "explicit is better than implicit" and you should have tests to begin with to catch when you forget to convert your `pathlib.Path` objects to `str` as needed. I'm just one of those programmers who's willing to type a bit more for easy-to-read code that's less error-prone within reason (and I obviously view this as reasonable).

But more importantly, why don't you work with the code and/or projects that are forcing you to convert to `str` to start accepting `pathlib` objects as well for those same APIs? If projects start to work on making `pathlib` objects acceptable anywhere a path is accepted then once Python 3.4 is the oldest version of Python that is supported by the project then they can start considering dropping support for `str` as paths in new releases (obviously this also includes dropping support for Python 2 unless people use [pathlib2](https://pypi.python.org/pypi/pathlib2/) https://pypi.python.org/pypi/pathlib2/%29). And I fully admit the stdlib is not exempt from a place that needs updating, so for `importlib` I have [opened an issue](http://bugs.python.org/issue26667) to update it to show this isn't just talk on my end (unfortunately there's some unique bootstrapping problems to import where stuff like `sys.path` probably can't be updated since that has to exist before you can import `pathlib` and that sort of thing ripples out, but I'm hoping I can update at least some things and this is a very unique case of potentially needing to stick with `str`). Hopefully if people start asking for `pathlib` support from projects they will add it, eventually leading to an alleviation of the desire to have `pathlib.Path` inherit from `str`.

I think updating the stdlib looks sufficiently simple to start hacking CPython. So, I would be willing to help here once the github repo is up and running. :)

I suspect simple fixes like updating the docs and using `getattr(p, 'path', p)` in the right places will be the kind of quick fixes the GitHub migration will encourage. -Brett

Nice post, Brett. Seems you covered the most important point quite well and also gave an explanation of why people request path->str.

Best, Sven

On Mar 30, 2016 01:07, "Ethan Furman" wrote:

On 03/29/2016 02:47 PM, Koos Zevenhoven wrote:

...

On Mar 30, 2016 00:39, "Ethan Furman" wrote:

...

On 03/29/2016 02:30 PM, Greg Ewing wrote:

...

...

...

If P were a suitable seed object, this could be

P/wherever/youwant/togo

An interesting idea, but how would you differentiate between relative and absolute paths?

Indeed interesting. It would currently work with P = Path(''), kinda like my earlier p'' / ... example. It works even with relative paths, although they would not look like relative ones because of the first /.

Please provide an example of both a relative and an absolute path

construction using `P`.

P/'relative/path' P/'/absolute/path' -K / mobile

...

On Mar 30, 2016 01:07, "Ethan Furman" wrote:

...

...

Please provide an example of both a relative and an absolute path construction using `P`.

P/'relative/path'

P/'/absolute/path'

Thank you.

So under this system a relative path appears with one leading slash, and an absolute path appears with two leading slashes.

Can't say I'm a fan. :(

I'll have to agree. But to be fair, Greg wrote something like 'a suitable seed object' so we could for ex. have P=PathSeed(), and use another operator. P@'rel/path' P@'/abs/path' Dunno. -K@mobile

On Tue, Mar 29, 2016, at 17:37, Brett Cannon wrote:

...

You can't add an int to a float and expect to get an integer back, either.

Sure, but that's because int and float are both part of the same type hierarchy (https://www.python.org/dev/peps/pep-3141/).

...so would str and path, in the scenario being discussed. The argument presented here against allowing paths to be strings is also an argument against allowing ints to be Numbers.

On 29.03.2016 23:57, Paul Moore wrote:

On 29 March 2016 at 22:00, Sven R. Kunze wrote:

...

However, something that I cannot leave uncommented is "suboptimal representation for paths". What would your optimal representation for paths look like? I cannot believe that the current representation is so bad and has been for so long and nobody, really nobody, has anything done about it. Well, pathlib.Path :-)

The point here is that C's char* representation is a serialisation of a path object, just like 123 is a serialisation of an integer object. We don't object to having to convert user input to a string if we need to, why object to having to convert it to a Path if appropriate?

I think there is a misunderstanding here. Let me quote myself: '''I think most "practicality beats purity" folks don't want that either. They are just bloody lazy. They actually want the benefits of both, the pure datastructure with its convenience methods and the dirty str-like thing with its convenience methods.''' The desire is not "str vs. Path"; its "Path + str". (at least from what I can tell) Best, Sven

On 30.03.2016 10:56, Wes Turner wrote:

so, like str but without +, %, .format ?

class pathstr(str): __add__ __mod__ format

python3: pathstr(str)? python2: pathstr(unicode)?

I am not sure I understand. Where is the Path in your code? To make it more explicit: like "all methods of Path" + "all methods of str". And no, format, % and + are nice to have in case you need them; so why removing them? Best, Sven

On 30.03.2016 00:03, Brett Cannon wrote:

I cannot believe that the current representation is so bad and has been for so long and nobody, really nobody, has anything done about it.

[...] pathlib.Path is a better representation of a path than a plain string and that strings on their own are nothing that special in terms of representing a string that demands we defend it on its own grounds as such.

I agree with the first part. I disagree with the second (assuming you mean "representing a path"). Strings are extremely useful for informal working like exploration, debugging etc.

I think most "practicality beats purity" folks don't want that either. They are just bloody lazy. They actually want the benefits of both, the pure datastructure with its convenience methods and the dirty str-like thing with its convenience methods.

People don't like it when somebody takes one bag of convenience away from them if they easily can have both. ;-)

That's fine, but I also don't want to have the worry of latent bugs lurking in code either due to implicit type compatibility and so that clashes in this instance.

Could you give an example here?

Sure, but that doesn't mean the situation can't be improved. You can also get the string out of Path objects to hack on, but that doesn't mean you should be shipping paths around in your code as strings either. I'm not saying you *can't* use strings to represent a path, just that it isn't the best option available from a higher-level perspective.

Let me think of a recommendation here. I personally find the proposed idiom quite cumbersome and ugly but that's just me. However, it is basically not better than "latent bugs lurking in code" due to inheritance. Wrapping stuff up just because we don't know if it's a string or a path is always a bad idea. In pathlibs current form, this would give use a recommendation for application developers: 1) pass a path if you need path methods 2) pass a path.path if you want to do string manipulations 3) don't try both at the same time Would this makes sense?

Selling pathlib with the "getattr(path, 'path', path)" idiom is not going to help those fellows. I mean, are you serious? It looks ridiculous. :-/

I disagree.

That's fine but that's something I would call toiled paper programming. Wrapping things up until it doesn't smell anymore. Seems like here I am the purist. ;-) So, having only one place (the stdlib) dealing with this inconvenience would make things less worse I suppose.

I suspect simple fixes like updating the docs and using `getattr(p, 'path', p)` in the right places will be the kind of quick fixes the GitHub migration will encourage.

I look forward to it. :) Best, Sven

On 30 March 2016 at 10:58, Sven R. Kunze wrote:

...

That's fine, but I also don't want to have the worry of latent bugs lurking in code either due to implicit type compatibility and so that clashes in this instance.

Could you give an example here?

def fix_path(p): # Old code, possibly in a module. # Quite possibly it does a lot more than # simply calling abspath, so it might not # be easy to review return os.path.abspath(p) # Elsewhere p = Path(sys.argv[1]) if not p.is_absolute(): p = fix_path(p) # Now a relative p is no longer a Path object. Later in your code, possibly somewhere # miles away if you rely heavily on paths being interchangeable with strings, you get # an AttributeError when you try to use a Path method on p. Obviously the issue here is trivial to spot. But in real code, it could be a mess of complex logic, with a single os.path call somewhere in an obscure conditional branch. Possibly fix_path has even been updated to work with pathlib, but that one os.path call was accidentally missed and there's no test covering it. Before you say so, of course this is a made up scenario, and of course it'll likely be rare. The question isn't whether such things are common, so much as whether the cost of requiring explicit conversion to string is worth the benefit of avoiding situations like this. And that's a hard judgement call. For casual scripters, and people doing research, the answer is "clearly not". For teaching new users, probably not (new users won't make this sort of mistake, but equally having to teach what a string subclass is relatively early is maybe an issue). For people writing mission critical software, very definitely yes (as a pip developer, I'd hate to switch pip over to a str-subclass path object, whereas switching to a non-subclass one is much more conceivable[1]). Paul [1] Backward compatibility, not wanting another dependency (on the pathlib backport) and the fact that it's change for no actual benefit mean we'd probably not bother anyway, but that's a different decision.

...

. They actually want the benefits of both, the pure datastructure with its convenience methods and the dirty str-like thing with its convenience methods.'''

The desire is not "str vs. Path"; its "Path + str". (at least from what I can tell)

I don't think so. ( though I agree with the lazy part). People want paths to be a strings so that they will work with all the code that already works with strings. That's the primary motivator. If early versions of Python had a path object, there'd be no problem. But whatever happened to duck typing? Paths don't need to BE strings. Rather, everything that needs a path needs to accept anything that acts like a path. I suppose a __path__ magic method would be the "proper" way to do this, but it's really not necessary, it can be handled by the libs themselves. I think there is a fine plan in place, but if more flexibility is required, something like numpy's asarray() would be handy -- it passes actual Numpy arrays through untouched, and makes various efforts to make an array out of anything else. It's remarkably easy and effective to write functions that take virtually anything remotely array like -- lists, tuples, array.arrays, custom objects, etc. -CHB

On 30 March 2016 at 15:26, Chris Barker - NOAA Federal wrote:

...

...

. They actually want the benefits of both, the pure datastructure with its convenience methods and the dirty str-like thing with its convenience methods.'''

The desire is not "str vs. Path"; its "Path + str". (at least from what I can tell)

I don't think so. ( though I agree with the lazy part).

I'm not sure I follow your point here.

People want paths to be a strings so that they will work with all the code that already works with strings.

Correct. That's the prime motivation. But you then say

But whatever happened to duck typing? Paths don't need to BE strings. Rather, everything that needs a path needs to accept anything that acts like a path.

But "all the code that already works with strings" doesn't do that. If we're allowed to change that code then a simple patharg = getattr(patharg, 'path', patharg) is sufficient to work with path objects or strings. Of course this doesn't address functions that *return* paths (as strings). There the caller has to wrap the return value in Path(). Or the function changes to return Path objects, which won't be backward compatible (whether that matters depends on what the code is).

I suppose a __path__ magic method would be the "proper" way to do this, but it's really not necessary, it can be handled by the libs themselves.

Well, the 'path' attribute is basically the same as a __path__ magic method. (That may be what you mean by "can be handled by the libs" - I'm not sure how you'd handle it *other* than in the libraries that currently assume a string).

I think there is a fine plan in place, but if more flexibility is required, something like numpy's asarray() would be handy -- it passes actual Numpy arrays through untouched, and makes various efforts to make an array out of anything else.

It's remarkably easy and effective to write functions that take virtually anything remotely array like -- lists, tuples, array.arrays, custom objects, etc.

Again, this seems to be backwards - the problem isn't treating things that aren't paths as paths (just do p = Path(p) and that's sorted - paths are immutable so there's no need for the complexity that I imagine asarray has to manage). The problem is existing code that expects and/or returns strings. Until that is changed or replaced with code that expects and returns Path objects, people will always need to convert to and from strings. And of course code that has to deal with *both* Path and string objects (for compatibility reasons) can easily enough handle anything it receives, but has a decision to make about what to return - if it returns Path objects, it won't be backward compatible. But if it returns string objects, we'll never get away from the need to convert strings to paths in our code at some point. As a simple case in point, what should the appdirs module (https://pypi.python.org/pypi/appdirs) do? Continue to return strings, and the user needs to wrap them in Path(), or switch to returning Path objects and break backward compatibility? Or maintain a messy API that caters (somehow) for both possibilities? It may be that we're talking at cross purposes here. I'm thinking very much of users working with library code on from PyPI and things like that. You may be looking at this from the perspective of a user who controls the majority of the code they are working with. I'm not sure. A transition like this is never simple, and libraries pretty much have to wait for end user code to switch first. And paths as string subclasses doesn't really count as switching, it simply makes it easier to ignore the issue. Arguably being able to pass a path object to code that expects a string means that you don't have to change your code twice, first to handle path objects because the library code doesn't, then again to switch back when the library code is fixed. But then there is zero motivation for the library code to change. And if code *accepting* paths doesn't change, then code *producing* paths won't either. And we're stuck where we are at the moment, with the string representation as the lowest common denominator representation of a path, and everyone converting back and forth if they want the richer Path interface (or deciding it's not worth it, and sticking to the old os.path routines...). Paul

On 03/30/2016 08:18 AM, Paul Moore wrote:

And of course code that has to deal with *both* Path and string objects (for compatibility reasons) can easily enough handle anything it receives, but has a decision to make about what to return - if it returns Path objects, it won't be backward compatible. But if it returns string objects, we'll never get away from the need to convert strings to paths in our code at some point.

A solution here is to return what you receive: - str coming in? str going out - path coming in? path going out Of course, this gets muddied when multiple path/str arguments are accepted coming in. -- ~Ethan~

On Wed, Mar 30, 2016 at 7:56 PM, Ethan Furman wrote:

On 03/30/2016 07:55 AM, Koos Zevenhoven wrote:

...

Sometimes, in interactive sessions, I have wished for this to work:

somedir(relative_file_path).method()

...which is kind of stupid ;)

Do you mean:

base = Path('source/antipathy') base('test').listdir()

Yes, I think. That is, equivalent to (base / 'test').listdir()

I don't know about pathlib, but with antipathy's Path [1] you can do:

base.listdir('test')

I have wished for base.mkdir("subdir") too, but the addition to pathlib would be backwards incompatible for people that give the mode argument as a positional argument.

[1] Yes, that one is mine. The step is to make it pathlib compatible, which will involve, at a minimum, renaming some attributes.

I wonder if it is still possible to rename pathlib attributes (and keep the old ones as deprecated aliases for a while), now that the adoption of pathlib is still limited. -Koos

On 03/30/2016 10:51 AM, Koos Zevenhoven wrote:

I wonder if it is still possible to rename pathlib attributes (and keep the old ones as deprecated aliases for a while), now that the adoption of pathlib is still limited.

The `.path` attribute, which seems a silly name, I look at as being necessary for compatibility with DirEntry. -- ~Ethan~

On Wed, 30 Mar 2016 at 11:10 Ethan Furman wrote:

On 03/30/2016 10:51 AM, Koos Zevenhoven wrote:

...

I wonder if it is still possible to rename pathlib attributes (and keep the old ones as deprecated aliases for a while), now that the adoption of pathlib is still limited.

The `.path` attribute, which seems a silly name, I look at as being necessary for compatibility with DirEntry.

I don't remember the exact issue that tracked the discussion, but there was back-and-forth on that attribute name and in the end Guido chose `path`.

On Wed, Mar 30, 2016 at 9:26 PM, Brett Cannon wrote:

...

The `.path` attribute, which seems a silly name, I look at as being necessary for compatibility with DirEntry.

I don't remember the exact issue that tracked the discussion, but there was back-and-forth on that attribute name and in the end Guido chose `path`.

This one: https://bugs.python.org/issue22570 See my comment below (from this thread, but there were no reactions). I'm not sure if that option was ever considered. Can we let DirEntry dictate the future of Python? On Sun, Mar 27, 2016 at 5:40 PM, Koos Zevenhoven wrote:

Actually, now that .path is not out yet, would it make sense to call it Path.str or Path.strpath instead, and introduce the same thing on DirEntry and guarantee a str (instead of str or bytes as DirEntry.path now does)? Maybe that would lead to fewer broken implementations in third-party libraries too?

-Koos

Koos Zevenhoven wrote:

See my comment below (from this thread, but there were no reactions). I'm not sure if that option was ever considered. Can we let DirEntry dictate the future of Python?

No. On the other hand, we'd need a /really/ good reason to change it now.

Actually, now that .path is not out yet, would it make sense to call it Path.str or Path.strpath instead, and introduce the same thing on DirEntry and guarantee a str (instead of str or bytes as DirEntry.path now does)? Maybe that would lead to fewer broken implementations in third-party libraries too?

DirEntry needs to return the same type it was given, and one can give it both bytes and strings. Your first point /may/ be valid. The big question is: is DirEntry provisional? I don't think it is. In which case, we're stuck with .path on DirEntry, and may as well use the same name for the same idea on pathlib.Path. -- ~Ethan~

On 30.03.2016 18:49, Ethan Furman wrote:

A solution here is to return what you receive:

- str coming in? str going out - path coming in? path going out

Of course, this gets muddied when multiple path/str arguments are accepted coming in.

I have bad memories about this kind of approach. It reminds me of the peculiarity of os(.path)-related functions in Python 2 that return either bytes or unicode depending on the argument's type. That's utterly confusing and I remember us hunting bugs related to this for hours (basically looking for "where does this byte string come from?"). Best, Sven

On 31.03.2016 11:15, Paul Moore wrote:

Any suggestions for a better design? It might be nice for someone to write a blog entry or posting somewhere on "how to upgrade your 3rd party library to support pathlib" and one of the harder questions will be "how do I decide whether to return a string or a path?" If we had a good recommendation for this, it might help adoption.

Good point. This discussion has been long already so it might make sense to write something down in a concise manner. I am not sure if I can make a good suggestion here because I am still trapped at the point where I need to sort out if a path more like a dict or another structured datatype, or if it is more a monolithic object. That will be the next blog post topic for me. "how do I decide whether to return a string or a path?" << very good question btw; my initial thought would be; make two functions (maybe the same name in two different namespaces), so it's crystal clear what this function takes a arguments and what it returns. But I need to think more about this. Best, Sven

On 31.03.2016 13:31, Koos Zevenhoven wrote:

On Thu, Mar 31, 2016 at 12:46 PM, Sven R. Kunze wrote:

...

I am not sure if I can make a good suggestion here because I am still trapped at the point where I need to sort out if a path more like a dict or another structured datatype, or if it is more a monolithic object. That will be the next blog post topic for me. While discussing, pondering and experimenting with this, I have formed a quite clear view on how I think we should move forward with this. I'm working on a proposal that attempts to not be in conflict with any of the goals. I hope it is not in conflict with your present thoughts about structured/vs monolithic objects either :)

Not at all. The following post it just a reflection of this whole discussion to make it clear to myself and hopefully to others about where several thoughts emerged from and why this whole topic started in the first place. Basically, a structured collection of thoughts, mainly mine: http://srkunze.blogspot.com/2016/03/what-is-path.html I pasted the interesting meat below: *What is a path in the first place?* Brett Cannon made a good point of why PEP 428 (that's the one which introduced pathlib as a stdlib module) deliberately chose Path not to inherit from string. I pondered over it for a while and saw that from his perspective a path is actually not a string but rather a complex data-structure consisting of parts with distinct meanings: literally the steps (of which a path consists) to a resource. The string which represents a path as most people know it is just that: a representation of a more complex object, just like a dict or a list. Let me make this a bit clearer. I think we agree on the following: writing down 21 characters in a row is a string, right? So, what about these 21 characters? {1: 11, 2: 12, 3: 13} If you see that in a Python program (and presumably in many other modern programming languages), you associate that with a dictionary, mapping, hash, etc. So, these 21 characters are a mere representation of a complex object with a very rich functionality. The following paragraphs summarizes what makes the discussion about paths and strings to hard. Depending on whom you ask there are different interpretations of what a path actually is. *Paths as complex objects and strings as their representation* Let's put this analogy to work with paths. If you come from a language that treats strings as file paths (like Python), you can imagine and categorize the facilities of pathlib like so: * pure path - operating on the path string * concrete path - operating on files corresponding to the given path The classic "extract the file extension" issue is done easily with the pure path methods. Writing to a file is also easily done with concrete path operations. So, it seems paths are pretty complex objects with some internal structure and a lot of functionality. *Paths as monolithic object for addressing resources* The previous interpretation is not the only one. Despite all the fine functionality of extracting file extensions, concatenating parts to a larger path, etc., building a path is not an end in itself. When you got a path, it addresses a resource on a machine. When doing so for reading or writing that resource, you actually don't care about whether the path consists of parts or not. To you, it's a monolithic structure, an address. But, you might say, each part of a path represents a directory in hierarchical file systems. Sure that is true for many file systems but not for all. Moreover, how often do you really care about the underlying directory structure? It needs to be there to make things work, of course. When it's there, you mostly don't care. How often do you need to create a subtree in an directory in order to create a single config file? I encounter this once in a while and to be honest: it sucks. ```touch /home/me/on/your/ssd.conf``` will fail if the directory "on/your/" has not been created by somebody before me. Especially for me, as a Web developer, it's quite hard to understand what purpose this restriction serves. Within a Web application the hierarchy of URLs is an emergent property not a prerequisite. Users of git are accustomed to not-committing directories in. Why? Because it's unnecessary and the directory structure is again emerging from the files names themselves (aka from the content). This said, it's rather cumbersome to attribute semantics to the parts of a string that happens to be separated by "/" or "\". At least to me, a path made of one piece. *What about security then?* One can further argue that Web development and git repositories are different here. There is a clear boundary where a path can lead. A URL path cannot address a foreign resource on another domain. git file paths are contained within the repository root. See the common theme? There is a container from where the path of a resource cannot escape. If you have a complete file system available at your fingertips, a lot harm can be done when malicious user input is concatenated unattendedly as a subpath; actually to address a resource within a container but misused to gain access to the complete file system. I cannot say if the container pattern would work for everybody but it's definitely worth exploring as there are some prominent working examples out there. *Conclusion* I really like pathlib since it solves many frequently asked questions in the right way once and for all. But I don't like using it as an argument again inheriting paths from strings saying paths have internal structure in contrast to strings. At least to me, they do not. That, on the other hand, does not necessarily mean inheriting path from string is a good idea but it makes it no worse one than it was before. That's it. Best, Sven

That's a very interesting perspective, and one I'd not fully appreciated. Thanks for posting it. There's a lot I still have to think about in your post, but there's one point I'd like to comment on already: On 31 March 2016 at 15:43, Sven R. Kunze wrote:

When doing so for reading or writing that resource, you actually don't care about whether the path consists of parts or not.

While that's true for *existing* resources (files) it's not so true for when you're creating a file. Like it or not, the filesystem imposes a set of container-containee relationships that is the directory structure. Your mental model (the application domain view) may not care about this, but at the "path" level, we're representing the filesystem structure, where directories are real, and so a/b/c being a set of 3 nested objects, 2 directories and a leaf object (which may be a file or itself a directory) is the reality being modeled. So I'd suggest that your "resource address" is a higher level abstraction, layered on top of the filesystem. In many ways, it's a URI - and as such may map to a filesystem path or to something else. This also clarifies (at least to me) why I am uncomfortable about the idea of merging pathlib and URIs - they are at 2 different abstraction levels. It's somewhat interesting to me that the "resource address" level of abstraction is a higher level that the "filesystem path" level, but has *less* structure. That seems reasonable to me (you abstract away the details). And it makes the string representation at the "lowest level" the really odd case, as it has *no* structure, and yet it's representing a detail level of object structure. Maybe that's an indication of why it's useful to have a structured Path object at this level rather than working solely with strings? Paul

On Mar 31, 2016 10:36 AM, "Wes Turner" wrote:

On Mar 31, 2016 10:02 AM, "Paul Moore" wrote:

...

That's a very interesting perspective, and one I'd not fully appreciated. Thanks for posting it.

There's a lot I still have to think about in your post, but there's one point I'd like to comment on already:

On 31 March 2016 at 15:43, Sven R. Kunze wrote:

...

When doing so for reading or writing that resource, you actually

don't care

...

...

about whether the path consists of parts or not.

While that's true for *existing* resources (files) it's not so true for when you're creating a file. Like it or not, the filesystem imposes a set of container-containee relationships that is the directory structure. Your mental model (the application domain view) may not care about this, but at the "path" level, we're representing the filesystem structure, where directories are real, and so a/b/c being a set of 3 nested objects, 2 directories and a leaf object (which may be a file or itself a directory) is the reality being modeled.

So I'd suggest that your "resource address" is a higher level abstraction, layered on top of the filesystem. In many ways, it's a URI - and as such may map to a filesystem path or to something else. This also clarifies (at least to me) why I am uncomfortable about the idea of merging pathlib and URIs - they are at 2 different abstraction levels.

It's somewhat interesting to me that the "resource address" level of abstraction is a higher level that the "filesystem path" level, but has *less* structure. That seems reasonable to me (you abstract away the details). And it makes the string representation at the "lowest level" the really odd case, as it has *no* structure, and yet it's representing a detail level of object structure. Maybe that's an indication of why it's useful to have a structured Path object at this level rather than working solely with strings?

it would seem that the utility here includes having a @memoize-d parsed copy of the path; and then not doubling forward/backslashes ... and then checking for '../' traversals and any chroot-like relpath predicates w/ e.g. fnmatch); and not shell quoting

None = gethostname() os.path.sep.join(['file:', None, '/home/user/']) os.path.sep.join(['file:', None, ('/home', '/user/')])

p = Path('/home/user/') p.parts

# ... URLObject + path.py

URLObject has a .path attr for getattr, too. - char*: bytestring, unicode/str

...

Paul _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On 03/31/2016 08:56 AM, Sven R. Kunze wrote:

That might also influence how we look at the other thread about including better URL handling to Python. It's just another abstraction level and the main question is: is Python ready for that level or better use a third party library for it.

The actual questions are: 1) Should URL abstraction be merged in to pathlib? 2) If no, does it /need/ to be in the stdlib? -- ~Ethan~

On 30 March 2016 at 06:51, Michael Selik wrote:

In days of yore, before package managers, people used to download source code and read it. Maybe not all of it, but enough to feel not terribly scared when running that code. In modern times, with centralized package repositories and convenient installer tools, we want a better way to know "what’s the right package to use for this task?" It's a "first-world problem" in a sense. There are too many products in my supermarket aisle! On a personal note, I have on occasion spent twenty minutes choosing a toothpaste at Target.

If I care enough, I'll take a moment to look at how many downloads have been counted recently, how many issues there are (usually on GitHub), how many contributors, etc. I'll read the docs. I might even poke around in the source. I'll also check Google rankings to see if people are chatting about the module and linking to it.

I'm not sure if there's a good centralized solution to this problem, but it's a question many people are asking: How do I know which non-stdlib module to use?

Back at Georgia Tech, my professor [0] once told me that the way to get rich is to invent an index. He was referring to Richard Florida's "Creative Class" book and the subsequent "Creativity Index" consulting that Florida provided to various municipalities. People who score high on the index pay you to speak. People who score low on the index pay you to consult.

There are a few companies who sell a Python package reputation service, along with some distribution tools. Continuum's Anaconda, Enthought's Canopy, and ActiveState's ActivePython come to mind. There's clearly value in helping people answer this question.

And so do Linux distributions. There's a reason there are so many of the latter: "good" choices depend a great deal on what you're doing, which means there's no point in trying to come up with the "one true reputation system" for software components. djangopackages.com does a relatively good job for the Django ecosystem, but the simple fact of using Django puts you far enough down the path towards solving a particular kind of problem (web service design with a rich user permission system backed by a relational database) that a community driven rating system can work. By contrast, I think Python itself covers too many domains for a common rating system to be feasible - "good for education" is not the same as "good for sysadmin tasks" is not the same as "good for data analysis" is not the same as "good for network service development", etc. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

2016-03-29 22:51 GMT+02:00, Michael Selik : [...]

Back at Georgia Tech, my professor [0] once told me that the way to get rich is to invent an index.

2016-04-01 16:42 GMT+02:00, Alexander Walters :

On 4/1/2016 09:07, Nick Coghlan wrote: [...]

...

By contrast, I think Python itself covers too many domains for a common rating system to be feasible - "good for education" is not the same as "good for sysadmin tasks" is not the same as "good for data analysis" is not the same as "good for network service development", etc.

Cheers, Nick. Not that this was the original proposal, but there can be such a thing as a universal 'bad' package, though. So about the only thing that a universal package rating system can do effectively is shame developers. I don't think we want that.

1. There is not only good-bad dimension. beauty-ugly, simple-complex, flat-nested (and others from PEP20) trusted-untrusted is one from others. 2. Art critics are not to shame artists. Constructive criticism could help authors too. It is not benefit only for "customers". One who wants invent index to get rich could probably invent this job (Python critic) too. 3. pypi has number of downloads for day, week and month. If there is public api to data - one could make graphs, trends, etc and public them. If they will be nice then they probably could be included on pypi site too.