On 11.04.2016 22:55, Alexander Walters wrote:

Every conceivable way to fix pathlib have already been argued. Are any of them worth doing? Can we get consensus enough to implement one of them? If not, we should consider either dropping the matter or dropping the module.

Right now, I don't see pathlib removed. Why? Because using strings alone has its caveats (we all know that). So, I cannot imagine an alternative concept to pathlib right now. We might call it differently, but the concept stays unchanged. MAYBE, if there's an alternative concept, I could be convinced to support dropping the module. Best, Sven PS: The only way out that I can imagine is to fix pathlib. I am not in favor of fixing functions of "os" and "os.path" to except "path" objects; which does the majority here discuss now with the new __fspath__ protocol. But shaping what we have is definitely worth it.

On 11.04.2016 23:08, Random832 wrote:

On Mon, Apr 11, 2016, at 17:04, Sven R. Kunze wrote:

...

PS: The only way out that I can imagine is to fix pathlib. I am not in favor of fixing functions of "os" and "os.path" to except "path" objects; Why not?

It occurred to me after pondering over Paul's comments. "os" and "os.path" is just a completely different level of abstraction. There is just no need to mess with them. The initial failure of my colleague and me of using pathlib can be solely attributed to pathlib's lack of functionality. Not to the incompatibility of "os" nor "os.path" with "Path" objects. Best, Sven

On 12.04.2016 12:41, Paul Moore wrote:

As your thoughts appear to have been triggered by my comments, I feel I should clarify.

1. I like pathlib even as it is right now, and I'm strongly -1 on removing it. 2. The "external dependency" aspect of 3rd party solutions makes them far less useful to me. 3. The work on improving integration with the stdlib (which is nearly sorted now, as far as I can see) is a big improvement, and I'm all in favour. But even without it, I wouldn't want pathlib to be removed. 4. There are further improvements that could be made to pathlib, certainly, but again they are optional, and pathlib is fine without them.

My conclusion is that these changes are not optional and tweaking os, io and shutil is just yet another workaround for a clean solution. :) Just my two cents.

5. I wish more 3rd party code integrated better with pathlib. The improved integration work might help with this. But ultimately, Python 2 compatibility is likely to be the biggest block (either perceived or real - we can make pathlib support as simple as possible, but some 3rd party authors will remain unwilling to add support for Python 3 only features in the short term). This isn't a pathlib problem. 6. There will probably always be a place for low-level os/os.path code. Adding support in those modules for pathlib doesn't affect that fact, but does make it easier to use pathlib "seamlessly", so why not do so?

tl; dr; I'm 100% in favour of pathlib, and in the direction the current discussion (excluding "let's give up on pathlib" digressions) is going.

Best, Sven

On 12.04.2016 18:04, Chris Barker wrote:

On Tue, Apr 12, 2016 at 7:54 AM, Sven R. Kunze mailto:srkunze@mail.de> wrote:

My conclusion is that these changes are not optional and tweaking os, io and shutil is just yet another workaround for a clean solution. :)

Is the clean solution to re-implement EVERYTHING in the stdlib that involves a path in a new, fancy pathlib way?

If we were starting from scratch, I _might_ like that idea, but we're not starting from scratch. And that would cement in pathlib itself, leaving no room for other path implementations. kind of like how the pre-__Index__ python cemented in python integers as the only objects once could use to index a sequence.

I cannot remember us using another datetime library. So, I don't value this "advantage" as much as you do. Best, Sven

one little note: On Tue, Apr 12, 2016 at 3:41 AM, Paul Moore wrote:

4. There are further improvements that could be made to pathlib, certainly, but again they are optional, and pathlib is fine without them.

Exactly -- "improvements to pathlib" and "make the stdlib pathlib compatible" are completely orthogonal.

5. I wish more 3rd party code integrated better with pathlib. The improved integration work might help with this. But ultimately, Python 2 compatibility is likely to be the biggest block (either perceived or real - we can make pathlib support as simple as possible, but some 3rd party authors will remain unwilling to add support for Python 3 only features in the short term). This isn't a pathlib problem.

true -- though the proposed protocol approach opens doors there -- any third party lib can check for a __whatever_it's_called__ and run fine in py2 or py3 or, indeed, any version of python. Also if you really don't like pathlib, then the protocol allows you to write/use a different path implementation -- really win-win. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On 04/11/2016 02:04 PM, Sven R. Kunze wrote:

On 11.04.2016 22:55, Alexander Walters wrote:

...

Every conceivable way to fix pathlib have already been argued. Are any of them worth doing? Can we get consensus enough to implement one of them? If not, we should consider either dropping the matter or dropping the module.

Right now, I don't see pathlib removed. Why? Because using strings alone has its caveats (we all know that). So, I cannot imagine an alternative concept to pathlib right now. We might call it differently, but the concept stays unchanged.

We've pretty decided that we have two options: 1. remove pathlib 2. make the stdlib work with pathlib So we're trying to make option 2 work before falling back to option 1. If you have a way to make pathlib work with the stdlib that doesn't involve "fixing" os and os.path, now is the time to speak up. -- ~Ethan~

On Mon, Apr 11, 2016, at 17:15, Ethan Furman wrote:

So we're trying to make option 2 work before falling back to option 1.

If you have a way to make pathlib work with the stdlib that doesn't involve "fixing" os and os.path, now is the time to speak up.

Fully general re-dispatch from argument types on any call to a function that raises TypeError or NotImplemented? [e.g. call Path.__missing_func__(os.open, path, mode)] Have pathlib monkey-patch things at import? On Mon, Apr 11, 2016, at 17:43, Sven R. Kunze wrote:

So, I might add:

3. add more high-level features to pathlib to prevent a downgrade to os or os.path

3. reimplement the entire ecosystem in every walled garden so no-one has to leave their walled gardens. What's the point of batteries being included if you can't wire them to anything? I don't get what you mean by this whole "different level of abstraction" thing, anyway. The fact that there is one obvious thing to want to do with open and a Path strongly suggests that that should be able to be done by passing the Path to open. Also, what level of abstraction is builtin open? Maybe we should _just_ leave os alone on the grounds of some holy sacred lowest-level-itude, but allow io and shutils to accept Path?

On Apr 12, 2016, at 10:52 AM, Sven R. Kunze wrote:

Path(...).open() is your friend then. I don't see why you need os.open.

Refusing to upgrade it like saying, everything was better in the old days. So let's use os.open instead of Path(...).open().

I think it was a mistake to have Path(…).open to be honest and I think the main reason it exists is because open(Path(…)) doesn’t work (yet!). You can’t hang every single thing you might ever want to do to a Path off the path object. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 12.04.2016 16:59, Random832 wrote:

...

Strings are strings. Paths are paths. That's were the difference is. Yes but why aren't these both "things that you may want to use to open a file"?

Because "things that you may want to use to open a file" is a bit vague and thus conceal the fact that we really need. As an example: time.sleep takes a number of seconds (notice the primitive datatype just like a string) and does not take timedelta. Why don't we add datetime.timedelta support to time.sleep? Very same thing.

...

...

The fact that there is one obvious thing to want to do with open and a Path strongly suggests that that should be able to be done by passing the Path to open. Path(...).open() is your friend then. I don't see why you need os.open. Because I'm passing it to modfoo.dosomethingwithafile() which takes a filename and passes it to shutils, which passes it to builtin open, which passes it to os.open.

Should Path grow a dosomethingwithmodfoo method?

Because we can argue here the other way round and say: "oh, pathlib can do things, I cannot do with os.path." Should os.path grow those things? Put differently, you cannot do everything. But the most common issues should be resolved in the correct module. This is no argument for or against either solution. I am sorry, if my contribution on the threads of python-ideas made it seem that I would always support this idea. I don't anymore. However, I will still be happy with the outcome even if not perfect, will help making the Python stdlib better. :) Best, Sven

On Mon, 11 Apr 2016 14:15:02 -0700 Ethan Furman wrote:

We've pretty decided that we have two options:

1. remove pathlib 2. make the stdlib work with pathlib

So we're trying to make option 2 work before falling back to option 1.

I have been doing a lot of porting to Python 3 and have really enjoyed having pathlib, even in its current state. In one of my previous projects using python 2 on linux we had to code to handle files with names that where not utf-8. (Users could FTP a file into the file system and it could end up non-utf-8). Today we would have used pathlib to represent paths in the app. But we would need to be able to detect the paths that do not following the fs encoding rules. I would suggest a predicate in Path to report that the path cannot be encoding without the use of surrogates. Not sure what to call the predicate. This can be used by code that cares to handle converting the path into a suitable presentation string for showing to a user. I'm assuming here that PEP383 may not provide an presentation string that is suitable for showing to users. In the case of our product we refused to use files that did not encode to utf-8 and had a UI to allow the user to fix the name. One reason for files that can only be represented as bytes() being detectable I suspect is to avoid security issues. I think if I have my black hat on I would probe a python3 app with filenames that are non-utf-8 and see if I can break the app. Barry

On 11.04.2016 23:05, Random832 wrote:

On Mon, Apr 11, 2016, at 16:48, Sven R. Kunze wrote:

...

...

If there is headway being made, I do not see it. Funny that you brought it up. I was about posting something myself. I cannot agree completely. But starting with a comment from Paul, I realized that pathlib is something different than a string. After doing

On 11.04.2016 22:33, Alexander Walters wrote: the research and our issues with pathlib, I found:

- pathlib just needs to be improved (see my 5 points) - os[.path] should not tinkered with I'm not so sure. Is there any particular reason os.path.join should require its arguments to be homogenous, rather than allowing os.path.join('a', b'b', Path('c')) to return 'a/b/c'?

Besides the fact, that I don't like mixing types (this was something that worried me about the discussion from the beginning), you can achieve the same using pathlib alone. There's no need of it let alone the maintenance and slowdown of these implicit conversions.

...

I know that all of those discussions of a new protocol (path->str, __fspath__ etc. etc.) might be rendered worthless by these two statements. But that's my conclusion.

"os" and "os.path" are just lower level. "pathlib" is a high-level, convenience library. When using it, I don't want to use "os" or "os.path" anymore. If I still do, "pathlib" needs improving. *Not "os" nor "os.path"*. The problem isn't you using os. It's you using other modules that use os. or io, shutil, or builtins.open. Or pathlib, if what *you're* using is some other path library. Are you content living in a walled garden where there is only your code and pathlib, and you never might want to pass a Path to some function someone else (who didn't use pathlib) wrote?

os is being used as an example because fixing os probably gets you most other things (that just pass it through to builtins.open which passes it through to os.open) for free.

Hypothetical assumptions meeting implicit type conversions. You might prefer those, I don't because of good reason. I was one of those starting the discussion around pathlib improvements. I understand now, that this is one of its minor issues. And btw. using some "other pathlib" is no argument for or against improving "THE pathlib". The .path attribute will do it from what I can see. Best, Sven

That is great news. I just couldn't see it myself in the threads On 4/11/2016 16:51, Ethan Furman wrote:

...

If there is headway being made, I do not see it.

It's being made, and I dare say we are close to the end.

On Mon, 11 Apr 2016 at 14:42 Ben Finney wrote:

Alexander Walters writes:

...

That is great news. I just couldn't see it myself in the threads

Agreed. A summary posting, from someone who has a good handle on the issue and outcome, would be very helpful.

- Guido has put Chris Angelico and myself in charge of drafting a proposal once we are done discussing things as a PEP (probably an amendment to the pathlib PEP where I will also explain why we are still not subclassing str) - Ethan Furman has volunteered to help out with code work (as have I) - Name bikeshedding never seems to end, but there seems to be coalescing around __fspath__ or __fspathname__ (I think, although __fspath__ seems to be what everyone has been typing today; I'm trying to stay out of it so as to not influence too much) - We are only discussing two things still (all going on in the threads relating to return values, arguments, types, etc. in their titles)... - Should path.__fspath__() be allowed to return bytes on top of strings? (we seem to have found an amicable way to allow os.fspath() to let a bytes argument pass through just like str in an explicit fashion) - Should we explicitly type check in os.fspath() what path.__fspath__() returns or just let it fall through and hope people do the right thing? That's pretty much it unless Chris or Ethan disagree. So I think pathlib is far from being as dead as a parrot. ;)

On Tue, Apr 12, 2016 at 7:55 AM, Brett Cannon wrote:

That's pretty much it unless Chris or Ethan disagree. So I think pathlib is far from being as dead as a parrot. ;)

That looks like an accurate summary! ChrisA