Mailman 3 April 2016 - Python-ideas

Re: [Python-ideas] Consistent programming error handling idiom
by Rian Hunter April 11, 2016

April 11, 2016

>> On Fri, Apr 08, 2016 at 11:24:15AM -0700, rian(a)thelig.ht wrote: >> I want to live in a world where I can do this: >> >> while cbs: >> cb = cbs.pop() >> try: >> cb() >> except Exception as e: >> logging.exception("In main loop") >> if is_a_bug(e): >> raise SystemExit() from e > > And I want a pony :-) > > So long as Python allows people to write code like this: > > # A deliberately silly example > if len(seq) == 0: > raise ImportError("assertion fails") > > you cannot expect to automatically know what an exception means without > any context of where it came from and why it happened. The above example > is silly, but it doesn't take much effort to come up with more serious > ones: > > - an interpreter written in Python may raise SyntaxError, which > is not a bug in the interpreter; > > - a test framework may raise AssertionError for a failed test, > which is not a bug in the framework; > > - a function may raise MemoryError if the call *would* run out of > memory, but without actually running out of memory; consequently > it is not a fatal error, while another function may raise the > same MemoryError because it actually did fatally run out of memory. > > Effectively, you want the compiler to Do What I Mean when it comes to > exceptions. DWIM may, occasionally, be a good idea in applications, but > I maintain it is never a good idea in a programming language. I think you're misinterpreting me. I don't want a pony and I don't want a sufficiently smart compiler. I want a consistent opt-in idiom with community consensus. I want a clear way to express that an exception is an error and not an internal bug. It doesn't have to catch 100% of cases, the idiom just needs to approach consistency across all Python libraries that I may import. If the programmer doesn't pay attention to the idiom, then is_a_bug() will never return true (or not True if it's is_not_a_bug()). AssertionError is already unambiguous, I'm sure there are other candidates as well. I'm not the first or only one to want something like this http://blog.tsunanet.net/2012/10/pythons-screwed-up-exception-hierarchy.html But I can see the pile on has begun and my point is lost. Maybe this will get added in ten or twenty years.

5 5

Boundaries for unpacking
by Michel Desmoulin April 10, 2016

April 10, 2016

Python is a lot about iteration, and I often have to get values from an iterable. For this, unpacking is fantastic: a, b, c = iterable One that problem arises is that you don't know when iterable will contain 3 items. In that case, this beautiful code becomes: iterator = iter(iterable) a = next(iterator, "default value") b = next(iterator, "default value") c = next(iterator, "default value") More often than not, I wish there were a way to specify a default value for unpacking. This would also come in handy to get the first or last item of an iterable that can be empty. Indeed: a, *b = iterable *a, b = iterable Would fail if the iterable's iterator cannot yield at least one item. I don't have a clean syntax in mind, so I hope some people will get inspired by it and will make suggestions. I add a couple of ideas: a, b, c with "default value" = iterable a, b, c except "default value" = iterable a, b, c | "default value" = iterable But none of them are great. Another problem is that if you can't use unpacking on generator yielding 1000000 values: a, b = iterable # error a, b, c* = iterable # consume the whole thing, and load it in memory So if I want the 2 first items, I have to go back to: iterator = iter(iterable) a = next(iterator) b = next(iterator) Now, I made a suggestion to allow slicing on any iterable (or at least on any iterator), but I'm not going to hold my breath on it: a, b = iterable[:2] # allow that on generators or a, b = iter(iterable)[:2] # allow that Some have suggested to add a new syntax: a, b = iterable(:2) But even just a way to allow to specify that you just want the 2 first items without unpacking the all iterable would work for me. Ideally though, the 2 options should be able to be used together.

6 8

Hijacking threads [was: Changing the meaning of bool.__invert__]
by Ethan Furman April 9, 2016

April 9, 2016

In the recent thread about changing the meaning of __invert__ on bools I was accused of an attempted hijack. Can anybody please enlighten me as to what, exactly, I did wrong? Original post follows: > I think the str() of a value, while possibly being the most > interesting piece of information (IntEnum, anyone?), is hardly the > most intrinsic. > > If we do make this change, besides needing a couple major versions to > make it happen, will anything else be different? > > - no longer subclass int? > - add an "unknown" value? > - how will indexing work? > - or any of the other operations? > - don't bother with any of the other mathematical operations? > - counting True's is not the same as adding True's > > I'm not firmly opposed, I just don't see a major issue here -- I've > needed an Unknown value for more often that I've needed ~True to be > False. -- ~Ethan~

6 7

New scope for exception handlers
by Joseph Jevnik April 9, 2016

April 9, 2016

I would like to propose a change to exception handlers to make it harder to accidently leak names defined only in the exception handler blocks. This change follows from the decision to delete the name of an exception at the end of a handler. The goal of this change is to prevent people from relying on names that are defined only in a handler. As an example, let's looks at a function with a try except: def f(): try: ... except: a = 1 return a This function will only work if the body raises some exception, otherwise we will get an UnBoundLocalError. I propose that we should make `a` fall out of scope when we leave the except handler regardless to prevent people from depending on this behavior. This will make it easier to catch bugs like this in testing. There is one case where I think the name should not fall out of scope, and that is when the name is already defined outside of the handler. For example: def g(): a = 1 try: ... except: a = 2 return a I think this code is well behaved and should continue to work as it already does. There are a couple of ways to implment this new behavior but I think the simplest way to do this would be to treat the handler as a closure where all the free variables defined as nonlocal. This would need to be a small change to the compiler but may have some performance implications for code that does not hit the except handler. If the handler is longer than the bytecode needed to create the inner closure then it may be faster to run the function when the except handler is not hit. This changes our definition of f from: 2 0 SETUP_EXCEPT 8 (to 11) 3 3 LOAD_CONST 1 (Ellipsis) 6 POP_TOP 7 POP_BLOCK 8 JUMP_FORWARD 14 (to 25) 4 >> 11 POP_TOP 12 POP_TOP 13 POP_TOP 5 14 LOAD_CONST 2 (1) 17 STORE_FAST 0 (a) 20 POP_EXCEPT 21 JUMP_FORWARD 1 (to 25) 24 END_FINALLY 6 >> 25 LOAD_FAST 0 (a) 28 RETURN_VALUE to something more like: f - 3 0 SETUP_EXCEPT 8 (to 11) 4 3 LOAD_CONST 0 (Ellipsis) 6 POP_TOP 7 POP_BLOCK 8 JUMP_FORWARD 20 (to 31) 5 >> 11 POP_TOP 12 POP_TOP 13 POP_TOP 14 LOAD_CONST 1 (<code object <excepthandler> at 0x7febcd6e2300, file "<code>", line 1>) 17 LOAD_CONST 2 ('<excepthandler>') 20 MAKE_FUNCTION 0 23 CALL_FUNCTION 0 (0 positional, 0 keyword pair) 26 POP_EXCEPT 27 JUMP_FORWARD 1 (to 31) 30 END_FINALLY 7 >> 31 LOAD_FAST 0 (a) 34 RETURN_VALUE f.<excepthandler> ----------------- 1 0 LOAD_CONST 0 (1) 3 STORE_FAST 0 (a) 6 LOAD_CONST 1 (None) 9 RETURN_VALUE This new code properly raises the unbound locals exception when executed. For g we could use a MAKE_CLOSURE instead of MAKE_FUNCTION.

5 5

Re: [Python-ideas] Consistent programming error handling idiom
by rian＠thelig.ht April 8, 2016

April 8, 2016

>> Oops, good call. That was a bad example. Maybe this one is a bit >> better: >> >> try: >> assert no_bugs() >> except AssertionError: >> # bugs are okay >> pass > > Not sure I understand this example. It's obviously a toy, because > you'd never put an 'assert' just to catch and discard the exception - > the net result is that you call no_bugs() if you're in debug mode and > don't if you're not, which is more cleanly spelled "if __debug__: > no_bugs()". The example is removed from its original context. The point was to show that there is no reason to locally catch AssertionError (contrast with AttributeError, ValueError, TypeError, etc.). AssertionError is *always* indicative of a bug (unlike AttributeError which only may be indicative of a bug). >>> This is what I'd call a boundary location. You have "outer" code and >>> "inner" code. Any uncaught exception in the inner code should get >>> logged rather than aborting the outer code. >> >> I'm very familiar with this pattern in Python and I've used it myself >> countless times. Unfortunately I've seen instances where it can lead >> to disastrous behavior, I think we all have. It seems to have limited >> usefulness in production and more usefulness in development. The point >> of my original email was precisely to put the legitimacy and >> usefulness of code like that into question. >> > > When does this lead to disaster? Is it because someone creates a > boundary that shouldn't exist, or because the code maintains global > state in bad ways? The latter is a problem even without exceptions; > imagine a programming error that doesn't cause an exception, but just > omits some crucial "un-modify global state" call. You have no way of > detecting that in the outer code, and your state is messed up. This leads to disaster in the case of buggy code. This leads to a greater disaster when the boundary allows the code to continue running. It's not a black and white thing, there is no 100% way to detect for buggy code but my argument is that you shouldn't ignore an assert when you're lucky enough to get one. >>> As to resetting stuff: I wouldn't bother; your functions should >>> already not mess with global state. The only potential mess you >>> should >>> consider dealing with is a database rollback; and actually, my >>> personal recommendation is to do that with a context manager inside >>> the inner code, rather than a reset in the exception handler in the >>> outer code. >> >> So I agree this pattern works if you assume all code is exception-safe >> (context managers will clean up intermediate state) and there are no >> programming errors. There are lots of things that good code should do >> but as we all well know good code doesn't always do those things. My >> point is that except-log loops are dangerous and careless in the face >> of programming errors. >> > > I don't think the except-log loop is the problem here. The problem is > the code that can go in part way, come out again, and leave itself in > a mess. I'm not saying except-log is to blame for buggy code. I'm saying that when an exception occurs, there's no way to tell whether it was caused by an internal bug or an external error. Except-log should normally be limited to exceptions caused by external errors in production code. >> Programming errors are unavoidable. In a large system they are a daily >> fact of life. When there is a bug in production it's very dangerous to >> re-enter a code block that has demonstrated itself to be buggy, else >> you risk corrupting data. For example: >> >> def random_code(state): >> assert is_valid(state.data) >> >> def user_code_a(state): >> state.data = "bad data" >> # the following throws >> random_code(state) >> >> def user_code_b(state): >> state.db.write(state.data) >> >> def main_loop(): >> state = State() >> loop = [user_code_a, >> user_code_b] >> for fn in loop: >> try: >> fn() >> except Exception: >> log_exception() >> >> This code allows user_code_b() to execute and corrupt data even though >> random_code() was lucky enough to be called and detect bad state early >> on. > > Can you give a non-toy example that has this kind of mutable state at > top level? I suspect it's bad design. If it's truly necessary, use a > context manager to guarantee the reset: > > def user_code_a(state): > with state.set_data("bad data"): > random_code(state) > Yes it may be bad design or the code may have bugs but that's precisely the point. By the time the exception hits your catch-all, there's no universal way of determining whether or not the exception was due to an internal bug (or bad design or whatever) or an external error. >> You may say the fix is to assert correct data before writing to the >> database and, yes, that would fix the problem for future executions in >> this instance. That's not the point, the point is that incorrect buggy >> code is running in production today and it's imperative to have >> multiple safeguards to limit its damage. For example, you wouldn't >> have an except-log loop in an airplane control system. >> > > Actually, yes I would. The alternative that you're suggesting is to > have any error immediately shut down the whole system. Is that really > better? To have the entire control system disabled? The alternative I'm suggesting is to reset that flight computer and switch to the backup one (potentially written by a different team). >> Sometimes an error is just an error but sometimes an error signifies >> the running system itself is in a bad state. It would be nice to >> distinguish between the two in a consistent way across all Python >> event loops. Halting on any escaped exception is inconvenient, but >> continuing after any escape exception is dangerous. >> > > There's no way for Python to be able to fix this for you. The tools > exist - most notably context managers - so the solution is to use > them. Context managers don't allow me to determine whether an exception is caused by an internal bug or external error, so it's not a solution to this problem. I want to live in a world where I can do this: while cbs: cb = cbs.pop() try: cb() except Exception as e: logging.exception("In main loop") if is_a_bug(e): raise SystemExit() from e Python may be able to do something here. One possible thing is a new exception hierarchy, there may be other solutions. This may be sufficient but I doubt it: def is_a_bug(e): return isinstance(e, AssertionError) The reason I am polling python-ideas is that I can't be the only one who has ever encountered this deficiency in Python. Rian

7 11

Re: [Python-ideas] Consistent programming error handling idiom
by Rian Hunter April 8, 2016

April 8, 2016

> From: Chris Angelico <rosuav(a)gmail.com> >> On Fri, Apr 8, 2016 at 12:17 PM, Rian Hunter <rian(a)thelig.ht> wrote: >> Contrast with the following code that never makes sense (and is why I >> said that NameError definitely signifies a programming error): >> >> try: >> foo = bar >> except NameError: >> foo = 0 > > This is exactly the idiom used to cope with builtins that may or may > not exist. If you want to support Python 2 as well as 3, you might use > something like this: > > try: > input = raw_input > except NameError: > raw_input = input Oops, good call. That was a bad example. Maybe this one is a bit better: try: assert no_bugs() except AssertionError: # bugs are okay pass > This is what I'd call a boundary location. You have "outer" code and > "inner" code. Any uncaught exception in the inner code should get > logged rather than aborting the outer code. I'd spell it like this: > > try: > response = ... > except BaseException as e: > logging.exception(...) > response = create_500_response() > > [snip] > > At this kind of boundary, you basically catch-and-log *all* > exceptions, handling them the same way. Doesn't matter whether it's > ValueError, SyntaxError, NameError, RuntimeError, GeneratorStop, or > SystemExit - they all get logged, the client gets a 500, and you go > back and look for another request. I'm very familiar with this pattern in Python and I've used it myself countless times. Unfortunately I've seen instances where it can lead to disastrous behavior, I think we all have. It seems to have limited usefulness in production and more usefulness in development. The point of my original email was precisely to put the legitimacy and usefulness of code like that into question. > As to resetting stuff: I wouldn't bother; your functions should > already not mess with global state. The only potential mess you should > consider dealing with is a database rollback; and actually, my > personal recommendation is to do that with a context manager inside > the inner code, rather than a reset in the exception handler in the > outer code. So I agree this pattern works if you assume all code is exception-safe (context managers will clean up intermediate state) and there are no programming errors. There are lots of things that good code should do but as we all well know good code doesn't always do those things. My point is that except-log loops are dangerous and careless in the face of programming errors. Programming errors are unavoidable. In a large system they are a daily fact of life. When there is a bug in production it's very dangerous to re-enter a code block that has demonstrated itself to be buggy, else you risk corrupting data. For example: def random_code(state): assert is_valid(state.data) def user_code_a(state): state.data = "bad data" # the following throws random_code(state) def user_code_b(state): state.db.write(state.data) def main_loop(): state = State() loop = [user_code_a, user_code_b] for fn in loop: try: fn() except Exception: log_exception() This code allows user_code_b() to execute and corrupt data even though random_code() was lucky enough to be called and detect bad state early on. You may say the fix is to assert correct data before writing to the database and, yes, that would fix the problem for future executions in this instance. That's not the point, the point is that incorrect buggy code is running in production today and it's imperative to have multiple safeguards to limit its damage. For example, you wouldn't have an except-log loop in an airplane control system. Sometimes an error is just an error but sometimes an error signifies the running system itself is in a bad state. It would be nice to distinguish between the two in a consistent way across all Python event loops. Halting on any escaped exception is inconvenient, but continuing after any escape exception is dangerous. Rian

2 1

Consistent programming error handling idiom
by Rian Hunter April 8, 2016

April 8, 2016

I like that in Python all errors are exceptions. This enables elegant code and provides a default error idiom for all Python code. An important distinction between exceptions arises when handling an exception in a top-level exception handler that doesn't have the context to properly handle it. In these situations some exceptions can be ignored (and hopefully logged) and some exceptions should the terminate/reset the program (not necessarily literally). I'd call exceptions that should terminate the program "programming errors” or “bugs.” It's practically impossible to gracefully recover from a programming error when encountered (you can potentially hot reload a bug fix but I digress). Examples of current exceptions that definitely represent programming errors include AssertionError, NameError and SyntaxError. In the case of AssertionError, it's very important the program terminates or the state is reset lest you run the risk of corrupting persistent data. I'd argue that other exceptions like AttributeError, ValueError, and TypeError can also represent programming errors depending on where they are caught. If one of these exceptions is caught near the top of the call stack where nothing useful can be done it's very likely a programming error. If it's caught locally where it can be predictably handled, no hard reset is necessary, e.g.: try: foo = val.bar except AttributeError: foo = 0 Contrast with the following code that never makes sense (and is why I said that NameError definitely signifies a programming error): try: foo = bar except NameError: foo = 0 Toy examples aside, this problem arises in real programs, like an extensible HTTP server: try: response = client_request_handler(global_state, connection_state, request) except Exception: response = create_500_response() # TODO: should the server be reset? # is global_state invalid? is connection_state invalid? In this case I think it’s polite to always send a “500” error (the HTTP status code for “internal server error”) but the question remains as to whether or not the server should reset its global state or close the connection. Something smells here and I don’t think Python currently has a solution to this problem. I don’t think it should be ambiguous by the time an exception is caught whether or not the program has a bug in it or whether it has simply run into an external error. I don’t know what the solution to this should be. I’ve seen proposals like a new exception hierarchy for “bad code” but I don’t think a good solution necessarily requires changes to the language. I think the solution could be as simple as a universally accepted PEP (like PEP 8) that discusses some of these issues, and related ones, and presents correct / pythonic ways of dealing with them. Maybe the answer is recommending that top-level exception handlers should only be used with extreme care and, unless you know what you’re doing, it’s best to let your program die (or affected state reset) and bias towards more fine-grained exception handling. Thoughts? Thanks for reading. Rian

2 1

Re: [Python-ideas] Dunder method to make object str-like
by Nathaniel Smith April 7, 2016

April 7, 2016

Why strings and not, say, floats or bools or lists or bytes? Is it because strings have a uniquely compelling use case, or...? (For context from the numpy side of things, since IIUC numpy's fixed-width integer types were the impetus for __index__: numpy actually has a shadow type system that has versions of all of those types above except list. It also has a general conversion/casting system with a concept of different levels of safety, so for us __index__ is like a "safe" cast to int, and __int__ is like a "unsafe", and we have versions of this for all pairs of internal types, which if you care about this in general then having a single parametrized concept of safe casting might make more sense then adding new methods one by one. I'm not aware of any particular pain points triggered by numpy's strings not being real strings, though, at least in numpy's current design.) -n On Apr 7, 2016 6:05 AM, "Chris Angelico" <rosuav(a)gmail.com> wrote: This is a spin-off from the __fspath__ discussion on python-dev, in which a few people said that a more general approach would be better. Proposal: Objects should be allowed to declare that they are "string-like" by creating a dunder method (analogously to __index__ for integers) which implies a loss-less conversion to str. This could supersede the __fspath__ "give me the string for this path" protocol, or could stand in parallel with it. Obviously str will have this dunder method, returning self. Most other core types (notably 'object') will not define it. Absence of this method implies that the object cannot be treated as a string. String methods will be defined as accepting string-like objects. For instance, "hello"+foo will succeed if foo is string-like. Downside: Two string-like objects may behave unexpectedly - foo+bar will concatenate strings if either is an actual string, but if both are other string-like objects, depends on the implementation of those objects. Bikeshedding: 1) What should the dunder method be named? __str_coerce__? __stringlike__? 2) Which standard types are sufficiently string-like to be used thus? 3) Should there be a bytes equivalent? 4) Should there be a format string "coerce to str"? "{}".format(x) is equivalent to str(x), but it might be nice to be able to assert that something's stringish already. Open season! _______________________________________________ Python-ideas mailing list Python-ideas(a)python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

2 1

unpacking in index expression
by Alexander Heger April 7, 2016

April 7, 2016

I was trying to unpack an array or tuple in index expression (python 3.5.1) but this failed. I think for consistency of language use it would be desirable to allow that as well. Below an example of my attempt indexing a numpy array, x: >>> import numpy as np >>> x = np.arange(12).reshape(3,-1) >>> x[*[0,1]] x[*[0,1]] ^ SyntaxError: invalid syntax >>> x[*(0,1)] x[*(0,1)] ^ SyntaxError: invalid syntax for the case of numpy, the latter works as intended w/o unpacking >>> x[(0,1)] 1 which is equivalent to the desired behaviour of unpacking >>> x[0,1] 1 but for the list case the behaviour is different (as intended by design) >>> x[[0,1]] array([[0, 1, 2, 3], [4, 5, 6, 7]]) My proposal hence is to allow unpacking in index expressions as well. I do not have a use case for dictionary / keyword unpacking; this would look awkward anyway. -Alexander

2 1

Making pathlib paths inherit from str
by Koos Zevenhoven April 6, 2016

April 6, 2016

Hi all, Recently, I have spent quite a lot of time thinking about what should be done to improve the situation of pathlib. Last week, after all kinds of wild ideas and trying hard every night to prove to myself that duck-typing-like compatibility with other path-representing objects is sufficient, I noticed that I had failed to prove that and that I had gone around a full circle back to where I started: "why can't paths subclass ``str``?" In the "Working with Path objects: p-strings?" thread, I said I was working on a proposal. Since it's been several days already, I think i should post it here and get some feedback before going any further. Maybe I should have done that even earlier. Anyway, there are some rough edges, and I will need to add links to references etc. So, do not hesitate to give feedback or criticism, which is especially appreciated it you take the time to read through the whole thing first :). You can read a github-rendered version here: https://github.com/k7hoven/strpathlib/blob/master/proposal.rst And the raw rst is here below: Proposal: Making path objects inherit from ``str``. =================================================== Abstract -------- This proposal addresses issues that limit the usability of the object-oriented filesystem path objects, most notably those of ``pathlib``, introduced in the standard library in Python 3.4 via PEP 0428. One issue being that the path objects are not directly compatible with nearly any libraries, including the standard library. A further goal of this proposal is to provide a smooth transition into a Python with better Path handling, while keeping backwards compatiblity concerns to a minimum. The approach involves making the path classes in ``pathlib`` (and optionally also DirEntry) subclasses of ``str``, but takes further measures to avoid problems and unnecessary additions. Introduction ------------ Filesystem paths are strings that give instructions for traversing a directory tree. In Python, they have traditionally been represented as byte strings, and more recently, unicode string. However, Python now has ``pathlib`` in the standard library, which is an object-oriented library for dealing with objects specialized in representing a path and working with it. In this proposal, such objects are generally referred to as *path objects*, or sometimes, in the specific context of instances of the ``pathlib`` path classes, they are explicitly referred to as ``pathlib`` objects. In ``pathlib`` there is a hierarchy of path classes with a common base class ``PurePath``. It has a subclass ``Path`` which essentially assumes the path is intended to represent a path on the current system. However, both of these classes, when called, instantiate a subclass of the ``Windows`` or ``Posix`` flavor, which have slightly different behavior. In total, there are thus five public classes: ``PurePath``, ``PurePosixPath``, ``PureWindowsPath``, ``Path``, ``PosixPath`` and ``WindowsPath``. Since Python 3.5 and the introduction of ``os.scandir``, the family of path classes has a new member, ``DirEntry``, which is a performance-oriented path object with significant duck-typing compatibility with ``pathlib`` objects. The adoption of the different types of path objects is still quite low, which is perhaps unsurprising, because they were only introduced very recently. However, it can also be inconvenient to work with these objects, because, they usually need to be explicitly converted into strings before passing them to functions, and path strings returned by functions need to be explicitly converted into path objects. Especially the latter issue is difficult in terms of backwards compatibility of APIs. While many things were recently discussed on Python ideas regarding the future of path-like objects, this proposal has a much more limited scope, to provide first steps in the right direction. However, the last part of this proposal considers possible future directions that this may optionally lead to. Rationale --------- Filesystem paths (or comparable things like URIs) are strings of characters that represent information needed to access a file or directory (or other resource). In other words, they form a subset of strings, involving specialized functionality such as joining absolute and relative paths together, accessing different parts of the path or file name, and even accessing the resources the path points to. In Python terms, for a path ``path``, one would have ``isinstance(path, str)``. It is also clear that not all strings are paths. On the one hand, this would make an ideal case for making all path-representing objects inherit from ``str``; while Python tries not to over-emphasize object-oriented programming and inheritance, it should not try to avoid class hierarchies when they are appropriate in terms of both purity and practicality. Regarding practicality, making specialized *path objects* also instances of ``str`` would make almost any stdlib or third-party function accept path objects as path arguments, assuming that they accept any instance of ``str``. Furthermore, functions now returning instances of ``str`` to represent paths could in future versions return path objects, with only minor backwards-incompatibility worries. On the other hand, strings are a very general concept, and the Python ``str`` class provides a large variety of methods to manipulate and work with them, including ``.split()``, ``.find()``, ``.isnumeric()`` and ``.join()``. These operations may be defined just as well for a string that represents a path than for any other string. In fact, this is the status quo in Python, as the adoption of ``pathlib`` is still quite limited and paths are in most cases represented as strings (sometimes byte strings). But while the string operations are *defined* on path-representing strings, the results of these operations may not be of any use in most cases, even if in some cases, they may be. While it is not the responsibility of the programming language to prevent doing things that are not useful, it may be practical in some cases. For instance, the string method ``.find()`` could be mistaken to mean finding files on the file system, while it in fact searches for a substring. String concatenation, in turn, can be a perfectly reasonable thing to do: ``show_msg("Data saved to file: " + file_path)``. The result of the concatenation of a string and a path is not a path, but a general string. Directly concatenating two path objects together as strings, however, likely has no sensible use cases. There is prior art in subclassing the Python ``str`` type to build a path object type. Packages on PyPI (TODO: list more?) that do this include ``path.py`` and ``antipathy``. The latter also supports ``bytes``-based paths by instantiating a different class, a subclass of ``bytes``. Since these libraries have existed for several years, experience from them is available for evaluating the potential benefits and weaknesses of this proposal (as well as other aspects regarding ``pathlib``). However, this proposal goes a step further to avoid potential problems and to provide a smooth transition plan that, if desired, can be followed to painlessly move towards a Python with a clear distinction between strings and paths. An optional long-term goal that this proposal facilitates may be to gradually move away from using strings (or even their subclasses) as paths. Specification of standard library changes ----------------------------------------- Making ``pathlib`` classes subclasses of ``str`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Assuming the present class hierarchy in ``pathlib``, inheritance from ``str`` will be introduced by making the base class ``pathlib.PurePath`` a subclass of ``str``. Methods will further be overridden in ``PurePath`` as described in the following. Overriding all ``str``-specific methods ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Since most of the ``str`` methods are not of any use on paths and can be confusing, leading to undesired behavior, *most* ``str`` methods (including magic methods, but excluding methods listed below) are overridden in ``PurePath`` with methods that by default raise ``TypeError("str method '<name>' is not available for paths."``. This will help programmers to immediately notice when they are using the wrong method. The perhaps unusual practice of disabling most base-class methods can be regarded as being conservative in adding ``str`` functionality to path objects. All methods, including double-underscore methods are overridden, except for the following, which are *not* overridden: - Methods of the ``str`` or ``object`` types that are already overridden by ``PurePath`` - Methods of the ``object`` type that are not overridden by ``str`` - ``__getattribute__`` - ``__len__`` (this could be debated, but not having it might be weird for a str instance) - ``encode`` - ``startswith`` and ``endswith`` (TODO: override these with case-insensitive behavior on the windows flavor) - ``__add__`` will be overriden separately, as described in later subsections. This will allow ``open(...)`` as well as most ``os`` and ``os.path`` functionality to work immediately, although there are cases that need special handling. Later, if shown to be desirable, some additional string methods may be enabled on paths. Overriding ``.__add__`` to disable adding two path objects together ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Overloading of the ``+`` operator in ``str`` will be overridden with a version which disables concatenation of two path objects together while allowing other type combinations (TODO: consider also fully disabling +): .. code:: python def __add__(self, other): if isinstance(other, PurePath): raise TypeError("Operator + for two paths is not defined; use / for joining paths.") return str.__add__(self, other) Optional enabling of string methods ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Since many APIs currently have functions or methods that return paths as strings, existing code may expect to have all string functionality available on the returned objects. While most users are unlikely to use much of the ``str`` functionality, a library function may want to explicitly allow these operations on a path object that it returns. Therefore, the overridden ``str`` methods can be enabled by setting a ``._enable_str_functionality`` method on a path object as follows: - ``pathobj._enable_str_functionality = True #`` -- Enable ``str`` methods - ``pathobj._enable_str_functionality = 'warn' #`` -- Enable ``str`` methods, but emit a ``FutureWarning`` with the message ``"str method '<name>' may be disabled on paths in future versions."`` The warning will help the API users notice that the return value is no longer a plain path. .. code:: python def <name>(self, *args, **kwargs): """Method of str, not for use with pathlib path objects.""" try: enable = self._enable_str_functionality except AttributeError: raise TypeError("str method '{}' is not available for paths." .format('<name>')) from None if enable == 'warn': warnings.warn("str method '{}' may be disabled on paths in future versions." .format('<name>'), FutureWarning, stacklevel = 2) elif enable is True: pass else: raise ValueError("_enable_str_functionality can be True or 'warn'") return getattr(str, name)(self, *args, **kwargs) New APIs, however, do not need to enable ``str`` functionality and may return default path objects. Helping interactive python tools and IDEs ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Interactive Python tools such as Jupyter are growing in popularity. When they use ``dir(...)`` to give suggestions for code completion, it is harmful to have all the disabled ``str`` methods show up in the list, even if they typically would raise exceptions. Therefore, the ``__dir__`` method should be overridden on ``PurePath`` to only show the methods that are meaningful for paths. Some tools used for code completion, such as ``rope`` and ``jedi`` may need some changes for optimal code completion. This in fact includes also the standard Python REPL, which currently does not respect ``__dir__`` in tab completion. Changes needed to other stdlib modules ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In stdlib modules other than ``pathlib``, mainly ``os``, ``ntpath`` and ``posixpath``, The stdlib functions in modules that use the methods/functionality listed below on path or file names, will be modified to explicitly convert the name ``name`` to a plain string first, e.g., using ``getattr(name, 'path', name)``, which also works for ``DirEntry`` but may return ``bytes``: - ``split`` - ``find`` - ``rfind`` - ``partition`` - ``__iter__`` - ``__getitem__`` (However, if ``DirEntry`` is not made to subclass ``str``, the idiom ``getattr(name, 'path', name)`` which is already supported in the development version, should be implemented in stdlib functions to accept not only ``str`` and path objects, but also DirEntry.) Guidelines for third-party package maintainers ---------------------------------------------- Libraries that take paths as arguments or return them ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Since all of the standard library will accept path objects as path arguments, most third-party libraries will automatically do so. However, those that directly manipulate or examine the path name using ``str`` methods may not work. Those libraries will not immediately be ``pathlib``-compatible. To achieve full ``pathlib``-compatiblity, the libraries are advised to: 1. Make sure they do not explicitly check the ``type(...)`` of arguments, but use ``isinstance(...)`` instead, if needed. 2. See if their functions use disabled ``str``/``bytes`` methods on paths that they take as arguments. If so, they should either: \* change their code to, achieve the same using ``os.path`` functions (*this is the preferred option*), or \* convert the argument first using ``name = getattr(name, 'path', name)``, which does not require importing pathlib 3. Consider, when returning a path or file name, to convert it to a path object first if a ``str``-subclassing ``pathlib`` is available. During a transition period, the attribute ``._enable_str_functionality = 'warn'`` should be set before returning the object. For an even softer, transition period it is also possible to set ``._enable_str_functionality = True``, which enables ``str`` methods with no warnings. Pathlib-compatible or near-compatible libraries ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To have the best level of compatibility, all path-like objects should preferably behave similarly to pathlib objects regarding subclassing ``str``. However, for the best level of *compatibility*, the safest options is to subclass ``str`` and *not* disable ``str`` functionality (which is already done by some known libraries). However, they may want to further disable methods of ``str`` to achieve the additional clarity that ``pathlib`` has regarding \* Having a ``.path`` attribute/property which gives a ``str`` (or ``bytes``) instance representing the path Older Python versions ~~~~~~~~~~~~~~~~~~~~~ The ``pathlib2`` module, which provides ``pathlib`` to pre-3.4 python versions, can also subclass ``str``, but it should by default have ``._enable_str_functionality = 'warn'`` or ``.enable_str_functionality = True``, because the stdlib in the older Python-versions is not compatible with paths that have ``str`` functionality disabled. Transition plans and future possibilities for long-term consideration --------------------------------------------------------------------- ``DirEntry`` ~~~~~~~~~~~~ ``DirEntry`` should also undergo a similar transition, which was, at first, part of this proposal, but it was removed to limit the scope (It could be added back, of course, if desired). Since ``DirEntry`` focuses on performance, it is important not to cause any significant performance drops. It would, however, simplify things if ``DirEntry`` did the same as ``pathlib`` regarding subclassing and disabling methods. A slight complication, however, arises from the fact that ``DirEntry`` may represent a path using ``bytes``, making the ``.path`` attribute also an instance of ``bytes`` instead of ``str``. This issue could be solved by at least two different approaches: 1. Make ``bytes``-kind DirEntry instances, interpreted as ``str`` instances, equivalent to ``os.fsdecode(direntry.path)``. 2. Instantiate a different ``DirEntry`` class for ``bytes`` paths, perhaps in a way similar to how the ``antipathy`` path library instantiates ``bPath`` when the ``bytes`` type is used. The future of plain string paths and ``os.path``? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It is possible to imagine having both ``os.path`` and ``pathlib`` coexist, as long as they co-operate well. Potentially, things like ``open("filename.txt")`` with a plain string argument will always be accepted. However, if regardless of what people use Python for, they slowly adopt path objects as the way to represent a path, the support for plain string paths may be deprecated and eventually dropped. On the one hand, to support the former situation, ``os.path`` functions can choose their return type to match the type of the arguments; with multiple different types in the arguments, ``pathlib`` might 'win' because it is already imported. On the other hand, to support the latter, all path-returning functions in the stdlib can begin to return pathlib objects, at first with ``str`` methods enabled with or without warning, and eventually, with ``str`` methods disabled. Literal syntax for paths: p-strings? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Should Python choose the *path* towards not allowing plain strings as paths, a convenient way to instantiate a path is desperately needed. As discussed in the recent python-ideas thread "Working with path objects: p-strings?", one possibility would be a new syntax like ``p"/path/to/file.ext"``, which would instantiate a path object. Another way of turning a string into a path could be to have a ``.path`` property on ``str`` objects that instantiates and returns a path object. It can be debated whether this 'Pythonic' or not. See also the next section. The ``.path`` attribute on path-like objects ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``DirEntry`` already had the ``.path`` attribute when it was introduced to the standard library in Python 3.5. It represents the absolute or relative path as a whole as a ``str`` or ``bytes`` instance. However, several people have raised the concern that the word ``path`` not referring to an actual path object may be misleading. However, if path objects are instances of str, the ``.path`` may in the future shift to mean the path object. In the case of ``pathlib`` paths, it would could be implemented as a property that returns ``self``, or during a transition phase, a path object with ``str`` functions enabled: .. code:: python @property def path(self): path = type(self)(self) path._enable_str_functionality = 'warn' return self ``DirEntry`` objects, on the other hand, could be converted to pathlib objects using the ``.path`` method. Similarly, ``str`` objects could have a similar property for conversion into a pathlib object (see previous section). Possibilities for making ``pathlib`` more lightweight ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If path objects were to become the norm in handling paths and file names, there may be a need for optimizations in terms of the speed and memory usage of path objects as well as the import time and memory footprint. Dependencies that are not always used by pathlib objects could also be imported lazily. Another base class for path-like objects ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Python already has multiple types that can represent paths-like objects. There could be one common base class for all of them, which would (at least at first) inherit from ``str``. ``DirEntry`` and ``PurePath`` would both be subclasses of this class. One would, however, need to answer the questions of what this class would be called, what it would look like, and what module would it be in (if not builtin). For now, let us call it ``PyRL`` for Python (Pyniversal?-) Resource Locator. This could also be a base class for URLs/URIs. Generalized Resource Locator addresses: a-strings? l-strings? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ... not to mention g-strings! A generalized concept may be valuable in the future, because the distinction between local and remote is getting more and more vague. As discussed in the python-ideas thread "URLs/URIs + pathlib.Path + literal syntax = ?", it is possible to quite reliably distinguish common types of URLs from filesystem paths. If this became the norm, many Python-written programs could 'magically' accept URLs as input file paths by simply calling the ``PyRL(...)``, which could be equivalent to some literal syntax for use in a scripting, testing or interactive setting, or when loading config files from fixed locations.

5 4