<div dir="ltr">Once you assign yourself a PEP number I'll do one more pass and then I expect to accept it -- the draft looks good to me!<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, May 16, 2016 at 1:00 PM, Brett Cannon <span dir="ltr"><<a href="mailto:brett@python.org" target="_blank">brett@python.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Recent discussions have been about type hints which are orthogonal to the PEP, so things have seemed to have reached a steady state.<div><br></div><div>Was there anything else that needed clarification, Guido, or are you ready to pronounce? Or did you want to wait until the language summit? Or did you want to assign a BDFL delegate?<div><div class="h5"><br><br><div class="gmail_quote"><div dir="ltr">On Fri, 13 May 2016 at 11:37 Brett Cannon <<a href="mailto:brett@python.org" target="_blank">brett@python.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Biggest changes since the second draft:<div><ol><li>Resolve __fspath__() from the type, not the instance (for Guido)</li><li>Updated the TypeError messages to say "os.PathLike object" instead of "path object" (implicitly for Steven)</li><li>TODO item to define "path-like" in the glossary (for Steven)</li><li>Various more things added to Rejected Ideas</li><li>Added Koos as a co-author (for Koos :)</li></ol><div>----------</div><div><div>PEP: NNN</div><div>Title: Adding a file system path protocol</div><div>Version: $Revision$</div><div>Last-Modified: $Date$</div><div>Author: Brett Cannon <<a href="mailto:brett@python.org" target="_blank">brett@python.org</a>>,</div><div> Koos Zevenhoven <<a href="mailto:k7hoven@gmail.com" target="_blank">k7hoven@gmail.com</a>></div><div>Status: Draft</div><div>Type: Standards Track</div><div>Content-Type: text/x-rst</div><div>Created: 11-May-2016</div><div>Post-History: 11-May-2016,</div><div> 12-May-2016,</div><div> 13-May-2016</div><div><br></div><div><br></div><div>Abstract</div><div>========</div><div><br></div><div>This PEP proposes a protocol for classes which represent a file system</div><div>path to be able to provide a ``str`` or ``bytes`` representation.</div><div>Changes to Python's standard library are also proposed to utilize this</div><div>protocol where appropriate to facilitate the use of path objects where</div><div>historically only ``str`` and/or ``bytes`` file system paths are</div><div>accepted. The goal is to facilitate the migration of users towards</div><div>rich path objects while providing an easy way to work with code</div><div>expecting ``str`` or ``bytes``.</div><div><br></div><div><br></div><div>Rationale</div><div>=========</div><div><br></div><div>Historically in Python, file system paths have been represented as</div><div>strings or bytes. This choice of representation has stemmed from C's</div><div>own decision to represent file system paths as</div><div>``const char *`` [#libc-open]_. While that is a totally serviceable</div><div>format to use for file system paths, it's not necessarily optimal. At</div><div>issue is the fact that while all file system paths can be represented</div><div>as strings or bytes, not all strings or bytes represent a file system</div><div>path. This can lead to issues where any e.g. string duck-types to a</div><div>file system path whether it actually represents a path or not.</div><div><br></div><div>To help elevate the representation of file system paths from their</div><div>representation as strings and bytes to a richer object representation,</div><div>the pathlib module [#pathlib]_ was provisionally introduced in</div><div>Python 3.4 through PEP 428. While considered by some as an improvement</div><div>over strings and bytes for file system paths, it has suffered from a</div><div>lack of adoption. Typically the key issue listed for the low adoption</div><div>rate has been the lack of support in the standard library. This lack</div><div>of support required users of pathlib to manually convert path objects</div><div>to strings by calling ``str(path)`` which many found error-prone.</div><div><br></div><div>One issue in converting path objects to strings comes from</div><div>the fact that the only generic way to get a string representation of</div><div>the path was to pass the object to ``str()``. This can pose a</div><div>problem when done blindly as nearly all Python objects have some</div><div>string representation whether they are a path or not, e.g.</div><div>``str(None)`` will give a result that</div><div>``builtins.open()`` [#builtins-open]_ will happily use to create a new</div><div>file.</div><div><br></div><div>Exacerbating this whole situation is the</div><div>``DirEntry`` object [#os-direntry]_. While path objects have a</div><div>representation that can be extracted using ``str()``, ``DirEntry``</div><div>objects expose a ``path`` attribute instead. Having no common</div><div>interface between path objects, ``DirEntry``, and any other</div><div>third-party path library has become an issue. A solution that allows</div><div>any path-representing object to declare that it is a path and a way</div><div>to extract a low-level representation that all path objects could</div><div>support is desired.</div><div><br></div><div>This PEP then proposes to introduce a new protocol to be followed by</div><div>objects which represent file system paths. Providing a protocol allows</div><div>for explicit signaling of what objects represent file system paths as</div><div>well as a way to extract a lower-level representation that can be used</div><div>with older APIs which only support strings or bytes.</div><div><br></div><div>Discussions regarding path objects that led to this PEP can be found</div><div>in multiple threads on the python-ideas mailing list archive</div><div>[#python-ideas-archive]_ for the months of March and April 2016 and on</div><div>the python-dev mailing list archives [#python-dev-archive]_ during</div><div>April 2016.</div><div><br></div><div><br></div><div>Proposal</div><div>========</div><div><br></div><div>This proposal is split into two parts. One part is the proposal of a</div><div>protocol for objects to declare and provide support for exposing a</div><div>file system path representation. The other part deals with changes to</div><div>Python's standard library to support the new protocol. These changes</div><div>will also lead to the pathlib module dropping its provisional status.</div><div><br></div><div>Protocol</div><div>--------</div><div><br></div><div>The following abstract base class defines the protocol for an object</div><div>to be considered a path object::</div><div><br></div><div> import abc</div><div> import typing as t</div><div><br></div><div><br></div><div> class PathLike(abc.ABC):</div><div><br></div><div> """Abstract base class for implementing the file system path protocol."""</div><div><br></div><div> @abc.abstractmethod</div><div> def __fspath__(self) -> t.Union[str, bytes]:</div><div> """Return the file system path representation of the object."""</div><div> raise NotImplementedError</div><div><br></div><div><br></div><div>Objects representing file system paths will implement the</div><div>``__fspath__()`` method which will return the ``str`` or ``bytes``</div><div>representation of the path. The ``str`` representation is the</div><div>preferred low-level path representation as it is human-readable and</div><div>what people historically represent paths as.</div><div><br></div><div><br></div><div>Standard library changes</div><div>------------------------</div><div><br></div><div>It is expected that most APIs in Python's standard library that</div><div>currently accept a file system path will be updated appropriately to</div><div>accept path objects (whether that requires code or simply an update</div><div>to documentation will vary). The modules mentioned below, though,</div><div>deserve specific details as they have either fundamental changes that</div><div>empower the ability to use path objects, or entail additions/removal</div><div>of APIs.</div><div><br></div><div><br></div><div>builtins</div><div>''''''''</div><div><br></div><div>``open()`` [#builtins-open]_ will be updated to accept path objects as</div><div>well as continue to accept ``str`` and ``bytes``.</div><div><br></div><div><br></div><div>os</div><div>'''</div><div><br></div><div>The ``fspath()`` function will be added with the following semantics::</div><div><br></div><div> import typing as t</div><div><br></div><div><br></div><div> def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str, bytes]:</div><div> """Return the string representation of the path.</div><div><br></div><div> If str or bytes is passed in, it is returned unchanged.</div><div> """</div><div> if isinstance(path, (str, bytes)):</div><div> return path</div><div><br></div><div> # Work from the object's type to match method resolution of other magic</div><div> # methods.</div><div> path_type = type(path)</div><div> try:</div><div> return path_type.__fspath__(path)</div><div> except AttributeError:</div><div> if hasattr(path_type, '__fspath__'):</div><div> raise</div><div><br></div><div> raise TypeError("expected str, bytes or os.PathLike object, not "</div><div> + path_type.__name__)</div><div><br></div><div>The ``os.fsencode()`` [#os-fsencode]_ and</div><div>``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept</div><div>path objects. As both functions coerce their arguments to</div><div>``bytes`` and ``str``, respectively, they will be updated to call</div><div>``__fspath__()`` if present to convert the path object to a ``str`` or</div><div>``bytes`` representation, and then perform their appropriate</div><div>coercion operations as if the return value from ``__fspath__()`` had</div><div>been the original argument to the coercion function in question.</div><div><br></div><div>The addition of ``os.fspath()``, the updates to</div><div>``os.fsencode()``/``os.fsdecode()``, and the current semantics of</div><div>``pathlib.PurePath`` provide the semantics necessary to</div><div>get the path representation one prefers. For a path object,</div><div>``pathlib.PurePath``/``Path`` can be used. To obtain the ``str`` or</div><div>``bytes`` representation without any coersion, then ``os.fspath()``</div><div>can be used. If a ``str`` is desired and the encoding of ``bytes``</div><div>should be assumed to be the default file system encoding, then</div><div>``os.fsdecode()`` should be used. If a ``bytes`` representation is</div><div>desired and any strings should be encoded using the default file</div><div>system encoding, then ``os.fsencode()`` is used. This PEP recommends</div><div>using path objects when possible and falling back to string paths as</div><div>necessary and using ``bytes`` as a last resort.</div><div><br></div><div>Another way to view this is as a hierarchy of file system path</div><div>representations (highest- to lowest-level): path → str → bytes. The</div><div>functions and classes under discussion can all accept objects on the</div><div>same level of the hierarchy, but they vary in whether they promote or</div><div>demote objects to another level. The ``pathlib.PurePath`` class can</div><div>promote a ``str`` to a path object. The ``os.fspath()`` function can</div><div>demote a path object to a ``str`` or ``bytes`` instance, depending</div><div>on what ``__fspath__()`` returns.</div><div>The ``os.fsdecode()`` function will demote a path object to</div><div>a string or promote a ``bytes`` object to a ``str``. The</div><div>``os.fsencode()`` function will demote a path or string object to</div><div>``bytes``. There is no function that provides a way to demote a path</div><div>object directly to ``bytes`` while bypassing string demotion.</div><div><br></div><div>The ``DirEntry`` object [#os-direntry]_ will gain an ``__fspath__()``</div><div>method. It will return the same value as currently found on the</div><div>``path`` attribute of ``DirEntry`` instances.</div><div><br></div><div>The Protocol_ ABC will be added to the ``os`` module under the name</div><div>``os.PathLike``.</div><div><br></div><div><br></div><div>os.path</div><div>'''''''</div><div><br></div><div>The various path-manipulation functions of ``os.path`` [#os-path]_</div><div>will be updated to accept path objects. For polymorphic functions that</div><div>accept both bytes and strings, they will be updated to simply use</div><div>``os.fspath()``.</div><div><br></div><div>During the discussions leading up to this PEP it was suggested that</div><div>``os.path`` not be updated using an "explicit is better than implicit"</div><div>argument. The thinking was that since ``__fspath__()`` is polymorphic</div><div>itself it may be better to have code working with ``os.path`` extract</div><div>the path representation from path objects explicitly. There is also</div><div>the consideration that adding support this deep into the low-level OS</div><div>APIs will lead to code magically supporting path objects without</div><div>requiring any documentation updated, leading to potential complaints</div><div>when it doesn't work, unbeknownst to the project author.</div><div><br></div><div>But it is the view of this PEP that "practicality beats purity" in</div><div>this instance. To help facilitate the transition to supporting path</div><div>objects, it is better to make the transition as easy as possible than</div><div>to worry about unexpected/undocumented duck typing support for</div><div>path objects by projects.</div><div><br></div><div>There has also been the suggestion that ``os.path`` functions could be</div><div>used in a tight loop and the overhead of checking or calling</div><div>``__fspath__()`` would be too costly. In this scenario only</div><div>path-consuming APIs would be directly updated and path-manipulating</div><div>APIs like the ones in ``os.path`` would go unmodified. This would</div><div>require library authors to update their code to support path objects</div><div>if they performed any path manipulations, but if the library code</div><div>passed the path straight through then the library wouldn't need to be</div><div>updated. It is the view of this PEP and Guido, though, that this is an</div><div>unnecessary worry and that performance will still be acceptable.</div><div><br></div><div><br></div><div>pathlib</div><div>'''''''</div><div><br></div><div>The constructor for ``pathlib.PurePath`` and ``pathlib.Path`` will be</div><div>updated to accept ``PathLike`` objects. Both ``PurePath`` and ``Path``</div><div>will continue to not accept ``bytes`` path representations, and so if</div><div>``__fspath__()`` returns ``bytes`` it will raise an exception.</div><div><br></div><div>The ``path`` attribute will be removed as this PEP makes it</div><div>redundant (it has not been included in any released version of Python</div><div>and so is not a backwards-compatibility concern).</div><div><br></div><div><br></div><div>C API</div><div>'''''</div><div><br></div><div>The C API will gain an equivalent function to ``os.fspath()``::</div><div><br></div><div> /*</div><div> Return the file system path of the object.</div><div><br></div><div> If the object is str or bytes, then allow it to pass through with</div><div> an incremented refcount. If the object defines __fspath__(), then</div><div> return the result of that method. All other types raise a TypeError.</div><div> */</div><div> PyObject *</div><div> PyOS_FSPath(PyObject *path)</div><div> {</div><div> if (PyUnicode_Check(path) || PyBytes_Check(path)) {</div><div> Py_INCREF(path);</div><div> return path;</div><div> }</div><div><br></div><div> if (PyObject_HasAttrString(path->ob_type, "__fspath__")) {</div><div> return PyObject_CallMethodObjArgs(path->ob_type, "__fspath__", path,</div><div> NULL);</div><div> }</div><div><br></div><div> return PyErr_Format(PyExc_TypeError,</div><div> "expected a str, bytes, or os.PathLike object, not %S",</div><div> path->ob_type);</div><div> }</div><div><br></div><div><br></div><div><br></div><div>Backwards compatibility</div><div>=======================</div><div><br></div><div>There are no explicit backwards-compatibility concerns. Unless an</div><div>object incidentally already defines a ``__fspath__()`` method there is</div><div>no reason to expect the pre-existing code to break or expect to have</div><div>its semantics implicitly changed.</div><div><br></div><div>Libraries wishing to support path objects and a version of Python</div><div>prior to Python 3.6 and the existence of ``os.fspath()`` can use the</div><div>idiom of</div><div>``path.__fspath__() if hasattr(path, "__fspath__") else path``.</div><div><br></div><div><br></div><div>Implementation</div><div>==============</div><div><br></div><div>This is the task list for what this PEP proposes:</div><div><br></div><div>#. Remove the ``path`` attribute from pathlib</div><div>#. Remove the provisional status of pathlib</div><div>#. Add ``os.PathLike``</div><div>#. Add ``os.fspath()``</div><div>#. Add ``PyOS_FSPath()``</div><div>#. Update ``os.fsencode()``</div><div>#. Update ``os.fsdecode()``</div><div>#. Update ``pathlib.PurePath`` and ``pathlib.Path``</div><div>#. Update ``builtins.open()``</div><div>#. Update ``os.DirEntry``</div><div>#. Update ``os.path``</div><div>#. Add a glossary entry for "path-like"</div><div><br></div><div><br></div><div>Rejected Ideas</div><div>==============</div><div><br></div><div>Other names for the protocol's method</div><div>-------------------------------------</div><div><br></div><div>Various names were proposed during discussions leading to this PEP,</div><div>including ``__path__``, ``__pathname__``, and ``__fspathname__``. In</div><div>the end people seemed to gravitate towards ``__fspath__`` for being</div><div>unambiguous without being unnecessarily long.</div><div><br></div><div><br></div><div>Separate str/bytes methods</div><div>--------------------------</div><div><br></div><div>At one point it was suggested that ``__fspath__()`` only return</div><div>strings and another method named ``__fspathb__()`` be introduced to</div><div>return bytes. The thinking is that by making ``__fspath__()`` not be</div><div>polymorphic it could make dealing with the potential string or bytes</div><div>representations easier. But the general consensus was that returning</div><div>bytes will more than likely be rare and that the various functions in</div><div>the os module are the better abstraction to promote over direct</div><div>calls to ``__fspath__()``.</div><div><br></div><div><br></div><div>Providing a ``path`` attribute</div><div>------------------------------</div><div><br></div><div>To help deal with the issue of ``pathlib.PurePath`` not inheriting</div><div>from ``str``, originally it was proposed to introduce a ``path``</div><div>attribute to mirror what ``os.DirEntry`` provides. In the end,</div><div>though, it was determined that a protocol would provide the same</div><div>result while not directly exposing an API that most people will never</div><div>need to interact with directly.</div><div><br></div><div><br></div><div>Have ``__fspath__()`` only return strings</div><div>------------------------------------------</div><div><br></div><div>Much of the discussion that led to this PEP revolved around whether</div><div>``__fspath__()`` should be polymorphic and return ``bytes`` as well as</div><div>``str`` or only return ``str``. The general sentiment for this view</div><div>was that ``bytes`` are difficult to work with due to their</div><div>inherent lack of information about their encoding and PEP 383 makes</div><div>it possible to represent all file system paths using ``str`` with the</div><div>``surrogateescape`` handler. Thus, it would be better to forcibly</div><div>promote the use of ``str`` as the low-level path representation for</div><div>high-level path objects.</div><div><br></div><div>In the end, it was decided that using ``bytes`` to represent paths is</div><div>simply not going to go away and thus they should be supported to some</div><div>degree. The hope is that people will gravitate towards path objects</div><div>like pathlib and that will move people away from operating directly</div><div>with ``bytes``.</div><div><br></div><div><br></div><div>A generic string encoding mechanism</div><div>-----------------------------------</div><div><br></div><div>At one point there was a discussion of developing a generic mechanism</div><div>to extract a string representation of an object that had semantic</div><div>meaning (``__str__()`` does not necessarily return anything of</div><div>semantic significance beyond what may be helpful for debugging). In</div><div>the end, it was deemed to lack a motivating need beyond the one this</div><div>PEP is trying to solve in a specific fashion.</div><div><br></div><div><br></div><div>Have __fspath__ be an attribute</div><div>-------------------------------</div><div><br></div><div>It was briefly considered to have ``__fspath__`` be an attribute</div><div>instead of a method. This was rejected for two reasons. One,</div><div>historically protocols have been implemented as "magic methods" and</div><div>not "magic methods and attributes". Two, there is no guarantee that</div><div>the lower-level representation of a path object will be pre-computed,</div><div>potentially misleading users that there was no expensive computation</div><div>behind the scenes in case the attribute was implemented as a property.</div><div><br></div><div>This also indirectly ties into the idea of introducing a ``path``</div><div>attribute to accomplish the same thing. This idea has an added issue,</div><div>though, of accidentally having any object with a ``path`` attribute</div><div>meet the protocol's duck typing. Introducing a new magic method for</div><div>the protocol helpfully avoids any accidental opting into the protocol.</div><div><br></div><div><br></div><div>Provide specific type hinting support</div><div>-------------------------------------</div><div><br></div><div>There was some consideration to provdinga generic ``typing.PathLike``</div><div>class which would allow for e.g. ``typing.PathLike[str]`` to specify</div><div>a type hint for a path object which returned a string representation.</div><div>While potentially beneficial, the usefulness was deemed too small to</div><div>bother adding the type hint class.</div><div><br></div><div>This also removed any desire to have a class in the ``typing`` module</div><div>which represented the union of all acceptable path-representing types</div><div>as that can be represented with</div><div>``typing.Union[str, bytes, os.PathLike]`` easily enough and the hope</div><div>is users will slowly gravitate to path objects only.</div><div><br></div><div><br></div><div>Provide ``os.fspathb()``</div><div>------------------------</div><div><br></div><div>It was suggested that to mirror the structure of e.g.</div><div>``os.getcwd()``/``os.getcwdb()``, that ``os.fspath()`` only return</div><div>``str`` and that another function named ``os.fspathb()`` be</div><div>introduced that only returned ``bytes``. This was rejected as the</div><div>purposes of the ``*b()`` functions are tied to querying the file</div><div>system where there is a need to get the raw bytes back. As this PEP</div><div>does not work directly with data on a file system (but which *may*</div><div>be), the view was taken this distinction is unnecessary. It's also</div><div>believed that the need for only bytes will not be common enough to</div><div>need to support in such a specific manner as ``os.fsencode()`` will</div><div>provide similar functionality.</div><div><br></div><div><br></div><div>Call ``__fspath__()`` off of the instance</div><div>-----------------------------------------</div><div><br></div><div>An earlier draft of this PEP had ``os.fspath()`` calling</div><div>``path.__fspath__()`` instead of ``type(path).__fspath__(path)``. The</div><div>changed to be consistent with how other magic methods in Python are</div><div>resolved.</div><div><br></div><div><br></div><div>Acknowledgements</div><div>================</div><div><br></div><div>Thanks to everyone who participated in the various discussions related</div><div>to this PEP that spanned both python-ideas and python-dev. Special</div><div>thanks to Stephen Turnbull for direct feedback on early drafts of this</div><div>PEP. More special thanks to Koos Zevenhoven and Ethan Furman for not</div><div>only feedback on early drafts of this PEP but also helping to drive</div><div>the overall discussion on this topic across the two mailing lists.</div><div><br></div><div><br></div><div>References</div><div>==========</div><div><br></div><div>.. [#python-ideas-archive] The python-ideas mailing list archive</div><div> (<a href="https://mail.python.org/pipermail/python-ideas/" target="_blank">https://mail.python.org/pipermail/python-ideas/</a>)</div><div><br></div><div>.. [#python-dev-archive] The python-dev mailing list archive</div><div> (<a href="https://mail.python.org/pipermail/python-dev/" target="_blank">https://mail.python.org/pipermail/python-dev/</a>)</div><div><br></div><div>.. [#libc-open] ``open()`` documention for the C standard library</div><div> (<a href="http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html" target="_blank">http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html</a>)</div><div><br></div><div>.. [#pathlib] The ``pathlib`` module</div><div> (<a href="https://docs.python.org/3/library/pathlib.html#module-pathlib" target="_blank">https://docs.python.org/3/library/pathlib.html#module-pathlib</a>)</div><div><br></div><div>.. [#builtins-open] The ``builtins.open()`` function</div><div> (<a href="https://docs.python.org/3/library/functions.html#open" target="_blank">https://docs.python.org/3/library/functions.html#open</a>)</div><div><br></div><div>.. [#os-fsencode] The ``os.fsencode()`` function</div><div> (<a href="https://docs.python.org/3/library/os.html#os.fsencode" target="_blank">https://docs.python.org/3/library/os.html#os.fsencode</a>)</div><div><br></div><div>.. [#os-fsdecode] The ``os.fsdecode()`` function</div><div> (<a href="https://docs.python.org/3/library/os.html#os.fsdecode" target="_blank">https://docs.python.org/3/library/os.html#os.fsdecode</a>)</div><div><br></div><div>.. [#os-direntry] The ``os.DirEntry`` class</div><div> (<a href="https://docs.python.org/3/library/os.html#os.DirEntry" target="_blank">https://docs.python.org/3/library/os.html#os.DirEntry</a>)</div><div><br></div><div>.. [#os-path] The ``os.path`` module</div><div> (<a href="https://docs.python.org/3/library/os.path.html#module-os.path" target="_blank">https://docs.python.org/3/library/os.path.html#module-os.path</a>)</div><div><br></div><div><br></div><div>Copyright</div><div>=========</div><div><br></div><div>This document has been placed in the public domain.</div><div><br></div><div><br></div><div> </div><div>..</div><div> Local Variables:</div><div> mode: indented-text</div><div> indent-tabs-mode: nil</div><div> sentence-end-double-space: t</div><div> fill-column: 70</div><div> coding: utf-8</div><div> End:</div></div><div><br></div></div></div></blockquote></div></div></div></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature">--Guido van Rossum (<a href="http://python.org/~guido" target="_blank">python.org/~guido</a>)</div>
</div>