file system path protocol PEP
**deep, calming breath** Here is the PEP for __fspath__(). The draft lives at https://github.com/brettcannon/path-pep so feel free to send me PRs for spelling mistakes, grammatical errors, etc. ----- PEP: NNN Title: Adding a file system path protocol Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon <brett@python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 11-May-2016 Post-History: 11-May-2016 Abstract ======== This PEP proposes a protocol for classes which represent a file system path to be able to provide a ``str`` or ``bytes`` representation. Changes to Python's standard library are also proposed to utilize this protocol where appropriate to facilitate the use of path objects where historically only ``str`` and/or ``bytes`` file system paths are accepted. The goal is to allow users to use the representation of a file system path that's easiest for them now as they migrate towards using path objects in the future. Rationale ========= Historically in Python, file system paths have been represented as strings or bytes. This choice of representation has stemmed from C's own decision to represent file system paths as ``const char *`` [#libc-open]_. While that is a totally serviceable format to use for file system paths, it's not necessarily optimal. At issue is the fact that while all file system paths can be represented as strings or bytes, not all strings or bytes represent a file system path. This can lead to issues where any e.g. string duck-types to a file system path whether it actually represents a path or not. To help elevate the representation of file system paths from their representation as strings and bytes to a more appropriate object representation, the pathlib module [#pathlib]_ was provisionally introduced in Python 3.4 through PEP 428. While considered by some as an improvement over strings and bytes for file system paths, it has suffered from a lack of adoption. Typically the key issue listed for the low adoption rate has been the lack of support in the standard library. This lack of support required users of pathlib to manually convert path objects to strings by calling ``str(path)`` which many found error-prone. One issue in converting path objects to strings comes from the fact that only generic way to get a string representation of the path was to pass the object to ``str()``. This can pose a problem when done blindly as nearly all Python objects have some string representation whether they are a path or not, e.g. ``str(None)`` will give a result that ``builtins.open()`` [#builtins-open]_ will happily use to create a new file. Exacerbating this whole situation is the ``DirEntry`` object [#os-direntry]_. While path objects have a representation that can be extracted using ``str()``, ``DirEntry`` objects expose a ``path`` attribute instead. Having no common interface between path objects, ``DirEntry``, and any other third-party path library had become an issue. A solution that allowed any path-representing object to declare that is was a path and a way to extract a low-level representation that all path objects could support was desired. This PEP then proposes to introduce a new protocol to be followed by objects which represent file system paths. Providing a protocol allows for clear signalling of what objects represent file system paths as well as a way to extract a lower-level representation that can be used with older APIs which only support strings or bytes. Discussions regarding path objects that led to this PEP can be found in multiple threads on the python-ideas mailing list archive [#python-ideas-archive]_ for the months of March and April 2016 and on the python-dev mailing list archives [#python-dev-archive]_ during April 2016. Proposal ======== This proposal is split into two parts. One part is the proposal of a protocol for objects to declare and provide support for exposing a file system path representation. The other part is changes to Python's standard library to support the new protocol. These changes will also have the pathlib module drop its provisional status. Protocol -------- The following abstract base class defines the protocol for an object to be considered a path object:: import abc import typing as t class PathLike(abc.ABC): """Abstract base class for implementing the file system path protocol.""" @abc.abstractmethod def __fspath__(self) -> t.Union[str, bytes]: """Return the file system path representation of the object.""" raise NotImplementedError Objects representing file system paths will implement the ``__fspath__()`` method which will return the ``str`` or ``bytes`` representation of the path. The ``str`` representation is the preferred low-level path representation as it is human-readable and what people historically represent paths as. Standard library changes ------------------------ It is expected that most APIs in Python's standard library that currently accept a file system path will be updated appropriately to accept path objects (whether that requires code or simply an update to documentation will vary). The modules mentioned below, though, deserve specific details as they have either fundamental changes that empower the ability to use path objects, or entail additions/removal of APIs. builtins '''''''' ``open()`` [#builtins-open]_ will be updated to accept path objects as well as continue to accept ``str`` and ``bytes``. os ''' The ``fspath()`` function will be added with the following semantics (although this is still an open issue: `Should os.fspath() return bytes?`_):: import typing as t def fspath(path: t.Union[PathLike, str]) -> str: """Return the string representation of the path. If a string is passed in then it is returned unchanged. """ if hasattr(path, '__fspath__'): path = path.__fspath__() if not isinstance(path, str): type_name = type(path).__name__ raise TypeError("expected a str or path object, not " + type_name) return path The ``os.fsencode()`` [#os-fsencode]_ and ``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept path objects. As both functions coerce their arguments to ``bytes`` and ``str``, respectively, they will be updated to call ``__fspath__()`` if present to convert the path object to a ``str`` or ``bytes`` representation, and then perform their appropriate coercion operations as if the return value from ``__fspath__()`` had been the original argument to the coercion function in question. The addition of ``os.fspath()``, the updates to ``os.fsencode()``/``os.fsdecode()``, and the current semantics of ``pathlib.PurePath`` provide the semantics necessary to get the path representation one prefers. For a path object, ``pathlib.PurePath``/``Path`` can be used. If ``str`` is desired and a ``bytes`` object is considered an error, then ``os.fspath()`` can be used. If a ``str`` is desired and the encoding of ``bytes`` should be assumed to be the default file system encoding, then ``os.fsdecode()`` should be used. Finally, if a ``bytes`` representation is desired and any strings should be encoded using the default file system encoding then ``os.fsencode()`` is used. This PEP recommends using path objects when possible and falling back to string paths as necessary. Therefore, no function is provided for the case of wanting a bytes representation but without any automatic encoding to help discourage the use of multiple bytes encodings on a single file system. If it is necessary to deal with an existing file system directory with entries in a non-default encoding, this can be done with low-level functions using ``str`` and the PEP 383 ``surrogateescape`` error handler, or by using ``bytes`` directly. Another way to view this is as a hierarchy of file system path representations (highest- to lowest-level): path -> str -> bytes. The functions and classes under discussion can all accept objects on the same level of the hierarchy, but they vary in whether they promote or demote objects to another level. The ``pathlib.PurePath`` class can promote a ``str`` to a path object. The ``os.fspath()`` function can demote a path object to a string, but only if ``__fspath__()`` returns a string. The ``os.fsdecode()`` function will demote a path object to a string or promote a ``bytes`` object to a ``str``. The ``os.fsencode()`` function will demote a path or string object to ``bytes``. There is no function that provides a way to demote a path object directly to ``bytes`` and not allow demoting strings. The ``DirEntry`` object [#os-direntry]_ will gain an ``__fspath__()`` method. It will return the same value as currently found on the ``path`` attribute of ``DirEntry`` instances. os.path ''''''' The various path-manipulation functions of ``os.path`` [#os-path]_ will be updated to accept path objects. For polymorphic functions that accept both bytes and strings, they will be updated to simply use code very much similar to ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This will allow for their pre-existing type-checking code to continue to function. During the discussions leading up to this PEP it was suggested that ``os.path`` not be updated using an "explicit is better than implicit" argument. The thinking was that since ``__fspath__()`` is polymorphic itself it may be better to have code working with ``os.path`` extract the path representation from path objects explicitly. There is also the consideration that adding support this deep into the low-level OS APIs will lead to code magically supporting path objects without requiring any documentation updated, leading to potential complaints when it doesn't work, unbeknownst to the project author. But it is the view of the authors that "practicality beats purity" in this instance. To help facilitate the transition to supporting path objects, it is better to make the transition as easy as possible than to worry about unexpected/undocumented duck typing support for projects. pathlib ''''''' The ``PathLike`` ABC as discussed in the Protocol_ section will be added to the pathlib module [#pathlib]_. The constructor for ``pathlib.PurePath`` and ``pathlib.Path`` will be updated to accept path objects. Both ``PurePath`` and ``Path`` will continue to not accept ``bytes`` path representations, and so if ``__fspath__()`` returns ``bytes`` it will raise an exception. The ``path`` attribute will be removed as this PEP makes its redundant (it has not been included in any released version of Python and so is not a backwards-compatibility concern). C API ''''' The C API will gain an equivalent function to ``os.fspath()`` that also allows bytes objects through:: /* Return the file system path of the object. If the object is str or bytes, then allow it to pass through with an incremented refcount. All other types raise a TypeError. */ PyObject * PyOS_RawFSPath(PyObject *path) { if (PyObject_HasAttrString(path, "__fspath__")) { path = PyObject_CallMethodObjArgs(path, "__fspath__", NULL); if (path == NULL) { return NULL; } } else { Py_INCREF(path); } if (!PyUnicode_Check(path) && !PyBytes_Check(path)) { Py_DECREF(path); return PyErr_Format(PyExc_TypeError, "expected a string, bytes, or path object, not %S", path->ob_type); } return path; } Backwards compatibility ======================= There are no explicit backwards-compatibility concerns. Unless an object incidentally already defines a ``__fspath__()`` method there is not reason to expect pre-existing code to break or expect to have their semantics implicitly changed. Libraries wishing to support path objects and a version of Python prior to Python 3.6 can use the idiom of ``path.__fspath__() if hasattr(path, '__fspath__') else path``. Open Issues =========== Should os.fspath() return bytes? -------------------------------- Some have argued that ``os.fspath()`` should be configurable so that the user can specify what types are acceptable (e.g. an argument to say that bytes are acceptable instead of strings, or both types). Others have suggested that ``os.fspath()`` match the proposed semantics of ``PyOS_RawFSPath()``. Both camps argue that use of ``os.fspath()`` will only be for a transitionary period while more libraries gain acceptance of path objects, and so being more flexible in what ``os.fspath()`` works with will help with the transition. The opponents to this -- which support the currently proposed sematnics -- worry that being so flexible with accepting bytes will lead to people not properly considering the ramifications of working with bytes, especially if bytes are transparently appearing in their code due to ``os.fspath()``. The name and location of the protocol's ABC ------------------------------------------- The name of the ABC being proposed to represent the protocol has not been discussed very much, not which module it should exist in. Names other than ``PathLike`` which are viable are ``PathABC`` and ``FSPathABC``. The name can't be ``Path`` if the ABC is put into the pathlib module. Type hint for path-like objects ------------------------------- Creating a proper type hint for APIs that accept path objects as well as strings and bytes will probably be needed. It could be as simple as defining ``typing.Path``/``typing.FSPath`` to correspond to the ABC and then having ``typing.PathLike = typing.Union[typing.Path, str, bytes]``. The type hint could also potentially be made to be generic to accept the specific low-level representation, e.g. ``typing.PathLike[str]``. In the end the type hinting solution should be properly discussed with the right type hinting experts if this is the best approach. Rejected Ideas ============== Other names for the protocol's function --------------------------------------- Various names were proposed during discussions leading to this PEP, including ``__path__``, ``__pathname__``, and ``__fspathname__``. In the end people seemed to gravitate towards ``__fspath__`` for being unambiguous without unnecessarily long. Separate str/bytes methods -------------------------- At one point it was suggested that ``__fspath__()`` only return strings and another method named ``__fspathb__()`` be introduced to return bytes. The thinking that by making ``__fspath__()`` not be polymorphic it could make dealing with the potential string or bytes representations easier. But the general consensus was that returning bytes will more than likely be rare and that the various functions in the os module are the better abstraction to be promoting over direct calls to ``__fspath__()``. Providing a path attribute -------------------------- To help deal with the issue of ``pathlib.PurePath`` not inheriting from ``str``, originally it was proposed to introduce a ``path`` attribute to mirror what ``os.DirEntry`` provides. In the end, though, it was determined that a protocol would provide the same result while not directly exposing an API that most people will never need to interact with directly. Have ``__fspath__()`` only return strings ------------------------------------------ Much of the discussion that led to this PEP revolved around whether ``__fspath__()`` should be polymorphic and return ``bytes`` as well as ``str`` instead of only ``str``. The general sentiment for this view was that ``bytes`` are difficult to work with due to their inherent lack of information about their encoding, and PEP 383 makes it possible to represent all file system paths using ``str`` with the ``surrogateescape`` handler. Thus it would be better to forcibly promote the use of ``str`` as the low-level path representation for high-level path objects. In the end it was decided that using ``bytes`` to represent paths is simply not going to go away and thus they should be supported to some degree. For those not wanting the hassle of working with ``bytes``, ``os.fspath()`` is provided. A generic string encoding mechanism ----------------------------------- At one point there was discussion of developing a generic mechanism to extract a string representation of an object that had semantic meaning (``__str__()`` does not necessarily return anything of semantic significance beyond what may be helpful for debugging). In the end it was deemed to lack a motivating need beyond the one this PEP is trying to solve in a specific fashion. Acknowledgements ================ Thanks to everyone who participated in the various discussions related to this PEP that spanned both python-ideas and python-dev. Special thanks to Koos Zevenhoven and Stephen Turnbull for direct feedback on early drafts of this PEP. References ========== .. [#python-ideas-archive] The python-ideas mailing list archive (https://mail.python.org/pipermail/python-ideas/) .. [#python-dev-archive] The python-dev mailing list archive (https://mail.python.org/pipermail/python-dev/) .. [#libc-open] ``open()`` documention for the C standard library ( http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.... ) .. [#pathlib] The ``pathlib`` module (https://docs.python.org/3/library/pathlib.html#module-pathlib) .. [#builtins-open] The ``builtins.open()`` function (https://docs.python.org/3/library/functions.html#open) .. [#os-fsencode] The ``os.fsencode()`` function (https://docs.python.org/3/library/os.html#os.fsencode) .. [#os-fsdecode] The ``os.fsdecode()`` function (https://docs.python.org/3/library/os.html#os.fsdecode) .. [#os-direntry] The ``os.DirEntry`` class (https://docs.python.org/3/library/os.html#os.DirEntry) .. [#os-path] The ``os.path`` module (https://docs.python.org/3/library/os.path.html#module-os.path) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
**another deep, calming breath** On Wed, May 11, 2016 at 7:43 PM, Brett Cannon <brett@python.org> wrote:
Open Issues ===========
Should os.fspath() return bytes? --------------------------------
In most cases, it of course should not. The section (or the title) do not represent my view on the topic, but bytes paths are not going away any time soon, so this requires considerations. Below is a copy paste of my comment in the discussion on my pull request ( https://github.com/brettcannon/path-pep/pull/2 ). It is about whether os.fspath should look like def fspath(path, *, type_constraint=str): ... This was already discussed before, receiving positive reactions. It would be an extension to what is in the draft Brett just posted. Note that the return type, however, does not depend on `type_constraint`. It fully depends on `path`. The constraint only determines when an exception is raised instead of returning a value. When using str-based paths normally, one would do os.fspath(path) Calling it with os.fspath(path, type_constraint=(str, bytes)) would turn off rejection of bytes paths (and be consistent with the rest of `os` and `os.path` where functions accept both types). But the default value for the keyword argument would be a good reminder that str-based paths should be used by default. This str-constraint is also present in the current drafted version, but it is not optional or visible in the signature. (See the diffs of my pull request for an example implementation of this.) So below is the copy-paste from the pull request discussion that I promised: """ Nobody wants to promote using `bytes` for paths without a proper reason. However, `os.fspath(path, type_contraint=str)` is already a compromise from the current `os.*` convention (of symmetrically supporting both `str` and `bytes`) towards enforcing `str`-based paths. As you know, the reason for using `os.fspath` is to switch to a lower-level representation of paths as strings. When you use it, you are already deciding that you are lower-level and want to examine or manipulate path strings manually. I think the `type_constraint=str` keyword-only argument (with the default) is a good way to remind the 'user' that `str` is the way to go unless you know what you are doing and, to `bytes` users, that `os.fspath` (by default) rejects bytes. Looking at the discussions on python-dev, one notices that the active people in the discussions were mostly in favor of exposing the bytes-supporting version in one way or the other. Some were even against bytes-rejecting versions. Most people (Python programmers), especially in the long term, should not be using `os.fspath` a lot. If they want to print a path, they should simply do so: `print(path_obj)`, or `print(f"The path is: {path_obj}")`. However, there will always be people that for whatever reason want to convert a pathlib object into a string, and without even consulting the docs, they may write `str(path_obj)`. And that's not the end of the world. It's not any worse than `str(integer)` for instance. After all, if the main point is to convert something into a generic string, the function should probably be called `str`, not `fspath`. But if the idea is to convert any path object into something that `os` and similar code internally understand, then `os.fspath` sounds fine to me. So I would not go as far as 'never use `str(path)`'. What we don't want is things like `open(str(path), 'w')`. I know you agree, because you wrote that yourself. But once this protocol is available, I see no reason why people would voluntarily wrap their path objects in `str(...)`, so I don't see that as a problem in the future (thanks to this PEP). So, especially in the long term, I don't expect `fspath` to be an everyday tool for people. Even in shorter term, people don't usually need it, because the stdlib will already support pathlib objects. Since it is not an everyday tool, I think it's ok to have that extra keyword-only argument. Even if it were an everyday tool, I can see the keyword-only argument in the signature as useful thing in a documentation sense. [As a side note, I could imagine someone convincing me that `fspath` should *always* accept both `str`- and `bytes`-based paths, because that would simplify it and align it with the rest of `os`. This would not be any worse than the status quo in terms of early failures. That would, however, still not detect accidental use of bytes paths (when forgotten to decode from the encoding it was received in).] """ -- Koos
A quick comment about sending me fixes. While I do appreciate them, sending them as a pull request is much easier for me as (a) I don't have to hunt the changes down in the text, and (b) you will see the fixes others have done already to the PEP and I then don't have to figure out what changes have not already been fixed. And honestly, reading the PEP in its rendered format on GitHub is easier IMO than in the text format unless you have something specific to respond to (and even if you do, you can copy and paste the relevant bits into an email reply). On Wed, 11 May 2016 at 09:43 Brett Cannon <brett@python.org> wrote:
**deep, calming breath**
Here is the PEP for __fspath__(). The draft lives at https://github.com/brettcannon/path-pep so feel free to send me PRs for spelling mistakes, grammatical errors, etc.
-----
PEP: NNN Title: Adding a file system path protocol Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon <brett@python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 11-May-2016 Post-History: 11-May-2016
Abstract ========
This PEP proposes a protocol for classes which represent a file system path to be able to provide a ``str`` or ``bytes`` representation. Changes to Python's standard library are also proposed to utilize this protocol where appropriate to facilitate the use of path objects where historically only ``str`` and/or ``bytes`` file system paths are accepted. The goal is to allow users to use the representation of a file system path that's easiest for them now as they migrate towards using path objects in the future.
Rationale =========
Historically in Python, file system paths have been represented as strings or bytes. This choice of representation has stemmed from C's own decision to represent file system paths as ``const char *`` [#libc-open]_. While that is a totally serviceable format to use for file system paths, it's not necessarily optimal. At issue is the fact that while all file system paths can be represented as strings or bytes, not all strings or bytes represent a file system path. This can lead to issues where any e.g. string duck-types to a file system path whether it actually represents a path or not.
To help elevate the representation of file system paths from their representation as strings and bytes to a more appropriate object representation, the pathlib module [#pathlib]_ was provisionally introduced in Python 3.4 through PEP 428. While considered by some as an improvement over strings and bytes for file system paths, it has suffered from a lack of adoption. Typically the key issue listed for the low adoption rate has been the lack of support in the standard library. This lack of support required users of pathlib to manually convert path objects to strings by calling ``str(path)`` which many found error-prone.
One issue in converting path objects to strings comes from the fact that only generic way to get a string representation of the path was to pass the object to ``str()``. This can pose a problem when done blindly as nearly all Python objects have some string representation whether they are a path or not, e.g. ``str(None)`` will give a result that ``builtins.open()`` [#builtins-open]_ will happily use to create a new file.
Exacerbating this whole situation is the ``DirEntry`` object [#os-direntry]_. While path objects have a representation that can be extracted using ``str()``, ``DirEntry`` objects expose a ``path`` attribute instead. Having no common interface between path objects, ``DirEntry``, and any other third-party path library had become an issue. A solution that allowed any path-representing object to declare that is was a path and a way to extract a low-level representation that all path objects could support was desired.
This PEP then proposes to introduce a new protocol to be followed by objects which represent file system paths. Providing a protocol allows for clear signalling of what objects represent file system paths as well as a way to extract a lower-level representation that can be used with older APIs which only support strings or bytes.
Discussions regarding path objects that led to this PEP can be found in multiple threads on the python-ideas mailing list archive [#python-ideas-archive]_ for the months of March and April 2016 and on the python-dev mailing list archives [#python-dev-archive]_ during April 2016.
Proposal ========
This proposal is split into two parts. One part is the proposal of a protocol for objects to declare and provide support for exposing a file system path representation. The other part is changes to Python's standard library to support the new protocol. These changes will also have the pathlib module drop its provisional status.
Protocol --------
The following abstract base class defines the protocol for an object to be considered a path object::
import abc import typing as t
class PathLike(abc.ABC):
"""Abstract base class for implementing the file system path protocol."""
@abc.abstractmethod def __fspath__(self) -> t.Union[str, bytes]: """Return the file system path representation of the object.""" raise NotImplementedError
Objects representing file system paths will implement the ``__fspath__()`` method which will return the ``str`` or ``bytes`` representation of the path. The ``str`` representation is the preferred low-level path representation as it is human-readable and what people historically represent paths as.
Standard library changes ------------------------
It is expected that most APIs in Python's standard library that currently accept a file system path will be updated appropriately to accept path objects (whether that requires code or simply an update to documentation will vary). The modules mentioned below, though, deserve specific details as they have either fundamental changes that empower the ability to use path objects, or entail additions/removal of APIs.
builtins ''''''''
``open()`` [#builtins-open]_ will be updated to accept path objects as well as continue to accept ``str`` and ``bytes``.
os '''
The ``fspath()`` function will be added with the following semantics (although this is still an open issue: `Should os.fspath() return bytes?`_)::
import typing as t
def fspath(path: t.Union[PathLike, str]) -> str: """Return the string representation of the path.
If a string is passed in then it is returned unchanged. """ if hasattr(path, '__fspath__'): path = path.__fspath__() if not isinstance(path, str): type_name = type(path).__name__ raise TypeError("expected a str or path object, not " + type_name) return path
The ``os.fsencode()`` [#os-fsencode]_ and ``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept path objects. As both functions coerce their arguments to ``bytes`` and ``str``, respectively, they will be updated to call ``__fspath__()`` if present to convert the path object to a ``str`` or ``bytes`` representation, and then perform their appropriate coercion operations as if the return value from ``__fspath__()`` had been the original argument to the coercion function in question.
The addition of ``os.fspath()``, the updates to ``os.fsencode()``/``os.fsdecode()``, and the current semantics of ``pathlib.PurePath`` provide the semantics necessary to get the path representation one prefers. For a path object, ``pathlib.PurePath``/``Path`` can be used. If ``str`` is desired and a ``bytes`` object is considered an error, then ``os.fspath()`` can be used. If a ``str`` is desired and the encoding of ``bytes`` should be assumed to be the default file system encoding, then ``os.fsdecode()`` should be used. Finally, if a ``bytes`` representation is desired and any strings should be encoded using the default file system encoding then ``os.fsencode()`` is used.
This PEP recommends using path objects when possible and falling back to string paths as necessary. Therefore, no function is provided for the case of wanting a bytes representation but without any automatic encoding to help discourage the use of multiple bytes encodings on a single file system. If it is necessary to deal with an existing file system directory with entries in a non-default encoding, this can be done with low-level functions using ``str`` and the PEP 383 ``surrogateescape`` error handler, or by using ``bytes`` directly.
Another way to view this is as a hierarchy of file system path representations (highest- to lowest-level): path -> str -> bytes. The functions and classes under discussion can all accept objects on the same level of the hierarchy, but they vary in whether they promote or demote objects to another level. The ``pathlib.PurePath`` class can promote a ``str`` to a path object. The ``os.fspath()`` function can demote a path object to a string, but only if ``__fspath__()`` returns a string. The ``os.fsdecode()`` function will demote a path object to a string or promote a ``bytes`` object to a ``str``. The ``os.fsencode()`` function will demote a path or string object to ``bytes``. There is no function that provides a way to demote a path object directly to ``bytes`` and not allow demoting strings.
The ``DirEntry`` object [#os-direntry]_ will gain an ``__fspath__()`` method. It will return the same value as currently found on the ``path`` attribute of ``DirEntry`` instances.
os.path '''''''
The various path-manipulation functions of ``os.path`` [#os-path]_ will be updated to accept path objects. For polymorphic functions that accept both bytes and strings, they will be updated to simply use code very much similar to ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This will allow for their pre-existing type-checking code to continue to function.
During the discussions leading up to this PEP it was suggested that ``os.path`` not be updated using an "explicit is better than implicit" argument. The thinking was that since ``__fspath__()`` is polymorphic itself it may be better to have code working with ``os.path`` extract the path representation from path objects explicitly. There is also the consideration that adding support this deep into the low-level OS APIs will lead to code magically supporting path objects without requiring any documentation updated, leading to potential complaints when it doesn't work, unbeknownst to the project author.
But it is the view of the authors that "practicality beats purity" in this instance. To help facilitate the transition to supporting path objects, it is better to make the transition as easy as possible than to worry about unexpected/undocumented duck typing support for projects.
pathlib '''''''
The ``PathLike`` ABC as discussed in the Protocol_ section will be added to the pathlib module [#pathlib]_. The constructor for ``pathlib.PurePath`` and ``pathlib.Path`` will be updated to accept path objects. Both ``PurePath`` and ``Path`` will continue to not accept ``bytes`` path representations, and so if ``__fspath__()`` returns ``bytes`` it will raise an exception.
The ``path`` attribute will be removed as this PEP makes its redundant (it has not been included in any released version of Python and so is not a backwards-compatibility concern).
C API '''''
The C API will gain an equivalent function to ``os.fspath()`` that also allows bytes objects through::
/* Return the file system path of the object.
If the object is str or bytes, then allow it to pass through with an incremented refcount. All other types raise a TypeError. */ PyObject * PyOS_RawFSPath(PyObject *path) { if (PyObject_HasAttrString(path, "__fspath__")) { path = PyObject_CallMethodObjArgs(path, "__fspath__", NULL); if (path == NULL) { return NULL; } } else { Py_INCREF(path); }
if (!PyUnicode_Check(path) && !PyBytes_Check(path)) { Py_DECREF(path); return PyErr_Format(PyExc_TypeError, "expected a string, bytes, or path object, not %S", path->ob_type); }
return path; }
Backwards compatibility =======================
There are no explicit backwards-compatibility concerns. Unless an object incidentally already defines a ``__fspath__()`` method there is not reason to expect pre-existing code to break or expect to have their semantics implicitly changed.
Libraries wishing to support path objects and a version of Python prior to Python 3.6 can use the idiom of ``path.__fspath__() if hasattr(path, '__fspath__') else path``.
Open Issues ===========
Should os.fspath() return bytes? --------------------------------
Some have argued that ``os.fspath()`` should be configurable so that the user can specify what types are acceptable (e.g. an argument to say that bytes are acceptable instead of strings, or both types). Others have suggested that ``os.fspath()`` match the proposed semantics of ``PyOS_RawFSPath()``. Both camps argue that use of ``os.fspath()`` will only be for a transitionary period while more libraries gain acceptance of path objects, and so being more flexible in what ``os.fspath()`` works with will help with the transition. The opponents to this -- which support the currently proposed sematnics -- worry that being so flexible with accepting bytes will lead to people not properly considering the ramifications of working with bytes, especially if bytes are transparently appearing in their code due to ``os.fspath()``.
The name and location of the protocol's ABC -------------------------------------------
The name of the ABC being proposed to represent the protocol has not been discussed very much, not which module it should exist in. Names other than ``PathLike`` which are viable are ``PathABC`` and ``FSPathABC``. The name can't be ``Path`` if the ABC is put into the pathlib module.
Type hint for path-like objects -------------------------------
Creating a proper type hint for APIs that accept path objects as well as strings and bytes will probably be needed. It could be as simple as defining ``typing.Path``/``typing.FSPath`` to correspond to the ABC and then having ``typing.PathLike = typing.Union[typing.Path, str, bytes]``. The type hint could also potentially be made to be generic to accept the specific low-level representation, e.g. ``typing.PathLike[str]``.
In the end the type hinting solution should be properly discussed with the right type hinting experts if this is the best approach.
Rejected Ideas ==============
Other names for the protocol's function ---------------------------------------
Various names were proposed during discussions leading to this PEP, including ``__path__``, ``__pathname__``, and ``__fspathname__``. In the end people seemed to gravitate towards ``__fspath__`` for being unambiguous without unnecessarily long.
Separate str/bytes methods --------------------------
At one point it was suggested that ``__fspath__()`` only return strings and another method named ``__fspathb__()`` be introduced to return bytes. The thinking that by making ``__fspath__()`` not be polymorphic it could make dealing with the potential string or bytes representations easier. But the general consensus was that returning bytes will more than likely be rare and that the various functions in the os module are the better abstraction to be promoting over direct calls to ``__fspath__()``.
Providing a path attribute --------------------------
To help deal with the issue of ``pathlib.PurePath`` not inheriting from ``str``, originally it was proposed to introduce a ``path`` attribute to mirror what ``os.DirEntry`` provides. In the end, though, it was determined that a protocol would provide the same result while not directly exposing an API that most people will never need to interact with directly.
Have ``__fspath__()`` only return strings ------------------------------------------
Much of the discussion that led to this PEP revolved around whether ``__fspath__()`` should be polymorphic and return ``bytes`` as well as ``str`` instead of only ``str``. The general sentiment for this view was that ``bytes`` are difficult to work with due to their inherent lack of information about their encoding, and PEP 383 makes it possible to represent all file system paths using ``str`` with the ``surrogateescape`` handler. Thus it would be better to forcibly promote the use of ``str`` as the low-level path representation for high-level path objects.
In the end it was decided that using ``bytes`` to represent paths is simply not going to go away and thus they should be supported to some degree. For those not wanting the hassle of working with ``bytes``, ``os.fspath()`` is provided.
A generic string encoding mechanism -----------------------------------
At one point there was discussion of developing a generic mechanism to extract a string representation of an object that had semantic meaning (``__str__()`` does not necessarily return anything of semantic significance beyond what may be helpful for debugging). In the end it was deemed to lack a motivating need beyond the one this PEP is trying to solve in a specific fashion.
Acknowledgements ================
Thanks to everyone who participated in the various discussions related to this PEP that spanned both python-ideas and python-dev. Special thanks to Koos Zevenhoven and Stephen Turnbull for direct feedback on early drafts of this PEP.
References ==========
.. [#python-ideas-archive] The python-ideas mailing list archive (https://mail.python.org/pipermail/python-ideas/)
.. [#python-dev-archive] The python-dev mailing list archive (https://mail.python.org/pipermail/python-dev/)
.. [#libc-open] ``open()`` documention for the C standard library ( http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.... )
.. [#pathlib] The ``pathlib`` module (https://docs.python.org/3/library/pathlib.html#module-pathlib)
.. [#builtins-open] The ``builtins.open()`` function (https://docs.python.org/3/library/functions.html#open)
.. [#os-fsencode] The ``os.fsencode()`` function (https://docs.python.org/3/library/os.html#os.fsencode)
.. [#os-fsdecode] The ``os.fsdecode()`` function (https://docs.python.org/3/library/os.html#os.fsdecode)
.. [#os-direntry] The ``os.DirEntry`` class (https://docs.python.org/3/library/os.html#os.DirEntry)
.. [#os-path] The ``os.path`` module (https://docs.python.org/3/library/os.path.html#module-os.path)
Copyright =========
This document has been placed in the public domain.
.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
On Wed, May 11, 2016 at 11:04 PM, Brett Cannon <brett@python.org> wrote:
A quick comment about sending me fixes. While I do appreciate them, sending them as a pull request is much easier for me as (a) I don't have to hunt the changes down in the text, and (b) you will see the fixes others have done already to the PEP and I then don't have to figure out what changes have not already been fixed. And honestly, reading the PEP in its rendered format on GitHub is easier IMO than in the text format unless you have something specific to respond to (and even if you do, you can copy and paste the relevant bits into an email reply).
Personally, I find it more important to settle the open issues first, so I will be looking for typos at a later stage. Besides, more typos may be added in the process. I previously left some typos uncorrected in attempt to keep the diffs of my commits cleaner. By the way, to see what additions and wordings I have suggested, the right way is to look at my PR(s). I might send more PRs to clarify open issues etc., but this will largely be based on my previous commits. -- Koos
On 11.05.16 19:43, Brett Cannon wrote:
os.path '''''''
The various path-manipulation functions of ``os.path`` [#os-path]_ will be updated to accept path objects. For polymorphic functions that accept both bytes and strings, they will be updated to simply use code very much similar to ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This will allow for their pre-existing type-checking code to continue to function.
I afraid that this will hit a performance. Some os.path functions are used in tight loops, they are hard optimized, and adding support of path protocol can have visible negative effect. I suggest first implement other changes and then look whether it is worth to add support of path protocol in os.path functions.
On 05/11/2016 01:44 PM, Serhiy Storchaka wrote:
os.path '''''''
The various path-manipulation functions of ``os.path`` [#os-path]_ will be updated to accept path objects. For polymorphic functions that accept both bytes and strings, they will be updated to simply use code very much similar to ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This will allow for their pre-existing type-checking code to continue to function.
I afraid that this will hit a performance. Some os.path functions are used in tight loops, they are hard optimized, and adding support of path protocol can have visible negative effect.
Do you have an example of os.path functions being used in a tight loop? -- ~Ethan~
On 05/11/2016 01:51 PM, Ethan Furman wrote:
On 05/11/2016 01:44 PM, Serhiy Storchaka wrote:
os.path '''''''
The various path-manipulation functions of ``os.path`` [#os-path]_ will be updated to accept path objects. For polymorphic functions that accept both bytes and strings, they will be updated to simply use code very much similar to ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This will allow for their pre-existing type-checking code to continue to function.
I afraid that this will hit a performance. Some os.path functions are used in tight loops, they are hard optimized, and adding support of path protocol can have visible negative effect.
Do you have an example of os.path functions being used in a tight loop?
Also, the C code for fspath can check types first and take the fast path if bytes/str are passed in, only falling back to the __fspath__ protocol if something else was passed in -- which should make any performance hits negligible. -- ~Ethan~
On Thu, May 12, 2016 at 12:15 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
On 05/11/2016 01:51 PM, Ethan Furman wrote:
On 05/11/2016 01:44 PM, Serhiy Storchaka wrote:
os.path '''''''
The various path-manipulation functions of ``os.path`` [#os-path]_ will be updated to accept path objects. For polymorphic functions that accept both bytes and strings, they will be updated to simply use code very much similar to ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This will allow for their pre-existing type-checking code to continue to function.
I afraid that this will hit a performance. Some os.path functions are used in tight loops, they are hard optimized, and adding support of path protocol can have visible negative effect.
Do you have an example of os.path functions being used in a tight loop?
I'd be interested in this too.
Also, the C code for fspath can check types first and take the fast path if bytes/str are passed in, only falling back to the __fspath__ protocol if something else was passed in -- which should make any performance hits negligible.
My suggestion for the *python version* already does this too (again, see my PR). This is a win-win, as it also improves error messages (as suggested by Nick in the earlier discussions). -- Koos
-- ~Ethan~
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
On 11 May 2016 at 22:51, Ethan Furman <ethan@stoneleaf.us> wrote:
On 05/11/2016 01:44 PM, Serhiy Storchaka wrote:
os.path
'''''''
The various path-manipulation functions of ``os.path`` [#os-path]_ will be updated to accept path objects. For polymorphic functions that accept both bytes and strings, they will be updated to simply use code very much similar to ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This will allow for their pre-existing type-checking code to continue to function.
I afraid that this will hit a performance. Some os.path functions are used in tight loops, they are hard optimized, and adding support of path protocol can have visible negative effect.
Do you have an example of os.path functions being used in a tight loop?
os.path.getmtime could be used in a tight loop, to sync directories with a lot of files for instance. % python3 -m timeit -s "import os.path; p = 'out'" "hasattr(p, '__fspath__'), os.path.getmtime(p)" 100000 loops, best of 3: 2.67 usec per loop % python3 -m timeit -s "import os.path; p = 'out'" "isinstance(p, (str, bytes)), os.path.getmtime(p)" 100000 loops, best of 3: 2.45 usec per loop % python3 -m timeit -s "import os.path; p = 'out'" "os.path.getmtime(p)" 100000 loops, best of 3: 2.02 usec per loop a 25% markup is a lot imo. a isinstance check prior to the hasattr might be a way to mitigate this a bit (but it doesn't help much) Granted, this example could be optimised by calling os.stat directly, which is not in os.path, but still, worth considering
On 05/11/2016 05:13 PM, Arthur Darcet wrote:
os.path.getmtime could be used in a tight loop, to sync directories with a lot of files for instance.
% python3 -m timeit -s "import os.path; p = 'out'" "hasattr(p, '__fspath__'), os.path.getmtime(p)" 100000 loops, best of 3: 2.67 usec per loop % python3 -m timeit -s "import os.path; p = 'out'" "isinstance(p, (str, bytes)), os.path.getmtime(p)" 100000 loops, best of 3: 2.45 usec per loop % python3 -m timeit -s "import os.path; p = 'out'" "os.path.getmtime(p)" 100000 loops, best of 3: 2.02 usec per loop
a 25% markup is a lot imo.
I don't think those results are very informative, since os.path.getmtime() accepts both str and bytes, so it must already have the str/bytes check. -- ~Ethan~
On 11.05.16 23:51, Ethan Furman wrote:
On 05/11/2016 01:44 PM, Serhiy Storchaka wrote:
I afraid that this will hit a performance. Some os.path functions are used in tight loops, they are hard optimized, and adding support of path protocol can have visible negative effect.
Do you have an example of os.path functions being used in a tight loop?
posixpath.realpath(), os.walk(), glob.glob() calls split() and join() for every path component. dirname() and basename() are also often called. I doesn't count functions like islink() and isfile() since they just pass the argument to underlying stat function and don't need conversion.
On Wed, 11 May 2016 at 13:45 Serhiy Storchaka <storchaka@gmail.com> wrote:
On 11.05.16 19:43, Brett Cannon wrote:
os.path '''''''
The various path-manipulation functions of ``os.path`` [#os-path]_ will be updated to accept path objects. For polymorphic functions that accept both bytes and strings, they will be updated to simply use code very much similar to ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This will allow for their pre-existing type-checking code to continue to function.
I afraid that this will hit a performance. Some os.path functions are used in tight loops, they are hard optimized, and adding support of path protocol can have visible negative effect.
As others have asked, what specific examples do you have that os.path is used in a tight loop w/o any I/O that would overwhelm the performance?
I suggest first implement other changes and then look whether it is worth to add support of path protocol in os.path functions.
I see this whole discussion breaking down into a few groups which changes what gets done upfront and what might be done farther down the line: 1. Maximum acceptance: do whatever we can to make all representation of paths just work, which means making all places working with a path in the stdlib accept path objects, str, and bytes. 2. Safely use path objects: __fspath__() is there to signal an object is a file system path and to get back a lower-level representation so people stop calling str() on everything, providing some interface signaling that someone doesn't misuse an object as a path and only changing path consumptions APIs -- e.g. open() -- and not path manipulation APIs -- e.g. os.path -- in the stdlib. 3. It ain't worth it: those that would rather just skip all of this and drop pathlib from the stdlib. Ethan and Koos are in group #1 and I'm personally in group #2 but I tried to compromise somewhat and find a middle ground in the PEP with the level of changes in the stdlib but being more restrictive with os.fspath(). If I were doing a pure group #2 PEP I would drop os.path changes and make os.fspath() do what Ethan and Koos have suggested and simply pass through without checks whatever path.__fspath__() returned if the argument wasn't str or bytes.
On 05/11/2016 03:13 PM, Brett Cannon wrote:
If [...] I would drop os.path changes and make os.fspath() do what Ethan and Koos have suggested and simply pass through without checks whatever path.__fspath__() returned if the argument wasn't str or bytes.
Not to derail the conversation too much, as I know we're all getting burned out on the topic, but that last bit is not accurate: my druthers are to have __fspath__ be able to return str /or/ bytes, and if anything else comes from the object in question an exception must be raised. Maybe a word got lost between your thoughts and your fingers -- happens to me all the time. :) -- ~Ethan~
On Thu, May 12, 2016 at 2:05 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
On 05/11/2016 03:13 PM, Brett Cannon wrote:
If [...] I would drop os.path changes and make os.fspath() do what
Ethan and Koos have suggested and simply pass through without checks
whatever path.__fspath__() returned if the argument wasn't str or bytes.
Not to derail the conversation too much, as I know we're all getting burned out on the topic, but that last bit is not accurate: my druthers are to have __fspath__ be able to return str /or/ bytes, and if anything else comes from the object in question an exception must be raised. Maybe a word got lost between your thoughts and your fingers -- happens to me all the time. :)
Yes. This would also be equivalent to my fspath(path, type_constraint=(str,bytes)). And if the compromise I mentioned about the rejecting (by default or optionally) is lifted, the keyword argument would not be needed. I might be ok with throwing away the isinstance check on the return value of __fspath__() if it has significant impact on performance in realistic cases (with DirEntry most likely, I suppose), but I doubt it. -- Koos
-- ~Ethan~
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
On Thu, May 12, 2016 at 2:49 AM, Koos Zevenhoven <k7hoven@gmail.com> wrote:
On Thu, May 12, 2016 at 2:05 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
On 05/11/2016 03:13 PM, Brett Cannon wrote:
If [...] I would drop os.path changes and make os.fspath() do what
Ethan and Koos have suggested and simply pass through without checks
whatever path.__fspath__() returned if the argument wasn't str or bytes.
Not to derail the conversation too much, as I know we're all getting burned out on the topic, but that last bit is not accurate: my druthers are to have __fspath__ be able to return str /or/ bytes, and if anything else comes from the object in question an exception must be raised. Maybe a word got lost between your thoughts and your fingers -- happens to me all the time. :)
Yes. This would also be equivalent to my fspath(path, type_constraint=(str,bytes)). And if the compromise I mentioned about the rejecting (by default or optionally) is lifted, the keyword
the rejecting -> rejecting bytes Good night. -- Koos
argument would not be needed. I might be ok with throwing away the isinstance check on the return value of __fspath__() if it has significant impact on performance in realistic cases (with DirEntry most likely, I suppose), but I doubt it.
-- Koos
-- ~Ethan~
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
On Thu, May 12, 2016 at 2:53 AM, Koos Zevenhoven <k7hoven@gmail.com> wrote:
On Thu, May 12, 2016 at 2:49 AM, Koos Zevenhoven <k7hoven@gmail.com> wrote:
On Thu, May 12, 2016 at 2:05 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
On 05/11/2016 03:13 PM, Brett Cannon wrote:
If [...] I would drop os.path changes and make os.fspath() do what
Ethan and Koos have suggested and simply pass through without checks
whatever path.__fspath__() returned if the argument wasn't str or bytes.
Not to derail the conversation too much, as I know we're all getting burned out on the topic, but that last bit is not accurate: my druthers are to have __fspath__ be able to return str /or/ bytes, and if anything else comes from the object in question an exception must be raised. Maybe a word got lost between your thoughts and your fingers -- happens to me all the time. :)
Yes. This would also be equivalent to my fspath(path, type_constraint=(str,bytes)). And if the compromise I mentioned about the rejecting (by default or optionally) is lifted, the keyword
the rejecting -> rejecting bytes
argument would not be needed. I might be ok with throwing away the isinstance check on the return value of __fspath__() if it has significant impact on performance in realistic cases (with DirEntry most likely, I suppose), but I doubt it.
-- Koos
I will send a pull request about this tomorrow. -- Koos
-- ~Ethan~
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
On Thu, May 12, 2016 at 1:13 AM, Brett Cannon <brett@python.org> wrote:
On Wed, 11 May 2016 at 13:45 Serhiy Storchaka <storchaka@gmail.com> wrote:
On 11.05.16 19:43, Brett Cannon wrote:
os.path '''''''
The various path-manipulation functions of ``os.path`` [#os-path]_ will be updated to accept path objects. For polymorphic functions that accept both bytes and strings, they will be updated to simply use code very much similar to ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This will allow for their pre-existing type-checking code to continue to function.
I afraid that this will hit a performance. Some os.path functions are used in tight loops, they are hard optimized, and adding support of path protocol can have visible negative effect.
As others have asked, what specific examples do you have that os.path is used in a tight loop w/o any I/O that would overwhelm the performance?
I suggest first implement other changes and then look whether it is worth to add support of path protocol in os.path functions.
I see this whole discussion breaking down into a few groups which changes what gets done upfront and what might be done farther down the line:
1. Maximum acceptance: do whatever we can to make all representation of paths just work, which means making all places working with a path in the stdlib accept path objects, str, and bytes.
Since you are putting me in this camp, there is at least one thing you are wrong about. I don't want all places that work with a path to accept bytes. Only those that already do so, including os/os.path. And yes, I think the stdlib should show a good example in accepting path types (especially those provided in the stdlib itself). Whether Ethan is fully in camp 1, I don't know. Not that I think he would be any closer to the other camps, though.
2. Safely use path objects: __fspath__() is there to signal an object is a file system path and to get back a lower-level representation so people stop calling str() on everything, providing some interface signaling that someone doesn't misuse an object as a path and only changing path consumptions APIs -- e.g. open() -- and not path manipulation APIs -- e.g. os.path -- in the stdlib.
3. It ain't worth it: those that would rather just skip all of this and drop pathlib from the stdlib.
Ethan and Koos are in group #1 and I'm personally in group #2 but I tried to compromise somewhat and find a middle ground in the PEP with the level of changes in the stdlib but being more restrictive with os.fspath(). If I were doing a pure group #2 PEP I would drop os.path changes and make os.fspath() do what Ethan and Koos have suggested and simply pass through without checks whatever path.__fspath__() returned if the argument wasn't str or bytes.
Related to this, based on the earlier discussions, I had the impression that you were largely in the same camp as me. In fact, I thought you had politely left some things out of the PEP draft so I could fill them in. It turned out I was wrong about that, because you didn't merge them. -- Koos
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
On Wed, 11 May 2016 at 16:08 Koos Zevenhoven <k7hoven@gmail.com> wrote:
On Thu, May 12, 2016 at 1:13 AM, Brett Cannon <brett@python.org> wrote:
On Wed, 11 May 2016 at 13:45 Serhiy Storchaka <storchaka@gmail.com>
On 11.05.16 19:43, Brett Cannon wrote:
os.path '''''''
The various path-manipulation functions of ``os.path`` [#os-path]_ will be updated to accept path objects. For polymorphic functions that accept both bytes and strings, they will be updated to simply use code very much similar to ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This will allow for their pre-existing type-checking code to continue to function.
I afraid that this will hit a performance. Some os.path functions are used in tight loops, they are hard optimized, and adding support of path protocol can have visible negative effect.
As others have asked, what specific examples do you have that os.path is used in a tight loop w/o any I/O that would overwhelm the performance?
I suggest first implement other changes and then look whether it is worth to add support of path protocol in os.path functions.
I see this whole discussion breaking down into a few groups which changes what gets done upfront and what might be done farther down the line:
1. Maximum acceptance: do whatever we can to make all representation of
wrote: paths
just work, which means making all places working with a path in the stdlib accept path objects, str, and bytes.
Since you are putting me in this camp, there is at least one thing you are wrong about. I don't want all places that work with a path to accept bytes. Only those that already do so, including os/os.path. And yes, I think the stdlib should show a good example in accepting path types (especially those provided in the stdlib itself).
That's actually what I meant. I'm not advocating widening the APIs that accept bytes at all. -Brett
Whether Ethan is fully in camp 1, I don't know. Not that I think he would be any closer to the other camps, though.
2. Safely use path objects: __fspath__() is there to signal an object is a file system path and to get back a lower-level representation so people stop calling str() on everything, providing some interface signaling that someone doesn't misuse an object as a path and only changing path consumptions APIs -- e.g. open() -- and not path manipulation APIs -- e.g. os.path -- in the stdlib.
3. It ain't worth it: those that would rather just skip all of this and drop pathlib from the stdlib.
Ethan and Koos are in group #1 and I'm personally in group #2 but I tried to compromise somewhat and find a middle ground in the PEP with the level of changes in the stdlib but being more restrictive with os.fspath(). If I were doing a pure group #2 PEP I would drop os.path changes and make os.fspath() do what Ethan and Koos have suggested and simply pass through without checks whatever path.__fspath__() returned if the argument wasn't str or bytes.
Related to this, based on the earlier discussions, I had the impression that you were largely in the same camp as me. In fact, I thought you had politely left some things out of the PEP draft so I could fill them in. It turned out I was wrong about that, because you didn't merge them.
On Wed, 11 May 2016 at 15:13 Brett Cannon <brett@python.org> wrote:
On Wed, 11 May 2016 at 13:45 Serhiy Storchaka <storchaka@gmail.com> wrote:
On 11.05.16 19:43, Brett Cannon wrote:
os.path '''''''
The various path-manipulation functions of ``os.path`` [#os-path]_ will be updated to accept path objects. For polymorphic functions that accept both bytes and strings, they will be updated to simply use code very much similar to ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This will allow for their pre-existing type-checking code to continue to function.
I afraid that this will hit a performance. Some os.path functions are used in tight loops, they are hard optimized, and adding support of path protocol can have visible negative effect.
As others have asked, what specific examples do you have that os.path is used in a tight loop w/o any I/O that would overwhelm the performance?
I suggest first implement other changes and then look whether it is worth to add support of path protocol in os.path functions.
I see this whole discussion breaking down into a few groups which changes what gets done upfront and what might be done farther down the line:
1. Maximum acceptance: do whatever we can to make all representation of paths just work, which means making all places working with a path in the stdlib accept path objects, str, and bytes. 2. Safely use path objects: __fspath__() is there to signal an object is a file system path and to get back a lower-level representation so people stop calling str() on everything, providing some interface signaling that someone doesn't misuse an object as a path and only changing path consumptions APIs -- e.g. open() -- and not path manipulation APIs -- e.g. os.path -- in the stdlib. 3. It ain't worth it: those that would rather just skip all of this and drop pathlib from the stdlib.
Ethan and Koos are in group #1 and I'm personally in group #2 but I tried to compromise somewhat and find a middle ground in the PEP with the level of changes in the stdlib but being more restrictive with os.fspath(). If I were doing a pure group #2 PEP I would drop os.path changes and make os.fspath() do what Ethan and Koos have suggested and simply pass through without checks whatever path.__fspath__() returned if the argument wasn't str or bytes.
I should mention there's also a side group who really wanted to either minimize bytes usage in all of this or not have a polymorphic os.fspath(), hence why that function is defined the way it is around only returning strings in the PEP. IOW the PEP is already an attempt at compromise and if we can't find a consensus somehow I'll update the PEP to directly reflect my personal preferences and let others write their own competing PEPs if they prefer (but I think it's in everyone's best interests if that's a last resort).
On 12.05.16 01:13, Brett Cannon wrote:
On Wed, 11 May 2016 at 13:45 Serhiy Storchaka <storchaka@gmail.com <mailto:storchaka@gmail.com>> wrote:
On 11.05.16 19:43, Brett Cannon wrote: > os.path > ''''''' > > The various path-manipulation functions of ``os.path`` [#os-path]_ > will be updated to accept path objects. For polymorphic functions that > accept both bytes and strings, they will be updated to simply use > code very much similar to > ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This > will allow for their pre-existing type-checking code to continue to > function.
I afraid that this will hit a performance. Some os.path functions are used in tight loops, they are hard optimized, and adding support of path protocol can have visible negative effect.
As others have asked, what specific examples do you have that os.path is used in a tight loop w/o any I/O that would overwhelm the performance?
Most examples does some I/O (like os.lstat()): posixpath.realpath(), os.walk(), glob.glob(). But for example os.walk() was significantly boosted with using os.scandir(), it would be sad to make it slower again. os.path is used in number of files, sometimes in loops, sometimes indirectly. It is hard to find all examples. Such functions as glob.glob() calls split() and join() for every component, but they also use string or bytes operations with paths. So they need to convert argument to str or bytes before start iteration, and always call os.path functions only with str or bytes. Additional conversion in every os.path function is redundant. I suppose most other high-level functions that manipulates paths in a loop also should convert arguments once at the start and don't need the support of path protocol in os.path functions.
I see this whole discussion breaking down into a few groups which changes what gets done upfront and what might be done farther down the line:
1. Maximum acceptance: do whatever we can to make all representation of paths just work, which means making all places working with a path in the stdlib accept path objects, str, and bytes. 2. Safely use path objects: __fspath__() is there to signal an object is a file system path and to get back a lower-level representation so people stop calling str() on everything, providing some interface signaling that someone doesn't misuse an object as a path and only changing path consumptions APIs -- e.g. open() -- and not path manipulation APIs -- e.g. os.path -- in the stdlib. 3. It ain't worth it: those that would rather just skip all of this and drop pathlib from the stdlib.
Ethan and Koos are in group #1 and I'm personally in group #2 but I tried to compromise somewhat and find a middle ground in the PEP with the level of changes in the stdlib but being more restrictive with os.fspath(). If I were doing a pure group #2 PEP I would drop os.path changes and make os.fspath() do what Ethan and Koos have suggested and simply pass through without checks whatever path.__fspath__() returned if the argument wasn't str or bytes.
I'm for adding conversions in C implemented path consuming APIs and may be in high-level path manipulation functions like os.walk(), but left low-level API of os.path, fnmatch and glob unchanged.
On 05/11/2016 09:55 PM, Serhiy Storchaka wrote:
[...] But for example os.walk() was significantly boosted with using os.scandir(), it would be sad to make it slower again.
scandir's speed improvement is due to not not throwing away data the OS was already giving us.
os.path is used in number of files, sometimes in loops, sometimes indirectly. It is hard to find all examples.
Currently, any of these functions that already take a string have to do a couple pointer comparisons to make sure they have a string; any of these functions that take both a string and a bytes have to do a couple pointer comparisons to make sure they have a string or a bytes; the only difference if this PEP is accepted is the fall-back path when those first checks fail. As an example: if os.walk is called with a Path, it converts the Path to a string (once!) and then uses that string to generate more strings and return strings. When os.walk calls os.path.join or os.path.split it will be with strings (or bytes), not with the original Path object.
Such functions as glob.glob() calls split() and join() for every component, but they also use string or bytes operations with paths. So they need to convert argument to str or bytes before start iteration, and always call os.path functions only with str or bytes.
Exactly.
Additional conversion in every os.path function is redundant.
And won't happen, since the fast-path checks will confirm that the argument is a string or bytes object.
I suppose most other high-level functions that manipulates paths in a loop also should convert arguments once at the start and don't need the support of path protocol in os.path functions.
Human's can call the os.path functions; therefore, the os.path functions need to support the __fspath__ protocol.
I'm for adding conversions in C implemented path consuming APIs and may be in high-level path manipulation functions like os.walk(), but left low-level API of os.path, fnmatch and glob unchanged.
So instead of the easy to remember "Path doesn't work with the rest of the standard library" we'll have "Path works with some APIs, but not others -- guess you better look it up" ? That is not an improvement. -- ~Ethan~
On 12.05.16 10:54, Ethan Furman wrote:
Currently, any of these functions that already take a string have to do a couple pointer comparisons to make sure they have a string; any of these functions that take both a string and a bytes have to do a couple pointer comparisons to make sure they have a string or a bytes; the only difference if this PEP is accepted is the fall-back path when those first checks fail.
This is cheap in C, but os.path functions are implemented in Python. They have to make at least one function call (os.fspath(), hasattr() or isinstance()), not counting a bytecode for retrieving arguments, resolving attributes, comparing, jumps. Currently os.path functions use tricks to avoid overheads Yet one problem is that currently many os,path functions work with duck-typed strings (e.g. UserString). Using os.fspath() likely limit supported types to str, bytes and types that support the path protocol.
On Thu, May 12, 2016 at 11:14 AM, Serhiy Storchaka <storchaka@gmail.com> wrote:
This is cheap in C, but os.path functions are implemented in Python. They have to make at least one function call (os.fspath(), hasattr() or isinstance()), not counting a bytecode for retrieving arguments, resolving attributes, comparing, jumps. Currently os.path functions use tricks to avoid overheads
I suppose a C-implemented version of fspath *called from python* might be the fastest option at least in some cases. After all, a function call (isinstance or hasattr) is likely anyway, unless of course `try: path.__fspath__` is used.
Yet one problem is that currently many os,path functions work with duck-typed strings (e.g. UserString). Using os.fspath() likely limit supported types to str, bytes and types that support the path protocol.
Something like path = path.__fspath__() if hasattr(path, '__fspath__') else path as currently in the PEP, would not have this problem. However, I wonder whether such duck string paths actually exist (although it does remind me of my earlier experiments for solving the pathlib compatibility problem ;-). -- Koos P.S: I think it's great that you are concerned about the performance, and I find it important. However, this does feel like premature optimization to me at this point. We should first decide what functions should support the protocol, and after that, find the performance concerns and fix them. I have the feeling that the cases where this would be a performance bottleneck are quite rare, but if they are not, they may well be fixable.
On 12.05.2016 00:13, Brett Cannon wrote:
I see this whole discussion breaking down into a few groups which changes what gets done upfront and what might be done farther down the line:
1. Maximum acceptance: do whatever we can to make all representation of paths just work, which means making all places working with a path in the stdlib accept path objects, str, and bytes. 2. Safely use path objects: __fspath__() is there to signal an object is a file system path and to get back a lower-level representation so people stop calling str() on everything, providing some interface signaling that someone doesn't misuse an object as a path and only changing path consumptions APIs -- e.g. open() -- and not path manipulation APIs -- e.g. os.path -- in the stdlib. 3. It ain't worth it: those that would rather just skip all of this and drop pathlib from the stdlib.
Sorry for being picky here. I think the last group needs to be split up: 3. It ain't worth it: those that would rather just skip all of this 4. drop pathlib from the stdlib. I put myself into camp 3, mostly because I don't consider the "wallet garden problem" a problem at all and I realized that our past issues with pathlib resulted from missing features in pathlib not in the rest of the stdlib. Sven
On May 11 2016, Brett Cannon <brett@python.org> wrote:
This PEP proposes a protocol for classes which represent a file system path to be able to provide a ``str`` or ``bytes`` representation. [...]
As I said before, to me this seems like a lot of effort for a very specific use-case. So let me put forward two hypothetical scenarios to better understand your position: - A new module for URL handling is added to the standard library (or urllib is suitably extended). There is a proposal to add a new protocol that allows classes to provide a ``str`` or ``bytes`` representation of URLs. - A new (third-party) library for natural language processing arises that exposes a specific class for representing audio data. Existing language processing code just uses bytes objects. To ease transition and interoperability, it is proposed to add a new protocol for classes that represend audio data to provide a bytes representation. Do you think you would you be in favor of adding these protocols to the stdlib/languange reference as well? If not, what's the crucial difference to file system paths? Thanks, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.«
On Thu, May 12, 2016 at 12:28 AM, Nikolaus Rath <Nikolaus@rath.org> wrote:
On May 11 2016, Brett Cannon <brett@python.org> wrote:
This PEP proposes a protocol for classes which represent a file system path to be able to provide a ``str`` or ``bytes`` representation. [...]
As I said before, to me this seems like a lot of effort for a very specific use-case. So let me put forward two hypothetical scenarios to better understand your position:
I think you are touching important points.
- A new module for URL handling is added to the standard library (or urllib is suitably extended). There is a proposal to add a new protocol that allows classes to provide a ``str`` or ``bytes`` representation of URLs.
This reminds me of the thread I recently started on python-ideas about extending the concept of paths to URLs. I don't know if you are referring to something like that or not. Anyway, it would be important to know whether the str or bytes representation is to be used a file system path or an URL, so that would need to be a separate protocol. But everyone would have the experience from these discussions, so hopefully less discussion then :). (By the way, this is one reason to have 'fs' in the name of the __fspath__ method, although I have not mentioned it before.)
- A new (third-party) library for natural language processing arises that exposes a specific class for representing audio data. Existing language processing code just uses bytes objects. To ease transition and interoperability, it is proposed to add a new protocol for classes that represend audio data to provide a bytes representation.
Do you think you would you be in favor of adding these protocols to the stdlib/languange reference as well? If not, what's the crucial difference to file system paths?
File system paths are very fundamental, and will probably be used in context of your natural language example too. -- Koos
Thanks, -Nikolaus
On 05/11/2016 02:28 PM, Nikolaus Rath wrote:
On May 11 2016, Brett Cannon wrote:
This PEP proposes a protocol for classes which represent a file system path to be able to provide a ``str`` or ``bytes`` representation. [...]
As I said before, to me this seems like a lot of effort for a very specific use-case. So let me put forward two hypothetical scenarios to better understand your position:
- A new module for URL handling is added to the standard library (or urllib is suitably extended). There is a proposal to add a new protocol that allows classes to provide a ``str`` or ``bytes`` representation of URLs.
- A new (third-party) library for natural language processing arises that exposes a specific class for representing audio data. Existing language processing code just uses bytes objects. To ease transition and interoperability, it is proposed to add a new protocol for classes that represend audio data to provide a bytes representation.
Do you think you would you be in favor of adding these protocols to the stdlib/languange reference as well? If not, what's the crucial difference to file system paths?
I think a crucial reason for this work is to unify the stdlib: we currently have four (?) different things that can be or represent a file-system path: - str - bytes - DirEntry - Path Half of those objects don't work well with the rest of the standard library. As for your second example, the protocol already exists: it's called __bytes__. ;) -- ~Ethan~
On Wed, 11 May 2016 at 14:29 Nikolaus Rath <Nikolaus@rath.org> wrote:
On May 11 2016, Brett Cannon <brett@python.org> wrote:
This PEP proposes a protocol for classes which represent a file system path to be able to provide a ``str`` or ``bytes`` representation. [...]
As I said before, to me this seems like a lot of effort for a very specific use-case. So let me put forward two hypothetical scenarios to better understand your position:
- A new module for URL handling is added to the standard library (or urllib is suitably extended). There is a proposal to add a new protocol that allows classes to provide a ``str`` or ``bytes`` representation of URLs.
- A new (third-party) library for natural language processing arises that exposes a specific class for representing audio data. Existing language processing code just uses bytes objects. To ease transition and interoperability, it is proposed to add a new protocol for classes that represend audio data to provide a bytes representation.
Do you think you would you be in favor of adding these protocols to the stdlib/languange reference as well?
Maybe for URLs, not for audio data (at least not in the stdlib; community can do what they want).
If not, what's the crucial difference to file system paths?
Nearly everyone uses file system paths on a regular basis, less so than URLs but still a good amount of people. Very few people work with audio data.
On 11.05.2016 23:57, Brett Cannon wrote:
On Wed, 11 May 2016 at 14:29 Nikolaus Rath <Nikolaus@rath.org <mailto:Nikolaus@rath.org>> wrote:
On May 11 2016, Brett Cannon <brett@python.org <mailto:brett@python.org>> wrote: > This PEP proposes a protocol for classes which represent a file system > path to be able to provide a ``str`` or ``bytes`` representation. [...]
As I said before, to me this seems like a lot of effort for a very specific use-case.
Exactly. Especially when considering what else can be done to improve the situation considerably.
So let me put forward two hypothetical scenarios to better understand your position:
- A new module for URL handling is added to the standard library (or urllib is suitably extended). There is a proposal to add a new protocol that allows classes to provide a ``str`` or ``bytes`` representation of URLs.
- A new (third-party) library for natural language processing arises that exposes a specific class for representing audio data. Existing language processing code just uses bytes objects. To ease transition and interoperability, it is proposed to add a new protocol for classes that represend audio data to provide a bytes representation.
You can even add the timedelta-to-seconds protocol that somebody thought would be good idea: https://mail.python.org/pipermail/python-dev/2016-April/144018.html https://mail.python.org/pipermail/python-ideas/2016-May/040226.html The generalization is straight-forward and a result of this discussion. If it works and is a good idea for pathlib, then there's absolutely no reason not to do this for the datetime lib and other rich-object libs. Same goes the other way round. Question still is: is it a good idea? Maybe, it will become a successful pattern. Maybe not.
Do you think you would you be in favor of adding these protocols to the stdlib/languange reference as well?
Maybe for URLs, not for audio data (at least not in the stdlib; community can do what they want).
If not, what's the crucial difference to file system paths?
Nearly everyone uses file system paths on a regular basis, less so than URLs but still a good amount of people. Very few people work with audio data.
Amount of usage should be taken into account of course. However, question remains if that suffices as a justification for the effort. Best, Sven
Thanks Brett for your hard work. My comments below: On 11.05.2016 18:43, Brett Cannon wrote:
Rationale =========
Historically in Python, file system paths have been represented as strings or bytes. This choice of representation has stemmed from C's own decision to represent file system paths as ``const char *`` [#libc-open]_. While that is a totally serviceable format to use for file system paths, it's not necessarily optimal. At issue is the fact that while all file system paths can be represented as strings or bytes, not all strings or bytes represent a file system path.
I can remember this argument being made during the discussion. I am not sure if that 100% correct as soon as we talk about PurePaths.
This can lead to issues where any e.g. string duck-types to a file system path whether it actually represents a path or not.
To help elevate the representation of file system paths from their representation as strings and bytes to a more appropriate object representation, the pathlib module [#pathlib]_ was provisionally introduced in Python 3.4 through PEP 428. While considered by some as an improvement over strings and bytes for file system paths, it has suffered from a lack of adoption. Typically the key issue listed for the low adoption rate has been the lack of support in the standard library. This lack of support required users of pathlib to manually convert path objects to strings by calling ``str(path)`` which many found error-prone.
One issue in converting path objects to strings comes from the fact that only generic way to get a string representation of the path was to pass the object to ``str()``. This can pose a problem when done blindly as nearly all Python objects have some string representation whether they are a path or not, e.g. ``str(None)`` will give a result that ``builtins.open()`` [#builtins-open]_ will happily use to create a new file.
Exacerbating this whole situation is the ``DirEntry`` object [#os-direntry]_. While path objects have a representation that can be extracted using ``str()``, ``DirEntry`` objects expose a ``path`` attribute instead. Having no common interface between path objects, ``DirEntry``, and any other third-party path library had become an issue. A solution that allowed any path-representing object to declare that is was a path and a way to extract a low-level representation that all path objects could support was desired.
I think the "Rationale" section ignores the fact the Path also supports the .path attribute now. Which indeed defines a common interface between path objects.
[...]
Proposal ========
This proposal is split into two parts. One part is the proposal of a protocol for objects to declare and provide support for exposing a file system path representation.
https://docs.python.org/3/whatsnew/changelog.html says: "Add ‘path’ attribute to pathlib.Path objects, returning the same as str(), to make it more similar to DirEntry. Library code can now write getattr(p, ‘path’, p) to get the path as a string from a Path, a DirEntry, or a plain string. This is essentially a small one-off protocol." So, in order to promote the "small one-off protocol" to a more broader protocol, this PEP proposes a simple rename of .path to .__fspath__, is that correct? The only issue I see with it is that it requires another function (os.fspath) to extract the "low-level representation". .path seems far easier to me.
The other part is changes to Python's standard library to support the new protocol.
I think this could be another PEP unrelated to the first part.
These changes will also have the pathlib module drop its provisional status.
Not sure if that should be part of the PEP, maybe yes.
[...]
The remainder of the PEP unfolds as a flawless implication of the rationale and the proposed idea. Unfortunately, I don't have anything to contribute to the open issues. All solutions have their pros and cons and everything that could be said has been said. I think you need to decide. Sven
On Thu, May 12, 2016 at 11:31 AM, Sven R. Kunze <srkunze@mail.de> wrote:
On 11.05.2016 18:43, Brett Cannon wrote:
Rationale =========
Historically in Python, file system paths have been represented as strings or bytes. This choice of representation has stemmed from C's own decision to represent file system paths as ``const char *`` [#libc-open]_. While that is a totally serviceable format to use for file system paths, it's not necessarily optimal. At issue is the fact that while all file system paths can be represented as strings or bytes, not all strings or bytes represent a file system path.
I can remember this argument being made during the discussion. I am not sure if that 100% correct as soon as we talk about PurePaths.
I had suggested an alternative wording for this (see my commit on the work on Rationale).
Proposal ========
This proposal is split into two parts. One part is the proposal of a protocol for objects to declare and provide support for exposing a file system path representation.
https://docs.python.org/3/whatsnew/changelog.html says:
"Add ‘path’ attribute to pathlib.Path objects, returning the same as str(), to make it more similar to DirEntry. Library code can now write getattr(p, ‘path’, p) to get the path as a string from a Path, a DirEntry, or a plain string. This is essentially a small one-off protocol."
So, in order to promote the "small one-off protocol" to a more broader protocol, this PEP proposes a simple rename of .path to .__fspath__, is that correct?
Well, I have brought this up previously several times. Indeed I see this as a further development of that duck-typing compatiblity approach. However, while the .path attribute is prior art, it has not been in a release yet.
Unfortunately, I don't have anything to contribute to the open issues. All solutions have their pros and cons and everything that could be said has been said. I think you need to decide.
Surprising enough, there are new things being said all the time. But luckily there seem to be signs of convergence. -- Koos
On 05/12/2016 01:31 AM, Sven R. Kunze wrote:
I think the "Rationale" section ignores the fact the Path also supports the .path attribute now. Which indeed defines a common interface between path objects.
The version of Python that has Path.path has not been released yet. And even so, .path is not a "common interface" as neither str nor bytes have it, and they also are used as path objects. And even given all that, for smoother interoperability with the rest of the stdlib, or at least the os.* portion, those functions would still need to be upgraded to check for .path on the incoming arguments -- at which point we may as well make a protocol to properly support file system paths instead of relying on the rather generic attribute name of 'path'. -- ~Ethan~
I am glad this is finally happening. There's quite a bit of noise in the thread which I have to ignore. The two issues that I want to respond to are speed and whether os.fspath() can return bytes. - Speed: We should trust our ability to optimize the implementations where necessary. First the API issues need to be settled. - Bytes: I strongly believe that os.fspath() should be a thin wrapper around the __fspath__ protocol, like next() wraps the .__next__ protocol. It should not get into bytes vs. string politics. If your app really needs strings, call os.fsdecode(). So this is my version (unoptimized): def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]: if isinstance(p, (str, bytes)): return p try: return p.__fspath__ except AttributeError: raise TypeError(...) Other than that I think the PEP is already in fine shape. -- --Guido van Rossum (python.org/~guido)
On Thu, 12 May 2016 at 09:25 Guido van Rossum <guido@python.org> wrote:
I am glad this is finally happening. There's quite a bit of noise in the thread which I have to ignore.
Don't worry, I'm not ignoring it on your behalf. :)
The two issues that I want to respond to are speed and whether os.fspath() can return bytes.
- Speed: We should trust our ability to optimize the implementations where necessary. First the API issues need to be settled.
Added a note to the PEP to say perf isn't a worry for os.path.
- Bytes: I strongly believe that os.fspath() should be a thin wrapper around the __fspath__ protocol, like next() wraps the .__next__ protocol. It should not get into bytes vs. string politics. If your app really needs strings, call os.fsdecode(). So this is my version (unoptimized):
def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]: if isinstance(p, (str, bytes)): return p try: return p.__fspath__ except AttributeError: raise TypeError(...)
Other than that I think the PEP is already in fine shape.
Just to double-check, did you mean for __fspath__ to only be an attribute in your example, or did you leave off the `()` by accident? As of right now the PEP is proposing a method for the protocol to follow common practice of using methods and in case the representation is not always pre-computed and thus not necessarily giving the wrong impression that the attribute access is cheap. But admittedly an attribute was previously proposed and there wasn't a terribly strong argument against it beyond "we historically haven't done it that way", so I'm open to swapping to an attribute if that's your preference.
On Thu, May 12, 2016 at 10:18 AM, Brett Cannon <brett@python.org> wrote:
On Thu, 12 May 2016 at 09:25 Guido van Rossum <guido@python.org> wrote:
def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]: if isinstance(p, (str, bytes)): return p try: return p.__fspath__ except AttributeError: raise TypeError(...)
Other than that I think the PEP is already in fine shape.
- Bytes: I strongly believe that os.fspath() should be a thin wrapper around the __fspath__ protocol, like next() wraps the .__next__ protocol. It should not get into bytes vs. string politics. If your app really needs strings, call os.fsdecode(). So this is my version (unoptimized):
Just to double-check, did you mean for __fspath__ to only be an attribute in your example, or did you leave off the `()` by accident? As of right now the PEP is proposing a method for the protocol to follow common practice of using methods and in case the representation is not always pre-computed and thus not necessarily giving the wrong impression that the attribute access is cheap. But admittedly an attribute was previously proposed and there wasn't a terribly strong argument against it beyond "we historically haven't done it that way", so I'm open to swapping to an attribute if that's your preference.
Whoops. Didn't mean to change that! Yes, __fspath__ should remain a method. You can breathe again. :-) -- --Guido van Rossum (python.org/~guido)
On Thu, 12 May 2016 at 11:36 Guido van Rossum <guido@python.org> wrote:
On Thu, May 12, 2016 at 10:18 AM, Brett Cannon <brett@python.org> wrote:
On Thu, 12 May 2016 at 09:25 Guido van Rossum <guido@python.org> wrote:
def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]:
if isinstance(p, (str, bytes)): return p try: return p.__fspath__ except AttributeError: raise TypeError(...)
Other than that I think the PEP is already in fine shape.
- Bytes: I strongly believe that os.fspath() should be a thin wrapper around the __fspath__ protocol, like next() wraps the .__next__ protocol. It should not get into bytes vs. string politics. If your app really needs strings, call os.fsdecode(). So this is my version (unoptimized):
Just to double-check, did you mean for __fspath__ to only be an attribute in your example, or did you leave off the `()` by accident? As of right now the PEP is proposing a method for the protocol to follow common practice of using methods and in case the representation is not always pre-computed and thus not necessarily giving the wrong impression that the attribute access is cheap. But admittedly an attribute was previously proposed and there wasn't a terribly strong argument against it beyond "we historically haven't done it that way", so I'm open to swapping to an attribute if that's your preference.
Whoops. Didn't mean to change that! Yes, __fspath__ should remain a method. You can breathe again. :-)
That's a mechanical change so not exactly the most stressful aspect of this PEP. :) I'll add the attribute angle to the Rejected Ideas, though. Anyway, with your strong preference of how to tweak os.fspath() what specifically would you like to see discussed at this point? Assuming the os.fspath() -> bytes discussion is dealt with, the only open issues listed in the PEP are the naming and placement of the ABC and how to do type hints for all of this (thanks to the dichotomy of path objects using the protocol and path-like objects which is the union of path object, str, and bytes and the joy of trying to name all of this well).
On Thu, May 12, 2016 at 11:51 AM, Brett Cannon <brett@python.org> wrote:
Anyway, with your strong preference of how to tweak os.fspath() what specifically would you like to see discussed at this point?
Preferably nothing. :-) There's been too much discussion already.
Assuming the os.fspath() -> bytes discussion is dealt with, the only open issues listed in the PEP are the naming and placement of the ABC and how to do type hints for all of this (thanks to the dichotomy of path objects using the protocol and path-like objects which is the union of path object, str, and bytes and the joy of trying to name all of this well).
Name and placement: I think it belongs in os, and os.PathLike sounds like a fine name. (I'm surprised that os.DirEntry doesn't exist.) Typing: do you want it to be a generic class? If not, the types can be left out of the stdlib and only put in the stub (though you can show them in the PEP of course). If you want it to be generic we have more work to do. I'm out of time now but we can discuss that after 3pm today. -- --Guido van Rossum (python.org/~guido)
On Thu, 12 May 2016 at 12:03 Guido van Rossum <guido@python.org> wrote:
On Thu, May 12, 2016 at 11:51 AM, Brett Cannon <brett@python.org> wrote:
Anyway, with your strong preference of how to tweak os.fspath() what specifically would you like to see discussed at this point?
Preferably nothing. :-) There's been too much discussion already.
Works for me. :) I'll update the PEP with the new semantics for os.fspath() and send out the updated version later today.
Assuming the os.fspath() -> bytes discussion is dealt with, the only open issues listed in the PEP are the naming and placement of the ABC and how to do type hints for all of this (thanks to the dichotomy of path objects using the protocol and path-like objects which is the union of path object, str, and bytes and the joy of trying to name all of this well).
Name and placement: I think it belongs in os, and os.PathLike sounds like a fine name. (I'm surprised that os.DirEntry doesn't exist.)
SGTM. And os.DirEntry doesn't exist simply because it's posix.DirEntry since it's implemented entirely in C. We could add an alias but since it isn't constructed from scratch I don't think it's worth it.
Typing: do you want it to be a generic class? If not, the types can be left out of the stdlib and only put in the stub (though you can show them in the PEP of course).
If we aren't going to restrict what os.fspath() returns then I don't see any need to make the type generic (I mean technically a generic version might be nice for e.g. the constructor of pathlib only taking strings, but it's probably overkill). I guess my real question is whether we want to create typing.PathLike to match os.PathLike? And it sounds like you don't want to bother with a potential second type that corresponds to Union[str, bytes, PathLike].
If you want it to be generic we have more work to do. I'm out of time now but we can discuss that after 3pm today.
Sure thing.
On Thu, May 12, 2016 at 7:24 PM, Guido van Rossum <guido@python.org> wrote:
I am glad this is finally happening. There's quite a bit of noise in the thread which I have to ignore. The two issues that I want to respond to are speed and whether os.fspath() can return bytes.
- Speed: We should trust our ability to optimize the implementations where necessary. First the API issues need to be settled.
- Bytes: I strongly believe that os.fspath() should be a thin wrapper around the __fspath__ protocol, like next() wraps the .__next__ protocol. It should not get into bytes vs. string politics. If your app really needs strings, call os.fsdecode(). So this is my version (unoptimized):
:) Thank you for this. I can breathe now. Some questions remain:
def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]: if isinstance(p, (str, bytes)): return p try: return p.__fspath__ except AttributeError: raise TypeError(...)
(I know Brett already posted this question, but somehow it did not show up in my mailbox before I had written this. I'm (re)posting because there is some stuff here that is not in Brett's email ) You might be suggesting that __fspath__ should be an attribute, not a method, or did you mean something like: def fspath(p): if isinstance(p, (str, bytes)): return p try: p.__fspath__ except AttributeError: raise TypeError(...) return p.__fspath__() IMO, either is fine, I suppose. As you know, it's mostly a question of whether __fspath__ will be a property or a method (on PurePath for instance). But if you meant the former, that would change also the ABC and the protocol description. -- Koos
On 12.05.2016 18:24, Guido van Rossum wrote:
def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]: if isinstance(p, (str, bytes)): return p try: return p.__fspath__ except AttributeError: raise TypeError(...)
@Brett Would you think it makes sense to swap the str/bytes check and the __fspath__ check? I just thought of a class subclassing str/bytes and defines __fspath__. Its __fspath__ method would be ignored currently. Best, Sven
This has just been discussed very recently in this thread (and earlier too). It may make sense, but it's not among our current worries. Besides, we already added the new fspath semantics to the PEP. While I hope Brett is asleep in his time zone, I'm guessing he will agree (just saying this because you write "@Brett"). -- Koos On Fri, May 13, 2016 at 10:58 AM, Sven R. Kunze <srkunze@mail.de> wrote:
On 12.05.2016 18:24, Guido van Rossum wrote:
def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]: if isinstance(p, (str, bytes)): return p try: return p.__fspath__ except AttributeError: raise TypeError(...)
@Brett Would you think it makes sense to swap the str/bytes check and the __fspath__ check?
I just thought of a class subclassing str/bytes and defines __fspath__. Its __fspath__ method would be ignored currently.
Best, Sven
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
On 13.05.2016 10:36, Koos Zevenhoven wrote:
This has just been discussed very recently in this thread (and earlier too).
Could you point me to that? It seems I missed that part. I only found posts related to performance degradation. However, the proposed semantics will change if the checks are swapped. So, my actual question is: Is that an intended API inconsistency or a known bug supposed to be resolved later?
It may make sense, but it's not among our current worries.
It might not be yours but mine. ;) That's why I was asking.
Besides, we already added the new fspath semantics to the PEP.
While I hope Brett is asleep in his time zone, I'm guessing he will agree (just saying this because you write "@Brett").
-- Koos
On Fri, May 13, 2016 at 10:58 AM, Sven R. Kunze <srkunze@mail.de> wrote:
On 12.05.2016 18:24, Guido van Rossum wrote:
def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]: if isinstance(p, (str, bytes)): return p try: return p.__fspath__ except AttributeError: raise TypeError(...)
@Brett Would you think it makes sense to swap the str/bytes check and the __fspath__ check?
I just thought of a class subclassing str/bytes and defines __fspath__. Its __fspath__ method would be ignored currently.
Best, Sven
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
On Fri, May 13, 2016 at 12:24 PM, Sven R. Kunze <srkunze@mail.de> wrote:
On 13.05.2016 10:36, Koos Zevenhoven wrote:
This has just been discussed very recently in this thread (and earlier too).
Could you point me to that? It seems I missed that part. I only found posts related to performance degradation.
This issue is coupled with the future optimization questions.
However, the proposed semantics will change if the checks are swapped. So, my actual question is:
Is that an intended API inconsistency or a known bug supposed to be resolved later?
Taking into account the description (and the drafted type hint), which the documentation will probably reflect, the semantic effects of that are very minor or nonexistent. I do think the documentation of the protocol should say that str or bytes subclasses should not implement __fspath__. So no API inconsistency there. -- Koos
On 13.05.2016 11:48, Koos Zevenhoven wrote:
This issue is coupled with the future optimization questions.
AFAIC coupling API design to optimization is called premature optimization.
However, the proposed semantics will change if the checks are swapped. So, my actual question is:
Is that an intended API inconsistency or a known bug supposed to be resolved later?
Taking into account the description (and the drafted type hint), which the documentation will probably reflect, the semantic effects of that are very minor or nonexistent.
From your perspective. As far as I remember, one goal of this proposal was to avoid wallet gardens. During the lengthy discussion on python-ideas people brought up that some third-party libs indeed subclass from str. They are currently locked out.
I do think the documentation of the protocol should say that str or bytes subclasses should not implement __fspath__.
Indeed. Just one minor note here: str or bytes subclasses *can* implement __fspath__ and currently it will be *ignored*. Maybe that changes in the future. So, that's the reason it should not be implemented.
So no API inconsistency there.
API consistency is not defined by docs-matching-implementation but by implementation-matching-expectations. Best, Sven
On 13 May 2016 at 17:06, Sven R. Kunze <srkunze@mail.de> wrote:
As far as I remember, one goal of this proposal was to avoid wallet gardens. During the lengthy discussion on python-ideas people brought up that some third-party libs indeed subclass from str. They are currently locked out.
Hardly. Libraries that subclass from str will provide their string representation precisely by being (a subclass of) a string. So they don't need the __fspath__ protocol, and can continue to work exactly as now. This PEP merely provides the same capability for libraries that prefer *not* to subclass str. Paul
On 05/13/2016 09:06 AM, Sven R. Kunze wrote:
On 13.05.2016 11:48, Koos Zevenhoven wrote:
Sven R Kunze had previously queried:
However, the proposed semantics will change if the checks are swapped. So, my actual question is:
Is that an intended API inconsistency or a known bug supposed to be resolved later?
Taking into account the description (and the drafted type hint), which the documentation will probably reflect, the semantic effects of that are very minor or nonexistent.
From your perspective. As far as I remember, one goal of this proposal was to avoid wallet gardens. During the lengthy discussion on python-ideas people brought up that some third-party libs indeed subclass from str. They are currently locked out.
Two points: 1) What is a wallet garden? 2) If a third-party path lib subclasses either str or bytes, and the os.* and other stdlib functions accept str or bytes, how are they locked out by not having an __fspath__? Assuming, of course, that those functions use isinstance and not type (if do use type a bug should be filed against them.
Indeed. Just one minor note here: str or bytes subclasses *can* implement __fspath__ and currently it will be *ignored*.
Sure. And int subclasses can implement __index__, which will likewise be ignored.
So no API inconsistency there.
API consistency is not defined by docs-matching-implementation but by implementation-matching-expectations.
Well, probably both. ;) -- ~Ethan~
On 13 May 2016 at 17:57, Ethan Furman <ethan@stoneleaf.us> wrote:
1) What is a wallet garden?
I assumed he meant "walled garden" (for people who don't know the phrase, it refers to an area or capability that's only made accessible to "favoured" clients - think of something like Apple's App Store, where you have to be approved to publish there). Paul
Greg Ewing writes:
Paul Moore wrote:
On 13 May 2016 at 17:57, Ethan Furman <ethan@stoneleaf.us> wrote:
1) What is a wallet garden?
I assumed he meant "walled garden"
Works either way -- you'd want a wall around your wallet garden to stop people stealing your wallets.
Actually, I expect it's a wall intended to keep your clients' wallets in a garden where you can harvest them at your convenience.
On Fri, May 13, 2016 at 7:06 PM, Sven R. Kunze <srkunze@mail.de> wrote:
On 13.05.2016 11:48, Koos Zevenhoven wrote:
This issue is coupled with the future optimization questions.
AFAIC coupling API design to optimization is called premature optimization.
I suppose so, but currently I consider the API not to make guarantees about what happens if you pass in something other than specified as valid input. I'm not sure what level of precision is usual at this point, but I think this is sufficient for now. I'm sure we will pay attention to this (as we have been doing, both while optimizing prematurely and for other reasons). -- Koos
On Fri, 13 May 2016 at 02:25 Sven R. Kunze <srkunze@mail.de> wrote:
On 13.05.2016 10:36, Koos Zevenhoven wrote:
This has just been discussed very recently in this thread (and earlier too).
Could you point me to that? It seems I missed that part. I only found posts related to performance degradation.
However, the proposed semantics will change if the checks are swapped. So, my actual question is:
Is that an intended API inconsistency or a known bug supposed to be resolved later?
Purposeful change. It was what I had in my head after I posted my "groups" breakdown email and what Guido suggested as well independently. This helps alleviate any perf worries as type checks in C are pointer checks that are cheap to make compared to attribute lookup. And even if we do get the community to move on to path objects the check is still cheap enough to not worry. And finally, this is simply how we have almost always coded this kind of special-casing at the C level throughout Python and the stdlib, so it now is more inline with standard coding practices than the original proposal in the PEP.
It may make sense, but it's not among our current worries.
It might not be yours but mine. ;) That's why I was asking.
Yes, I was very much asleep. :) -Brett
Besides, we already added the new fspath semantics to the PEP.
While I hope Brett is asleep in his time zone, I'm guessing he will agree (just saying this because you write "@Brett").
-- Koos
On Fri, May 13, 2016 at 10:58 AM, Sven R. Kunze <srkunze@mail.de> wrote:
On 12.05.2016 18:24, Guido van Rossum wrote:
def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]: if isinstance(p, (str, bytes)): return p try: return p.__fspath__ except AttributeError: raise TypeError(...)
@Brett Would you think it makes sense to swap the str/bytes check and the __fspath__ check?
I just thought of a class subclassing str/bytes and defines __fspath__. Its __fspath__ method would be ignored currently.
Best, Sven
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
On 13.05.2016 17:29, Brett Cannon wrote:
Purposeful change. It was what I had in my head after I posted my "groups" breakdown email and what Guido suggested as well independently. This helps alleviate any perf worries as type checks in C are pointer checks that are cheap to make compared to attribute lookup. And even if we do get the community to move on to path objects the check is still cheap enough to not worry. And finally, this is simply how we have almost always coded this kind of special-casing at the C level throughout Python and the stdlib, so it now is more inline with standard coding practices than the original proposal in the PEP.
Maybe, we have a misunderstanding here. Does "PyUnicode_Check(path)" respect subclass hierarchies such as isinstance(path, str) does? Could not find the definition of it in the source. You said it's just a pointer comparison. To me that implies it could not respect subclasses as isinstance does. If that is the case, the C implementation differs from the Python version in semantics. I'm sorry if I misunderstood the whole implementation here. Best, Sven
On Sat, May 14, 2016 at 2:33 AM, Sven R. Kunze <srkunze@mail.de> wrote:
On 13.05.2016 17:29, Brett Cannon wrote:
Purposeful change. It was what I had in my head after I posted my "groups" breakdown email and what Guido suggested as well independently. This helps alleviate any perf worries as type checks in C are pointer checks that are cheap to make compared to attribute lookup. And even if we do get the community to move on to path objects the check is still cheap enough to not worry. And finally, this is simply how we have almost always coded this kind of special-casing at the C level throughout Python and the stdlib, so it now is more inline with standard coding practices than the original proposal in the PEP.
Maybe, we have a misunderstanding here. Does "PyUnicode_Check(path)" respect subclass hierarchies such as isinstance(path, str) does? Could not find the definition of it in the source.
You said it's just a pointer comparison. To me that implies it could not respect subclasses as isinstance does. If that is the case, the C implementation differs from the Python version in semantics. I'm sorry if I misunderstood the whole implementation here.
https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_Check https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_CheckExact "Check" accepts subclasses; "CheckExact" doesn't (it's like "type(x) is str"). The question is, which one SHOULD be being done? What should this do: class TmpPath(str): def __fspath__(self): return "/tmp/"+self x = TmpPath("foo/bar") open(x, "w") Does that respect __fspath__, or respect the fact that it's a string? ChrisA
On 13.05.2016 18:43, Chris Angelico wrote:
https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_Check
https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_CheckExact
Thanks for pointing me at this. I searched via github and found usages only: https://github.com/python/cpython/search?utf8=%E2%9C%93&q=PyUnicode_Check
"Check" accepts subclasses; "CheckExact" doesn't (it's like "type(x) is str"). The question is, which one SHOULD be being done? What should this do:
class TmpPath(str): def __fspath__(self): return "/tmp/"+self x = TmpPath("foo/bar") open(x, "w")
Does that respect __fspath__, or respect the fact that it's a string?
Fine example. Thinking naively, I would say, when somebody made an effort to write __fspath__, it should be respected. Maybe, that's just me. Not sure if that can be a source of errors, when somebody adds str as baseclass later because the team needs str functionality. At least I would expect __fspath__ to still work . Best, Sven
On 14 May 2016 at 19:36, Sven R. Kunze <srkunze@mail.de> wrote:
Fine example. Thinking naively, I would say, when somebody made an effort to write __fspath__, it should be respected. Maybe, that's just me.
Over time, we've generally moved towards treating subclasses as if they were the base class where protocols are concerned, and encouraging the use of composition if people don't actually want that "is-a" behaviour. For example, compare the int protocol to the last major type protocol we added, index:
from operator import index class MyInt(int): ... def __int__(self): ... return 42 ... def __index__(self): ... return 42 ... x = MyInt(5) x 5 int(x) 42 index(x) 5
In the case of retroactively added protocols like operator.index() and os.fspath(), where inheritance from the relevant builtin type was previously the only option, retaining that behaviour is also the only way to avoid a performance regression for those subclasses. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, May 13, 2016 at 7:43 PM, Chris Angelico <rosuav@gmail.com> wrote: [...]
"Check" accepts subclasses; "CheckExact" doesn't (it's like "type(x) is str"). The question is, which one SHOULD be being done?
Indeed it should do "Check", so that path libraries that do inherit from str will still work (see also below).
What should this do:
class TmpPath(str): def __fspath__(self): return "/tmp/"+self x = TmpPath("foo/bar") open(x, "w")
Does that respect __fspath__, or respect the fact that it's a string?
It should not respect __fspath__ because that would mean the behavior is different in older versions of Python(!). This also means that such code should never be written (except for that example you wrote ;-). So indeed, composition would be the way to go. So yes, I still think str subclasses should not implement __fspath__, and that, at the very least, the docs should say that :). -- Koos
ChrisA _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
On 12.05.2016 17:42, Ethan Furman wrote:
On 05/12/2016 01:31 AM, Sven R. Kunze wrote:
I think the "Rationale" section ignores the fact the Path also supports the .path attribute now. Which indeed defines a common interface between path objects.
The version of Python that has Path.path has not been released yet. And even so, .path is not a "common interface" as neither str nor bytes have it, and they also are used as path objects.
str and bytes will receive the __fspath__ attribute when this PEP is accepted?
And even given all that, for smoother interoperability with the rest of the stdlib, or at least the os.* portion, those functions would still need to be upgraded to check for .path on the incoming arguments -- at which point we may as well make a protocol to properly support file system paths instead of relying on the rather generic attribute name of 'path'.
Just so, if you accept changing os.* as a necessary solution. If not, keeping .path would suffice and would be much simpler. Best, Sven
On 05/12/2016 09:26 AM, Sven R. Kunze wrote:
On 12.05.2016 17:42, Ethan Furman wrote:
On 05/12/2016 01:31 AM, Sven R. Kunze wrote:
I think the "Rationale" section ignores the fact the Path also supports the .path attribute now. Which indeed defines a common interface between path objects.
The version of Python that has Path.path has not been released yet. And even so, .path is not a "common interface" as neither str nor bytes have it, and they also are used as path objects.
str and bytes will receive the __fspath__ attribute when this PEP is accepted?
No, they won't. The __fspath__ protocol will reduce the rich path object down to a str/bytes object. One could argue that a .path attribute is similar, but consider: if you are handed a random object with a .path attribute, how certain can you be that it represents a file system path? Contrariwise, how certain can you be of an object that has __fspath__? At any rate, we seem to be down to the details of os.fspath() so I don't see any reason to discuss .path any further. -- ~Ethan~
On 12.05.2016 18:56, Ethan Furman wrote:
On 05/12/2016 09:26 AM, Sven R. Kunze wrote:
str and bytes will receive the __fspath__ attribute when this PEP is accepted?
No, they won't. The __fspath__ protocol will reduce the rich path object down to a str/bytes object.
Would this make the implementation of os.fspath simpler? After all, str and bytes are to some extend path-like objects. Best, Sven
On 05/12/2016 10:21 AM, Sven R. Kunze wrote:
On 12.05.2016 18:56, Ethan Furman wrote:
On 05/12/2016 09:26 AM, Sven R. Kunze wrote:
str and bytes will receive the __fspath__ attribute when this PEP is accepted?
No, they won't. The __fspath__ protocol will reduce the rich path object down to a str/bytes object.
Would this make the implementation of os.fspath simpler?
Maybe, but a bad idea for two reasons: 1) Reducing a str to the exact same str is silly; and, more importantly 2) not every str/bytes is a path -- ~Ethan~
On 05/12/2016 05:42 PM, Ethan Furman wrote:
And even given all that, for smoother interoperability with the rest of the stdlib, or at least the os.* portion, those functions would still need to be upgraded to check for .path on the incoming arguments -- at which point we may as well make a protocol to properly support file system paths instead of relying on the rather generic attribute name of 'path'.
FWIW, this shouldn't be very difficult. Paths are already complicated enough that the parsing has been factored out into a PyArg_Parse "converter" called path_converter. path_converter isn't used everywhere in posixmodule yet, but it would probably make sense to convert everything to use it anyway. And then we would only need to implement Path object support once. It's remotely possible that converting functions to use path_converter will create slight incompatibilities (though I don't know how, this is just FUD really) so 3.6 would be a good opportunity to make this change. Optimistically, //arry/
On 12 May 2016 at 02:43, Brett Cannon <brett@python.org> wrote:
**deep, calming breath**
Here is the PEP for __fspath__(). The draft lives at https://github.com/brettcannon/path-pep so feel free to send me PRs for spelling mistakes, grammatical errors, etc.
Thanks for putting this together :)
C API '''''
The C API will gain an equivalent function to ``os.fspath()`` that also allows bytes objects through::
/* Return the file system path of the object.
If the object is str or bytes, then allow it to pass through with an incremented refcount. All other types raise a TypeError. */ PyObject * PyOS_RawFSPath(PyObject *path) { if (PyObject_HasAttrString(path, "__fspath__")) { path = PyObject_CallMethodObjArgs(path, "__fspath__", NULL); if (path == NULL) { return NULL; } } else { Py_INCREF(path); }
if (!PyUnicode_Check(path) && !PyBytes_Check(path)) { Py_DECREF(path); return PyErr_Format(PyExc_TypeError, "expected a string, bytes, or path object, not %S", path->ob_type); }
return path; }
I'd still like to see this exposed to Python code as os._raw_fspath() (with the leading underscore just meaning "this probably isn't the API you want" rather than indicating a private or unstable API), and then fspath() defined as a wrapper around it which disallows bytes as output. However, I don't have a specific use case, and it would be straightforward to add later, so the overall PEP gets a +1 from me. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, May 12, 2016 at 3:04 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I'd still like to see this exposed to Python code as os._raw_fspath() (with the leading underscore just meaning "this probably isn't the API you want" rather than indicating a private or unstable API), and then fspath() defined as a wrapper around it which disallows bytes as output.
I don't remember (should probably check) if you previously proposed implementing exactly that in C, but I indeed agree with what you write above, except that I don't like the "_raw_" prefix in the name. I would be willing to call that simply fspath though, since as mentioned before in this thread (I think by Brett and me), the reasons for rejecting bytes in fspath are really quite minor.
On 12 May 2016 at 22:40, Koos Zevenhoven <k7hoven@gmail.com> wrote:
On Thu, May 12, 2016 at 3:04 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I'd still like to see this exposed to Python code as os._raw_fspath() (with the leading underscore just meaning "this probably isn't the API you want" rather than indicating a private or unstable API), and then fspath() defined as a wrapper around it which disallows bytes as output.
I don't remember (should probably check) if you previously proposed implementing exactly that in C, but I indeed agree with what you write above, except that I don't like the "_raw_" prefix in the name. I would be willing to call that simply fspath though, since as mentioned before in this thread (I think by Brett and me), the reasons for rejecting bytes in fspath are really quite minor.
It's not unusual for me to encounter "POSIX oughtta be enough for anyone" folks that are not yet entirely convinced that bytes-are-not-text, so I'm actually in favour of making the default Python-level API str-only as a healthy nudge away from the "text-is-just-bytes-with-an-encoding!" school of thought. However, in terms of the three groups Brett articulated (maximum flexibility, encouraging cross-platform correctness, and forgetting the whole idea), I'm in both camps 1 & 2 - I work with POSIX enough that I'm entirely on board with the notion that if you're specifically modelling *POSIX* paths, then bytes-with-an-assumed-encoding is frequently a good enough representation, but also deal with other environments (like Windows, the JVM and the CLR) enough to know that that particular representation of filesystem paths breaks down the moment you expand your scope of interest beyond *nix platforms. Hence the suggestion of having os.fspath() be the group 2 guaranteed-to-only-emit-str API, with os._raw_fspath() as the lower level "I know I'm potentially being POSIX-centric here, but that's OK for my use case" group 1 interface. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, May 12, 2016 at 4:20 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
It's not unusual for me to encounter "POSIX oughtta be enough for anyone" folks that are not yet entirely convinced that bytes-are-not-text, so I'm actually in favour of making the default Python-level API str-only as a healthy nudge away from the "text-is-just-bytes-with-an-encoding!" school of thought.
This was also how I convinced myself about the default str constraint. However, I'm afraid it would be a weak weapon against using bytes paths, since the people using bytes paths would not be likely to call it, regardless of whether it supports bytes or not. The nice thing is that pathlib is str-only and *that* will push people away from bytes paths.
However, in terms of the three groups Brett articulated (maximum flexibility, encouraging cross-platform correctness, and forgetting the whole idea), I'm in both camps 1 & 2 - I work with POSIX enough that I'm entirely on board with the notion that if you're specifically modelling *POSIX* paths, then bytes-with-an-assumed-encoding is frequently a good enough representation, but also deal with other environments (like Windows, the JVM and the CLR) enough to know that that particular representation of filesystem paths breaks down the moment you expand your scope of interest beyond *nix platforms.
I also agree with parts about Brett's "camp 2". -- Koos
I would like to make just 1 comment regarding the question of accepting (or not) bytes as output of `os.fspath`. The whole point of adding `os.fspath` is to make it easier to use Path objects. This is in an effort to gain greater adoption of pathlib in libraries. Now, this is an excellent idea. However, if it were to reject bytes, that would mean that when libraries start to use pathlib, it would suddenly become harder for people that actually need bytes-support to use pathlib. Now, the claim 'if you need bytes, you should not be using pathlib` is a reasonable one. But what if I need bytes *and* a specific library (say, image handling, or a web framework, or ...). It's not up to me if that library uses pathlib or plain old os.path.join. Is using surrogate-escapes enough for this case? I myself am not sure, (and also not affected), but it sounds to me that rejecting bytes is a wrong approach if there is no proper workaround (assuming the use-case of pathlib is somewhere deep in library code).
On Thu, May 12, 2016 at 8:22 PM, Sjoerd Job Postmus <sjoerdjob@sjoerdjob.com> wrote:
I would like to make just 1 comment regarding the question of accepting (or not) bytes as output of `os.fspath`.
The whole point of adding `os.fspath` is to make it easier to use Path objects. This is in an effort to gain greater adoption of pathlib in libraries. Now, this is an excellent idea.
However, if it were to reject bytes, that would mean that when libraries start to use pathlib, it would suddenly become harder for people that actually need bytes-support to use pathlib.
Now, the claim 'if you need bytes, you should not be using pathlib` is a reasonable one. But what if I need bytes *and* a specific library (say, image handling, or a web framework, or ...). It's not up to me if that library uses pathlib or plain old os.path.join.
Is using surrogate-escapes enough for this case? I myself am not sure, (and also not affected), but it sounds to me that rejecting bytes is a wrong approach if there is no proper workaround (assuming the use-case of pathlib is somewhere deep in library code).
This is out of the scope of this PEP and probably a very insignificant issue (luckily, this is not the pathlib PEP). Surrogates will probably work and if not, on can "blaim" broken filenames ;). -- Koos
On 05/12/2016 10:22 AM, Sjoerd Job Postmus wrote:
However, if it were to reject bytes, that would mean that when libraries start to use pathlib, it would suddenly become harder for people that actually need bytes-support to use pathlib.
pathlib is not about bytes support. While bytes are necessary in this digital world we live in, most things that look like text should be text, and that includes most paths.
Now, the claim 'if you need bytes, you should not be using pathlib` is a reasonable one. But what if I need bytes *and* a specific library (say, image handling, or a web framework, or ...). It's not up to me if that library uses pathlib or plain old os.path.join.
If you need bytes support for your paths, there's at least one [1] that has that support. -- ~Ethan~ [1] Yeah, it's mine [2]. ;) I haven't checked if the other third-party libs do or not. [2] <shameless plug> https://pypi.python.org/pypi/antipathy </shameless plug>
On 12 May 2016, at 21:30, Ethan Furman <ethan@stoneleaf.us> wrote:
If you need bytes support for your paths, there's at least one [1] that has that support.
So if I would need bytes support, I should submit a pull request to <arbitrary but awesome library> which replaces usage of the stdlib pathlib with another variant, upon which they will decline the pull request because it introduces another "useless" dependency. Good to know.
On Thu, 12 May 2016 at 14:04 Sjoerd Job Postmus <sjoerdjob@sjoerdjob.com> wrote:
On 12 May 2016, at 21:30, Ethan Furman <ethan@stoneleaf.us> wrote:
If you need bytes support for your paths, there's at least one [1] that has that support.
So if I would need bytes support, I should submit a pull request to <arbitrary but awesome library> which replaces usage of the stdlib pathlib with another variant, upon which they will decline the pull request because it introduces another "useless" dependency.
No, what you should do is ask them to create the pathlib instance lazily and only when duck typing shows they weren't given a compatible object. Then you could pass in some bytes-based solution like Ethan's and not worry about pathlib's refusal to work with bytes. Or you simply ask them to work with os.path after calling os.fspath() and be very careful to not use any strings with the functions.
On 05/12/2016 01:59 PM, Sjoerd Job Postmus wrote:
On 12 May 2016, at 21:30, Ethan Furman <ethan@stoneleaf.us> wrote:
If you need bytes support for your paths, there's at least one [1] that has that support.
So if I would need bytes support, I should submit a pull request to <arbitrary but awesome library> which replaces usage of the stdlib pathlib with another variant, upon which they will decline the pull request because it introduces another "useless" dependency.
My apologies, I thought you were talking about your own code. As far as <arbitrary but awesome library> -- it wouldn't be awesome to me if I couldn't pass in the data types I need. -- ~Ethan~
On 13 May 2016, at 08:07, Ethan Furman <ethan@stoneleaf.us> wrote:
On 05/12/2016 01:59 PM, Sjoerd Job Postmus wrote:
On 12 May 2016, at 21:30, Ethan Furman <ethan@stoneleaf.us> wrote:
If you need bytes support for your paths, there's at least one [1] that has that support.
So if I would need bytes support, I should submit a pull request to <arbitrary but awesome library> which replaces usage of the stdlib pathlib with another variant, upon which they will decline the pull request because it introduces another "useless" dependency.
My apologies, I thought you were talking about your own code.
As far as <arbitrary but awesome library> -- it wouldn't be awesome to me if I couldn't pass in the data types I need.
-- ~Ethan~
I just hope you are right. Thank you (and Brett) for clearing things up for me. My apologies if I came off as (a bit) harsh.
On Thu, May 12, 2016 at 07:22:25PM +0200, Sjoerd Job Postmus wrote:
The whole point of adding `os.fspath` is to make it easier to use Path objects. This is in an effort to gain greater adoption of pathlib in libraries. Now, this is an excellent idea.
However, if it were to reject bytes, that would mean that when libraries start to use pathlib, it would suddenly become harder for people that actually need bytes-support to use pathlib.
I'm not sure how it would be *harder*. Currently, if you need bytes paths, you can't use pathlib. That won't change, so it will be exactly as hard as it is now.
Now, the claim 'if you need bytes, you should not be using pathlib` is a reasonable one. But what if I need bytes *and* a specific library (say, image handling, or a web framework, or ...). It's not up to me if that library uses pathlib or plain old os.path.join.
I feel this is Not Our Problem. Surely the stdlib cannot be held responsible for all the poor decisions made by third-party libraries? If a library hard-codes "\\" as their directory separator, because everyone uses Windows, you'd simply say "this library is not very good for my use-case". Likewise if it rejects bytes paths, then: - perhaps that library is not fit for your purpose; - perhaps you can use it, but with the limitation that you can no longer support all paths, but only the ones which can be accessed via a string path; - perhaps you can build a compatibility layer that lets you use both bytes and the library. I don't doubt that all three options are unpleasant, but I don't see how they are the fault of this PEP. This PEP supports Path-Like objects that return bytes paths. Its up to library authors to support bytes paths, or not, as they see fit. -- Steve
On Fri, May 13, 2016 at 8:19 PM, Steven D'Aprano <steve@pearwood.info> wrote:
I feel this is Not Our Problem. Surely the stdlib cannot be held responsible for all the poor decisions made by third-party libraries? If a library hard-codes "\\" as their directory separator, because everyone uses Windows, you'd simply say "this library is not very good for my use-case". Likewise if it rejects bytes paths, then:
- perhaps that library is not fit for your purpose;
- perhaps you can use it, but with the limitation that you can no longer support all paths, but only the ones which can be accessed via a string path;
- perhaps you can build a compatibility layer that lets you use both bytes and the library.
AFAICT, the compatibility layer would simply decode the bytes using surrogateescape handling, which should round-trip anything. Or am I wrong here somewhere? ChrisA
Chris Angelico writes:
AFAICT, the compatibility layer would simply decode the bytes using surrogateescape handling, which should round-trip anything.
By design. See PEP 383. Or rather, the OP should; he has not done his homework and is confused by his own FUD. This whole subthread is really python-list territory. Since a lot of people I respect seem uncertain about the facts, for the record, let's lay out the (putative) issues remaining for post-PEP-383 Python vs. str-y path objects. (0) "Can't work with some POSIX (bytes) paths" is closed by PEP 383, forget it. os.fsdecode(bytespath) as soon as you get one, os.fsencode(strpath) just before you need one, done. Surrogates embedded in strpath may need special handling depending on the application (see (1)). (1) str.encode(errors='strict') (the default) will blow up on embedded surrogates. Yes, but that's a *good* thing if you're mixing str derived from filesystem paths with other text. There's no way to avoid it. If you're just passing it back to open(), it Just Works, done. (2) You're using bytes as text a la 2.x for "efficiency's" sake, and you're worried that you might pass a str-y Path deep into bytes territory and it will explode there. I don't think there is any sympathy left for that use case on Python dev channels. Define a clear boundary with well-defined entry and exit gates, and convert there. Then you can get some sleep. (How-to example: your "compatibility layer".) (3) You're worried about inefficiency of decoding/encoding to the same or trivially changed bytes (ie, you didn't need pathlib in the first place, but you got it anyway) -- this especially matters for 2.7, but is significant for 3.x too, if you're using a bunch of paths in a tight loop. I don't have sympathy for that use case, but Brett and Guido do, and Brett's PEP handles it by making __fspath__ polymorphic in the usual os.path-y way, with Guido's modification. This is always a tradeoff. If you know your JPEGs all have extension '.JPG' and png_path = jpeg_path[:-4] + b'.png' is readable enough for you, use that, not pathlib or Antipathy, and you get your efficiency. (Doing jpeg_path.rindex(b'.') is left as an exercise for the reader. Part (i): Is it really worth it?) If you want the readability of a rich path library and the efficiency of bytes, you *may* have the option of using Ethan's Antipathy (or whatever). If you can't use Antipathy, use bytes methods directly, or accept that it isn't *that* inefficient and use pathlib. At this point, I think this subcase is just FUD, no real examples were presented where the efficiency hit of encoding/decoding gets in the way of getting work done using pathlib. If you need to stick to stdlib for some reason (eg, to use a higher-level library that uses pathlib), live with the "compatibility layer"'s inefficiency. Decoding and encoding are actually rather low-cost operations at path lengths (PATHMAX=256 was common, not so long ago!). Most high-level libraries will impose a lot more overhead elsewhere, and calling into pathlib by itself will add a certain amount of overhead as well. (4) Lack of transparency/readability for "simple" operations. If Antipathy is something you can use, I agree it's plausible that avoiding a few os.fsdecode and os.fsencode calls would look nicer, but this is really a style question. My take: I think of paths as human-readable, so presenting them as str (not bytes) is important to me, important enough that I advocate that position to other developers. If you do the conversion at the boundary between a bytes-y module and pathlib ("compatibility layer") I don't see how it affects readability of the path manipulation code, while data marshaling at boundaries is a expected fact of software development. YMMV. (0) is thus a non-issue. (1) is not something that can be addressed by general principles, let alone language design. (2)-(4) are all real issues regardless of how I feel they should be resolved :-), but they're all design trade-offs, not things that can completely block you from getting some kinds of work done in your own style (eg, the situation str-minded people were in before PEP 383). Python 3 is an example of how language design can help alleviate issues like (2), by discouraging that use case in various ways. Brett's PEP is an example of how language design can help alleviate issues like (3) and (4). In particular, it helps us to interface pathlib to open() and friends in a very natural, readable way, without explicit conversions that should be unnecessary by the nature of the operation and its arguments. By contrast, the conversion of bytes to str is important to do explicitly because they are different representations of the same thing, and it's important that readers be notified of that change of representation.
Or am I wrong here somewhere?
Well, considering the length of this irrelevant-to-the-PEP subthread, arguably you are feeding a successful troll. I hope that having posted the above, in the future there will be *one*, *short* reply to such questions: Not a problem. Read PEP 383. and the thread will end there. Steve
participants (16)
-
Arthur Darcet -
Brett Cannon -
Chris Angelico -
Ethan Furman -
Greg Ewing -
Guido van Rossum -
Koos Zevenhoven -
Larry Hastings -
Nick Coghlan -
Nikolaus Rath -
Paul Moore -
Serhiy Storchaka -
Sjoerd Job Postmus -
Stephen J. Turnbull -
Steven D'Aprano -
Sven R. Kunze