Mailman 3 May 2016 - Python-Dev

Buildbots now always run tests in subprocesses
by Victor Stinner 20 May '16

20 May '16

Hi, I modified regrtest to not ignore -j1 anymore (issue #25285). I also modified Tools/buildbot/test.bat to add the -j1 option (-j1 was already used on UNIX on "make buildbottest"). All buildbots will now run each test file in a fresh subprocess. It helps to restrict all kinds of side effects of unit tests to a single file. The drawback is that buildbots will be less efficient to detect side effects of unit tests. By the way, I also changed regrtest output in "multiprocess mode" (when -jN is used) to add "passed" to test names: --- $ ./python -m test -j1 ... Run tests in parallel using 1 child processes 0:00:00 [ 1/401] test_grammar passed 0:00:00 [ 2/401] test_opcodes passed 0:00:01 [ 3/401] test_dict passed --- When tests are run sequentially, the output is different: --- $ ./python -u -m test ... Run tests sequentially 0:00:00 [ 1/401] test_grammar 0:00:00 [ 2/401] test_opcodes 0:00:00 [ 3/401] test_dict ... --- In multiprocess mode, regrtest displays the name of the test which just completed. In sequential mode, it displays the name of the test which is currently running. Maybe the output of sequential mode can also be enhanced, but I made the minimum change to avoid breaking regrtest. Some parts of the code rely on the regrtest output format. I added many unit tests to test_regrtest, but not everything is tested yet. Victor

1 0

Re: [Python-Dev] Speeding up CPython 5-10%
by zreed＠fastmail.com 18 May '16

18 May '16

In the project https://github.com/zachariahreed/byteasm I mentioned on the list earlier this month, I have a pass that to computes stack usage for a given sequence of bytecodes. It seems to be a fair bit more agressive than cpython. Maybe it's more generally useful. It's pure python rather than C though.

2 4

Python2.7.11+ as in Debian testing and Ubuntu 16.04 LTS crashes with segfault
by Dr. Ján Jockusch 17 May '16

17 May '16

Hello Python Developers, I would like to draw your attention to a segmentation fault bug which concerns the packaged versions of Python 2.7.11 as seen in Ubuntu's latest 16.04 release and in Debian testing. I have reported this bug in Ubuntu's tracker here: https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/1580577 The report contains a script which can reproduce the error for both distributions' python2.7 packages. The plain python.org version 2.7.11 does not exhibit this problem. Matthias Klose, Debian maintainer for the package, has asked me to report this problem upstream, because otherwise we will invariably run into this regression in 2.7.12. I'd like to help in tracking down this bug, please advise. Thank you, - Ján Jockusch

2 1

File system path PEP, 3rd draft
by Brett Cannon 17 May '16

17 May '16

Biggest changes since the second draft: 1. Resolve __fspath__() from the type, not the instance (for Guido) 2. Updated the TypeError messages to say "os.PathLike object" instead of "path object" (implicitly for Steven) 3. TODO item to define "path-like" in the glossary (for Steven) 4. Various more things added to Rejected Ideas 5. Added Koos as a co-author (for Koos :) ---------- PEP: NNN Title: Adding a file system path protocol Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon <brett(a)python.org>, Koos Zevenhoven <k7hoven(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 11-May-2016 Post-History: 11-May-2016, 12-May-2016, 13-May-2016 Abstract ======== This PEP proposes a protocol for classes which represent a file system path to be able to provide a ``str`` or ``bytes`` representation. Changes to Python's standard library are also proposed to utilize this protocol where appropriate to facilitate the use of path objects where historically only ``str`` and/or ``bytes`` file system paths are accepted. The goal is to facilitate the migration of users towards rich path objects while providing an easy way to work with code expecting ``str`` or ``bytes``. Rationale ========= Historically in Python, file system paths have been represented as strings or bytes. This choice of representation has stemmed from C's own decision to represent file system paths as ``const char *`` [#libc-open]_. While that is a totally serviceable format to use for file system paths, it's not necessarily optimal. At issue is the fact that while all file system paths can be represented as strings or bytes, not all strings or bytes represent a file system path. This can lead to issues where any e.g. string duck-types to a file system path whether it actually represents a path or not. To help elevate the representation of file system paths from their representation as strings and bytes to a richer object representation, the pathlib module [#pathlib]_ was provisionally introduced in Python 3.4 through PEP 428. While considered by some as an improvement over strings and bytes for file system paths, it has suffered from a lack of adoption. Typically the key issue listed for the low adoption rate has been the lack of support in the standard library. This lack of support required users of pathlib to manually convert path objects to strings by calling ``str(path)`` which many found error-prone. One issue in converting path objects to strings comes from the fact that the only generic way to get a string representation of the path was to pass the object to ``str()``. This can pose a problem when done blindly as nearly all Python objects have some string representation whether they are a path or not, e.g. ``str(None)`` will give a result that ``builtins.open()`` [#builtins-open]_ will happily use to create a new file. Exacerbating this whole situation is the ``DirEntry`` object [#os-direntry]_. While path objects have a representation that can be extracted using ``str()``, ``DirEntry`` objects expose a ``path`` attribute instead. Having no common interface between path objects, ``DirEntry``, and any other third-party path library has become an issue. A solution that allows any path-representing object to declare that it is a path and a way to extract a low-level representation that all path objects could support is desired. This PEP then proposes to introduce a new protocol to be followed by objects which represent file system paths. Providing a protocol allows for explicit signaling of what objects represent file system paths as well as a way to extract a lower-level representation that can be used with older APIs which only support strings or bytes. Discussions regarding path objects that led to this PEP can be found in multiple threads on the python-ideas mailing list archive [#python-ideas-archive]_ for the months of March and April 2016 and on the python-dev mailing list archives [#python-dev-archive]_ during April 2016. Proposal ======== This proposal is split into two parts. One part is the proposal of a protocol for objects to declare and provide support for exposing a file system path representation. The other part deals with changes to Python's standard library to support the new protocol. These changes will also lead to the pathlib module dropping its provisional status. Protocol -------- The following abstract base class defines the protocol for an object to be considered a path object:: import abc import typing as t class PathLike(abc.ABC): """Abstract base class for implementing the file system path protocol.""" @abc.abstractmethod def __fspath__(self) -> t.Union[str, bytes]: """Return the file system path representation of the object.""" raise NotImplementedError Objects representing file system paths will implement the ``__fspath__()`` method which will return the ``str`` or ``bytes`` representation of the path. The ``str`` representation is the preferred low-level path representation as it is human-readable and what people historically represent paths as. Standard library changes ------------------------ It is expected that most APIs in Python's standard library that currently accept a file system path will be updated appropriately to accept path objects (whether that requires code or simply an update to documentation will vary). The modules mentioned below, though, deserve specific details as they have either fundamental changes that empower the ability to use path objects, or entail additions/removal of APIs. builtins '''''''' ``open()`` [#builtins-open]_ will be updated to accept path objects as well as continue to accept ``str`` and ``bytes``. os ''' The ``fspath()`` function will be added with the following semantics:: import typing as t def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str, bytes]: """Return the string representation of the path. If str or bytes is passed in, it is returned unchanged. """ if isinstance(path, (str, bytes)): return path # Work from the object's type to match method resolution of other magic # methods. path_type = type(path) try: return path_type.__fspath__(path) except AttributeError: if hasattr(path_type, '__fspath__'): raise raise TypeError("expected str, bytes or os.PathLike object, not " + path_type.__name__) The ``os.fsencode()`` [#os-fsencode]_ and ``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept path objects. As both functions coerce their arguments to ``bytes`` and ``str``, respectively, they will be updated to call ``__fspath__()`` if present to convert the path object to a ``str`` or ``bytes`` representation, and then perform their appropriate coercion operations as if the return value from ``__fspath__()`` had been the original argument to the coercion function in question. The addition of ``os.fspath()``, the updates to ``os.fsencode()``/``os.fsdecode()``, and the current semantics of ``pathlib.PurePath`` provide the semantics necessary to get the path representation one prefers. For a path object, ``pathlib.PurePath``/``Path`` can be used. To obtain the ``str`` or ``bytes`` representation without any coersion, then ``os.fspath()`` can be used. If a ``str`` is desired and the encoding of ``bytes`` should be assumed to be the default file system encoding, then ``os.fsdecode()`` should be used. If a ``bytes`` representation is desired and any strings should be encoded using the default file system encoding, then ``os.fsencode()`` is used. This PEP recommends using path objects when possible and falling back to string paths as necessary and using ``bytes`` as a last resort. Another way to view this is as a hierarchy of file system path representations (highest- to lowest-level): path → str → bytes. The functions and classes under discussion can all accept objects on the same level of the hierarchy, but they vary in whether they promote or demote objects to another level. The ``pathlib.PurePath`` class can promote a ``str`` to a path object. The ``os.fspath()`` function can demote a path object to a ``str`` or ``bytes`` instance, depending on what ``__fspath__()`` returns. The ``os.fsdecode()`` function will demote a path object to a string or promote a ``bytes`` object to a ``str``. The ``os.fsencode()`` function will demote a path or string object to ``bytes``. There is no function that provides a way to demote a path object directly to ``bytes`` while bypassing string demotion. The ``DirEntry`` object [#os-direntry]_ will gain an ``__fspath__()`` method. It will return the same value as currently found on the ``path`` attribute of ``DirEntry`` instances. The Protocol_ ABC will be added to the ``os`` module under the name ``os.PathLike``. os.path ''''''' The various path-manipulation functions of ``os.path`` [#os-path]_ will be updated to accept path objects. For polymorphic functions that accept both bytes and strings, they will be updated to simply use ``os.fspath()``. During the discussions leading up to this PEP it was suggested that ``os.path`` not be updated using an "explicit is better than implicit" argument. The thinking was that since ``__fspath__()`` is polymorphic itself it may be better to have code working with ``os.path`` extract the path representation from path objects explicitly. There is also the consideration that adding support this deep into the low-level OS APIs will lead to code magically supporting path objects without requiring any documentation updated, leading to potential complaints when it doesn't work, unbeknownst to the project author. But it is the view of this PEP that "practicality beats purity" in this instance. To help facilitate the transition to supporting path objects, it is better to make the transition as easy as possible than to worry about unexpected/undocumented duck typing support for path objects by projects. There has also been the suggestion that ``os.path`` functions could be used in a tight loop and the overhead of checking or calling ``__fspath__()`` would be too costly. In this scenario only path-consuming APIs would be directly updated and path-manipulating APIs like the ones in ``os.path`` would go unmodified. This would require library authors to update their code to support path objects if they performed any path manipulations, but if the library code passed the path straight through then the library wouldn't need to be updated. It is the view of this PEP and Guido, though, that this is an unnecessary worry and that performance will still be acceptable. pathlib ''''''' The constructor for ``pathlib.PurePath`` and ``pathlib.Path`` will be updated to accept ``PathLike`` objects. Both ``PurePath`` and ``Path`` will continue to not accept ``bytes`` path representations, and so if ``__fspath__()`` returns ``bytes`` it will raise an exception. The ``path`` attribute will be removed as this PEP makes it redundant (it has not been included in any released version of Python and so is not a backwards-compatibility concern). C API ''''' The C API will gain an equivalent function to ``os.fspath()``:: /* Return the file system path of the object. If the object is str or bytes, then allow it to pass through with an incremented refcount. If the object defines __fspath__(), then return the result of that method. All other types raise a TypeError. */ PyObject * PyOS_FSPath(PyObject *path) { if (PyUnicode_Check(path) || PyBytes_Check(path)) { Py_INCREF(path); return path; } if (PyObject_HasAttrString(path->ob_type, "__fspath__")) { return PyObject_CallMethodObjArgs(path->ob_type, "__fspath__", path, NULL); } return PyErr_Format(PyExc_TypeError, "expected a str, bytes, or os.PathLike object, not %S", path->ob_type); } Backwards compatibility ======================= There are no explicit backwards-compatibility concerns. Unless an object incidentally already defines a ``__fspath__()`` method there is no reason to expect the pre-existing code to break or expect to have its semantics implicitly changed. Libraries wishing to support path objects and a version of Python prior to Python 3.6 and the existence of ``os.fspath()`` can use the idiom of ``path.__fspath__() if hasattr(path, "__fspath__") else path``. Implementation ============== This is the task list for what this PEP proposes: #. Remove the ``path`` attribute from pathlib #. Remove the provisional status of pathlib #. Add ``os.PathLike`` #. Add ``os.fspath()`` #. Add ``PyOS_FSPath()`` #. Update ``os.fsencode()`` #. Update ``os.fsdecode()`` #. Update ``pathlib.PurePath`` and ``pathlib.Path`` #. Update ``builtins.open()`` #. Update ``os.DirEntry`` #. Update ``os.path`` #. Add a glossary entry for "path-like" Rejected Ideas ============== Other names for the protocol's method ------------------------------------- Various names were proposed during discussions leading to this PEP, including ``__path__``, ``__pathname__``, and ``__fspathname__``. In the end people seemed to gravitate towards ``__fspath__`` for being unambiguous without being unnecessarily long. Separate str/bytes methods -------------------------- At one point it was suggested that ``__fspath__()`` only return strings and another method named ``__fspathb__()`` be introduced to return bytes. The thinking is that by making ``__fspath__()`` not be polymorphic it could make dealing with the potential string or bytes representations easier. But the general consensus was that returning bytes will more than likely be rare and that the various functions in the os module are the better abstraction to promote over direct calls to ``__fspath__()``. Providing a ``path`` attribute ------------------------------ To help deal with the issue of ``pathlib.PurePath`` not inheriting from ``str``, originally it was proposed to introduce a ``path`` attribute to mirror what ``os.DirEntry`` provides. In the end, though, it was determined that a protocol would provide the same result while not directly exposing an API that most people will never need to interact with directly. Have ``__fspath__()`` only return strings ------------------------------------------ Much of the discussion that led to this PEP revolved around whether ``__fspath__()`` should be polymorphic and return ``bytes`` as well as ``str`` or only return ``str``. The general sentiment for this view was that ``bytes`` are difficult to work with due to their inherent lack of information about their encoding and PEP 383 makes it possible to represent all file system paths using ``str`` with the ``surrogateescape`` handler. Thus, it would be better to forcibly promote the use of ``str`` as the low-level path representation for high-level path objects. In the end, it was decided that using ``bytes`` to represent paths is simply not going to go away and thus they should be supported to some degree. The hope is that people will gravitate towards path objects like pathlib and that will move people away from operating directly with ``bytes``. A generic string encoding mechanism ----------------------------------- At one point there was a discussion of developing a generic mechanism to extract a string representation of an object that had semantic meaning (``__str__()`` does not necessarily return anything of semantic significance beyond what may be helpful for debugging). In the end, it was deemed to lack a motivating need beyond the one this PEP is trying to solve in a specific fashion. Have __fspath__ be an attribute ------------------------------- It was briefly considered to have ``__fspath__`` be an attribute instead of a method. This was rejected for two reasons. One, historically protocols have been implemented as "magic methods" and not "magic methods and attributes". Two, there is no guarantee that the lower-level representation of a path object will be pre-computed, potentially misleading users that there was no expensive computation behind the scenes in case the attribute was implemented as a property. This also indirectly ties into the idea of introducing a ``path`` attribute to accomplish the same thing. This idea has an added issue, though, of accidentally having any object with a ``path`` attribute meet the protocol's duck typing. Introducing a new magic method for the protocol helpfully avoids any accidental opting into the protocol. Provide specific type hinting support ------------------------------------- There was some consideration to provdinga generic ``typing.PathLike`` class which would allow for e.g. ``typing.PathLike[str]`` to specify a type hint for a path object which returned a string representation. While potentially beneficial, the usefulness was deemed too small to bother adding the type hint class. This also removed any desire to have a class in the ``typing`` module which represented the union of all acceptable path-representing types as that can be represented with ``typing.Union[str, bytes, os.PathLike]`` easily enough and the hope is users will slowly gravitate to path objects only. Provide ``os.fspathb()`` ------------------------ It was suggested that to mirror the structure of e.g. ``os.getcwd()``/``os.getcwdb()``, that ``os.fspath()`` only return ``str`` and that another function named ``os.fspathb()`` be introduced that only returned ``bytes``. This was rejected as the purposes of the ``*b()`` functions are tied to querying the file system where there is a need to get the raw bytes back. As this PEP does not work directly with data on a file system (but which *may* be), the view was taken this distinction is unnecessary. It's also believed that the need for only bytes will not be common enough to need to support in such a specific manner as ``os.fsencode()`` will provide similar functionality. Call ``__fspath__()`` off of the instance ----------------------------------------- An earlier draft of this PEP had ``os.fspath()`` calling ``path.__fspath__()`` instead of ``type(path).__fspath__(path)``. The changed to be consistent with how other magic methods in Python are resolved. Acknowledgements ================ Thanks to everyone who participated in the various discussions related to this PEP that spanned both python-ideas and python-dev. Special thanks to Stephen Turnbull for direct feedback on early drafts of this PEP. More special thanks to Koos Zevenhoven and Ethan Furman for not only feedback on early drafts of this PEP but also helping to drive the overall discussion on this topic across the two mailing lists. References ========== .. [#python-ideas-archive] The python-ideas mailing list archive (https://mail.python.org/pipermail/python-ideas/) .. [#python-dev-archive] The python-dev mailing list archive (https://mail.python.org/pipermail/python-dev/) .. [#libc-open] ``open()`` documention for the C standard library ( http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files… ) .. [#pathlib] The ``pathlib`` module (https://docs.python.org/3/library/pathlib.html#module-pathlib) .. [#builtins-open] The ``builtins.open()`` function (https://docs.python.org/3/library/functions.html#open) .. [#os-fsencode] The ``os.fsencode()`` function (https://docs.python.org/3/library/os.html#os.fsencode) .. [#os-fsdecode] The ``os.fsdecode()`` function (https://docs.python.org/3/library/os.html#os.fsdecode) .. [#os-direntry] The ``os.DirEntry`` class (https://docs.python.org/3/library/os.html#os.DirEntry) .. [#os-path] The ``os.path`` module (https://docs.python.org/3/library/os.path.html#module-os.path) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

7 15

Speeding up CPython 5-10%
by Yury Selivanov 17 May '16

17 May '16

Hi, tl;dr The summary is that I have a patch that improves CPython performance up to 5-10% on macro benchmarks. Benchmarks results on Macbook Pro/Mac OS X, desktop CPU/Linux, server CPU/Linux are available at [1]. There are no slowdowns that I could reproduce consistently. There are twodifferent optimizations that yield this speedup: LOAD_METHOD/CALL_METHOD opcodes and per-opcode cache in ceval loop. LOAD_METHOD & CALL_METHOD ------------------------- We had a lot of conversations with Victor about his PEP 509, and he sent me a link to his amazing compilation of notes about CPython performance [2]. One optimization that he pointed out to me was LOAD/CALL_METHOD opcodes, an idea first originated in PyPy. There is a patch that implements this optimization, it's tracked here: [3]. There are some low level details that I explained in the issue, but I'll go over the high level design in this email as well. Every time you access a method attribute on an object, a BoundMethod object is created. It is a fairly expensive operation, despite a freelist of BoundMethods (so that memory allocation is generally avoided). The idea is to detect what looks like a method call in the compiler, and emit a pair of specialized bytecodes for that. So instead of LOAD_GLOBAL/LOAD_ATTR/CALL_FUNCTION we will have LOAD_GLOBAL/LOAD_METHOD/CALL_METHOD. LOAD_METHOD looks at the object on top of the stack, and checks if the name resolves to a method or to a regular attribute. If it's a method, then we push the unbound method object and the object to the stack. If it's an attribute, we push the resolved attribute and NULL. When CALL_METHOD looks at the stack it knows how to call the unbound method properly (pushing the object as a first arg), or how to call a regular callable. This idea does make CPython faster around 2-4%. And it surely doesn't make it slower. I think it's a safe bet to at least implement this optimization in CPython 3.6. So far, the patch only optimizes positional-only method calls. It's possible to optimize all kind of calls, but this will necessitate 3 more opcodes (explained in the issue). We'll need to do some careful benchmarking to see if it's really needed. Per-opcode cache in ceval ------------------------- While reading PEP 509, I was thinking about how we can use dict->ma_version in ceval to speed up globals lookups. One of the key assumptions (and this is what makes JITs possible) is that real-life programs don't modify globals and rebind builtins (often), and that most code paths operate on objects of the same type. In CPython, all pure Python functions have code objects. When you call a function, ceval executes its code object in a frame. Frames contain contextual information, including pointers to the globals and builtins dict. The key observation here is that almost all code objects always have same pointers to the globals (the module they were defined in) and to the builtins. And it's not a good programming practice to mutate globals or rebind builtins. Let's look at this function: def spam(): print(ham) Here are its opcodes: 2 0 LOAD_GLOBAL 0 (print) 3 LOAD_GLOBAL 1 (ham) 6 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 9 POP_TOP 10 LOAD_CONST 0 (None) 13 RETURN_VALUE The opcodes we want to optimize are LAOD_GLOBAL, 0 and 3. Let's look at the first one, that loads the 'print' function from builtins. The opcode knows the following bits of information: - its offset (0), - its argument (0 -> 'print'), - its type (LOAD_GLOBAL). And these bits of information will *never* change. So if this opcode could resolve the 'print' name (from globals or builtins, likely the latter) and save the pointer to it somewhere, along with globals->ma_version and builtins->ma_version, it could, on its second call, just load this cached info back, check that the globals and builtins dict haven't changed and push the cached ref to the stack. That would save it from doing two dict lookups. We can also optimize LOAD_METHOD. There are high chances, that 'obj' in 'obj.method()' will be of the same type every time we execute the code object. So if we'd have an opcodes cache, LOAD_METHOD could then cache a pointer to the resolved unbound method, a pointer to obj.__class__, and tp_version_tag of obj.__class__. Then it would only need to check if the cached object type is the same (and that it wasn't modified) and that obj.__dict__ doesn't override 'method'. Long story short, this caching really speeds up method calls on types implemented in C. list.append becomes very fast, because list doesn't have a __dict__, so the check is very cheap (with cache). A straightforward way to implement such a cache is simple, but consumes a lot of memory, that would be just wasted, since we only need such a cache for LOAD_GLOBAL and LOAD_METHOD opcodes. So we have to be creative about the cache design. Here's what I came up with: 1. We add a few fields to the code object. 2. ceval will count how many times each code object is executed. 3. When the code object is executed over ~900 times, we mark it as "hot". We also create an 'unsigned char' array "MAPPING", with length set to match the length of the code object. So we have a 1-to-1 mapping between opcodes and MAPPING array. 4. Next ~100 calls, while the code object is "hot", LOAD_GLOBAL and LOAD_METHOD do "MAPPING[opcode_offset()]++". 5. After 1024 calls to the code object, ceval loop will iterate through the MAPPING, counting all opcodes that were executed more than 50 times. 6. We then create an array of cache structs "CACHE" (here's a link to the updated code.h file: [6]). We update MAPPING to be a mapping between opcode position and position in the CACHE. The code object is now "optimized". 7. When the code object is "optimized", LOAD_METHOD and LOAD_GLOBAL use the CACHE array for fast path. 8. When there is a cache miss, i.e. the builtins/global/obj.__dict__ were mutated, the opcode marks its entry in 'CACHE' as deoptimized, and it will never try to use the cache again. Here's a link to the issue tracker with the first version of the patch: [5]. I'm working on the patch in a github repo here: [4]. Summary ------- There are many things about this algorithm that we can improve/tweak. Perhaps we should profile code objects longer, or account for time they were executed. Maybe we shouldn't deoptimize opcodes on their first cache miss. Maybe we can come up with better data structures. We also need to profile the memory and see how much more this cache will require. One thing I'm certain about, is that we can get a 5-10% speedup of CPython with relatively low memory impact. And I think it's worth exploring that! If you're interested in these kind of optimizations, please help with code reviews, ideas, profiling and benchmarks. The latter is especially important, I'd never imagine how hard it is to come up with a good macro benchmark. I also want to thank my company MagicStack (magic.io) for sponsoring this work. Thanks, Yury [1] https://gist.github.com/1st1/aed69d63a2ff4de4c7be [2] http://faster-cpython.readthedocs.org/index.html [3] http://bugs.python.org/issue26110 [4] https://github.com/1st1/cpython/tree/opcache2 [5] http://bugs.python.org/issue26219 [6] https://github.com/python/cpython/compare/master...1st1:opcache2?expand=1#d…

12 26

file system path protocol PEP
by Brett Cannon 15 May '16

15 May '16

**deep, calming breath** Here is the PEP for __fspath__(). The draft lives at https://github.com/brettcannon/path-pep so feel free to send me PRs for spelling mistakes, grammatical errors, etc. ----- PEP: NNN Title: Adding a file system path protocol Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon <brett(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 11-May-2016 Post-History: 11-May-2016 Abstract ======== This PEP proposes a protocol for classes which represent a file system path to be able to provide a ``str`` or ``bytes`` representation. Changes to Python's standard library are also proposed to utilize this protocol where appropriate to facilitate the use of path objects where historically only ``str`` and/or ``bytes`` file system paths are accepted. The goal is to allow users to use the representation of a file system path that's easiest for them now as they migrate towards using path objects in the future. Rationale ========= Historically in Python, file system paths have been represented as strings or bytes. This choice of representation has stemmed from C's own decision to represent file system paths as ``const char *`` [#libc-open]_. While that is a totally serviceable format to use for file system paths, it's not necessarily optimal. At issue is the fact that while all file system paths can be represented as strings or bytes, not all strings or bytes represent a file system path. This can lead to issues where any e.g. string duck-types to a file system path whether it actually represents a path or not. To help elevate the representation of file system paths from their representation as strings and bytes to a more appropriate object representation, the pathlib module [#pathlib]_ was provisionally introduced in Python 3.4 through PEP 428. While considered by some as an improvement over strings and bytes for file system paths, it has suffered from a lack of adoption. Typically the key issue listed for the low adoption rate has been the lack of support in the standard library. This lack of support required users of pathlib to manually convert path objects to strings by calling ``str(path)`` which many found error-prone. One issue in converting path objects to strings comes from the fact that only generic way to get a string representation of the path was to pass the object to ``str()``. This can pose a problem when done blindly as nearly all Python objects have some string representation whether they are a path or not, e.g. ``str(None)`` will give a result that ``builtins.open()`` [#builtins-open]_ will happily use to create a new file. Exacerbating this whole situation is the ``DirEntry`` object [#os-direntry]_. While path objects have a representation that can be extracted using ``str()``, ``DirEntry`` objects expose a ``path`` attribute instead. Having no common interface between path objects, ``DirEntry``, and any other third-party path library had become an issue. A solution that allowed any path-representing object to declare that is was a path and a way to extract a low-level representation that all path objects could support was desired. This PEP then proposes to introduce a new protocol to be followed by objects which represent file system paths. Providing a protocol allows for clear signalling of what objects represent file system paths as well as a way to extract a lower-level representation that can be used with older APIs which only support strings or bytes. Discussions regarding path objects that led to this PEP can be found in multiple threads on the python-ideas mailing list archive [#python-ideas-archive]_ for the months of March and April 2016 and on the python-dev mailing list archives [#python-dev-archive]_ during April 2016. Proposal ======== This proposal is split into two parts. One part is the proposal of a protocol for objects to declare and provide support for exposing a file system path representation. The other part is changes to Python's standard library to support the new protocol. These changes will also have the pathlib module drop its provisional status. Protocol -------- The following abstract base class defines the protocol for an object to be considered a path object:: import abc import typing as t class PathLike(abc.ABC): """Abstract base class for implementing the file system path protocol.""" @abc.abstractmethod def __fspath__(self) -> t.Union[str, bytes]: """Return the file system path representation of the object.""" raise NotImplementedError Objects representing file system paths will implement the ``__fspath__()`` method which will return the ``str`` or ``bytes`` representation of the path. The ``str`` representation is the preferred low-level path representation as it is human-readable and what people historically represent paths as. Standard library changes ------------------------ It is expected that most APIs in Python's standard library that currently accept a file system path will be updated appropriately to accept path objects (whether that requires code or simply an update to documentation will vary). The modules mentioned below, though, deserve specific details as they have either fundamental changes that empower the ability to use path objects, or entail additions/removal of APIs. builtins '''''''' ``open()`` [#builtins-open]_ will be updated to accept path objects as well as continue to accept ``str`` and ``bytes``. os ''' The ``fspath()`` function will be added with the following semantics (although this is still an open issue: `Should os.fspath() return bytes?`_):: import typing as t def fspath(path: t.Union[PathLike, str]) -> str: """Return the string representation of the path. If a string is passed in then it is returned unchanged. """ if hasattr(path, '__fspath__'): path = path.__fspath__() if not isinstance(path, str): type_name = type(path).__name__ raise TypeError("expected a str or path object, not " + type_name) return path The ``os.fsencode()`` [#os-fsencode]_ and ``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept path objects. As both functions coerce their arguments to ``bytes`` and ``str``, respectively, they will be updated to call ``__fspath__()`` if present to convert the path object to a ``str`` or ``bytes`` representation, and then perform their appropriate coercion operations as if the return value from ``__fspath__()`` had been the original argument to the coercion function in question. The addition of ``os.fspath()``, the updates to ``os.fsencode()``/``os.fsdecode()``, and the current semantics of ``pathlib.PurePath`` provide the semantics necessary to get the path representation one prefers. For a path object, ``pathlib.PurePath``/``Path`` can be used. If ``str`` is desired and a ``bytes`` object is considered an error, then ``os.fspath()`` can be used. If a ``str`` is desired and the encoding of ``bytes`` should be assumed to be the default file system encoding, then ``os.fsdecode()`` should be used. Finally, if a ``bytes`` representation is desired and any strings should be encoded using the default file system encoding then ``os.fsencode()`` is used. This PEP recommends using path objects when possible and falling back to string paths as necessary. Therefore, no function is provided for the case of wanting a bytes representation but without any automatic encoding to help discourage the use of multiple bytes encodings on a single file system. If it is necessary to deal with an existing file system directory with entries in a non-default encoding, this can be done with low-level functions using ``str`` and the PEP 383 ``surrogateescape`` error handler, or by using ``bytes`` directly. Another way to view this is as a hierarchy of file system path representations (highest- to lowest-level): path -> str -> bytes. The functions and classes under discussion can all accept objects on the same level of the hierarchy, but they vary in whether they promote or demote objects to another level. The ``pathlib.PurePath`` class can promote a ``str`` to a path object. The ``os.fspath()`` function can demote a path object to a string, but only if ``__fspath__()`` returns a string. The ``os.fsdecode()`` function will demote a path object to a string or promote a ``bytes`` object to a ``str``. The ``os.fsencode()`` function will demote a path or string object to ``bytes``. There is no function that provides a way to demote a path object directly to ``bytes`` and not allow demoting strings. The ``DirEntry`` object [#os-direntry]_ will gain an ``__fspath__()`` method. It will return the same value as currently found on the ``path`` attribute of ``DirEntry`` instances. os.path ''''''' The various path-manipulation functions of ``os.path`` [#os-path]_ will be updated to accept path objects. For polymorphic functions that accept both bytes and strings, they will be updated to simply use code very much similar to ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This will allow for their pre-existing type-checking code to continue to function. During the discussions leading up to this PEP it was suggested that ``os.path`` not be updated using an "explicit is better than implicit" argument. The thinking was that since ``__fspath__()`` is polymorphic itself it may be better to have code working with ``os.path`` extract the path representation from path objects explicitly. There is also the consideration that adding support this deep into the low-level OS APIs will lead to code magically supporting path objects without requiring any documentation updated, leading to potential complaints when it doesn't work, unbeknownst to the project author. But it is the view of the authors that "practicality beats purity" in this instance. To help facilitate the transition to supporting path objects, it is better to make the transition as easy as possible than to worry about unexpected/undocumented duck typing support for projects. pathlib ''''''' The ``PathLike`` ABC as discussed in the Protocol_ section will be added to the pathlib module [#pathlib]_. The constructor for ``pathlib.PurePath`` and ``pathlib.Path`` will be updated to accept path objects. Both ``PurePath`` and ``Path`` will continue to not accept ``bytes`` path representations, and so if ``__fspath__()`` returns ``bytes`` it will raise an exception. The ``path`` attribute will be removed as this PEP makes its redundant (it has not been included in any released version of Python and so is not a backwards-compatibility concern). C API ''''' The C API will gain an equivalent function to ``os.fspath()`` that also allows bytes objects through:: /* Return the file system path of the object. If the object is str or bytes, then allow it to pass through with an incremented refcount. All other types raise a TypeError. */ PyObject * PyOS_RawFSPath(PyObject *path) { if (PyObject_HasAttrString(path, "__fspath__")) { path = PyObject_CallMethodObjArgs(path, "__fspath__", NULL); if (path == NULL) { return NULL; } } else { Py_INCREF(path); } if (!PyUnicode_Check(path) && !PyBytes_Check(path)) { Py_DECREF(path); return PyErr_Format(PyExc_TypeError, "expected a string, bytes, or path object, not %S", path->ob_type); } return path; } Backwards compatibility ======================= There are no explicit backwards-compatibility concerns. Unless an object incidentally already defines a ``__fspath__()`` method there is not reason to expect pre-existing code to break or expect to have their semantics implicitly changed. Libraries wishing to support path objects and a version of Python prior to Python 3.6 can use the idiom of ``path.__fspath__() if hasattr(path, '__fspath__') else path``. Open Issues =========== Should os.fspath() return bytes? -------------------------------- Some have argued that ``os.fspath()`` should be configurable so that the user can specify what types are acceptable (e.g. an argument to say that bytes are acceptable instead of strings, or both types). Others have suggested that ``os.fspath()`` match the proposed semantics of ``PyOS_RawFSPath()``. Both camps argue that use of ``os.fspath()`` will only be for a transitionary period while more libraries gain acceptance of path objects, and so being more flexible in what ``os.fspath()`` works with will help with the transition. The opponents to this -- which support the currently proposed sematnics -- worry that being so flexible with accepting bytes will lead to people not properly considering the ramifications of working with bytes, especially if bytes are transparently appearing in their code due to ``os.fspath()``. The name and location of the protocol's ABC ------------------------------------------- The name of the ABC being proposed to represent the protocol has not been discussed very much, not which module it should exist in. Names other than ``PathLike`` which are viable are ``PathABC`` and ``FSPathABC``. The name can't be ``Path`` if the ABC is put into the pathlib module. Type hint for path-like objects ------------------------------- Creating a proper type hint for APIs that accept path objects as well as strings and bytes will probably be needed. It could be as simple as defining ``typing.Path``/``typing.FSPath`` to correspond to the ABC and then having ``typing.PathLike = typing.Union[typing.Path, str, bytes]``. The type hint could also potentially be made to be generic to accept the specific low-level representation, e.g. ``typing.PathLike[str]``. In the end the type hinting solution should be properly discussed with the right type hinting experts if this is the best approach. Rejected Ideas ============== Other names for the protocol's function --------------------------------------- Various names were proposed during discussions leading to this PEP, including ``__path__``, ``__pathname__``, and ``__fspathname__``. In the end people seemed to gravitate towards ``__fspath__`` for being unambiguous without unnecessarily long. Separate str/bytes methods -------------------------- At one point it was suggested that ``__fspath__()`` only return strings and another method named ``__fspathb__()`` be introduced to return bytes. The thinking that by making ``__fspath__()`` not be polymorphic it could make dealing with the potential string or bytes representations easier. But the general consensus was that returning bytes will more than likely be rare and that the various functions in the os module are the better abstraction to be promoting over direct calls to ``__fspath__()``. Providing a path attribute -------------------------- To help deal with the issue of ``pathlib.PurePath`` not inheriting from ``str``, originally it was proposed to introduce a ``path`` attribute to mirror what ``os.DirEntry`` provides. In the end, though, it was determined that a protocol would provide the same result while not directly exposing an API that most people will never need to interact with directly. Have ``__fspath__()`` only return strings ------------------------------------------ Much of the discussion that led to this PEP revolved around whether ``__fspath__()`` should be polymorphic and return ``bytes`` as well as ``str`` instead of only ``str``. The general sentiment for this view was that ``bytes`` are difficult to work with due to their inherent lack of information about their encoding, and PEP 383 makes it possible to represent all file system paths using ``str`` with the ``surrogateescape`` handler. Thus it would be better to forcibly promote the use of ``str`` as the low-level path representation for high-level path objects. In the end it was decided that using ``bytes`` to represent paths is simply not going to go away and thus they should be supported to some degree. For those not wanting the hassle of working with ``bytes``, ``os.fspath()`` is provided. A generic string encoding mechanism ----------------------------------- At one point there was discussion of developing a generic mechanism to extract a string representation of an object that had semantic meaning (``__str__()`` does not necessarily return anything of semantic significance beyond what may be helpful for debugging). In the end it was deemed to lack a motivating need beyond the one this PEP is trying to solve in a specific fashion. Acknowledgements ================ Thanks to everyone who participated in the various discussions related to this PEP that spanned both python-ideas and python-dev. Special thanks to Koos Zevenhoven and Stephen Turnbull for direct feedback on early drafts of this PEP. References ========== .. [#python-ideas-archive] The python-ideas mailing list archive (https://mail.python.org/pipermail/python-ideas/) .. [#python-dev-archive] The python-dev mailing list archive (https://mail.python.org/pipermail/python-dev/) .. [#libc-open] ``open()`` documention for the C standard library ( http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files… ) .. [#pathlib] The ``pathlib`` module (https://docs.python.org/3/library/pathlib.html#module-pathlib) .. [#builtins-open] The ``builtins.open()`` function (https://docs.python.org/3/library/functions.html#open) .. [#os-fsencode] The ``os.fsencode()`` function (https://docs.python.org/3/library/os.html#os.fsencode) .. [#os-fsdecode] The ``os.fsdecode()`` function (https://docs.python.org/3/library/os.html#os.fsdecode) .. [#os-direntry] The ``os.DirEntry`` class (https://docs.python.org/3/library/os.html#os.DirEntry) .. [#os-path] The ``os.path`` module (https://docs.python.org/3/library/os.path.html#module-os.path) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

16 76

Wordcode: new regular bytecode using 16-bit units
by Victor Stinner 15 May '16

15 May '16

Hi, In the middle of recent discussions about Python performance, it was discussed to change the Python bytecode. Serhiy proposed to reuse MicroPython short bytecode to reduce the disk space and reduce the memory footprint. Demur Rumed proposes a different change to use a regular bytecode using 16-bit units: an instruction has always one 8-bit argument, it's zero if the instruction doesn't have an argument: http://bugs.python.org/issue26647 According to benchmarks, it looks faster: http://bugs.python.org/issue26647#msg263339 IMHO it's a nice enhancement: it makes the code simpler. The most interesting change is made in Python/ceval.c: - if (HAS_ARG(opcode)) - oparg = NEXTARG(); + oparg = NEXTARG(); This code is the very hot loop evaluating Python bytecode. I expect that removing a conditional branch here can reduce the CPU branch misprediction. I reviewed first versions of the change, and IMHO it's almost ready to be merged. But I would prefer to have a review from a least a second core reviewer. Can someone please review the change? -- The side effect of wordcode is that arguments in 0..255 now uses 2 bytes per instruction instead of 3, so it also reduce the size of bytecode for the most common case. Larger argument, 16-bit argument (0..65,535), now uses 4 bytes instead of 3. Arguments are supported up to 32-bit: 24-bit uses 3 units (6 bytes), 32-bit uses 4 units (8 bytes). MAKE_FUNCTION uses 16-bit argument for keyword defaults and 24-bit argument for annotations. Other common instruction known to use large argument are jumps for bytecode longer than 256 bytes. -- Right now, ceval.c still fetchs opcode and then oparg with two 8-bit instructions. Later, we can discuss if it would be possible to ensure that the bytecode is always aligned to 16-bit in memory to fetch the two bytes using a uint16_t* pointer. Maybe we can overallocate 1 byte in codeobject.c and align manually the memory block if needed. Or ceval.c should maybe copy the code if it's not aligned? Raymond Hettinger proposes something like that, but it looks like there are concerns about non-aligned memory accesses: http://bugs.python.org/issue25823 The cost of non-aligned memory accesses depends on the CPU architecture, but it can raise a SIGBUS on some arch (MIPS and SPARC?). Victor

9 19

Wordcode v2
by Demur Rumed 15 May '16

15 May '16

Saw recent discussion: https://mail.python.org/pipermail/python-dev/2016-February/143013.html I remember trying WPython; it was fast. Unfortunately it feels it came at the wrong time when development was invested in getting py3k out the door. It also had a lot of other ideas like *_INT instructions which allowed having oparg to be a constant int rather than needing to LOAD_CONST one. Anyways I'll stop reminiscing abarnert has started an experiment with wordcode: https://github.com/abarnert/cpython/blob/c095a32f2a68ac708466b9c64906cc4d0f… I've personally benchmarked this fork with positive results. This experiment seeks to be conservative-- it doesn't seek to introduce new opcodes or combine BINARY_OP's all into a single op where the currently unused-in-wordcode arg then states the kind of binary op (à la COMPARE_OP). I've submitted a pull request which is working on fixing tests & updating peephole.c Bringing this up on the list to figure out if there's interest in a basic wordcode change. It feels like there's no downsides: faster code, smaller bytecode, simpler interpretation of bytecode (The Nth instruction starts at the 2Nth byte if you count EXTENDED_ARG as an instruction). The only downside is the transitional cost What'd be necessary for this to be pulled upstream?

7 8

First 3.6.0 alpha release coming up: 2016-05-16 12:00 UTC
by Ned Deily 13 May '16

13 May '16

Hi everyone! Thank you for all of your work so far on our next feature release cycle, Python 3.6. Code checkins for 3.6 began 12 months ago, after the feature code cutoff and beta phase for Python 3.5.0. We are now about to do 3.6.0 alpha 1, our first of 4 alpha and 4 beta releases planned for the next 6 months. As a reminder, alpha releases are intended to make it easier for the wider community to test the current state of new features and bug fixes for an upcoming Python release as a whole and for us to test the release process. The start of the alpha phase does *not* mean feature development is complete. During the alpha phase, features may be added, modified, or deleted up until the start of the beta phase. Alpha users beware! 3.6.0 beta 1, currently scheduled for 2016-09-07, marks feature code cutoff, at which time new feature development is expected to be complete and no additional features are to be added to 3.6. At the same time, feature development will begin for the following release cycle, 3.7. During the 3.6.0 beta phase, which is planned to end on 2016-12-04 with the first final release candidate, the focus will be on bug fixes and regressions and preparing for the final release. The goal is for the first release candidate to be the same as the final release: that is, no changes will be accepted for 3.6.0 after release candidate 1 without discussion, review, and release manager approval. Please plan accordingly! Although I've been involved behind the scenes in most releases over the past several years, 3.6.0a1 will be my first as a release manager so no doubt there will be bumps along the way. Many thanks to my fellow release managers, particularly Benjamin, Georg, Larry, and Barry, for their help and support. Your understanding, help, and feedback will be greatly appreciated! For this first alpha release at least, the release team (primarily Steve Dower and myself) are going to try a slightly different timetable for producing the release. We are planning for the code cutoff for alpha 1 to be on 2016-05-16 around 12:00 UTC; that allows for code checkins to continue through Sunday almost anywhere on the planet. We will then go off and manufacture the release bits and make them available via the python.org website as soon as possible thereafter, assuming no major glitches, that should be no later than 24 hours thereafter. In the remaining hours until the code cutoff, please get your commits in as soon as possible and please pay attention to the buildbots for possible regressions. Looking ahead, the next alpha release, 3.6.0a2, will follow in about a month on 2016-06-12, which is a week after the end of the development sprints at US PyCon 2016 in Portland, Oregon. Hope to see you there! To summarize: 2016-05-16 ~12:00 UTC: code snapshot for 3.6.0 alpha 1 now to 2016-09-07: Alpha phase (unrestricted feature development) 2016-09-07: 3.6.0 feature code freeze, 3.7.0 feature development begins 2016-09-07 to 2016-12-04: 3.6.0 beta phase (bug and regression fixes, no new features) 2016-12-04 3.6.0 release candidate 1 (3.6.0 code freeze) 2016-12-16 3.6.0 release (3.6.0rc1 plus, if necessary, any dire emergency fixes) Let me know if you have questions or comments. And thanks again for all of your work towards making Python even better! --Ned D AKA The Other Ned https://www.python.org/dev/peps/pep-0494/ https://docs.python.org/devguide/devcycle.html#stages https://en.wikipedia.org/wiki/Software_release_life_cycle#Alpha https://docs.python.org/devguide/buildbots.html -- Ned Deily nad(a)python.org -- [] Freenode IRC: ned_deily, #python-dev

1 0

File system path PEP, part 2
by Brett Cannon 13 May '16

13 May '16

Second draft that takes Guido's comments into consideration. The biggest change is os.fspath() now returns whatever path.__fspath__() returns instead of restricting it to only str. Minor changes: - Renamed the C function to PyOS_FSPath() - Added an Implementation section with a TODO list - Bunch of things added to the Rejected Ideas section ---------- PEP: NNN Title: Adding a file system path protocol Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon <brett(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 11-May-2016 Post-History: 11-May-2016, 12-May-2016 Abstract ======== This PEP proposes a protocol for classes which represent a file system path to be able to provide a ``str`` or ``bytes`` representation. Changes to Python's standard library are also proposed to utilize this protocol where appropriate to facilitate the use of path objects where historically only ``str`` and/or ``bytes`` file system paths are accepted. The goal is to facilitate the migration of users towards rich path objects while providing an easy way to work with code expecting ``str`` or ``bytes``. Rationale ========= Historically in Python, file system paths have been represented as strings or bytes. This choice of representation has stemmed from C's own decision to represent file system paths as ``const char *`` [#libc-open]_. While that is a totally serviceable format to use for file system paths, it's not necessarily optimal. At issue is the fact that while all file system paths can be represented as strings or bytes, not all strings or bytes represent a file system path. This can lead to issues where any e.g. string duck-types to a file system path whether it actually represents a path or not. To help elevate the representation of file system paths from their representation as strings and bytes to a richer object representation, the pathlib module [#pathlib]_ was provisionally introduced in Python 3.4 through PEP 428. While considered by some as an improvement over strings and bytes for file system paths, it has suffered from a lack of adoption. Typically the key issue listed for the low adoption rate has been the lack of support in the standard library. This lack of support required users of pathlib to manually convert path objects to strings by calling ``str(path)`` which many found error-prone. One issue in converting path objects to strings comes from the fact that the only generic way to get a string representation of the path was to pass the object to ``str()``. This can pose a problem when done blindly as nearly all Python objects have some string representation whether they are a path or not, e.g. ``str(None)`` will give a result that ``builtins.open()`` [#builtins-open]_ will happily use to create a new file. Exacerbating this whole situation is the ``DirEntry`` object [#os-direntry]_. While path objects have a representation that can be extracted using ``str()``, ``DirEntry`` objects expose a ``path`` attribute instead. Having no common interface between path objects, ``DirEntry``, and any other third-party path library has become an issue. A solution that allows any path-representing object to declare that it is a path and a way to extract a low-level representation that all path objects could support is desired. This PEP then proposes to introduce a new protocol to be followed by objects which represent file system paths. Providing a protocol allows for explicit signaling of what objects represent file system paths as well as a way to extract a lower-level representation that can be used with older APIs which only support strings or bytes. Discussions regarding path objects that led to this PEP can be found in multiple threads on the python-ideas mailing list archive [#python-ideas-archive]_ for the months of March and April 2016 and on the python-dev mailing list archives [#python-dev-archive]_ during April 2016. Proposal ======== This proposal is split into two parts. One part is the proposal of a protocol for objects to declare and provide support for exposing a file system path representation. The other part deals with changes to Python's standard library to support the new protocol. These changes will also lead to the pathlib module dropping its provisional status. Protocol -------- The following abstract base class defines the protocol for an object to be considered a path object:: import abc import typing as t class PathLike(abc.ABC): """Abstract base class for implementing the file system path protocol.""" @abc.abstractmethod def __fspath__(self) -> t.Union[str, bytes]: """Return the file system path representation of the object.""" raise NotImplementedError Objects representing file system paths will implement the ``__fspath__()`` method which will return the ``str`` or ``bytes`` representation of the path. The ``str`` representation is the preferred low-level path representation as it is human-readable and what people historically represent paths as. Standard library changes ------------------------ It is expected that most APIs in Python's standard library that currently accept a file system path will be updated appropriately to accept path objects (whether that requires code or simply an update to documentation will vary). The modules mentioned below, though, deserve specific details as they have either fundamental changes that empower the ability to use path objects, or entail additions/removal of APIs. builtins '''''''' ``open()`` [#builtins-open]_ will be updated to accept path objects as well as continue to accept ``str`` and ``bytes``. os ''' The ``fspath()`` function will be added with the following semantics:: import typing as t def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str, bytes]: """Return the string representation of the path. If str or bytes is passed in, it is returned unchanged. """ if isinstance(path, (str, bytes)): return path try: return path.__fspath__() except AttributeError: if hasattr(path, '__fspath__'): raise raise TypeError("expected str, bytes or path object, not " + type(path).__name__) The ``os.fsencode()`` [#os-fsencode]_ and ``os.fsdecode()`` [#os-fsdecode]_ functions will be updated to accept path objects. As both functions coerce their arguments to ``bytes`` and ``str``, respectively, they will be updated to call ``__fspath__()`` if present to convert the path object to a ``str`` or ``bytes`` representation, and then perform their appropriate coercion operations as if the return value from ``__fspath__()`` had been the original argument to the coercion function in question. The addition of ``os.fspath()``, the updates to ``os.fsencode()``/``os.fsdecode()``, and the current semantics of ``pathlib.PurePath`` provide the semantics necessary to get the path representation one prefers. For a path object, ``pathlib.PurePath``/``Path`` can be used. To obtain the ``str`` or ``bytes`` representation without any coersion, then ``os.fspath()`` can be used. If a ``str`` is desired and the encoding of ``bytes`` should be assumed to be the default file system encoding, then ``os.fsdecode()`` should be used. If a ``bytes`` representation is desired and any strings should be encoded using the default file system encoding, then ``os.fsencode()`` is used. This PEP recommends using path objects when possible and falling back to string paths as necessary and using ``bytes`` as a last resort. Another way to view this is as a hierarchy of file system path representations (highest- to lowest-level): path → str → bytes. The functions and classes under discussion can all accept objects on the same level of the hierarchy, but they vary in whether they promote or demote objects to another level. The ``pathlib.PurePath`` class can promote a ``str`` to a path object. The ``os.fspath()`` function can demote a path object to a ``str`` or ``bytes`` instance, depending on what ``__fspath__()`` returns. The ``os.fsdecode()`` function will demote a path object to a string or promote a ``bytes`` object to a ``str``. The ``os.fsencode()`` function will demote a path or string object to ``bytes``. There is no function that provides a way to demote a path object directly to ``bytes`` while bypassing string demotion. The ``DirEntry`` object [#os-direntry]_ will gain an ``__fspath__()`` method. It will return the same value as currently found on the ``path`` attribute of ``DirEntry`` instances. The Protocol_ ABC will be added to the ``os`` module under the name ``os.PathLike``. os.path ''''''' The various path-manipulation functions of ``os.path`` [#os-path]_ will be updated to accept path objects. For polymorphic functions that accept both bytes and strings, they will be updated to simply use ``os.fspath()``. During the discussions leading up to this PEP it was suggested that ``os.path`` not be updated using an "explicit is better than implicit" argument. The thinking was that since ``__fspath__()`` is polymorphic itself it may be better to have code working with ``os.path`` extract the path representation from path objects explicitly. There is also the consideration that adding support this deep into the low-level OS APIs will lead to code magically supporting path objects without requiring any documentation updated, leading to potential complaints when it doesn't work, unbeknownst to the project author. But it is the view of this PEP that "practicality beats purity" in this instance. To help facilitate the transition to supporting path objects, it is better to make the transition as easy as possible than to worry about unexpected/undocumented duck typing support for path objects by projects. There has also been the suggestion that ``os.path`` functions could be used in a tight loop and the overhead of checking or calling ``__fspath__()`` would be too costly. In this scenario only path-consuming APIs would be directly updated and path-manipulating APIs like the ones in ``os.path`` would go unmodified. This would require library authors to update their code to support path objects if they performed any path manipulations, but if the library code passed the path straight through then the library wouldn't need to be updated. It is the view of this PEP and Guido, though, that this is an unnecessary worry and that performance will still be acceptable. pathlib ''''''' The constructor for ``pathlib.PurePath`` and ``pathlib.Path`` will be updated to accept ``PathLike`` objects. Both ``PurePath`` and ``Path`` will continue to not accept ``bytes`` path representations, and so if ``__fspath__()`` returns ``bytes`` it will raise an exception. The ``path`` attribute will be removed as this PEP makes it redundant (it has not been included in any released version of Python and so is not a backwards-compatibility concern). C API ''''' The C API will gain an equivalent function to ``os.fspath()``:: /* Return the file system path of the object. If the object is str or bytes, then allow it to pass through with an incremented refcount. If the object defines __fspath__(), then return the result of that method. All other types raise a TypeError. */ PyObject * PyOS_FSPath(PyObject *path) { if (PyUnicode_Check(path) || PyBytes_Check(path)) { Py_INCREF(path); return path; } if (PyObject_HasAttrString(path, "__fspath__")) { return PyObject_CallMethodObjArgs(path, "__fspath__", NULL); } return PyErr_Format(PyExc_TypeError, "expected a str, bytes, or path object, not %S", path->ob_type); } Backwards compatibility ======================= There are no explicit backwards-compatibility concerns. Unless an object incidentally already defines a ``__fspath__()`` method there is no reason to expect the pre-existing code to break or expect to have its semantics implicitly changed. Libraries wishing to support path objects and a version of Python prior to Python 3.6 and the existence of ``os.fspath()`` can use the idiom of ``path.__fspath__() if hasattr(path, "__fspath__") else path``. Implementation ============== This is the task list for what this PEP proposes: #. Remove the ``path`` attribute from pathlib #. Remove the provisional status of pathlib #. Add ``os.PathLike`` #. Add ``os.fspath()`` #. Add ``PyOS_FSPath()`` #. Update ``os.fsencode()`` #. Update ``os.fsdecode()`` #. Update ``pathlib.PurePath`` and ``pathlib.Path`` #. Update ``builtins.open()`` #. Update ``os.DirEntry`` #. Update ``os.path`` Rejected Ideas ============== Other names for the protocol's method ------------------------------------- Various names were proposed during discussions leading to this PEP, including ``__path__``, ``__pathname__``, and ``__fspathname__``. In the end people seemed to gravitate towards ``__fspath__`` for being unambiguous without being unnecessarily long. Separate str/bytes methods -------------------------- At one point it was suggested that ``__fspath__()`` only return strings and another method named ``__fspathb__()`` be introduced to return bytes. The thinking is that by making ``__fspath__()`` not be polymorphic it could make dealing with the potential string or bytes representations easier. But the general consensus was that returning bytes will more than likely be rare and that the various functions in the os module are the better abstraction to promote over direct calls to ``__fspath__()``. Providing a ``path`` attribute ------------------------------ To help deal with the issue of ``pathlib.PurePath`` not inheriting from ``str``, originally it was proposed to introduce a ``path`` attribute to mirror what ``os.DirEntry`` provides. In the end, though, it was determined that a protocol would provide the same result while not directly exposing an API that most people will never need to interact with directly. Have ``__fspath__()`` only return strings ------------------------------------------ Much of the discussion that led to this PEP revolved around whether ``__fspath__()`` should be polymorphic and return ``bytes`` as well as ``str`` or only return ``str``. The general sentiment for this view was that ``bytes`` are difficult to work with due to their inherent lack of information about their encoding and PEP 383 makes it possible to represent all file system paths using ``str`` with the ``surrogateescape`` handler. Thus, it would be better to forcibly promote the use of ``str`` as the low-level path representation for high-level path objects. In the end, it was decided that using ``bytes`` to represent paths is simply not going to go away and thus they should be supported to some degree. The hope is that people will gravitate towards path objects like pathlib and that will move people away from operating directly with ``bytes``. A generic string encoding mechanism ----------------------------------- At one point there was a discussion of developing a generic mechanism to extract a string representation of an object that had semantic meaning (``__str__()`` does not necessarily return anything of semantic significance beyond what may be helpful for debugging). In the end, it was deemed to lack a motivating need beyond the one this PEP is trying to solve in a specific fashion. Have __fspath__ be an attribute ------------------------------- It was briefly considered to have ``__fspath__`` be an attribute instead of a method. This was rejected for two reasons. One, historically protocols have been implemented as "magic methods" and not "magic methods and attributes". Two, there is no guarantee that the lower-level representation of a path object will be pre-computed, potentially misleading users that there was no expensive computation behind the scenes in case the attribute was implemented as a property. This also indirectly ties into the idea of introducing a ``path`` attribute to accomplish the same thing. This idea has an added issue, though, of accidentally having any object with a ``path`` attribute meet the protocol's duck typing. Introducing a new magic method for the protocol helpfully avoids any accidental opting into the protocol. Provide specific type hinting support ------------------------------------- There was some consideration to provdinga generic ``typing.PathLike`` class which would allow for e.g. ``typing.PathLike[str]`` to specify a type hint for a path object which returned a string representation. While potentially beneficial, the usefulness was deemed too small to bother adding the type hint class. This also removed any desire to have a class in the ``typing`` module which represented the union of all acceptable path-representing types as that can be represented with ``typing.Union[str, bytes, os.PathLike]`` easily enough and the hope is users will slowly gravitate to path objects only. Acknowledgements ================ Thanks to everyone who participated in the various discussions related to this PEP that spanned both python-ideas and python-dev. Special thanks to Stephen Turnbull for direct feedback on early drafts of this PEP. More special thanks to Koos Zevenhoven and Ethan Furman for not only feedback on early drafts of this PEP but also helping to drive the overall discussion on this topic across the two mailing lists. References ========== .. [#python-ideas-archive] The python-ideas mailing list archive (https://mail.python.org/pipermail/python-ideas/) .. [#python-dev-archive] The python-dev mailing list archive (https://mail.python.org/pipermail/python-dev/) .. [#libc-open] ``open()`` documention for the C standard library ( http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files… ) .. [#pathlib] The ``pathlib`` module (https://docs.python.org/3/library/pathlib.html#module-pathlib) .. [#builtins-open] The ``builtins.open()`` function (https://docs.python.org/3/library/functions.html#open) .. [#os-fsencode] The ``os.fsencode()`` function (https://docs.python.org/3/library/os.html#os.fsencode) .. [#os-fsdecode] The ``os.fsdecode()`` function (https://docs.python.org/3/library/os.html#os.fsdecode) .. [#os-direntry] The ``os.DirEntry`` class (https://docs.python.org/3/library/os.html#os.DirEntry) .. [#os-path] The ``os.path`` module (https://docs.python.org/3/library/os.path.html#module-os.path) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

10 18