Mailman 3 September 2021 - Python-Dev

Should I care what -3.14 // inf produces?
by Jeff Allen Oct. 2, 2021

Oct. 2, 2021

Is an implementation of Python free to make up its own answers to division and modulus operatons in the case of inf and nan arguments? The standard library documentation says only that // is "rounded towards minus infinity". The language reference says that ||: 1. |x == (x//y)*y + (x%y)|, 2. the modulus has the same sign as y, and 3. division by (either kind of) zero raises |ZeroDivisionError| . It's consistent, but it doesn't define the operator over the full range of potential … [View More]

3 3

Python 3.10 release party
by Pablo Galindo Salgado Sept. 29, 2021

Sept. 29, 2021

Hi, As it seems that I didn't have enough work dealing with the multiple release blockers that have been raised these weeks I am also preparing a Python 3.10 release party. We will be doing an stream, co-hosted by the good people of Python discord channel. The URL for the stream is this one: https://youtu.be/AHT2l3hcIJg The release party will start on October 4th at 17:00 UTC+0. In this stream, some core developers (Irit, Carol, Brandt and Łukasz) will be talking about Python 3.10 features … [View More]

1 0

PEP: Taking the Python C API to the Next Level
by Victor Stinner Sept. 28, 2021

Sept. 28, 2021

Hi, I would like to change the Python C API. I failed to write a single document listing all constraints and proposing all changes that I would to do. For example, my previous PEP 620 contains too many changes and is too long. Here is my attempt to focus on the bare minimum and (what I consider as) the least controversial part: list current usages of C API and constraints of these usages. This *informal* PEP should be the base of future PEP changing the C API. The current draft lives at: … [View More]https://github.com/vstinner/misc/blob/main/cpython/pep-c-api-next-level.rst My PEP is based on HPy Next Level Manifesto written by Simon Cross: https://github.com/hpyproject/hpy/wiki/c-api-next-level-manifesto To reach most users of the C API, I cross-posted this email to python-dev, capi-sig and hpy-dev. +++++++++++++++++++++++++++++++++++++++++ Taking the Python C API to the Next Level +++++++++++++++++++++++++++++++++++++++++ :: PEP: xxx Title: Taking the Python C API to the Next Level Author: Victor Stinner <vstinner(a)python.org> Status: Draft Type: Informational Content-Type: text/x-rst Created: 28-Sep-2021 Python-Version: 3.11 While the C API is a key of the Python popularity, it causes multiple subtle and complex issues. There are different ways to use the C API, each usage has its own constraints, and some constraints are exclusive. This document lists constraints but doesn't propose changes, it only gives vague ideas how to solve some issues. More concrete C API changes will require writing separated PEPs. C extensions are a key component of the Python popularity ========================================================= The Python popularity comes from its great programming language and from its wide collection of modules freely available on PyPI. Many of the most popular Python modules rely directly or indirectly on C extensions written with the C API. The Python C API is a key component of the Python popularity. For example, the numpy project is now a common dependency on many scientific projects and a large part of the project is written by hand with the C API. **Abandoning or removing the C API** is out of question. Years ago, the incomplete C API support was the main drawback of PyPy, since PyPy only supported a minority of C extensions. Today, CPython still have a similar issue. **When Cython or numpy don't support a new Python version** (because of incompatible C API changes), many Python projects depending on them are cannot be installed, especially during the development phase of the next Python version. Backward compatibility and unmaintained C extensions ==================================================== One important property of the C API is the backward compatibility. Developers expect that if their C extension works on Python 3.10, it will work unmodified in Python 3.11: building the C extension with Python 3.11 should be enough. This property is even more important for unmaintained C extensions. Sometimes, unmaintained just means that the only maintainer is busy or overwhelmed for a few months. Sometimes, the project has no activity for longer than 5 years. When an incompatible change is introduced in the C API, like removing a function or changing a function behavior, there is a **risk of breaking an unknown number of C extensions**. One option can be to update old C extensions when they are built on recent Python versions, to adapt them to incompatible changes. This conversion is non trivial and cannot handle all kinds of incompatible changes. Migration plan for incompatible changes ======================================= There should be a **sensible migration path** for large C extensions (e.g. numpy) when incompatible changes are introduced. Whenever possible, it should be possible to write a **single code base** compatible with old and new Python versions. A **compatibility layer** can be maintained externally. Cython and numpy have their own internal compatibility layer. There should be a way to easily pick up common errors introduced by migrating. One practical way to **minimize the number of broken projects** is to attempt to check in advance if an incompatible change is going to break popular C extensions. For broken C extensions, propose a fix and wait until a new release includes the fix, before introducing the change in Python. Obviously, it doesn't solve the problem of less popular C extensions and private C extensions. Obtain the best possible performance ==================================== There are two main reasons for writing a C extension: implement a function which cannot be written in pure Python, or write a **C accelerator**: rewrite the 10% of an application in C where 90% of the CPU time is spent. About the former use case, the intent is to obtain the best possible performance. Tradeoffs are made with portability: it is acceptable to only support a limited number of Python versions and to only support a limited number of Python implementations (usually only CPython). Cython is a good example of accelerator. It is able to support a large number of Python versions and multiple Python implementation with compatibility layers and ``#ifdef``. The main drawback is that it is common that Cython is **broken by incompatible changes made at each Python release**. It happens because Cython relies on many implementation details. On the other side, the **limited C API** is a small as possible, excludes implementation details on purpose, and provides a stable ABI. Building a C extension with the limited C API only once produces a binary wheel package usable on many Python versions, but each platform still requires its own binary wheel package. Emulating the current C API is inefficient ========================================== The PyPy project is a Python implementation written from scratch, it was not created as a CPython fork. It made many implementation choices different than CPython: no reference counting, moving garbage collector, JIT compiler, etc. To support C extensions, PyPy emulates the Python C API in its cpyext module. When the C API access an object, cpyext has to convert the PyPy object to a CPython object (``PyObject``). CPython objects are less efficient than PyPy objects with the PyPy JIT compiler and conversions from PyPy objects to CPython objects are also inefficient. PyPy has to reimplement every single detail of the CPython implementation to be as much compatible as possible. The C API exposes multiple implementation details: * Reference counting, borrowed references, stealing references. * Objects location in memory. * Rely on pointers for object identity: Python 3.10 adds the ``Py_Is()`` function to solve this problem. * Expose the memory layout of Python objects as part of the API. * Expose static types. * Implicit execution context. * etc. The C API of Python 3.10 is made of around 15 000 lines of C header files, 1500 functions and 100 structures. Supporting the full C API is a significant amount of work. **Freezing the C API** for a few Python releases would help other Python implementations to catch up with the latest Python version, but it doesn't solve the efficiency problem. Moreover, it is common that adding a new feature to Python requires to change the C API, even if it is just to add new functions. Not adding new features to Python for a few Python releases is out of question. The C API prevents optimizing CPython ===================================== It is challenging to evolve the C API to optimize CPython without breaking the backward compatibility. Emulating the old C API is an option, but it is inefficient. If everything above is achievable -- and we believe it is! -- we'll arrive in a wonderful new future where Python implementations can experiment with all sorts of amazing new features: * tracing garbage collectors; * nurseries for short-lived objects; * sub-interpreters with separate contexts; * specialised implementations of lists; * removing the GIL; * avoiding the boxing of primitive types; * just-in-time compilation; * ... and many other things you can imagine that we haven't! No one can guarantee that a particular new idea will work out, but exposing fewer implementation details via the C API will make it possible to try many new things. -- Night gathers, and now my watch begins. It shall not end until my death. [View Less]

1 0

The Default for python -X frozen_modules.
by Eric Snow Sept. 28, 2021

Sept. 28, 2021

We've frozen most of the stdlib modules imported during "python -c pass" [1][2], to make startup a bit faster. Import of those modules is controlled by "-X frozen_modules=[on|off]". Currently it defaults to "off" but we'd like to default to "on". The blocker is the impact on contributors. I expect many will make changes to a stdlib module and then puzzle over why those changes aren't getting used. That's an annoyance we can avoid, which is the point of this thread. Possible solutions: 1. … [View More]

16 41

Re: The Default for python -X frozen_modules.
by Patrick Reader Sept. 27, 2021

Sept. 27, 2021

I accidentally a word. "... module's hash and/or *timestamp* and comparing it ..." On 27/09/2021 18:38, Patrick Reader wrote: > How about checking each non-frozen module's hash and/or and comparing it to that of the frozen module? Would that defeat the performance improvement of freezing? Is it just a terrible idea? > > On 27/09/2021 17:51, Eric Snow wrote: >> We've frozen most of the stdlib modules imported during "python -c >> pass" [1][2], to make startup a bit faster. … [View More]

2 1

Changing exception text in micro releases
by Eric V. Smith Sept. 25, 2021

Sept. 25, 2021

This PR https://github.com/python/cpython/pull/28310 changes the message for some exceptions. Currently: >>> format('', '%M') Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: Invalid format specifier With the proposed change: >>> format('', '%M') Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: Invalid format specifier '%M' for object of type 'str' This also … [View More]

6 7

Accepting PEP 654.
by Thomas Wouters Sept. 24, 2021

Sept. 24, 2021

Irit, Guido, Yuri, Nathaniel, (In the interest of visibility, this was also posted to discuss.python.org <https://discuss.python.org/t/accepting-pep-654-exception-groups-and-except/…> .) After careful consideration and rather lengthy discussion, the Steering Council has decided to accept PEP 654 (Exception Groups and except*) <https://www.python.org/dev/peps/pep-0654/>. We do have two requests, one of which we mentioned before. We’re very pleased with the amount of care and … [View More]discussion that went into this PEP, and we’re satisfied that it solves a real-world problem in the least disruptive way. We do remain open for further tweaking of the design, incorporating some of Nathaniel’s ideas, or new ones, but we want to move forward with PEP 654’s proposal and let people get hands-on experience with it. To that end, our request is that we get at least one non-toy use of ExceptionGroups somewhere in the standard library (targeted to 3.11, like PEP 654), and preferably get some third-party experimentation with it as well. The other request is that PEP 654 be updated with a reference to Nathaniel’s ideas (a link to the Discourse discussion), outlining why PEP 654 rejects those ideas (it could just be a copy-paste of some of the discussion, just so long as it’s recorded in the PEP). Regarding Nathaniel’s alternative proposal <https://discuss.python.org/t/flat-exception-groups-alternative-to-pep-654>, I want to be clear on the main reason why we’re accepting PEP 654 rather than waiting for a more complete alternative PEP with concrete examples. The SC agrees that exception groups solve a real problem, albeit a relatively rare and low-level one. We don’t think the problem it solves is big enough that it warrants fundamental changes to the language; changes that would affect the way everyone would write code, or that would make learning Python or reasoning about Python code harder. PEP 654 just about squeaks by those criteria: the new syntax is something new to learn about Python, but it’s not something that’s *required *unless consuming an API that uses ExceptionGroups. ExceptionGroups behave like regular exceptions in code that does not expect them, and do not change the behaviour of existing try/except. In code that potentially produces ExceptionGroups (or its documentation, examples, tutorials, etc) there is enough opportunity to document that, show examples of how to use it, explain the implications and refer to documentation about the concept of ExceptionGroups. By contrast, Nathaniel’s proposal that ExceptionGroups affect how *existing *try/except statements behave, fails those criteria. By sheer nature of doing something meaningful (if confusing) in try/except, it would encourage API designs that let ExceptionGroups escape to unsuspecting code. It would introduce subtle bugs in code that -- entirely legitimately -- expects only one except block to be executed, or only executed once. These problems would be very hard to detect since they would only show up if called code could *in practice* raise ExceptionGroups. We don’t consider this a viable design choice. The SC does remain open to proposals to extend PEP 654 (like adding an API to iterate over ExceptionGroups), or change some of the semantics (like flat versus nested ExceptionGroups), and we do not want to discourage discussion on any of this. We do want to move forward with PEP 654 as-is in the meantime, in particular to get practical experience in the standard library. With thanks for all your hard work, For the whole SC, Thomas. -- Thomas Wouters <thomas(a)python.org> Hi! I'm an email virus! Think twice before sending your email to help me spread! [View Less]

2 2

Summary of Python tracker Issues
by Python tracker Sept. 24, 2021

Sept. 24, 2021

ACTIVITY SUMMARY (2021-09-17 - 2021-09-24) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7333 (-48) closed 49684 (+97) total 57017 (+49) Open issues with patches: 2907 Issues opened (37) ================== #34451: docs: tutorial/introduction doesn't mention toggle of prompts https://bugs.python.org/issue34451 reopened by rhettinger #45237: … [View More]

1 0

RFC: New/update the Docs Sphinx theme: responsive, edit page links,
by Wes Turner Sept. 24, 2021

Sept. 24, 2021

Where would be the best place to discuss or just improve the CPython docs sphinx theme? What would Python need in a {PyData,} Sphinx theme fork: - [ ] ENH: add <edit on github> links to every docs page - [ ] ENH: responsive breakpoints for mobile devices - [ ] ENH: responsive fonts - [ ] ENH: toggle-able dark theme - [ ] ENH: interactive code examples (in WASM with pyodide like JupyterLite) - [ ] ENH?: Two sidebars: {Global TOC} {Content} {Document TOC} The PyData theme [bandwagon, … [View More]

2 1

f-strings in the grammar
by Pablo Galindo Salgado Sept. 24, 2021

Sept. 24, 2021

Hi, I have started a project to move the parsing off-strings to the parser and the grammar. Appart from some maintenance improvements (we can drop a considerable amount of hand-written code), there are some interesting things we **could** (emphasis on could) get out of this and I wanted to discuss what people think about them. * The parser will likely have "\n" characters and backslashes in f-strings expressions, which currently is impossible: >>> f"blah blah {'\n'} blah" File "&… [View More]

15 28