Mailman 3 February 2022 - Python-Dev

"immortal" objects and how they would help per-interpreter GIL
by Eric Snow 08 Feb '22

08 Feb '22

Most of the work toward interpreter isolation and a per-interpreter GIL involves moving static global variables to _PyRuntimeState or PyInterpreterState (or module state). Through the effort of quite a few people, we've made good progress. However, many globals still remain, with the majority being objects and most of those being static strings (e.g. _Py_Identifier), static types (incl. exceptions), and singletons. On top of that, a number of those objects are exposed in the public C-API and even in the limited API. :( Dealing with this specifically is probably the trickiest thing I've had to work through in this project. There is one solution that would help both of the above in a nice way: "immortal" objects. The idea of objects that never get deallocated isn't new and has been explored here several times. Not that long ago I tried it out by setting the refcount really high. That worked. Around the same time Eddie Elizondo at Facebook did something similar but modified Py_INCREF() and Py_DECREF() to keep the refcount from changing. Our solutions were similar but with different goals in mind. (Facebook wants to avoid copy-on-write in their pre-fork model.) A while back I concluded that neither approach would work for us. The approach I had taken would have significant cache performance penalties in a per-interpreter GIL world. The approach that modifies Py_INCREF() has a significant performance penalty due to the extra branch on such a frequent operation. Recently I've come back to the idea of immortal objects because it's much simpler than the alternate (working) solution I found. So how do we get around that performance penalty? Let's say it makes CPython 5% slower. We have some options: * live with the full penalty * make other changes to reduce the penalty to a more acceptable threshold than 5% * eliminate the penalty (e.g. claw back 5% elsewhere) * abandon all hope Mark Shannon suggested to me some things we can do. Also, from a recent conversation with Dino Viehland it sounds like Eddie was able to reach performance-neutral with a few techniques. So here are some things we can do to reduce or eliminate that penalty: * reduce refcount operations on high-activity objects (e.g. None, True, False) * reduce refcount operations in general * walk the heap at the end of runtime initialization and mark all objects as immortal * mark all global objects as immortal (statics or in _PyRuntimeState; for PyInterpreterState not needed) What do you think? Does this sound realistic? Are there additional things we can do to counter that penalty? -eric

16 35

Moving away from _Py_IDENTIFIER().
by Eric Snow 08 Feb '22

08 Feb '22

I'm planning on moving us to a simpler, more efficient alternative to _Py_IDENTIFIER(), but want to see if there are any objections first before moving ahead. Also see https://bugs.python.org/issue46541. _Py_IDENTIFIER() was added in 2011 to replace several internal string object caches and to support cleaning up the cached objects during finalization. A number of "private" functions (each with a _Py_Identifier param) were added at that time, mostly corresponding to existing functions that take PyObject* or char*. Note that at present there are several hundred uses of _Py_IDENTIFIER(), including a number of duplicates. My plan is to replace our use of _Py_IDENTIFIER() with statically initialized string objects (as fields under _PyRuntimeState). That involves the following: * add a PyUnicodeObject field (not a pointer) to _PyRuntimeState for each string that currently uses _Py_IDENTIFIER() (or _Py_static_string()) * statically initialize each object as part of the initializer for _PyRuntimeState * add a macro to look up a given global string * update each location that currently uses _Py_IDENTIFIER() to use the new macro instead Pros: * reduces indirection (and extra calls) for C-API functions that need the strings (making the code a little easier to understand and speeding it up) * the objects are referenced from a fixed address in the static data section instead of the heap (speeding things up and allowing the C compiler to optimize better) * there is no lazy allocation (or lookup, etc.) so there are fewer possible failures when the objects get used (thus less error return checking) * saves memory (at little, at least) * if needed, the approach for per-interpreter is simpler * helps us get rid of several hundred static variables throughout the code base * allows us to get rid of _Py_IDENTIFIER() and a bunch of related C-API functions * "deep frozen" modules can use the global strings * commonly-used strings could be pre-allocated by adding _PyRuntimeState fields for them Cons: * a little less convenient: adding a global string requires modifying a separate file from the one where you actually want to use the string * strings can get "orphaned" (I'm planning on checking in CI) * some strings may never get used for any given ./python invocation (not that big a difference though) I have a PR up (https://github.com/python/cpython/pull/30928) that adds the global strings and replaces use of _Py_IDENTIFIER() in our code base, except for in non-builtin stdlib extension modules. (Those will be handled separately if we proceed.) The PR also adds a CI check for "orphaned" strings. It leaves _Py_IDENTIFIER() for now, but disallows any Py_BUILD_CORE code from using it. With that change I'm seeing a 1% improvement in performance (see https://github.com/faster-cpython/ideas/issues/230). I'd also like to actually get rid of _Py_IDENTIFIER(), along with other related API including ~14 (private) C-API functions. Dropping all that helps reduce maintenance costs. However, at least one PyPI project (blender) is using _Py_IDENTIFIER(). So, before we could get rid of it, we'd first have to deal with that project (and any others). To sum up, I wanted to see if there are any objections before I start merging anything. Thanks! -eric

12 36

Python-Dev] Replace debug runtime checks in release mode, with assertions in debug mode
by Matti Picus 07 Feb '22

07 Feb '22

On 7/2/22 17:55, Victor Stinner wrote: > End users should not have to > pay the price of these runtime checks. Is the cost of the checks measurable and significant? Surely sanitizing arguments to prevent segfaults is worth a few nano-seconds? Matti

1 0

PEP 674 – Disallow using macros as l-value (version 2)
by Victor Stinner 07 Feb '22

07 Feb '22

Hi, I made the following changes to the PEP since the version 1 (November 30): * Elaborate the HPy section and add a new "Relationship with the HPy project" section * Elaborate which projects are impacted and which changes are needed * Remove the PyPy section The PEP can be read online: https://python.github.io/peps/ep-0674/ And you can find the plain text below. Victor PEP: 674 Title: Disallow using macros as l-value Author: Victor Stinner <vstinner(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30-Nov-2021 Python-Version: 3.11 Abstract ======== Incompatible C API change disallowing using macros as l-value to: * Allow evolving CPython internals; * Ease the C API implementation on other Python implementation; * Help migrating existing C extensions to the HPy API. On the PyPI top 5000 projects, only 14 projects (0.3%) are affected by 4 macro changes. Moreover, 24 projects just have to regenerate their Cython code to use ``Py_SET_TYPE()``. In practice, the majority of affected projects only have to make two changes: * Replace ``Py_TYPE(obj) = new_type;`` with ``Py_SET_TYPE(obj, new_type);``. * Replace ``Py_SIZE(obj) = new_size;`` with ``Py_SET_SIZE(obj, new_size);``. Rationale ========= Using a macro as a l-value -------------------------- In the Python C API, some functions are implemented as macro because writing a macro is simpler than writing a regular function. If a macro exposes directly a structure member, it is technically possible to use this macro to not only get the structure member but also set it. Example with the Python 3.10 ``Py_TYPE()`` macro:: #define Py_TYPE(ob) (((PyObject *)(ob))->ob_type) This macro can be used as a **r-value** to **get** an object type:: type = Py_TYPE(object); It can also be used as **l-value** to **set** an object type:: Py_TYPE(object) = new_type; It is also possible to set an object reference count and an object size using ``Py_REFCNT()`` and ``Py_SIZE()`` macros. Setting directly an object attribute relies on the current exact CPython implementation. Implementing this feature in other Python implementations can make their C API implementation less efficient. CPython nogil fork ------------------ Sam Gross forked Python 3.9 to remove the GIL: the `nogil branch <https://github.com/colesbury/nogil/>`_. This fork has no ``PyObject.ob_refcnt`` member, but a more elaborated implementation for reference counting, and so the ``Py_REFCNT(obj) = new_refcnt;`` code fails with a compiler error. Merging the nogil fork into the upstream CPython main branch requires first to fix this C API compatibility issue. It is a concrete example of a Python optimization blocked indirectly by the C API. This issue was already fixed in Python 3.10: the ``Py_REFCNT()`` macro has been already modified to disallow using it as a l-value. These statements are endorsed by Sam Gross (nogil developer). HPy project ----------- The `HPy project <https://hpyproject.org/>`_ is a brand new C API for Python using only handles and function calls: handles are opaque, structure members cannot be accessed directly, and pointers cannot be dereferenced. Searching and replacing ``Py_SET_SIZE()`` is easier and safer than searching and replacing some strange macro uses of ``Py_SIZE()``. ``Py_SIZE()`` can be semi-mechanically replaced by ``HPy_Length()``, whereas seeing ``Py_SET_SIZE()`` would immediately make clear that the code needs bigger changes in order to be ported to HPy (for example by using ``HPyTupleBuilder`` or ``HPyListBuilder``). The fewer internal details exposed via macros, the easier it will be for HPy to provide direct equivalents. Any macro that references "non-public" interfaces effectively exposes those interfaces publicly. These statements are endorsed by Antonio Cuni (HPy developer). GraalVM Python -------------- In GraalVM, when a Python object is accessed by the Python C API, the C API emulation layer has to wrap the GraalVM objects into wrappers that expose the internal structure of the CPython structures (PyObject, PyLongObject, PyTypeObject, etc). This is because when the C code accesses it directly or via macros, all GraalVM can intercept is a read at the struct offset, which has to be mapped back to the representation in GraalVM. The smaller the "effective" number of exposed struct members (by replacing macros with functions), the simpler GraalVM wrappers can be. This PEP alone is not enough to get rid of the wrappers in GraalVM, but it is a step towards this long term goal. GraalVM already supports HPy which is a better solution in the long term. These statements are endorsed by Tim Felgentreff (GraalVM Python developer). Specification ============= Disallow using macros as l-value -------------------------------- PyObject and PyVarObject macros ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * ``Py_TYPE()``: ``Py_SET_TYPE()`` must be used instead * ``Py_SIZE()``: ``Py_SET_SIZE()`` must be used instead "GET" macros ^^^^^^^^^^^^ * ``PyByteArray_GET_SIZE()`` * ``PyBytes_GET_SIZE()`` * ``PyCFunction_GET_CLASS()`` * ``PyCFunction_GET_FLAGS()`` * ``PyCFunction_GET_FUNCTION()`` * ``PyCFunction_GET_SELF()`` * ``PyCell_GET()`` * ``PyCode_GetNumFree()`` * ``PyDict_GET_SIZE()`` * ``PyFunction_GET_ANNOTATIONS()`` * ``PyFunction_GET_CLOSURE()`` * ``PyFunction_GET_CODE()`` * ``PyFunction_GET_DEFAULTS()`` * ``PyFunction_GET_GLOBALS()`` * ``PyFunction_GET_KW_DEFAULTS()`` * ``PyFunction_GET_MODULE()`` * ``PyHeapType_GET_MEMBERS()`` * ``PyInstanceMethod_GET_FUNCTION()`` * ``PyList_GET_SIZE()`` * ``PyMemoryView_GET_BASE()`` * ``PyMemoryView_GET_BUFFER()`` * ``PyMethod_GET_FUNCTION()`` * ``PyMethod_GET_SELF()`` * ``PySet_GET_SIZE()`` * ``PyTuple_GET_SIZE()`` * ``PyUnicode_GET_DATA_SIZE()`` * ``PyUnicode_GET_LENGTH()`` * ``PyUnicode_GET_LENGTH()`` * ``PyUnicode_GET_SIZE()`` * ``PyWeakref_GET_OBJECT()`` "AS" macros ^^^^^^^^^^^ * ``PyByteArray_AS_STRING()`` * ``PyBytes_AS_STRING()`` * ``PyFloat_AS_DOUBLE()`` * ``PyUnicode_AS_DATA()`` * ``PyUnicode_AS_UNICODE()`` PyUnicode macros ^^^^^^^^^^^^^^^^ * ``PyUnicode_1BYTE_DATA()`` * ``PyUnicode_2BYTE_DATA()`` * ``PyUnicode_4BYTE_DATA()`` * ``PyUnicode_DATA()`` * ``PyUnicode_IS_ASCII()`` * ``PyUnicode_IS_COMPACT()`` * ``PyUnicode_IS_READY()`` * ``PyUnicode_KIND()`` * ``PyUnicode_READ()`` * ``PyUnicode_READ_CHAR()`` PyDateTime "GET" macros ^^^^^^^^^^^^^^^^^^^^^^^ * ``PyDateTime_DATE_GET_FOLD()`` * ``PyDateTime_DATE_GET_HOUR()`` * ``PyDateTime_DATE_GET_MICROSECOND()`` * ``PyDateTime_DATE_GET_MINUTE()`` * ``PyDateTime_DATE_GET_SECOND()`` * ``PyDateTime_DATE_GET_TZINFO()`` * ``PyDateTime_DELTA_GET_DAYS()`` * ``PyDateTime_DELTA_GET_MICROSECONDS()`` * ``PyDateTime_DELTA_GET_SECONDS()`` * ``PyDateTime_GET_DAY()`` * ``PyDateTime_GET_MONTH()`` * ``PyDateTime_GET_YEAR()`` * ``PyDateTime_TIME_GET_FOLD()`` * ``PyDateTime_TIME_GET_HOUR()`` * ``PyDateTime_TIME_GET_MICROSECOND()`` * ``PyDateTime_TIME_GET_MINUTE()`` * ``PyDateTime_TIME_GET_SECOND()`` * ``PyDateTime_TIME_GET_TZINFO()`` PyDescr macros ^^^^^^^^^^^^^^ * ``PyDescr_NAME()`` * ``PyDescr_TYPE()`` Port C extensions to Python 3.11 -------------------------------- In practice, the majority of projects affected by these PEP only have to make two changes: * Replace ``Py_TYPE(obj) = new_type;`` with ``Py_SET_TYPE(obj, new_type);``. * Replace ``Py_SIZE(obj) = new_size;`` with ``Py_SET_SIZE(obj, new_size);``. The `pythoncapi_compat project <https://github.com/pythoncapi/pythoncapi_compat>`_ can be used to update automatically C extensions: add Python 3.11 support without losing support with older Python versions. The project provides a header file which provides ``Py_SET_REFCNT()``, ``Py_SET_TYPE()`` and ``Py_SET_SIZE()`` functions to Python 3.8 and older. PyTuple_GET_ITEM() and PyList_GET_ITEM() ---------------------------------------- The ``PyTuple_GET_ITEM()`` and ``PyList_GET_ITEM()`` macros are left unchanged. The code pattern ``&PyTuple_GET_ITEM(tuple, 0)`` and ``&PyList_GET_ITEM(list, 0)`` is still commonly used to get access to the inner ``PyObject**`` array. Changing these macros is out of the scope of this PEP. Backwards Compatibility ======================= The proposed C API changes are backward incompatible on purpose. At December 1, 2021, a code search on the PyPI top 5000 projects (4760 projects in practice, others don't have a source archive) found that `only 14 projects are affected <https://bugs.python.org/issue45476#msg407456>`_ (0.3%): * datatable (1.0.0) * frozendict (2.1.1) * guppy3 (3.1.2) * M2Crypto (0.38.0) * mecab-python3 (1.0.4) * mypy (0.910) * Naked (0.1.31) * pickle5 (0.0.12) * pysha3 (1.0.2) * python-snappy (0.6.0) * recordclass (0.16.3) * scipy (1.7.3) * zodbpickle (2.2.0) * zstd (1.5.0.2) These 14 projects only use 4 macros as l-value: * ``PyDescr_NAME()`` and ``PyDescr_TYPE()`` (2 projects) * ``Py_SIZE()`` (8 projects) * ``Py_TYPE()`` (4 projects) Moreover, `24 projects just have to regenerate their Cython code <https://bugs.python.org/issue45476#msg407416>`_ to use ``Py_SET_TYPE()``. This change does not follow the PEP 387 deprecation process. There is no known way to emit a deprecation warning only when a macro is used as a l-value, but not when it's used differently (ex: as a r-value). Relationship with the HPy project ================================= The HPy project --------------- The hope with the HPy project is indeed to provide a C API that is close to the original API to make porting easy and have it perform as close to the existing API as possible. At the same time, HPy is sufficently removed to be a good "C extension API" (as opposed to a stable subset of the CPython implementation API) that does not leak implementation details. To ensure this latter property is why the HPy project tries to develop everything in parallel for CPython, PyPy, and GraalVM Python. HPy is still evolving very fast. Issues are still being solved while migrating NumPy, and work has begun on adding support for HPy to Cython. Work on pybind11 is starting soon. Tim Felgentreff believes by the time HPy has these users of the existing C API working, HPy should be in a state where it is generally useful and can be deemed stable enough that further development can follow a more stable process. In the long run the HPy project would like to become a promoted API to write Python C extensions. The HPy project is a good solution for the long term. It has the advantage of being developed outside Python and it doesn't require any C API change. The C API is here is stay for a few more years ---------------------------------------------- The first concern about HPy is that right now, HPy is not mature nor widely used, and CPython still has to continue supporting a large amount of C extensions which are not likely to be ported to HPy soon. The second concern is the inability to evolve CPython internals to implement new optimizations, and the inefficient implementation of the current C API in PyPy, GraalPython, etc. Sadly, HPy will only solve these problems when most C extensions will be fully ported to HPy: when it will become reasonable to consider dropping the "legacy" Python C API. While porting a C extension to HPy can be done incrementally on CPython, it requires to modify a lot of code and takes time. Porting most C extensions to HPy is expected to take a few years. This PEP proposes to make the C API "less bad" by fixing one problem which is clearily identified as causing practical issues: macros used as l-values. This PEP only requires to update a minority of C extensions, and usually only a few lines need to be changed in impacted extensions. For example, numpy 1.22 is made of 307,300 lines of C code, and adapting numpy to the this PEP only modified 11 lines (use Py_SET_TYPE and Py_SET_SIZE) and adding 4 lines (to define Py_SET_TYPE and Py_SET_SIZE for Python 3.8 and older). Porting numpy to HPy is expected to require modifying more lines than that. Right now, it's hard to bet which approach is the best: fixing the current C API, or focusing on HPy. It would be risky to only focus on HPy. Rejected Idea: Leave the macros as they are =========================================== The documentation of each function can discourage developers to use macros to modify Python objects. If these is a need to make an assignment, a setter function can be added and the macro documentation can require to use the setter function. For example, a ``Py_SET_TYPE()`` function has been added to Python 3.9 and the ``Py_TYPE()`` documentation now requires to use the ``Py_SET_TYPE()`` function to set an object type. If developers use macros as l-value, it's their responsibility when their code breaks, not the Python responsibility. We are operating under the consenting adults principle: we expect users of the Python C API to use it as documented and expect them to take care of the fallout, if things break when they don't. This idea was rejected because only few developers read the documentation, and only a minority is tracking changes of the Python C API documentation. The majority of developers are only using CPython and so are not aware of compatibility issues with other Python implementations. Moreover, continuing to allow using macros as l-value does not help the HPy project and leaves the burden of emulating them on GraalVM's Python implementation. Macros already modified ======================= The following C API macros have already been modified to disallow using them as l-value: * ``PyCell_SET()`` * ``PyList_SET_ITEM()`` * ``PyTuple_SET_ITEM()`` * ``Py_REFCNT()`` (Python 3.10): ``Py_SET_REFCNT()`` must be used * ``_PyGCHead_SET_FINALIZED()`` * ``_PyGCHead_SET_NEXT()`` * ``asdl_seq_GET()`` * ``asdl_seq_GET_UNTYPED()`` * ``asdl_seq_LEN()`` * ``asdl_seq_SET()`` * ``asdl_seq_SET_UNTYPED()`` For example, ``PyList_SET_ITEM(list, 0, item) < 0`` now fails with a compiler error as expected. Discussion ========== * `PEP 674: Disallow using macros as l-value <https://mail.python.org/archives/list/python-dev@python.org/thread/KPIJPPJ6…>`_ (Nov 30, 2021) References ========== * `Python C API: Add functions to access PyObject <https://vstinner.github.io/c-api-abstract-pyobject.html>`_ (October 2021) article by Victor Stinner * `[C API] Disallow using PyFloat_AS_DOUBLE() as l-value <https://bugs.python.org/issue45476>`_ (October 2021) * `[capi-sig] Py_TYPE() and Py_SIZE() become static inline functions <https://mail.python.org/archives/list/capi-sig@python.org/thread/WGRLTHTHC3…>`_ (September 2021) * `[C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type <https://bugs.python.org/issue39573>`__ (February 2020) * `bpo-30459: PyList_SET_ITEM could be safer <https://bugs.python.org/issue30459>`_ (May 2017) Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

3 7

I want to contribute to Python.
by Ezekiel Adetoro 06 Feb '22

06 Feb '22

Hello, My name is Ezekiel, and it is my desire to start contributing to Python, be part of the core development of Python. I have forked the CPython and cloned it. What is the next step I need to do?

3 2

PyPy on PySide6 is there: PyPy with a Gui
by Christian Tismer-Sperling 05 Feb '22

05 Feb '22

Hi Guido et. al., since May 2021 I have been working at running PyPy on PySide6, which was a difficult undertaking, since PyPy internally is quite a bit different. I declared the project to be ready-to-use when the Mandelbrot example of the PySide examples (examples/corelib/threads/mandelbrot.py) is working. This was finally solved this week on 2022-02-01, so we have the first advanced Gui working with PyPy and with the amazing result of speed: PyPy 3.8 works 10 times faster than the identical code on Python 3.10 and 5.5 times slower than the same example in C++ Qt. I think to send an official announce when this is available on pip. This effort marks the completion of my PyPy support, which began in 2003 and ended involuntarily in 2006 due to a stroke. All the best -- Chris -- Christian Tismer-Sperling :^) tismer(a)stackless.com Software Consulting : http://www.stackless.com/ Strandstraße 37 : https://github.com/PySide 24217 Schönberg : GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023

3 3

Summary of Python tracker Issues
by Python tracker 04 Feb '22

04 Feb '22

ACTIVITY SUMMARY (2022-01-28 - 2022-02-04) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7144 (-13) closed 51222 (+84) total 58366 (+71) Open issues with patches: 2890 Issues opened (44) ================== #45711: Simplify the interpreter's (type, val, tb) exception represent https://bugs.python.org/issue45711 reopened by vstinner #45773: Compiler hangs during jump elimination https://bugs.python.org/issue45773 reopened by brandtbucher #46570: Windows support for OpenSSL 3.0 https://bugs.python.org/issue46570 opened by jay0lee #46571: Strange `(a)typing.no_type_check` behavior for class variables https://bugs.python.org/issue46571 opened by sobolevn #46575: One-off errors in hashlib.scrypt error messages https://bugs.python.org/issue46575 opened by ron_kaminsky #46577: Hostname spoofing via backslashes in URL https://bugs.python.org/issue46577 opened by meetdash #46578: cant DEBUG os.spawnv() https://bugs.python.org/issue46578 opened by michaellongge #46580: email.utils.unquote strips too many slashes https://bugs.python.org/issue46580 opened by hbandi #46581: _typevar_types and _paramspec_tvars are missing from _GenericA https://bugs.python.org/issue46581 opened by posita #46585: Should we re-export `PyObj_FromPtr` in `ctypes`? https://bugs.python.org/issue46585 opened by sobolevn #46586: In documentation contents enum.property erroneously links to b https://bugs.python.org/issue46586 opened by Dutcho #46587: datetime and time tests use non-portal "%4Y" format https://bugs.python.org/issue46587 opened by christian.heimes #46589: Improve documentation for typing._GenericAlias https://bugs.python.org/issue46589 opened by matthew.rahtz #46592: Undocumented behavior in strptime for ISO week dates https://bugs.python.org/issue46592 opened by Jonatan Skogsfors #46593: memoryview lacks support for half floats https://bugs.python.org/issue46593 opened by pitrou #46594: Windows "Edit with IDLE >" only has one selection https://bugs.python.org/issue46594 opened by terry.reedy #46596: PyLineTable_InitAddressRange isn't exported - causing C Extens https://bugs.python.org/issue46596 opened by nathan3 #46598: ElementTree: wrong XML prolog for the utf-8-sig encoding https://bugs.python.org/issue46598 opened by prikryl #46600: Python built with clang -O0 allocates 10x more stack memory th https://bugs.python.org/issue46600 opened by vstinner #46601: macOS installer "Install Certificates.command" fails if pip is https://bugs.python.org/issue46601 opened by cryptophoto #46603: `typing._strip_annotations` is not fully covered https://bugs.python.org/issue46603 opened by sobolevn #46604: Documentation fix in ssl module https://bugs.python.org/issue46604 opened by glk0 #46605: Py_XDECREF() module on fail in Py_mod_exec https://bugs.python.org/issue46605 opened by ov2k #46606: Large C stack usage of os.getgroups() and os.setgroups() https://bugs.python.org/issue46606 opened by methane #46607: Add DeprecationWarning to configparser's LegacyInterpolation https://bugs.python.org/issue46607 opened by hugovk #46608: Exclude marshalled-frozen data if deep-freezing to save 300 KB https://bugs.python.org/issue46608 opened by kumaraditya303 #46609: Generator-based coroutines in Python 3.10 docs https://bugs.python.org/issue46609 opened by srittau #46611: Improve coverage of `__instancecheck__` and `__subclasscheck__ https://bugs.python.org/issue46611 opened by sobolevn #46613: Add PyType_GetModuleByDef to the public & limited API https://bugs.python.org/issue46613 opened by petr.viktorin #46614: Add option to output UTC datetimes as "Z" in `.isoformat()` https://bugs.python.org/issue46614 opened by p-ganssle #46615: Use-after-free by mutating set during set operations https://bugs.python.org/issue46615 opened by Dennis Sweeney #46619: lazy module property not recognized by doctests https://bugs.python.org/issue46619 opened by jaraco #46620: Documentation of ipaddress behavior for prefix length with lea https://bugs.python.org/issue46620 opened by lay20114 #46621: Should map(function, iterable, ...) replace StopIteration with https://bugs.python.org/issue46621 opened by xavieryao #46622: Support decorating a coroutine with functools.cached_property https://bugs.python.org/issue46622 opened by uranusjr #46623: test_zlib: test_pair() and test_speech128() fail with s390x ha https://bugs.python.org/issue46623 opened by vstinner #46625: timeout option of socket.create_connection is not respected https://bugs.python.org/issue46625 opened by Nicolas SURRIBAS #46631: Implement a "strict" mode for getpass.getuser() https://bugs.python.org/issue46631 opened by eryksun #46632: test_ssl: 2 tests fail on cstratak-CentOS9-fips-x86_64 https://bugs.python.org/issue46632 opened by vstinner #46633: AddressSanitizer: Skip tests directly in Python, not with exte https://bugs.python.org/issue46633 opened by vstinner #46634: [sqlite3]��speed up cursor.execute*() https://bugs.python.org/issue46634 opened by erlendaasland #46635: unittest.defaultTestLoader.discover fails for namespace packag https://bugs.python.org/issue46635 opened by tpapastylianou #46637: Incorrect error message: "missing 1 required positional argume https://bugs.python.org/issue46637 opened by Anders.Hovm��ller #46638: Inconsistent registry virtualization in Windows Store package https://bugs.python.org/issue46638 opened by steve.dower Most recent 15 issues with no replies (15) ========================================== #46635: unittest.defaultTestLoader.discover fails for namespace packag https://bugs.python.org/issue46635 #46634: [sqlite3]��speed up cursor.execute*() https://bugs.python.org/issue46634 #46632: test_ssl: 2 tests fail on cstratak-CentOS9-fips-x86_64 https://bugs.python.org/issue46632 #46631: Implement a "strict" mode for getpass.getuser() https://bugs.python.org/issue46631 #46625: timeout option of socket.create_connection is not respected https://bugs.python.org/issue46625 #46623: test_zlib: test_pair() and test_speech128() fail with s390x ha https://bugs.python.org/issue46623 #46622: Support decorating a coroutine with functools.cached_property https://bugs.python.org/issue46622 #46620: Documentation of ipaddress behavior for prefix length with lea https://bugs.python.org/issue46620 #46619: lazy module property not recognized by doctests https://bugs.python.org/issue46619 #46614: Add option to output UTC datetimes as "Z" in `.isoformat()` https://bugs.python.org/issue46614 #46613: Add PyType_GetModuleByDef to the public & limited API https://bugs.python.org/issue46613 #46611: Improve coverage of `__instancecheck__` and `__subclasscheck__ https://bugs.python.org/issue46611 #46609: Generator-based coroutines in Python 3.10 docs https://bugs.python.org/issue46609 #46606: Large C stack usage of os.getgroups() and os.setgroups() https://bugs.python.org/issue46606 #46605: Py_XDECREF() module on fail in Py_mod_exec https://bugs.python.org/issue46605 Most recent 15 issues waiting for review (15) ============================================= #46638: Inconsistent registry virtualization in Windows Store package https://bugs.python.org/issue46638 #46634: [sqlite3]��speed up cursor.execute*() https://bugs.python.org/issue46634 #46623: test_zlib: test_pair() and test_speech128() fail with s390x ha https://bugs.python.org/issue46623 #46615: Use-after-free by mutating set during set operations https://bugs.python.org/issue46615 #46613: Add PyType_GetModuleByDef to the public & limited API https://bugs.python.org/issue46613 #46611: Improve coverage of `__instancecheck__` and `__subclasscheck__ https://bugs.python.org/issue46611 #46608: Exclude marshalled-frozen data if deep-freezing to save 300 KB https://bugs.python.org/issue46608 #46607: Add DeprecationWarning to configparser's LegacyInterpolation https://bugs.python.org/issue46607 #46606: Large C stack usage of os.getgroups() and os.setgroups() https://bugs.python.org/issue46606 #46604: Documentation fix in ssl module https://bugs.python.org/issue46604 #46603: `typing._strip_annotations` is not fully covered https://bugs.python.org/issue46603 #46600: Python built with clang -O0 allocates 10x more stack memory th https://bugs.python.org/issue46600 #46589: Improve documentation for typing._GenericAlias https://bugs.python.org/issue46589 #46581: _typevar_types and _paramspec_tvars are missing from _GenericA https://bugs.python.org/issue46581 #46571: Strange `(a)typing.no_type_check` behavior for class variables https://bugs.python.org/issue46571 Top 10 most discussed issues (10) ================================= #45711: Simplify the interpreter's (type, val, tb) exception represent https://bugs.python.org/issue45711 13 msgs #46600: Python built with clang -O0 allocates 10x more stack memory th https://bugs.python.org/issue46600 13 msgs #46571: Strange `(a)typing.no_type_check` behavior for class variables https://bugs.python.org/issue46571 11 msgs #29688: Add support for Path.absolute() https://bugs.python.org/issue29688 7 msgs #46521: codeop._maybe_compile passes code with error + triple quotes https://bugs.python.org/issue46521 7 msgs #46555: Unicode-mangled names refer inconsistently to constants https://bugs.python.org/issue46555 7 msgs #46615: Use-after-free by mutating set during set operations https://bugs.python.org/issue46615 7 msgs #42238: Deprecate suspicious.py? https://bugs.python.org/issue42238 6 msgs #45975: Simplify some while-loops with walrus operator https://bugs.python.org/issue45975 6 msgs #33222: Various test failures if PYTHONUSERBASE is not canonicalized https://bugs.python.org/issue33222 5 msgs Issues closed (73) ================== #6682: Default traceback does not handle PEP302 loaded modules https://bugs.python.org/issue6682 closed by iritkatriel #12010: Compile fails when sizeof(wchar_t) == 1 https://bugs.python.org/issue12010 closed by iritkatriel #26276: Inconsistent behaviour of PEP 3101 formatting between versions https://bugs.python.org/issue26276 closed by eric.smith #26552: Failing ensure_future still creates a Task https://bugs.python.org/issue26552 closed by gvanrossum #27101: Compilation of python (modules) for foreign target platform pr https://bugs.python.org/issue27101 closed by iritkatriel #29155: test.test_spwd.TestSpwdNonRoot failure with FileNotFoundError https://bugs.python.org/issue29155 closed by iritkatriel #29172: blake2: Use lowest-common denominator signature of #pragma pac https://bugs.python.org/issue29172 closed by iritkatriel #30210: No Documentation on tkinter dnd module https://bugs.python.org/issue30210 closed by iritkatriel #30393: test_readline hangs https://bugs.python.org/issue30393 closed by iritkatriel #36346: Prepare for removing the legacy Unicode C API https://bugs.python.org/issue36346 closed by methane #37588: Py_DEPRECATED and unavoidable warnings https://bugs.python.org/issue37588 closed by iritkatriel #38682: struct timeval is not declared https://bugs.python.org/issue38682 closed by iritkatriel #38902: image/webp support in mimetypes https://bugs.python.org/issue38902 closed by eric.araujo #39339: ProcessPoolExecutor() Exception in thread QueueManagerThread https://bugs.python.org/issue39339 closed by iritkatriel #39711: SIGBUS and core dumped during tests of 3.8.1 https://bugs.python.org/issue39711 closed by iritkatriel #40746: test_gdb failing on 32-bit armv7l when built with GCC -Og: <cl https://bugs.python.org/issue40746 closed by gregory.p.smith #42784: issues with object.h includes https://bugs.python.org/issue42784 closed by iritkatriel #43012: Remove pathlib accessors https://bugs.python.org/issue43012 closed by eric.araujo #43320: test test_webbrowser "can't locate runnable browser" with enab https://bugs.python.org/issue43320 closed by iritkatriel #43359: Dead assignment in Py_UniversalNewlineFgets https://bugs.python.org/issue43359 closed by iritkatriel #43540: importlib: Document how to replace load_module() in What's New https://bugs.python.org/issue43540 closed by vstinner #43583: make test failures, 2 tests failed: test_embed test_tabnan https://bugs.python.org/issue43583 closed by iritkatriel #44031: test_embed and test_tabnanny fails if the current directory is https://bugs.python.org/issue44031 closed by iritkatriel #44359: test_ftplib.test_makeport() fails as "env changes" if a socket https://bugs.python.org/issue44359 closed by vstinner #44977: Deprecate delegation of int to __trunc__ https://bugs.python.org/issue44977 closed by serhiy.storchaka #45459: Limited API support for Py_buffer https://bugs.python.org/issue45459 closed by christian.heimes #45555: Object stays alive for weak reference if an exception happens https://bugs.python.org/issue45555 closed by iritkatriel #45703: importlib.invalidate_caches() does not invalidate _NamespacePa https://bugs.python.org/issue45703 closed by petr.viktorin #45925: Upgrade macOS and Windows installers to use SQLite 3.37.2 https://bugs.python.org/issue45925 closed by ned.deily #45990: Exception notes need more documentation https://bugs.python.org/issue45990 closed by iritkatriel #46101: argparse: using parents & subcommands, options can be ignored https://bugs.python.org/issue46101 closed by paul.j3 #46440: ArgumentParser.parse_args exits on missing required argument w https://bugs.python.org/issue46440 closed by paul.j3 #46451: Tracing causes module globals to be mutated when calling funct https://bugs.python.org/issue46451 closed by seberg #46458: Optimise try-except code generation for the happy path https://bugs.python.org/issue46458 closed by iritkatriel #46482: `typing.Annotation.__new__` is not covered https://bugs.python.org/issue46482 closed by sobolevn #46483: `pathlib.PurePath.__class_getitem__` does not return `GenericA https://bugs.python.org/issue46483 closed by serhiy.storchaka #46487: `_SSLProtocolTransport` doesn't have the `get_write_buffer_lim https://bugs.python.org/issue46487 closed by asvetlov #46489: webbrowser crashes Ubuntu kernel https://bugs.python.org/issue46489 closed by dizcza #46524: test_peg_generator takes 8 minutes on Windows https://bugs.python.org/issue46524 closed by gregory.p.smith #46540: dylibs not loading properly from NFS mounts https://bugs.python.org/issue46540 closed by ned.deily #46542: test_json and test_lib2to3 crash on s390x Fedora Clang 3.x bui https://bugs.python.org/issue46542 closed by vstinner #46550: __slots__ updates despite being read-only https://bugs.python.org/issue46550 closed by ronaldoussoren #46560: ParamSpec: Typo in doc string https://bugs.python.org/issue46560 closed by kj #46563: Add tests regarding context of exception in a finally block https://bugs.python.org/issue46563 closed by iritkatriel #46568: non awaited coroutines on a IsolatedAsyncioTestCase results on https://bugs.python.org/issue46568 closed by asvetlov #46569: final note on StrEnum documentation incorrectly refers to int. https://bugs.python.org/issue46569 closed by ethan.furman #46572: Unicode identifiers not necessarily unique https://bugs.python.org/issue46572 closed by eryksun #46573: Python modules such as pyglet or pygame crash Python when tkin https://bugs.python.org/issue46573 closed by ronaldoussoren #46574: itertools.count should work with non-number types https://bugs.python.org/issue46574 closed by rhettinger #46576: test_peg_generator is extremely slow https://bugs.python.org/issue46576 closed by gregory.p.smith #46579: Compiling from source prints ��ModuleNotFoundError: No module https://bugs.python.org/issue46579 closed by christian.heimes #46582: Windows builds fail: fatal error RC1116: RC terminating after https://bugs.python.org/issue46582 closed by vstinner #46583: Modernize `selectors.py` by removing unused `sys.version_info` https://bugs.python.org/issue46583 closed by asvetlov #46584: Modernize `ctypes/test_python_api` by removing old version che https://bugs.python.org/issue46584 closed by zach.ware #46588: IDLE fix typo in test_calltip.py https://bugs.python.org/issue46588 closed by caioagiani #46590: I suggests to modify Include/object.h in 98 line https://bugs.python.org/issue46590 closed by 4d656f77 #46591: Make About IDLE links clickable https://bugs.python.org/issue46591 closed by terry.reedy #46595: [doc] Undocumented behavior of importlib.metadata.version https://bugs.python.org/issue46595 closed by cowlinator #46597: Remove Python 3.3 compatibility code from overlapped.c https://bugs.python.org/issue46597 closed by kumaraditya303 #46599: Objects/object.c:767:24: runtime error: member access within n https://bugs.python.org/issue46599 closed by christian.heimes #46602: Subtle trouble with heredoc append in configure. https://bugs.python.org/issue46602 closed by ned.deily #46610: assertCountEqual doesn't work as expected for dictionary eleme https://bugs.python.org/issue46610 closed by serhiy.storchaka #46612: Unclear behavior of += operator https://bugs.python.org/issue46612 closed by eric.smith #46616: test_importlib leaves stray registry entries on Windows https://bugs.python.org/issue46616 closed by steve.dower #46617: CSV Creation occasional off by one error https://bugs.python.org/issue46617 closed by matthewstidham #46618: Exponent operator(**) interpreter issue https://bugs.python.org/issue46618 closed by tim.peters #46624: random.randrange removed support for non-integer types after j https://bugs.python.org/issue46624 closed by rhettinger #46626: expose IP_BIND_ADDRESS_NO_PORT linux socket option https://bugs.python.org/issue46626 closed by benjamin.peterson #46627: Regex hangs indefinitely https://bugs.python.org/issue46627 closed by tim.peters #46628: Can't install YARL https://bugs.python.org/issue46628 closed by eric.smith #46629: Cannot sideload MSIX package on Windows https://bugs.python.org/issue46629 closed by steve.dower #46630: IDLE: Set query focus to entry box on Windows https://bugs.python.org/issue46630 closed by terry.reedy #46636: Bugs of 2to3 on built-in function or types(classes) https://bugs.python.org/issue46636 closed by gregory.p.smith

1 0

[RELEASE] Python 3.11.0a5 is available
by Pablo Galindo Salgado 04 Feb '22

04 Feb '22

We needed to tame some angry buildbots, but after a small fight, we won with just some scratches! Here you have a shiny new alpha release: Python 3.11.0a5. https://www.python.org/downloads/release/python-3110a5/ **This is an early developer preview of Python 3.11** # Major new features of the 3.11 series, compared to 3.10 Python 3.11 is still in development. This release, 3.11.0a5 is the fifth of seven planned alpha releases. Alpha releases are intended to make it easier to test the current state of new features and bug fixes and to test the release process. During the alpha phase, features may be added up until the start of the beta phase (2022-05-06) and, if necessary, may be modified or deleted up until the release candidate phase (2022-08-01). Please keep in mind that this is a preview release and its use is **not** recommended for production environments. Many new features for Python 3.11 are still being planned and written. Among the new major new features and changes so far: * [PEP 657](https://www.python.org/dev/peps/pep-0657/) -- Include Fine-Grained Error Locations in Tracebacks * [PEP 654](https://www.python.org/dev/peps/pep-0654/) -- Exception Groups and except* * [PEP 673](https://www.python.org/dev/peps/pep-0673/) -- Self Type * [PEP 646](https://www.python.org/dev/peps/pep-0646/)-- Variadic Generics * The [Faster Cpython Project](https://github.com/faster-cpython) is already yielding some exciting results: this version of CPython 3.11 is ~12% faster on the geometric mean of the [PyPerformance benchmarks]( speed.python.org), compared to 3.10.0. * Hey, **fellow core developer,** if a feature you find important is missing from this list, let me know. The next pre-release of Python 3.11 will be 3.11.0a6, currently scheduled for Monday, 2022-02-28. # More resources * [Online Documentation](https://docs.python.org/3.11/) * [PEP 664](https://www.python.org/dev/peps/pep-0664/), 3.11 Release Schedule * Report bugs at [https://bugs.python.org](https://bugs.python.org). * [Help fund Python and its community](/psf/donations/). # And now for something completely different In physics, the Poynting vector (Umov-Poynting vector) represents the directional energy flux (the energy transfer per unit area per unit time) or power flow of an electromagnetic field. It is named after its discoverer John Henry Poynting who first derived it in 1884. Oliver Heaviside also discovered it independently in the more general form that recognises the freedom of adding the curl of an arbitrary vector field to the definition. The Poynting vector is used throughout electromagnetics in conjunction with Poynting’s theorem, the continuity equation expressing conservation of electromagnetic energy, to calculate the power flow in electromagnetic fields. # We hope you enjoy those new releases! Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. Your friendly release team, Pablo Galindo @pablogsal Ned Deily @nad Steve Dower @steve.dower

1 0

Re: Please update Cython *before* introcuding C API incompatible changes in Python
by Thomas Caswell 02 Feb '22

02 Feb '22

I disagree with point (3). I think it would be better to discourage projects from including the output of cython in their sdists. They should either have cython as a build-time requirement or provide built wheels (which are specific a platform and CPython version). The middle ground of not expecting the user to have cython while expecting them to have a working c-complier is a very narrow case and I think asking those users to install cython is worth the forward compatibility for Python versions you get by requiring people installing from source to re-cythonize. Tom -- Thomas Caswell tcaswell(a)gmail.com

5 7

Increase of Spammy PRs and PR reviews
by Lrupert 01 Feb '22

01 Feb '22

As someone who is watching the python/cpython repository, I'm very used to see lots of traffic. But lately there have been a surge of spammy PRs which are about the same, generally very trivial subject but individually fixing each occurrence one by one: - Add coverage to X (tens of them, as separate PRs) - Make `test_xxx` executable with direct invocation (tens of them, as separate PRs) - Lint source with flake8, fix linting errors (stylistic changes) And lots of non-committer PR reviews that only approve. Am I the only one who feels like this is only done to 'grind' commits (get a higher number of commits into the repository, and boast about it)? In the past there were one or two people who would submit typo fixes, but most of them weren't making it continuously. The situation right now feels much worse than those.

21 29