Mailman 3 January 2022 - Python-Dev

Proposal: Allow non-default after default arguments
by Sebastian Rittau Oct. 16, 2023

Oct. 16, 2023

Currently, Python doesn't allow non-default arguments after default arguments: >>> def foo(x=None, y): pass File "<stdin>", line 1 def foo(x=None, y): pass ^ SyntaxError: non-default argument follows default argument I believe that at the time this was introduced, no use cases for this were known and this is is supposed to … [View More]prevent a source of bugs. I have two use cases for this, one fringe, but valid, the other more important: The fringe use case: Suppose you have a function that takes a 2D coordinate value as separate "x" and "y" arguments. The "x" argument is optional, the "y" argument isn't. Currently there are two ways to do this, none of them particularly great: def foo(y, x): # reverse x and y arguments, confusing ... def foo(x, y=None): # Treat the x argument as y if only one argument is provided if y is None: x, y = y, x ... To me, the "natural" solution looks like this: def foo(x=None, y): ... # Called like this: foo(1, 2) foo(y=2) This could also be useful when evolving APIs. For example there is a function "bar" that takes two required arguments. In a later version, the first argument gains a useful default, the second doesn't. There is no sensible way to evolve the API at the moment. The more important use case involves @overloads. A condensed example of a function where the return type depends on an "encoding" parameter, followed by further parameters that could be called like this: foo(123, "utf-8") # would return bytes foo(encoding="utf-8") foo(123, None) # would return str foo(encoding=None) foo(x=123) # would return str foo() This could ideally be written as: @overload def foo(x: int = ..., encoding: None = ...) -> str: ... @overload def foo(x: int = ..., encoding: str) -> bytes: ... # plus the actual implementation But due to the syntax constraint, this needs to be hacked around with a third overload: @overload def foo(x: int = ... encoding: None = ...) -> str: ... @overload def foo(x: int, encoding: str) -> bytes: ... # for foo(123, "utf-8") @overload def foo(*, encoding: str) -> bytes: ... # for foo(encoding="utf-8") Not only is this hard to read, real examples in typeshed are usually more complex, with many arguments before or after the affected argument or even multiple affected arguments. This often becomes too complex to write or maintain. Here is one example from the wild: https://github.com/python/typeshed/blob/b95b729b9e07ab21d252701af0f5b740467… Allowing non-default arguments after default arguments would solve both use cases above and eliminates a special case. I'm also not sure what exactly the current SyntaxError really protects us from. Adding a non-default after a default argument can't really lead bugs. - Sebastian [View Less]

11 14

Python multithreading without the GIL
by Sam Gross July 31, 2023

July 31, 2023

Hi, I've been working on changes to CPython to allow it to run without the global interpreter lock. I'd like to share a working proof-of-concept that can run without the GIL. The proof-of-concept involves substantial changes to CPython internals, but relatively few changes to the C-API. It is compatible with many C extensions: extensions must be rebuilt, but usually require small or no modifications to source code. I've built compatible versions of packages from the scientific Python ecosystem,… [View More]

24 48

Mixed Python/C debugging
by Skip Montanaro April 26, 2023

April 26, 2023

Having tried comp.lang.python with no response, I turn here... After at least ten years away from Python's run-time interpreter & byte code compiler, I'm getting set to familiarize myself with that again. This will, I think, entail debugging a mixed Python/C environment. I'm an Emacs user and am aware that GDB since 7.0 has support for debugging at the Python code level. Is Emacs+GDB my best bet? Are there any Python IDEs which support C-level breakpoints and debugging? Thanks, Skip

5 6

PEP 448 review
by Guido van Rossum March 29, 2023

March 29, 2023

I'm back, I've re-read the PEP, and I've re-read the long thread with "(no subject)". I think Georg Brandl nailed it: """ *I like the "sequence and dict flattening" part of the PEP, mostly because itis consistent and should be easy to understand, but the comprehension syntaxenhancements seem to be bad for readability and "comprehending" what the codedoes.The call syntax part is a mixed bag on the one hand it is nice to be consistent with the extended possibilities in literals (… [View More]flattening), but on the other hand there would be small but annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* """ Greg Ewing followed up explaining that the inconsistency between dict flattening and call syntax is inherent in the pre-existing different rules for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, **{'a': 1}) is a TypeError.) For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations for function calls proposed by the PEP is an easy +1 -- it's a straightforward extension of the existing pattern, and anybody who knows what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, **e) means shouldn't be hard either. Understanding the edge case for duplicate keys with f(**d, **e) is a little harder, but the error messages are pretty clear, and it is not a new edge case. The sequence and dict flattening syntax proposals are also clean and logical -- we already have *-unpacking on the receiving side, so allowing *x in tuple expressions reads pretty naturally (and the similarity with *a in argument lists certainly helps). From here, having [a, *x, b, *y] is also natural, and then the extension to other displays is natural: {a, *x, b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. So that leaves comprehensions. IIRC, during the development of the patch we realized that f(*x for x in xs) is sufficiently ambiguous that we decided to disallow it -- note that f(x for x in xs) is already somewhat of a special case because an argument can only be a "bare" generator expression if it is the only argument. The same reasoning doesn't apply (in that form) to list, set and dict comprehensions -- while f(x for x in xs) is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the same as [(x for x in xs)] (that's a list of one element, and the element is a generator expression). The basic premise of this part of the proposal is that if you have a few iterables, the new proposal (without comprehensions) lets you create a list or generator expression that iterates over all of them, essentially flattening them: >>> xs = [1, 2, 3] >>> ys = ['abc', 'def'] >>> zs = [99] >>> [*xs, *ys, *zs] [1, 2, 3, 'abc', 'def', 99] >>> But now suppose you have a list of iterables: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xss[0], *xss[1], *xss[2]] [1, 2, 3, 'abc', 'def', 99] >>> Wouldn't it be nice if you could write the latter using a comprehension? >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs for xs in xss] [1, 2, 3, 'abc', 'def', 99] >>> This is somewhat seductive, and the following is even nicer: the *xs position may be an expression, e.g.: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs[:2] for xs in xss] [1, 2, 'abc', 'def', 99] >>> On the other hand, I had to explore the possibilities here by experimenting in the interpreter, and I discovered some odd edge cases (e.g. you can parenthesize the starred expression, but that seems a syntactic accident). All in all I am personally +0 on the comprehension part of the PEP, and I like that it provides a way to "flatten" a sequence of sequences, but I think very few people in the thread have supported this part. Therefore I would like to ask Neil to update the PEP and the patch to take out the comprehension part, so that the two "easy wins" can make it into Python 3.5 (basically, I am accepting two-thirds of the PEP :-). There is some time yet until alpha 2. I would also like code reviewers (Benjamin?) to start reviewing the patch <http://bugs.python.org/issue2292>, taking into account that the comprehension part needs to be removed. -- --Guido van Rossum (python.org/~guido) [View Less]

7 17

PEP 638: Syntactic macros
by Mark Shannon Feb. 5, 2023

Feb. 5, 2023

Hi everyone, I've submitted my PEP on syntactic macros as PEP 638. https://www.python.org/dev/peps/pep-0638/ All comments and suggestions are welcome. Cheers, Mark

16 25

How about using modern C++ in development of CPython ?
by redradist＠gmail.com Oct. 19, 2022

Oct. 19, 2022

Hi all, What about of using modern C++ in developing CPython ? With C++ we can get the following benefits: 1) Readability - less code, most code is hidden by abstraction without losing performance 2) Maintainability - no code duplication in favor of using reusable classes 3) RAII - Resource Acquisition Is Initialization, predictable allocation and free resources ...

23 51

PEP 505 (None-aware operators) for Python 3.11
by Doug Swarin Sept. 21, 2022

Sept. 21, 2022

Hello, I've been following PEP 505 (https://www.python.org/dev/peps/pep-0505/) since it was first proposed for Python 3.8. It's been deferred for some time and I'm interested in seeing it in Python 3.11, but I know there were also a number of objections which resulted in it being deferred (including by one of the original authors, Mr. Dower). I did email both Mr. Dower and Mr. Haase and they graciously gave me their permission to bring it up on this list for discussion and hopefully final … [View More]

22 53

PEP 563 and Python 3.10.
by Thomas Wouters July 22, 2022

July 22, 2022

(Starting a new thread so as not to derail any of the ongoing discussions.) Thanks, everyone, for your thoughts on Python 3.10 and the impact of PEP 563 (postponed evaluation of annotations) becoming the default. The Steering Council has considered the issue carefully, along with many of the proposed alternatives and solutions, and we’ve decided that at this point, we simply can’t risk the compatibility breakage of PEP 563. We need to roll back the change that made stringified annotations the … [View More]default, at least for 3.10. (Pablo is already working on this.) To be clear, we are not reverting PEP 563 itself. The future import will keep working like it did since Python 3.7. We’re delaying making PEP 563 string-based annotations the default until Python 3.11. This will give us time to find a solution that works for everyone (or to find a feasible upgrade path for users who currently rely on evaluated annotations). Some considerations that led us to this decision: - PEP 563’s default change is clearly too disruptive to downstream users and third-party libraries to happen right now. We can’t risk breaking even a small subset of the FastAPI/pydantic users, not to mention other uses of evaluated type annotations that we’re not aware of yet. - PEP 563 provides no warning to users of the feature it’s disabling. Without that, we can’t expect users to be aware of the upcoming breakage. The lack of a warning was by design, and made sense in a world where type annotations were only consumed by static type checkers --- but that’s not actually the situation we’re in. There are clearly existing real-world, run-time uses of type annotations that would be adversely affected by this change. - Originally, PEP 563 was scheduled to take effect in Python 4, and this changed recently (after the discussion in the Language Summit of 2020). It's possible that third-party libraries and users didn’t plan to react in the current time frame as they were not aware of this change in timing. - There isn’t enough time to properly discuss PEP 649 or any of the alternatives before the beta 1 deadline, and we really need to make sure we don’t compound errors here. We need to look for a long term solution, which isn’t possible while still maintaining the release deadlines of Python 3.10. That means we’re also deferring PEP 649 to Python 3.11. In the Steering Council’s unanimous opinion, rolling back the default flip for stringified annotations in Python 3.10 is the least disruptive of all the options. We need to continue discussing the issue and potential solutions, since this merely postpones the problem until 3.11. (For the record, postponing the change further is not off the table, either, for example if the final decision is to treat evaluated annotations as a deprecated feature, with warnings on use.) For what it’s worth, the SC is also considering what we can do to reduce the odds of something like this happening again, but that’s a separate consideration, and a multi-faceted one at that. For the Steering Council, Thomas. -- Thomas Wouters <thomas(a)python.org> Hi! I'm an email virus! Think twice before sending your email to help me spread! [View Less]

6 5

compiled python3.10 is unable to find _ssl
by Sandeep Gupta May 29, 2022

May 29, 2022

I am having compilation issues again with python3.10 with ssl . The ./configure was invoked with ssl options and ssl modules seems to be build successfully. """ The following modules found by detect_modules() in setup.py, have been built by the Makefile instead, as configured by the Setup files: _abc _hashlib _ssl pwd time """ However, when I do import ssl from python, I get the error: Traceback (most recent call last): File "<stdin>",… [View More]

6 12

Slowly bend the C API towards the limited API to get a stable ABI for everyone
by Victor Stinner April 20, 2022

April 20, 2022

Hi, There is a reason why I'm bothering C extensions maintainers and Python core developers with my incompatible C API changes since Python 3.8. Let me share my plan with you :-) In 2009 (Python 3.2), Martin v. Löwis did an amazing job with the PEP 384 "Defining a Stable ABI" to provide a "limited C API" and a "stable ABI" for C extensions: build an extension once, use it on multiple Python versions. Some projects like PyQt5 and cryptograpy use it, but it is just a drop in the PyPI ocean (… [View More]353,084 projects). I'm trying to bend the "default" C API towards this "limited C API" to make it possible tomorrow to build *more* C extensions for the stable ABI. My goal is that the stable ABI would be the default, and only a minority of C extensions would opt-out because they need to access to more functions for best performance. The basic problem is that at the ABI level, C extensions must only call functions, rather than getting and setting directly to structure members. Structures changes frequently in Python (look at changes between Python 3.2 and Python 3.11), and any minor structure change breaks the ABI. The limited C API hides structures and only use function calls to solve this problem. Since 2020, I'm modifying the C API, one function by one, to slowly hide implementations (prepare the API to make strutures opaque). I focused on the following structures: * PyObject and PyVarObject (bpo-39573) * PyTypeObject (bpo-40170) * PyFrameObject (bpo-40421) * PyThreadState (bpo-39947) The majority of C extensions use functions and macros, they don't access directly structure members. There are a few members which are sometimes accessed directly which prevents making these structures opaque. For example, some old C extensions use obj->ob_type rather than Py_TYPE(obj). Fixing the minority of C extensions should benefit to the majority which may become compatible with the stable ABI. I am also converting macros to static inline functions to fix their API: define parameter types, result type and avoid surprising macros side effects ("macro pitfalls"). I wrote the PEP 670 "Convert macros to functions in the Python C API" for these changes. I wrote the upgrade_pythoncapi.py tool in my pythoncapi_project (*) which modify C code to use Py_TYPE(), Py_SIZE() and Py_REFCNT() rather than accessing directly PyObject and PyVarObject members. (*) https://github.com/pythoncapi/pythoncapi_compat In this tool, I also added "Borrow" variant of functions like PyFrame_GetCode() which returns a strong reference, to replace frame->f_code with _PyFrame_GetCodeBorrow(). In Python 3.11, you cannot use the frame->f_code member anymore, since it has been removed! You must call PyFrame_GetCode() (or pythoncapi_compat _PyFrame_GetCodeBorrow() variant). There are also a few macros which can be used as l-values like Py_TYPE(): "Py_TYPE(type1) = type2" must now be written "Py_SET_TYPE(type1, type2)" to avoid setting directly the tp_type type at the ABI level. I proposed the PEP 674 "Disallow using Py_TYPE() and Py_SIZE() macros as l-values" to solve these issues. Currently, many "functions" are still implemented as macros or static inline functions, so C extensions still access structure members at the ABI level for best Python performance. Converting these to regular functions has an impact on performance and I would prefer to first write a PEP giving the rationale for that. Today, it is not possible yet to build numpy for the stable ABI. The gap is just too large for this big C extension. But step by step, the C API becomes closer to the limited API, and more and more code is ready to be built for the stable ABI. Well, these C API changes have other advantages, like preparing Python for further optimizations, ease Python maintenance, clarify the seperation between the limited C API and the default C API, etc. ;-) Victor -- Night gathers, and now my watch begins. It shall not end until my death. [View Less]

12 38

PEP 678: Enriching Exceptions with Notes
by Zac Hatfield Dodds April 12, 2022

April 12, 2022

Hi all, I've written PEP 678, proposing the addition of a __note__ attribute which can be used to enrich displayed exceptions. This is particularly useful with ExceptionGroup, or where exception chaining is unsuitable, and ensures that 'orphan' output is not produced if the annotated exception is caught. Link to the PEP: https://www.python.org/dev/peps/pep-0678/ *Please, redirect all discussions to:* https://discuss.python.org/t/pep-678-enriching-exceptions-with-notes/13374 Thanks for … [View More]

2 1

RFC on PEP 655: Required[] and NotRequired[] for TypedDict
by David Foster March 10, 2022

March 10, 2022

This PEP [1] introduces syntax to mark individual keys of a TypedDict as either required or potentially-missing. Previously the only way to have a TypedDict with mixed required and non-required keys was to define two TypedDicts - one with total=True and one with total=False - and have one of those TypedDicts inherit from the other. This PEP introduces special forms Required[] and NotRequired[] to the "typing" module that can be used to directly mark individual items of a TypedDict as … [View More]

4 6

Suggestion: a little language for type definitions
by jack.jansen＠cwi.nl Feb. 10, 2022

Feb. 10, 2022

I posted this suggestion earlier in the callable type syntax discussion, at which point it was completely ignored. Possibly because it’s a really stupid idea, but let me post it again on the off chance that it isn’t a stupid idea but was overlooked. > If I can make a wild suggestion: why not create a little language for type specifications? > > If you look at other programming languages you’ll see that the “type definition sub-language” is often completely different from the “… [View More]

15 35

"immortal" objects and how they would help per-interpreter GIL
by Eric Snow Feb. 8, 2022

Feb. 8, 2022

Most of the work toward interpreter isolation and a per-interpreter GIL involves moving static global variables to _PyRuntimeState or PyInterpreterState (or module state). Through the effort of quite a few people, we've made good progress. However, many globals still remain, with the majority being objects and most of those being static strings (e.g. _Py_Identifier), static types (incl. exceptions), and singletons. On top of that, a number of those objects are exposed in the public C-API and … [View More]even in the limited API. :( Dealing with this specifically is probably the trickiest thing I've had to work through in this project. There is one solution that would help both of the above in a nice way: "immortal" objects. The idea of objects that never get deallocated isn't new and has been explored here several times. Not that long ago I tried it out by setting the refcount really high. That worked. Around the same time Eddie Elizondo at Facebook did something similar but modified Py_INCREF() and Py_DECREF() to keep the refcount from changing. Our solutions were similar but with different goals in mind. (Facebook wants to avoid copy-on-write in their pre-fork model.) A while back I concluded that neither approach would work for us. The approach I had taken would have significant cache performance penalties in a per-interpreter GIL world. The approach that modifies Py_INCREF() has a significant performance penalty due to the extra branch on such a frequent operation. Recently I've come back to the idea of immortal objects because it's much simpler than the alternate (working) solution I found. So how do we get around that performance penalty? Let's say it makes CPython 5% slower. We have some options: * live with the full penalty * make other changes to reduce the penalty to a more acceptable threshold than 5% * eliminate the penalty (e.g. claw back 5% elsewhere) * abandon all hope Mark Shannon suggested to me some things we can do. Also, from a recent conversation with Dino Viehland it sounds like Eddie was able to reach performance-neutral with a few techniques. So here are some things we can do to reduce or eliminate that penalty: * reduce refcount operations on high-activity objects (e.g. None, True, False) * reduce refcount operations in general * walk the heap at the end of runtime initialization and mark all objects as immortal * mark all global objects as immortal (statics or in _PyRuntimeState; for PyInterpreterState not needed) What do you think? Does this sound realistic? Are there additional things we can do to counter that penalty? -eric [View Less]

16 35

PEP 674 – Disallow using macros as l-value (version 2)
by Victor Stinner Feb. 7, 2022

Feb. 7, 2022

Hi, I made the following changes to the PEP since the version 1 (November 30): * Elaborate the HPy section and add a new "Relationship with the HPy project" section * Elaborate which projects are impacted and which changes are needed * Remove the PyPy section The PEP can be read online: https://python.github.io/peps/ep-0674/ And you can find the plain text below. Victor PEP: 674 Title: Disallow using macros as l-value Author: Victor Stinner <vstinner(a)python.org> Status: Draft … [View More]Type: Standards Track Content-Type: text/x-rst Created: 30-Nov-2021 Python-Version: 3.11 Abstract ======== Incompatible C API change disallowing using macros as l-value to: * Allow evolving CPython internals; * Ease the C API implementation on other Python implementation; * Help migrating existing C extensions to the HPy API. On the PyPI top 5000 projects, only 14 projects (0.3%) are affected by 4 macro changes. Moreover, 24 projects just have to regenerate their Cython code to use ``Py_SET_TYPE()``. In practice, the majority of affected projects only have to make two changes: * Replace ``Py_TYPE(obj) = new_type;`` with ``Py_SET_TYPE(obj, new_type);``. * Replace ``Py_SIZE(obj) = new_size;`` with ``Py_SET_SIZE(obj, new_size);``. Rationale ========= Using a macro as a l-value -------------------------- In the Python C API, some functions are implemented as macro because writing a macro is simpler than writing a regular function. If a macro exposes directly a structure member, it is technically possible to use this macro to not only get the structure member but also set it. Example with the Python 3.10 ``Py_TYPE()`` macro:: #define Py_TYPE(ob) (((PyObject *)(ob))->ob_type) This macro can be used as a **r-value** to **get** an object type:: type = Py_TYPE(object); It can also be used as **l-value** to **set** an object type:: Py_TYPE(object) = new_type; It is also possible to set an object reference count and an object size using ``Py_REFCNT()`` and ``Py_SIZE()`` macros. Setting directly an object attribute relies on the current exact CPython implementation. Implementing this feature in other Python implementations can make their C API implementation less efficient. CPython nogil fork ------------------ Sam Gross forked Python 3.9 to remove the GIL: the `nogil branch <https://github.com/colesbury/nogil/>`_. This fork has no ``PyObject.ob_refcnt`` member, but a more elaborated implementation for reference counting, and so the ``Py_REFCNT(obj) = new_refcnt;`` code fails with a compiler error. Merging the nogil fork into the upstream CPython main branch requires first to fix this C API compatibility issue. It is a concrete example of a Python optimization blocked indirectly by the C API. This issue was already fixed in Python 3.10: the ``Py_REFCNT()`` macro has been already modified to disallow using it as a l-value. These statements are endorsed by Sam Gross (nogil developer). HPy project ----------- The `HPy project <https://hpyproject.org/>`_ is a brand new C API for Python using only handles and function calls: handles are opaque, structure members cannot be accessed directly, and pointers cannot be dereferenced. Searching and replacing ``Py_SET_SIZE()`` is easier and safer than searching and replacing some strange macro uses of ``Py_SIZE()``. ``Py_SIZE()`` can be semi-mechanically replaced by ``HPy_Length()``, whereas seeing ``Py_SET_SIZE()`` would immediately make clear that the code needs bigger changes in order to be ported to HPy (for example by using ``HPyTupleBuilder`` or ``HPyListBuilder``). The fewer internal details exposed via macros, the easier it will be for HPy to provide direct equivalents. Any macro that references "non-public" interfaces effectively exposes those interfaces publicly. These statements are endorsed by Antonio Cuni (HPy developer). GraalVM Python -------------- In GraalVM, when a Python object is accessed by the Python C API, the C API emulation layer has to wrap the GraalVM objects into wrappers that expose the internal structure of the CPython structures (PyObject, PyLongObject, PyTypeObject, etc). This is because when the C code accesses it directly or via macros, all GraalVM can intercept is a read at the struct offset, which has to be mapped back to the representation in GraalVM. The smaller the "effective" number of exposed struct members (by replacing macros with functions), the simpler GraalVM wrappers can be. This PEP alone is not enough to get rid of the wrappers in GraalVM, but it is a step towards this long term goal. GraalVM already supports HPy which is a better solution in the long term. These statements are endorsed by Tim Felgentreff (GraalVM Python developer). Specification ============= Disallow using macros as l-value -------------------------------- PyObject and PyVarObject macros ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * ``Py_TYPE()``: ``Py_SET_TYPE()`` must be used instead * ``Py_SIZE()``: ``Py_SET_SIZE()`` must be used instead "GET" macros ^^^^^^^^^^^^ * ``PyByteArray_GET_SIZE()`` * ``PyBytes_GET_SIZE()`` * ``PyCFunction_GET_CLASS()`` * ``PyCFunction_GET_FLAGS()`` * ``PyCFunction_GET_FUNCTION()`` * ``PyCFunction_GET_SELF()`` * ``PyCell_GET()`` * ``PyCode_GetNumFree()`` * ``PyDict_GET_SIZE()`` * ``PyFunction_GET_ANNOTATIONS()`` * ``PyFunction_GET_CLOSURE()`` * ``PyFunction_GET_CODE()`` * ``PyFunction_GET_DEFAULTS()`` * ``PyFunction_GET_GLOBALS()`` * ``PyFunction_GET_KW_DEFAULTS()`` * ``PyFunction_GET_MODULE()`` * ``PyHeapType_GET_MEMBERS()`` * ``PyInstanceMethod_GET_FUNCTION()`` * ``PyList_GET_SIZE()`` * ``PyMemoryView_GET_BASE()`` * ``PyMemoryView_GET_BUFFER()`` * ``PyMethod_GET_FUNCTION()`` * ``PyMethod_GET_SELF()`` * ``PySet_GET_SIZE()`` * ``PyTuple_GET_SIZE()`` * ``PyUnicode_GET_DATA_SIZE()`` * ``PyUnicode_GET_LENGTH()`` * ``PyUnicode_GET_LENGTH()`` * ``PyUnicode_GET_SIZE()`` * ``PyWeakref_GET_OBJECT()`` "AS" macros ^^^^^^^^^^^ * ``PyByteArray_AS_STRING()`` * ``PyBytes_AS_STRING()`` * ``PyFloat_AS_DOUBLE()`` * ``PyUnicode_AS_DATA()`` * ``PyUnicode_AS_UNICODE()`` PyUnicode macros ^^^^^^^^^^^^^^^^ * ``PyUnicode_1BYTE_DATA()`` * ``PyUnicode_2BYTE_DATA()`` * ``PyUnicode_4BYTE_DATA()`` * ``PyUnicode_DATA()`` * ``PyUnicode_IS_ASCII()`` * ``PyUnicode_IS_COMPACT()`` * ``PyUnicode_IS_READY()`` * ``PyUnicode_KIND()`` * ``PyUnicode_READ()`` * ``PyUnicode_READ_CHAR()`` PyDateTime "GET" macros ^^^^^^^^^^^^^^^^^^^^^^^ * ``PyDateTime_DATE_GET_FOLD()`` * ``PyDateTime_DATE_GET_HOUR()`` * ``PyDateTime_DATE_GET_MICROSECOND()`` * ``PyDateTime_DATE_GET_MINUTE()`` * ``PyDateTime_DATE_GET_SECOND()`` * ``PyDateTime_DATE_GET_TZINFO()`` * ``PyDateTime_DELTA_GET_DAYS()`` * ``PyDateTime_DELTA_GET_MICROSECONDS()`` * ``PyDateTime_DELTA_GET_SECONDS()`` * ``PyDateTime_GET_DAY()`` * ``PyDateTime_GET_MONTH()`` * ``PyDateTime_GET_YEAR()`` * ``PyDateTime_TIME_GET_FOLD()`` * ``PyDateTime_TIME_GET_HOUR()`` * ``PyDateTime_TIME_GET_MICROSECOND()`` * ``PyDateTime_TIME_GET_MINUTE()`` * ``PyDateTime_TIME_GET_SECOND()`` * ``PyDateTime_TIME_GET_TZINFO()`` PyDescr macros ^^^^^^^^^^^^^^ * ``PyDescr_NAME()`` * ``PyDescr_TYPE()`` Port C extensions to Python 3.11 -------------------------------- In practice, the majority of projects affected by these PEP only have to make two changes: * Replace ``Py_TYPE(obj) = new_type;`` with ``Py_SET_TYPE(obj, new_type);``. * Replace ``Py_SIZE(obj) = new_size;`` with ``Py_SET_SIZE(obj, new_size);``. The `pythoncapi_compat project <https://github.com/pythoncapi/pythoncapi_compat>`_ can be used to update automatically C extensions: add Python 3.11 support without losing support with older Python versions. The project provides a header file which provides ``Py_SET_REFCNT()``, ``Py_SET_TYPE()`` and ``Py_SET_SIZE()`` functions to Python 3.8 and older. PyTuple_GET_ITEM() and PyList_GET_ITEM() ---------------------------------------- The ``PyTuple_GET_ITEM()`` and ``PyList_GET_ITEM()`` macros are left unchanged. The code pattern ``&PyTuple_GET_ITEM(tuple, 0)`` and ``&PyList_GET_ITEM(list, 0)`` is still commonly used to get access to the inner ``PyObject**`` array. Changing these macros is out of the scope of this PEP. Backwards Compatibility ======================= The proposed C API changes are backward incompatible on purpose. At December 1, 2021, a code search on the PyPI top 5000 projects (4760 projects in practice, others don't have a source archive) found that `only 14 projects are affected <https://bugs.python.org/issue45476#msg407456>`_ (0.3%): * datatable (1.0.0) * frozendict (2.1.1) * guppy3 (3.1.2) * M2Crypto (0.38.0) * mecab-python3 (1.0.4) * mypy (0.910) * Naked (0.1.31) * pickle5 (0.0.12) * pysha3 (1.0.2) * python-snappy (0.6.0) * recordclass (0.16.3) * scipy (1.7.3) * zodbpickle (2.2.0) * zstd (1.5.0.2) These 14 projects only use 4 macros as l-value: * ``PyDescr_NAME()`` and ``PyDescr_TYPE()`` (2 projects) * ``Py_SIZE()`` (8 projects) * ``Py_TYPE()`` (4 projects) Moreover, `24 projects just have to regenerate their Cython code <https://bugs.python.org/issue45476#msg407416>`_ to use ``Py_SET_TYPE()``. This change does not follow the PEP 387 deprecation process. There is no known way to emit a deprecation warning only when a macro is used as a l-value, but not when it's used differently (ex: as a r-value). Relationship with the HPy project ================================= The HPy project --------------- The hope with the HPy project is indeed to provide a C API that is close to the original API to make porting easy and have it perform as close to the existing API as possible. At the same time, HPy is sufficently removed to be a good "C extension API" (as opposed to a stable subset of the CPython implementation API) that does not leak implementation details. To ensure this latter property is why the HPy project tries to develop everything in parallel for CPython, PyPy, and GraalVM Python. HPy is still evolving very fast. Issues are still being solved while migrating NumPy, and work has begun on adding support for HPy to Cython. Work on pybind11 is starting soon. Tim Felgentreff believes by the time HPy has these users of the existing C API working, HPy should be in a state where it is generally useful and can be deemed stable enough that further development can follow a more stable process. In the long run the HPy project would like to become a promoted API to write Python C extensions. The HPy project is a good solution for the long term. It has the advantage of being developed outside Python and it doesn't require any C API change. The C API is here is stay for a few more years ---------------------------------------------- The first concern about HPy is that right now, HPy is not mature nor widely used, and CPython still has to continue supporting a large amount of C extensions which are not likely to be ported to HPy soon. The second concern is the inability to evolve CPython internals to implement new optimizations, and the inefficient implementation of the current C API in PyPy, GraalPython, etc. Sadly, HPy will only solve these problems when most C extensions will be fully ported to HPy: when it will become reasonable to consider dropping the "legacy" Python C API. While porting a C extension to HPy can be done incrementally on CPython, it requires to modify a lot of code and takes time. Porting most C extensions to HPy is expected to take a few years. This PEP proposes to make the C API "less bad" by fixing one problem which is clearily identified as causing practical issues: macros used as l-values. This PEP only requires to update a minority of C extensions, and usually only a few lines need to be changed in impacted extensions. For example, numpy 1.22 is made of 307,300 lines of C code, and adapting numpy to the this PEP only modified 11 lines (use Py_SET_TYPE and Py_SET_SIZE) and adding 4 lines (to define Py_SET_TYPE and Py_SET_SIZE for Python 3.8 and older). Porting numpy to HPy is expected to require modifying more lines than that. Right now, it's hard to bet which approach is the best: fixing the current C API, or focusing on HPy. It would be risky to only focus on HPy. Rejected Idea: Leave the macros as they are =========================================== The documentation of each function can discourage developers to use macros to modify Python objects. If these is a need to make an assignment, a setter function can be added and the macro documentation can require to use the setter function. For example, a ``Py_SET_TYPE()`` function has been added to Python 3.9 and the ``Py_TYPE()`` documentation now requires to use the ``Py_SET_TYPE()`` function to set an object type. If developers use macros as l-value, it's their responsibility when their code breaks, not the Python responsibility. We are operating under the consenting adults principle: we expect users of the Python C API to use it as documented and expect them to take care of the fallout, if things break when they don't. This idea was rejected because only few developers read the documentation, and only a minority is tracking changes of the Python C API documentation. The majority of developers are only using CPython and so are not aware of compatibility issues with other Python implementations. Moreover, continuing to allow using macros as l-value does not help the HPy project and leaves the burden of emulating them on GraalVM's Python implementation. Macros already modified ======================= The following C API macros have already been modified to disallow using them as l-value: * ``PyCell_SET()`` * ``PyList_SET_ITEM()`` * ``PyTuple_SET_ITEM()`` * ``Py_REFCNT()`` (Python 3.10): ``Py_SET_REFCNT()`` must be used * ``_PyGCHead_SET_FINALIZED()`` * ``_PyGCHead_SET_NEXT()`` * ``asdl_seq_GET()`` * ``asdl_seq_GET_UNTYPED()`` * ``asdl_seq_LEN()`` * ``asdl_seq_SET()`` * ``asdl_seq_SET_UNTYPED()`` For example, ``PyList_SET_ITEM(list, 0, item) < 0`` now fails with a compiler error as expected. Discussion ========== * `PEP 674: Disallow using macros as l-value <https://mail.python.org/archives/list/python-dev@python.org/thread/KPIJPPJ6…>`_ (Nov 30, 2021) References ========== * `Python C API: Add functions to access PyObject <https://vstinner.github.io/c-api-abstract-pyobject.html>`_ (October 2021) article by Victor Stinner * `[C API] Disallow using PyFloat_AS_DOUBLE() as l-value <https://bugs.python.org/issue45476>`_ (October 2021) * `[capi-sig] Py_TYPE() and Py_SIZE() become static inline functions <https://mail.python.org/archives/list/capi-sig@python.org/thread/WGRLTHTHC3…>`_ (September 2021) * `[C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type <https://bugs.python.org/issue39573>`__ (February 2020) * `bpo-30459: PyList_SET_ITEM could be safer <https://bugs.python.org/issue30459>`_ (May 2017) Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. [View Less]

3 7

Increase of Spammy PRs and PR reviews
by Lrupert Feb. 1, 2022

Feb. 1, 2022

As someone who is watching the python/cpython repository, I'm very used to see lots of traffic. But lately there have been a surge of spammy PRs which are about the same, generally very trivial subject but individually fixing each occurrence one by one: - Add coverage to X (tens of them, as separate PRs) - Make `test_xxx` executable with direct invocation (tens of them, as separate PRs) - Lint source with flake8, fix linting errors (stylistic changes) And lots of non-committer PR reviews that … [View More]

21 29

PEP 651, Robust Stack Overflow Handling, Rejection notice
by Python Steering Council Jan. 31, 2022

Jan. 31, 2022

Hi Mark, Thank you for submitting PEP 651. The Steering Council has spent the past two weeks reviewing PEP 651. After careful consideration, we have decided to reject the PEP. The following were the key points that led us to this decision: * The benefits are not compelling enough. Deep recursion is not a common tool in Python, and even with PEP 651 it would not be efficient enough to make it a common tool. * The benefit of PEP 651 is negated as soon as a non-Python function is … [View More]

6 8

Blocking the main branch due to too many buildbots failing
by Pablo Galindo Salgado Jan. 29, 2022

Jan. 29, 2022

Hi everyone, We have a huge amount of buildbots failing and seems to be related to different issues that keep piling up. To prevent this from going worse,* I am blocking the main branch* *until these issues are resolved.* Only release managers and the developer in residence will be able to merge pull requests until these issues are fixed. Please, ping me if you have a pull request for fixing any of these issues so we can merge. I apologize for the inconvenience. Thanks for your … [View More]

2 1

Summary of Python tracker Issues
by Python tracker Jan. 28, 2022

Jan. 28, 2022

ACTIVITY SUMMARY (2022-01-21 - 2022-01-28) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7157 (-18) closed 51138 (+124) total 58295 (+106) Open issues with patches: 2888 Issues opened (71) ================== #33205: GROWTH_RATE prevents dict shrinking https://bugs.python.org/issue33205 reopened by rhettinger #38195: A bug in the multiprocessing … [View More]

1 0

Request to revert unittest and configparser incompatible changes in Python 3.11
by Victor Stinner Jan. 28, 2022

Jan. 28, 2022

Hi, My colleagues Tomáš Hrnčiar and Miro Hrončok made good progress on updating Python 3.10 to Python 3.11 in Fedora, but some specific Python 3.11 incompatible changes are causing more troubles than others: https://discuss.python.org/t/experience-with-python-3-11-in-fedora/12911 We propose to revert the following 2 changes in Python 3.11 and postpone them in a later Python version, once most projects will be compatible with these changes: * Removal of unittest aliases (bpo-45162): it broke … [View More]61 Fedora packages * Removals from configparser module (bpo-45173) - broke 28 Fedora packages -- We reported the issue to many affected projects, or when possible, we also wrote a pull request to make the code compatible (lot of those were made by others, e.g. Hugo van Kemenade, thank you!). The problem is that fixing a Fedora package requires multiple steps: * (1) Propose a pull request upstream * (2) Get the pull request merged upstream * (3) Wait for a new release upstream * (4) Update the Fedora package downstream, or backport the change in Fedora (only needed by Fedora) Identifying the Python incompatible changes causing most troubles took us a lot of time, but we did this work. Reverting the two Python 3.11 incompatible changes (causing most troubles) will save us "bug triage" time, to get more time on updating projects to Python 3.11 for the other remaining incompatible changes. We are not saying that these incompatible changes are bad, it's just a matter of getting most projects ready before Python 3.11 final will be released. -- By the way, before making a change known to be incompatible, it would be nice to run a code search on PyPI top 5000 projects to estimate how many projects will be broken, and try to update these projects *before* making the change. For example, only introduce an incompatible change into Python once less than 15 projects are affected. Getting a pull request merged is nice, but a release including the fix is way better for practical reasons: people use "pip install <project name>". -- Fedora work on Python 3.11 is public and tracked at: https://bugzilla.redhat.com/show_bug.cgi?id=PYTHON3.11 Click on "depends on" to see current issues: https://bugzilla.redhat.com/buglist.cgi?bug_id=2016048&bug_id_type=anddepen… Example of bz #2025600: mom fails to build with Python 3.11: AttributeError: module 'configparser' has no attribute 'SafeConfigParser'. Victor Stinner -- in the name of the Python Red Hat / Fedora maintenance team -- Night gathers, and now my watch begins. It shall not end until my death. [View Less]

12 16

Python no longer leaks memory at exit
by Victor Stinner Jan. 28, 2022

Jan. 28, 2022

Hi, tl; dr Python no longer leaks memory at exit on the "python -c pass" command ;-) == Bug report == In 2007, the bpo-1635741 issue was reported on SourceForge: "Interpreter seems to leak references after finalization". This bug is 15 years old. It saw the bugs migration from SourceForge to Roundup (bugs.python.org) in 2007, the code migration from Subversion to Mercurial in 2009, and the second code migration from Mercurial to Git in 2016! Link to the main issue: https://bugs.python.… [View More]org/issue1635741 Some people reported similar issues seen by WinCRT debug (bpo-6741), MSVC Debug C Runtime (bpo-32026), GCC AddressSanitzer (bpo-26888), or LeakSanitizer and Valgrind (bpo-21387). In general, "leaking memory" at Python exit doesn't matter, since the operating system releases all memory when a process completes. It matters when Python is embedded in an application. For example, Python can be used by a plugin which is unloaded and reloaded multiple times. It also matters for sub-interpreters. == Tedious task == In the last 3 years, many people helped fixing this old issue by converting static types to heap types, adding a module state to C extension modules, converting extensions to the multi-phase initialization API (PEP 489), fix many memory leaks, fix bugs, etc. When it became possible to cleanly unload an extension module (thanks for the multi-phase init), tests on sub-intepreters (which load and unload extension modules) showed many *old* reference leaks: all of them have been fixed! When a test is run in sub-interpreters, Python is able to detect leaks, whereas currently it doesn't check for memory leaks at Python exit. == Python 3.10 regressions == During the Python 3.10 development, we identified and fixed 3 major regressions caused by this work: * Converting static types to heap types make them mutable: the Py_TPFLAGS_IMMUTABLETYPE flag was added; 68 types use it in Python 3.11 (bpo-43908). * It became possible to create uninitialized objects by instanciating types (which should not be instanciated directly): the Py_TPFLAGS_DISALLOW_INSTANTIATION was added; 41 types use it in Python 3.11 (bpo-43916). For example, you cannot create an instance of type(sys.flags). * Heap types must implement the full GC protocol: Py_TPFLAGS_HAVE_GC flag, traverse and clear functions. Otherwise, the GC is unable to break reference cycles, whereas a type contains (multiple) strong references to itself (in the MRO tuple and in methods). All heap types have been fixed to fully implement the GC protocol (bpo-40217). == PEPs == The work relies on multiple PEPs: * PEP 489: Multi-phase extension module initialization * PEP 573: Module State Access from C Extension Methods * PEP 630: Isolating Extension Modules == Persons who helped fixing the issue == Incomplete list of people who helped to fix this issue: * Christian Heimes (modules: symtable, _hashlib, ,_random, grp, pwd, _queue, spwd, _struct, gc, _posixshmem, _posixsubprocess, select, resource) * Dong-hee Na (modules: _statistics, itertools, _heapq, _collections, _uuid, math, _stat, syslog, errno, fcntl, mmap, _dbm, _gdbm, _lzma, faulthandler, _bisect) * Eric Snow (PEP 573) * Erlend Egeberg Aasland (modules: _sqlite3, _sre, _multibytecodec) * Hai Shi (modules: _json, _codecs, _crypt, _contextvars, _abc, _bz2, _locale, audioop, _ctypes_test, _weakref, _locale) * Marcel Plch (PEP 573) * Martin von Löwis (PEP 3121) * Mohamed Koubaa (modules: sha256, multiprocessing, _winapi, _blake2, _sha3, _signal, _sha1, _sha512, _md5, zlib, _opcode, _overlapped, _curses_panel, termios, _sha256, scproxy, cmath, _lsprof, unicodedata) * Nick Coghlan (PEP 489, PEP 573) * Paulo Henrique Silva (modules: time, operator, _functools) * Petr Viktorin (PEP 489, PEP 573, PEP 630) * Stefan Behnel (PEP 489) * Victor Stinner (modules _string, mashal,_imp, _warnings, _thread) The work was scatted into many sub-issues, it was hard for me to track all persons who contributed, sorry about that! I only searched for "New changeset" in bpo-1635741. For me, it was a very pleasant collaborative work :-) Contributors wrote pull requests, I reviewed and merged them. Slowly, contributors started to review each others and shared some recipes for these tasks. == Close the very old bpo-1635741 issue == Today, in the main development branch, Python no longer leaks memory at exit for the simplest command: "python3 -c pass"! Using a Python debug build, you can check the "python -I -X showrefcount -c pass" command (if you get a negative reference count, see bpo-46449), or you can use a memory debugger like Valgrind. While the work is not 100% done, it's a great milestone (at least for me ;-)! 15 years after bpo-1635741 was reported, finally I can close it! Sadly, this bug will no see the bugs migration from Roundup (bugs.python.org) to GitHub ;-) There are still some static types which should be converted to heap types and some extensions which should be be ported to the multi-phase initialization API. But this work can be done in existing or new specific issues. Again, a *big* thanks to every single person who helped directly and indirectly on fixing this issue! Victor -- Night gathers, and now my watch begins. It shall not end until my death. [View Less]

4 3

RFC on PEP 680 -- tomllib: Support for Parsing TOML in the Standard Library
by Shantanu Jain Jan. 28, 2022

Jan. 28, 2022

Hello all, Link to the PEP: https://www.python.org/dev/peps/pep-0680/ Please redirect discussion to: https://discuss.python.org/t/13040 PEP 680 proposes adding the tomllib module to the standard library for parsing TOML <https://toml.io>. The main motivation is to address bootstrapping problems with pyproject.toml based builds. Thank you to everyone who has already left comments, suggestions and reacts on the Discuss thread! We welcome any further suggestions before submitting to the … [View More]

1 0

Dropping AC_C_CHAR_UNSIGNED from configure.ac?
by Miro Hrončok Jan. 25, 2022

Jan. 25, 2022

Hello Pythonistas. In (development branch of) Fedora, we have juts upgraded to GCC 12. It seems that the presence of AC_C_CHAR_UNSIGNED in Python's autotools files (configure.ac?) is causing the __CHAR_UNSIGNED__ symbol to be defined in pyconfig.h and that breaks some other packages with GCC 12. The GCC maintainers told us it is a reserved symbol, see https://bugzilla.redhat.com/2043555 for details. It seems that using AC_C_CHAR_UNSIGNED is not recommended and also not required, but I … [View More]

2 2

SC Acceptance: PEP 673 -- Self Type
by Petr Viktorin Jan. 24, 2022

Jan. 24, 2022

Hello Pradeep, James and Jelle, The Steering Council discussed PEP 673 -- Self Type, and unanimously decided to accept it. Congratulations! Please change the PEP status to Accepted, and merge the change to Python 3.11, at your convenience. Happy typing! - Petr (on behalf of the Python Steering Council)

1 0

The current state of typing PEPs
by Barry Warsaw Jan. 24, 2022

Jan. 24, 2022

Does PEP 563 or 649 satisfy static and dynamic typing needs? In the interest of full transparency, we want to let the Python community know that the Steering Council continues to discuss PEP 563 (Postponed Evaluation of Annotations) and PEP 649 (Deferred Evaluation Of Annotations Using Descriptors). We have concluded that we lack enough detailed information to make a decision in favor of either PEP. As we see it, adopting either PEP 563 or PEP 649 as the default would be insufficient. They … [View More]will not fully resolve the existing problems these PEPs intend to fix, will break some existing code, and likely don’t address all the use cases and requirements across the static and dynamic typing constituents. We are also uncertain as to the best migration path from the current state of affairs. Defer decision on PEP 563 and 649 in 3.11 As such, at this time, the only reasonable path forward that the SC sees is to defer the decision in Python 3.11 again, essentially keeping the 3.10 status quo. We know that this is far from ideal, but it’s also the safest path since we can’t clearly make the situation better, and we don’t have confidence that either PEP solves the problems once and for all. Pragmatically, we don’t want to make the situation worse, and we really don’t want to find ourselves back here again in a couple of releases because we overlooked an important requirement for a set of users. Calling for help from static and dynamic typing users This is also a call to you, the folks who both care deeply about typing in Python, and have a solid understanding of the subject matter (or willingness to learn) to help us! There are use cases we don’t know about, and unclear requirements from both the static and dynamic typing users. We need help in defining those requirements clearly, in uncovering the use cases we’re not aware of, and most importantly, being an interface to typing enthusiasts to help build consensus with either of the proposed PEPs, or some other solution that doesn’t yet have a PEP (such as suggestions made by Łukasz in his blog post here: https://lukasz.langa.pl/61df599c-d9d8-4938-868b-36b67fdb4448/). Volunteer to be a typing PEP shepherd We’re looking for someone(s) who can be neutral as to the solution, knowledgeable enough to understand all sides of the problem, willing to serve as not quite a PEP author, not a PEP Delegate, but more than a PEP shepherd. If you are that person, please let us know! Cheers, -Barry (on behalf of the Python Steering Council) [View Less]

29 81

Summary of Python tracker Issues
by Python tracker Jan. 21, 2022

Jan. 21, 2022

ACTIVITY SUMMARY (2022-01-14 - 2022-01-21) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7175 (-27) closed 51014 (+108) total 58189 (+81) Open issues with patches: 2878 Issues opened (59) ================== #24711: Document getpass.getpass behavior on ^C https://bugs.python.org/issue24711 reopened by iritkatriel #44133: Some C-API symbols (e.g. … [View More]

1 0

Discussion about crash tolerance feature for gdbm module
by Dong-hee Na Jan. 21, 2022

Jan. 21, 2022

Hi folks, From gdbm 1.21, gdbm supports the crash tolerance feature. see: https://www.gnu.org.ua/software/gdbm/manual/Crash-Tolerance.html I would like to introduce this feature since python standard library is the only gdbm binding library that is available for end-users. And this is also effort for end-users to provide the latest new features. If this feature is added following methods will be added. - 'x' flag will be added for extended gdbm format. - gdbm.gdbm_failure_atomic('path1', '… [View More]

2 6

Adding an `inspect` function to return the signature of a type?
by Steven Troxler Jan. 21, 2022

Jan. 21, 2022

I've been wondering whether it would make sense to have a function in `inspect` that returns the signature of a type, rather than the signature of a specific callable object. I'm not attached to any name for such a function, two ideas are `inspect.signature_of` or `inspect.signature_of_type`. ## Context: the behavior of `inspect.signature` The existing `inspect.signature` function specified in PEP 362 [0] returns the call signature of a particular object, in other words given ``` class Dog: … [View More] def __init__(self, color: str): self.color = color def __call__(self, command: str): print(command) ``` we will get ``` >>> import inspect >>> inspect.signature(Dog) <Signature (color: str)> >>> inspect.signature(Dog("brown")) <Signature (command: str)> ``` ## Outline of the idea In some cases it might be nice to be able to access the signature of an instance of `Dog` given the type `Dog`, without actually needing to instantiate it. Moreover, there are some cases where it isn't even possible to get a callable object, but it still might be nice to have access to an `inspect.Signature` object: - protocols cannot be instantiated. This is particularly relevant to getting Signatures because of callback protocols [1] - typing.Callable types describe a signature, but again can't be instantiated If if we had a function to operate on types, it would treat normal classes as if calling `inspect.signature` on the `__call__` method of an instance. For callable types we'd want to do something similar, although this brings up some issues: - the existing Signature requires parameter names, but a `typing.Callable` doesn't have any. - One option would be to invent names `__0`, `__1,` etc. Otherwise we'd have to make the name optional. - it's not obvious how to support PEP 612 ParamSpecs, and even less obvious how to support PEP 646 in full generality I have some code snippets running through how simple cases might work, as well as the edge cases I've thought of thus far: https://gist.github.com/stroxler/4760cbc5e49df2295cd0b524328b1c73 I'm undecided about whether the edge cases are an indication that this idea isn't worth considering. It's worth noting that most of the edge cases involve PEP 612 / 646 generics which are pretty rare. ## Why did I start thinking about this? In discussions of PEP 677 [2], a few folks suggested using `inspect.Signature` as inspiration for the runtime API (which we did). This got me thinking that it might be nice for code that uses runtime types, regardless of what we decide about PEP 677, to have an API that makes accessing signature information from a type easier. I don't personally have much need for this, but - when I chatted with Carl Meyer from the static python team at Meta he thought it could be handy - I could imagine it being useful for tools like ML frameworks, which I've worked on in the past, that interpret type annotations to auto-generate logic -- [0] PEP 362 Function Signatures https://www.python.org/dev/peps/pep-0362/ [1] Callback protocols https://mypy.readthedocs.io/en/stable/protocols.html#callback-protocols [2] PEP 677 Callable Type Syntax https://www.python.org/dev/peps/pep-0677/ [View Less]

1 0

RFC on PEP 673: Self Type
by Pradeep Kumar Srinivasan Jan. 20, 2022

Jan. 20, 2022

This PEP [1] introduces a simple and intuitive way to annotate methods and classmethods that return an instance of their class. Such methods and classmethods occur quite frequently, but the existing way to annotate them correctly is quite arcane and error-prone. The PEP introduces a special type `Self` to represent the type of the `self` parameter, similar to the `this` type in TypeScript and the `Self` type in Rust. We have implementations for mypy and pyright. The PEP does not affect CPython … [View More]

4 4

3.11 enhanced error location - can it be smarter?
by Mats Wichmann Jan. 20, 2022

Jan. 20, 2022

"When printing tracebacks, the interpreter will now point to the exact expression that caused the error instead of just the line." I get the motivation for better error messages, but there are scenarios where you can't provide useful new information. I've been lax in getting to testing the project I work on with 3.11, but here's some spew from one testcase (this is an intentionally triggered error - the testcase is testing predictable behavior in the face of this error, 3.11 didn't *cause* … [View More]

10 16

Restated typing issue for class fields.
by Vincent Risi Jan. 19, 2022

Jan. 19, 2022

I have written an xsd parser to generate python classes. I also have written a utility to read in an xml file to populate the python classes. As well as a utility to write xml from the python classes thus generated. For example the code below that I include here shows a small sample of the generated code. The __annotations__ for these are very easy to use by the utility code for doing their job. Using the typing list[x] and tuple[a,b] do not work as well as - [x] instead of list[x] - (a, b) … [View More]

2 1

SC Acceptance: PEP 646 -- Variadic Generics
by Barry Warsaw Jan. 19, 2022

Jan. 19, 2022

Hello Mark, Matthew, Pradeep, Vincent, and Guido, The Python Steering Council discussed the latest version of PEP 646 (Variadic Generics) at our last meeting, and have unanimously decided to accept the PEP. Congratulations! We want to specifically mention that we appreciate the way you called out the Python grammar changes required by the typing features you proposed. As we’ve said before, the Steering Council strongly believes that the typing language and the “general” Python programming … [View More]

3 2

Minor inconvenience: f-string not recognized as docstring
by Antoine Pitrou Jan. 19, 2022

Jan. 19, 2022

Hello, Currently, a f-string is not recognized as a docstring: >>> class C: f"foo" >>> C.__doc__ >>> This means you need to use a (admittedly easy) workaround: >>> class C: __doc__ = f"foo" >>> C.__doc__ 'foo' Shouldn't the former be allowed for convenience? Regards Antoine.

9 11

PEP 646 (Variadic Generics): final call for comments
by Matthew Rahtz Jan. 18, 2022

Jan. 18, 2022

Hi everyone, We've got to the stage now with PEP 646 that we're feeling pretty happy with it. So far though we've mainly been workshopping it in typing-sig, so as PEP 1 requires we're asking for some feedback here too before submitting it to the steering council. If you have time over the next couple of weeks, please take a look at the current draft and let us know your thoughts: https://www.python.org/dev/peps/pep-0646/ (Note that the final couple of sections are out of date; https://github.… [View More]

9 19

PEP 677 (Callable Type Syntax): Final Call for Comments
by Steven Troxler Jan. 18, 2022

Jan. 18, 2022

Hello all, We’d like to make a last call for discussion on PEP 677: Callable Type Syntax [1] before sending this to the Steering Council. For context, PEP 677 proposes an arrow-like callable syntax for python: ``` (int, str) -> bool # equivalent to Callable[[int, str], bool] ``` Thanks everyone for comments on our earlier thread [2]. We made some changes to the PEP, but not the proposal itself, in response to that discussion: - More discussion about readability of return types and … [View More]

7 10

Is anyone using 15-bit PyLong digits (PYLONG_BITS_IN_DIGIT=15)?
by Mark Dickinson Jan. 18, 2022

Jan. 18, 2022

tl;dr: I'd like to deprecate and eventually remove the option to use 15-bit digits in the PyLong implementation. Before doing so, I'd like to find out whether there's anyone still using 15-bit PyLong digits, and if so, why they're doing so. History: the use of 30-bit digits in PyLong was introduced for Python 3.1 and Python 2.7, to improve performance of int (Python 3) / long (Python 2) arithmetic. At that time, we retained the option to use 15-bit digits, for two reasons: - (1) use of 30-bit … [View More]

10 32

Changing PySequence and PyMapping checks
by Bar Harel Jan. 15, 2022

Jan. 15, 2022

There's a long time issue of trying to differentiate mappings and sequences in the C-API in a fast and reliable way. Due to recent changes, we might be able to do so at last, by checking tp_flags + str/bytes/bytearray which are considered unique. This might be a breaking change in the Stable ABI promise but one that can be considered a bug fix as the current behavior is lacking. What do you think? bpo: https://bugs.python.org/issue46376 Best regards, Bar Harel

3 2

[RELEASE] Python 3.10.2, 3.9.10, and 3.11.0a4 are now available
by Łukasz Langa Jan. 14, 2022

Jan. 14, 2022

Hi there, before we begin the usual round of release notes, please do note that the three new versions of Python released today do not contain Windows installers yet. This is temporary, due to a more complex than expected code signing certificate renewal. We’ve held the releases all week while the situation is getting resolved but the urgency of 3.10.2 in particular made us release without the Windows installers after all. We apologize for the inconvenience and are doing everything we can to … [View More]put the Windows installer in place as soon as possible. We’re rooting for both Ee Durbin and Steve Dower who are helping us resolve this. Thanks for your hard work! Hopefully, by this time next week, this will only be a footnote in release management history. The releases you’re looking at were all cursed in some way. What a way to start 2022! Besides the certificate hold up, Python 3.10.2 is an expedited release (you’ll want to upgrade, read below!), Python 3.11.0a4 had almost 20 (sic, twenty!) release blockers before being finally green, and Python 3.9.10 was made from a new M1 Mac on macOS Monterey which made the usually boring process quite a ride. We’re hoping 2022 won’t be this intense all year! <https://discuss.python.org/t/python-3-10-2-3-9-10-and-3-11-0a4-are-now-avai…>Python 3.10.2 Get it here: https://www.python.org/downloads/release/python-3102/ <https://www.python.org/downloads/release/python-3102/> <https://www.blogger.com/#> This is a special bugfix release ahead of schedule to address a memory leak that was happening on certain function calls when using Cython <https://github.com/cython/cython>. The memory leak consisted of a small constant amount of bytes in certain function calls from Cython code. Although in most cases this was not very noticeable, it was very impactful for long-running applications and certain usage patterns. Check bpo-46347 <https://bugs.python.org/issue46347> for more information. Upgrading existing Python 3.10 installations is highly recommended. Even though this is an expedited release, it still contains over 100 other bug fixes. See the change log <https://docs.python.org/release/3.10.2/whatsnew/changelog.html> for details. The next Python 3.10 maintenance release will be 3.10.3, currently scheduled for 2022-04-04. <https://discuss.python.org/t/python-3-10-2-3-9-10-and-3-11-0a4-are-now-avai…>Python 3.9.10 Get it here: https://www.python.org/downloads/release/python-3910/ <https://www.blogger.com/#> Python 3.9.10 is the ninth maintenance release of the legacy 3.9 series. Note: Python 3.10 is now the latest feature release series of Python 3. Python 3.9 micro-releases enter double digits! There’s been 130 commits since 3.9.9 which is a higher number of fixes for this stage of the life cycle compared to 3.8. See the changelog <https://docs.python.org/release/3.9.10/whatsnew/changelog.html> for details on what changed. As a reminder, on macOS, the default installer is now the new universal2 variant. It’s compatible with Mac OS X 10.9 and newer, including macOS 11 Big Sur and macOS 12 Monterey. Python installed with this variant will work natively on Apple Silicon processors. The next Python 3.9 maintenance release will be 3.9.11, currently scheduled for Pi Day '22 (2022-03-14). <https://discuss.python.org/t/python-3-10-2-3-9-10-and-3-11-0a4-are-now-avai…>Python 3.11.0a4 Get it here: https://www.python.org/downloads/release/python-3110a4/ <https://www.blogger.com/#> Python 3.11 is still in development. This release, 3.11.0a4, is the fourth of seven planned alpha releases. Alpha releases are intended to make it easier to test the current state of new features and bug fixes by the community, as well as to test the release process. During the alpha phase, features may be added up until the start of the beta phase (2022-05-06) and, if necessary, may be modified or deleted up until the release candidate phase (2022-08-01). Please keep in mind that this is a preview release and its use is not recommended for production environments. Many new features for Python 3.11 are still being planned and written. Among the new major new features and changes so far: PEP 657 <https://www.python.org/dev/peps/pep-0657/> – Include Fine-Grained Error Locations in Tracebacks PEP 654 <https://www.python.org/dev/peps/pep-0654/> – Exception Groups and except* The Faster CPython Project <https://github.com/faster-cpython> is already yielding some exciting results: this version of CPython 3.11 is ~ 19% faster on the geometric mean of the PyPerformance benchmarks <>, compared to 3.10.0. (Hey, fellow core developer, if a feature you find important is missing from this list, let Pablo know <mailto:pablogsal@python.org>.) The next pre-release of Python 3.11 will be 3.11.0a5, currently scheduled for Wednesday, 2022-02-02. <https://discuss.python.org/t/python-3-10-2-3-9-10-and-3-11-0a4-are-now-avai…>Python 3.6 is pining for the fjords Python 3.6 is no more. It’s an ex-Python. It has ceased to be. On December 23rd 2021 is has reached its end-of-life phase <https://www.python.org/dev/peps/pep-0494/> after five successful years. It’s been the first truly popular Python 3 release, introducing f-strings to the world and making big improvements to both asyncio (async generators!) and typing (variable annotations!). We’d like to congratulate Ned Deily @nad <https://discuss.python.org/u/nad> on successfully driving the 3.6 series to completion as Release Manager. He’s not fully retired yet, as 3.7, which he is also managing, is still receiving security patches until June 2023. <https://discuss.python.org/t/python-3-10-2-3-9-10-and-3-11-0a4-are-now-avai…>We hope you enjoy the new releases Your friendly release team, Pablo Galindo Salgado @pablogsal <https://discuss.python.org/u/pablogsal> Łukasz Langa @ambv <https://discuss.python.org/u/ambv> [View Less]

1 0

Summary of Python tracker Issues
by Python tracker Jan. 14, 2022

Jan. 14, 2022

ACTIVITY SUMMARY (2022-01-07 - 2022-01-14) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7202 ( +3) closed 50906 (+81) total 58108 (+84) Open issues with patches: 2886 Issues opened (48) ================== #32876: HTMLParser raises exception on some inputs https://bugs.python.org/issue32876 reopened by iritkatriel #46298: Automatically check … [View More]

1 0

[PSA] OpenSSL 3.0 support is preliminary and experimental
by Christian Heimes Jan. 12, 2022

Jan. 12, 2022

Hi, I would like to remind everybody that Python's support for OpenSSL 3.0 is preliminary [1]. Python compiles with OpenSSL 3.0.0 and simple code kinda works. However there are known performance regressions, missing features (e.g. usedforsecurity flag), and potential bugs cause by API incompatibilities. Due to the experimental state I advise against using Python with OpenSSL 3.0 in production. It may take a while until Python gains full support for the next version of OpenSSL. I have … [View More]

4 5

PEP 674: Disallow using macros as l-value
by Victor Stinner Jan. 12, 2022

Jan. 12, 2022

Hi, I propose to disallow using macros as l-value. Read and comment the plain text below, or read the PEP 674 online, https://python.github.io/peps/pep-0674/ While I'm not sure that the proposed changes are really controversial, I decided to write a formal PEP since the incompatible changes are not following the PEP 387 deprecation process and so a format PEP 387 exception is better. Well, the list of modified macros is also quite long. Moreover, a PEP is a way to document and announce the … [View More]changes ;-) The Py_TYPE() and Py_SIZE() changes are already approved the Steering Council, but I prefer to list them in the PEP: https://github.com/python/steering-council/issues/79 In practice, I'm only aware of 4 projects impacted by these changes, and I wrote the pythoncapi_compat project which updates automatically C extensions: add Python 3.11 support, without losing support for older Python versions. I already prepared major projects like Cython and numpy for these changes (in total, 14 impacted projects have already been updated). Victor --- PEP: 674 Title: Disallow using macros as l-value Author: Victor Stinner <vstinner(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30-Oct-2021 Python-Version: 3.11 Abstract ======== Incompatible C API change disallowing using macros as l-value to allow evolving CPython internals and to ease the C API implementation on other Python implementation. In practice, the majority of projects impacted by these incompatible changes should only have to make two changes: * Replace ``Py_TYPE(obj) = new_type;`` with ``Py_SET_TYPE(obj, new_type);``. * Replace ``Py_SIZE(obj) = new_size;`` with ``Py_SET_SIZE(obj, new_size);``. Rationale ========= Using a macro as a l-value -------------------------- In the Python C API, some functions are implemented as macro because writing a macro is simpler than writing a regular function. If a macro exposes directly a struture member, it is technically possible to use this macro to not only get the structure member but also set it. Example with the Python 3.10 ``Py_TYPE()`` macro:: #define Py_TYPE(ob) (((PyObject *)(ob))->ob_type) This macro can be used as a **r-value** to **get** an object type:: type = Py_TYPE(object); It can also be used as **l-value** to **set** an object type:: Py_TYPE(object) = new_type; It is also possible to set an object reference count and an object size using ``Py_REFCNT()`` and ``Py_SIZE()`` macros. Setting directly an object attribute relies on the current exact CPython implementation. Implementing this feature in other Python implementations can make their C API implementation less efficient. CPython nogil fork ------------------ Sam Gross forked Python 3.8 to remove the GIL: the `nogil branch <https://github.com/colesbury/nogil/>`_. This fork has no ``PyObject.ob_refcnt`` member, but a more elaborated implementation for reference counting, and so the ``Py_REFCNT(obj) = new_refcnt;`` code fails with a compiler error. Merging the nogil fork into the upstream CPython main branch requires first to fix this C API compatibility issue. It is a concrete example of a Python optimization blocked indirectly by the C API. This issue was already fixed in Python 3.10: the ``Py_REFCNT()`` macro has been already modified to disallow using it as a l-value. HPy project ----------- The `HPy project <https://hpyproject.org/>`_ is a brand new C API for Python using only handles and function calls: handles are opaque, structure members cannot be accessed directly,and pointers cannot be dereferenced. Disallowing the usage of macros as l-value helps the migration of existing C extensions to HPy by reducing differences between the C API and the HPy API. PyPy cpyext module ------------------ In PyPy, when a Python object is accessed by the Python C API, the PyPy ``cpyext`` module has to convert PyPy object to a CPython object. While PyPy objects are designed to be efficient with the PyPy JIT compiler, CPython objects are less efficient and increase the memory usage. This PEP alone is not enough to get rid of the CPython objects in the PyPy ``cpyext`` module, but it is a step towards this long term goal. PyPy already supports HPy which is a better solution in the long term. Specification ============= Disallow using macros as l-value -------------------------------- PyObject and PyVarObject macros ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * ``Py_TYPE()``: ``Py_SET_TYPE()`` must be used instead * ``Py_SIZE()``: ``Py_SET_SIZE()`` must be used instead "GET" macros ^^^^^^^^^^^^ * ``PyByteArray_GET_SIZE()`` * ``PyBytes_GET_SIZE()`` * ``PyCFunction_GET_CLASS()`` * ``PyCFunction_GET_FLAGS()`` * ``PyCFunction_GET_FUNCTION()`` * ``PyCFunction_GET_SELF()`` * ``PyCell_GET()`` * ``PyCode_GetNumFree()`` * ``PyDict_GET_SIZE()`` * ``PyFunction_GET_ANNOTATIONS()`` * ``PyFunction_GET_CLOSURE()`` * ``PyFunction_GET_CODE()`` * ``PyFunction_GET_DEFAULTS()`` * ``PyFunction_GET_GLOBALS()`` * ``PyFunction_GET_KW_DEFAULTS()`` * ``PyFunction_GET_MODULE()`` * ``PyHeapType_GET_MEMBERS()`` * ``PyInstanceMethod_GET_FUNCTION()`` * ``PyList_GET_SIZE()`` * ``PyMemoryView_GET_BASE()`` * ``PyMemoryView_GET_BUFFER()`` * ``PyMethod_GET_FUNCTION()`` * ``PyMethod_GET_SELF()`` * ``PySet_GET_SIZE()`` * ``PyTuple_GET_SIZE()`` * ``PyUnicode_GET_DATA_SIZE()`` * ``PyUnicode_GET_LENGTH()`` * ``PyUnicode_GET_LENGTH()`` * ``PyUnicode_GET_SIZE()`` * ``PyWeakref_GET_OBJECT()`` "AS" macros ^^^^^^^^^^^ * ``PyByteArray_AS_STRING()`` * ``PyBytes_AS_STRING()`` * ``PyFloat_AS_DOUBLE()`` * ``PyUnicode_AS_DATA()`` * ``PyUnicode_AS_UNICODE()`` PyUnicode macros ^^^^^^^^^^^^^^^^ * ``PyUnicode_1BYTE_DATA()`` * ``PyUnicode_2BYTE_DATA()`` * ``PyUnicode_4BYTE_DATA()`` * ``PyUnicode_DATA()`` * ``PyUnicode_IS_ASCII()`` * ``PyUnicode_IS_COMPACT()`` * ``PyUnicode_IS_READY()`` * ``PyUnicode_KIND()`` * ``PyUnicode_READ()`` * ``PyUnicode_READ_CHAR()`` PyDateTime "GET" macros ^^^^^^^^^^^^^^^^^^^^^^^ * ``PyDateTime_DATE_GET_FOLD()`` * ``PyDateTime_DATE_GET_HOUR()`` * ``PyDateTime_DATE_GET_MICROSECOND()`` * ``PyDateTime_DATE_GET_MINUTE()`` * ``PyDateTime_DATE_GET_SECOND()`` * ``PyDateTime_DATE_GET_TZINFO()`` * ``PyDateTime_DELTA_GET_DAYS()`` * ``PyDateTime_DELTA_GET_MICROSECONDS()`` * ``PyDateTime_DELTA_GET_SECONDS()`` * ``PyDateTime_GET_DAY()`` * ``PyDateTime_GET_MONTH()`` * ``PyDateTime_GET_YEAR()`` * ``PyDateTime_TIME_GET_FOLD()`` * ``PyDateTime_TIME_GET_HOUR()`` * ``PyDateTime_TIME_GET_MICROSECOND()`` * ``PyDateTime_TIME_GET_MINUTE()`` * ``PyDateTime_TIME_GET_SECOND()`` * ``PyDateTime_TIME_GET_TZINFO()`` PyDescr macros ^^^^^^^^^^^^^^ * ``PyDescr_NAME()`` * ``PyDescr_TYPE()`` Port C extensions to Python 3.11 -------------------------------- In practice, the majority of projects impacted by these PEP incompatible changes should only have to make two changes: * Replace ``Py_TYPE(obj) = new_type;`` with ``Py_SET_TYPE(obj, new_type);``. * Replace ``Py_SIZE(obj) = new_size;`` with ``Py_SET_SIZE(obj, new_size);``. The `pythoncapi_compat project <https://github.com/pythoncapi/pythoncapi_compat>`_ can be used to update automatically C extensions: add Python 3.11 support without losing support with older Python versions. The project provides a header file which provides ``Py_SET_REFCNT()``, ``Py_SET_TYPE()`` and ``Py_SET_SIZE()`` functions to Python 3.8 and older. PyTuple_GET_ITEM() and PyList_GET_ITEM() ---------------------------------------- The ``PyTuple_GET_ITEM()`` and ``PyList_GET_ITEM()`` macros are left unchanged. The code pattern ``&PyTuple_GET_ITEM(tuple, 0)`` and ``&PyList_GET_ITEM(list, 0)`` is still commonly used to get access to the inner ``PyObject**`` array. Changing these macros would require to add a new API to get access to the inner array which is out of the scope of this PEP. Backwards Compatibility ======================= The proposed C API changes are backward incompatible on purpose. In practice, only a minority of third party projects are affected (16 projects are known to be broken) and `most of them have already been updated for these changes <https://bugs.python.org/issue39573#msg401378>`__ (12 on 16). Most projects are broken by ``Py_TYPE()`` and ``Py_SIZE()`` changes. These two macros have been converted to static inline macro in Python 3.10 alpha versions, but the change had to be reverted since it broke too many projects. In the meanwhile, many projects, like Cython, have been prepared for this change by using ``Py_SET_TYPE()`` and ``Py_SET_SIZE()``. For example, projects using Cython only have to regenerate their outdated generated C code to become compatible. For the "GET" functions like ``PyDict_GET_SIZE()``, no project in the PyPI top 5000 projects use these functions as l-value. The ``PyFloat_AS_DOUBLE()`` function is not used as a l-value in the PyPI top 5000 projects. The ``PyBytes_AS_STRING()`` and ``PyByteArray_AS_STRING()`` are used as l-value but only to modify string characters, not to override the ``PyBytesObject.ob_sval`` or ``PyByteArrayObject.ob_start`` member. For example, Cython uses the following code which remains valid:: PyByteArray_AS_STRING(string)[i] = (char) v; This change does not follow the PEP 387 deprecation process. There is no known way to emit a deprecation warning when a macro is used as a l-value, but not when it's used differently (ex: r-value). Rejected Idea: Leave the macros as they are =========================================== The documentation of each function can discourage developers to use macros to modify Python objects. If these is a need to make an assignment, a setter function can be added and the macro documentation can require to use the setter function. For example, a ``Py_SET_TYPE()`` function has been added to Python 3.9 and the ``Py_TYPE()`` documentation now requires to use the ``Py_SET_TYPE()`` function to set an object type. If developers use macros as l-value, it's their responsibility when their code breaks, not the Python responsibility. We are operating under the consenting adults principle: we expect users of the Python C API to use it as documented and expect them to take care of the fallout, if things break when they don't. This idea was rejected because only few developers read the documentation, and only a minority is tracking changes of the Python C API documentation. The majority of developers are only using CPython and so are not aware of compatibility issues with other Python implementations. Moreover, continuing to allow using macros as l-value does not solve issues of the nogil, PyPy and HPy projects. Macros already modified ======================= The following C API macros have already been modified to disallow using them as l-value: * ``PyCell_SET()`` * ``PyList_SET_ITEM()`` * ``PyTuple_SET_ITEM()`` * ``Py_REFCNT()`` (Python 3.10): ``Py_SET_REFCNT()`` must be used * ``_PyGCHead_SET_FINALIZED()`` * ``_PyGCHead_SET_NEXT()`` * ``asdl_seq_GET()`` * ``asdl_seq_GET_UNTYPED()`` * ``asdl_seq_LEN()`` * ``asdl_seq_SET()`` * ``asdl_seq_SET_UNTYPED()`` For example, ``PyList_SET_ITEM(list, 0, item) < 0`` now fails with a compiler error as expected. References ========== * `Python C API: Add functions to access PyObject <https://vstinner.github.io/c-api-abstract-pyobject.html>`_ (October 2021) article by Victor Stinner * `[C API] Disallow using PyFloat_AS_DOUBLE() as l-value <https://bugs.python.org/issue45476>`_ (October 2021) * `[capi-sig] Py_TYPE() and Py_SIZE() become static inline functions <https://mail.python.org/archives/list/capi-sig@python.org/thread/WGRLTHTHC3…>`_ (September 2021) * `[C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type <https://bugs.python.org/issue39573>`__ (February 2020) * `bpo-30459: PyList_SET_ITEM could be safer <https://bugs.python.org/issue30459>`_ (May 2017) Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. [View Less]

10 25

Status of Python 3.11.0a4
by Pablo Galindo Salgado Jan. 11, 2022

Jan. 11, 2022

Hi everyone, I want to report on the status of Python 3.11.0a4. We had a ton of release blockers (some extra ones since I reported the last time) and it seems that we managed to fix them all (thanks to Mark Shannon, Christian Heimes, Gregory P. Smith, Neil Schemenauer, Steve Dower and many others that helped with the blockers. Unfortunately it seems that the release is cursed and we are having some problems with the certificates to create and sign the Windows artefacts so we are waiting for … [View More]

1 0

PEP 676 -- PEP Infrastructure Process
by python＠quite.org.uk Jan. 10, 2022

Jan. 10, 2022

Hi, I would like to announce PEP 676 to python-dev. It is a meta-PEP focussed on modernising the PEP build infrastructure. From the abstract, "This PEP addresses the infrastructure around rendering PEP files from reStructuredText files to HTML webpages. We aim to specify a self-contained and maintainable solution for PEP readers, authors, and editors." Link: https://www.python.org/dev/peps/pep-0676/ Rendered through the PEP 676 system: https://python.github.io/peps/pep-0676/ Please see … [View More]

4 3

Core review request: bpo-33178: Add BigEndianUnion, LittleEndianUnion classes to ctypes #25480
by Dave Goncalves Jan. 10, 2022

Jan. 10, 2022

Apologies for one of these review request emails. Anybody with ctypes experience in the core reviewer team available to review and comment/merge? This has gotten reviewed and passes checks. Python bug: https://bugs.python.org/issue33178 Current PR: https://github.com/python/cpython/pull/25480 Thanks, David Goncalves

1 0

PEP 679 – Allow parentheses in assert statements
by Pablo Galindo Salgado Jan. 10, 2022

Jan. 10, 2022

Hi everyone, I would like to start a discussion about a small PEP proposal to allow parentheses in assert statements to fix a common gotcha with assert statements. Link to the PEP: https://www.python.org/dev/peps/pep-0679/ *Please, redirect all discussions to: * https://discuss.python.org/t/pep-679-allow-parentheses-in-assert-statements… as I will not be monitoring answers to this thread. Thanks, everyone for your time! Regards from cloudy London, Pablo Galindo Salgado

6 7

About vulnerabilities in Cpython native code
by lxr1210＠mail.ustc.edu.cn Jan. 9, 2022

Jan. 9, 2022

Hi all, I am currently doing some research on the security of CPython. I used the open source vulnerability analysis engine, Infer(https://fbinfer.com/), to scan the native code of CPython 3.10.0. The scan results show that there are still a number of vulnerabilities in the CPython native code, such as Null dereference, Uninitialized variable, Resource/Memory leak, etc. Moreover, I found that some of the vulnerabilities are related to Python/C API. I enclose the vulnerability report for … [View More]

6 10

Python 3.6 rides into the sunset
by Ned Deily Jan. 8, 2022

Jan. 8, 2022

While there are many reasons to welcome the end of 2021, I would like to give a shout-out to Python 3.6 which officially reached end-of-life on 2021-12-23, 6.5 years after its development began and exactly five years after its initial release. Building on the success of previous Python 3 releases, 3.6 added many new features and improvements too numerous to list: over 10000 commits by some 160 contributors, including one of the most popular features in recent Python releases, f-strings (thanks,… [View More] EVS!). I think it is fair to say that, with Python 3.6, the long transition from Python 2 was finally settled. As release manager for 3.6, I would like to personally thank all those contributors, and mostly volunteers: you made my job an easy one with your overwhelmingly positive attitude and support. I would also like to thank the authors and maintainers of the many third-party packages that were updated to support 3.6 as well as the downstream distributors of Python whose rapid uptake and release of 3.6 in their distributions was crucial to its success. I would also like to thank those who helped get the 3.6 releases out the door, in particular, Steve Dower for manufacturing the Windows packages, Julien Palard for managing our on-line documentation build process, Elvis Pranskevichus and Yury Selivanov for taking on the thankless task of assembling and editing the 3.6 "What's New" document, my fellow release managers for their encouragement and support from the start to finish of 3.6's life, the Steering Councils, the PSF Infrastructure Team, those individuals and organizations who contribute resources (money, people, time, facilities, services) to the PSF, making Python development possible. And, I suppose I should thank that git who produces the macOS packages. Thanks again to you all for making 3.6 so successful! --Ned P.S. As a reminder, with 3.6 having reached end-of-life, we no longer accept bug reports of any type against 3.6 and the 3.6 source code is now frozen. There is no longer a 3.6 branch in the GitHub cpython repository; the final state of the branch is captured in the repo as tag "3.6" and, as always, the source code for any release can be checked out using its tag; for example, the source for the final release of 3.6 can be obtained with "git checkout v3.6.15". Pro tip: if you haven't already, you may want to update your repo clones with "git fetch --tags upstream" (if you use the recommended naming convention) to get the latest tags and branches. -- Ned Deily nad(a)python.org -- [] [View Less]

1 0

Summary of Python tracker Issues
by Python tracker Jan. 7, 2022

Jan. 7, 2022

ACTIVITY SUMMARY (2021-12-31 - 2022-01-07) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7199 ( +4) closed 50825 (+78) total 58024 (+82) Open issues with patches: 2886 Issues opened (50) ================== #34931: os.path.splitext with more dots https://bugs.python.org/issue34931 reopened by xnovakj #46009: sending non-None values makes … [View More]

1 0

Sanity check about ctypes
by Yonatan Zunger Jan. 6, 2022

Jan. 6, 2022

Hey everyone. Quick sanity check: The ctypes docs <https://docs.python.org/3.10/library/ctypes.html#ctypes._CData> refer to _CData as a non-public class which is in the module, but _ctypes.c doesn't actually export it <https://github.com/python/cpython/blob/main/Modules/_ctypes/_ctypes.c#L5680>. (I discovered this because it turns out that typeshed *is* referencing _CData, e.g. in its type annotations for RawIOBase <https://github.com/python/typeshed/blob/master/stdlib/_typeshed/… [View More]

2 2

Python 3.11.0a4 is blocked
by Pablo Galindo Salgado Jan. 6, 2022

Jan. 6, 2022

Hi everyone, I am writing this to notify you that unfortunately the release of 3.11.0a4 is blocked as there are a bunch of release blockers (some of them affect Python 3.10): https://bugs.python.org/issue46263 https://bugs.python.org/issue46208 https://bugs.python.org/issue46006 https://bugs.python.org/issue43683 If this was a single release blocker I would think about moving forward but unfortunately, there are several of them and one of them is that Python fails to compile FreeBSD, so I … [View More]

3 3

my plans for subinterpreters (and a per-interpreter GIL)
by Eric Snow Jan. 6, 2022

Jan. 6, 2022

Hi all, I'm still hoping to land a per-interpreter GIL for 3.11. There is still a decent amount of work to be done but little of it will require solving any big problems: * pull remaining static globals into _PyRuntimeState and PyInterpreterState * minor updates to PEP 554 * finish up the last couple pieces of the PEP 554 implementation * maybe publish a companion PEP about per-interpreter GIL There are also a few decisions to be made. I'll open a couple of other threads to get feedback on … [View More]

14 29

Function Prototypes
by asleep.cult＠gmail.com Jan. 5, 2022

Jan. 5, 2022

Hello In thread about PEP 677, I (with the help of another proposal) came up with an alternative to typing.Callable called function prototypes. The basic concept is to create a function prototype object when the body of a function is omitted. The thread can be found here: https://mail.python.org/archives/list/python-dev@python.org/thread/OGACYN2X… Mark Shannon initially proposed that functions be used as types and provided this example: @Callable def IntToIntFunc(a:int)->int: pass … [View More]

14 27

RFC on Callable Syntax PEP
by Steven Troxler Jan. 4, 2022

Jan. 4, 2022

Hello all, Thanks everyone for comments on our earlier thread [1] about callable type syntax. We now have a draft PEP [2] proposing an arrow-based syntax for callable types, for example: ``` (int, str) -> bool # equivalent to Callable[[int, str], bool] ``` In support of the PEP we also have: - a reference implementation of the parser [3] to ensure the grammar is correct (tests [5], [6], [7]) - a detailed specification of planned runtime behavior [4], which is not yet in the … [View More]

13 22