first of all, thanks for working on variadic generics! I look forward to them.
My question: all the examples in https://www.python.org/dev/peps/pep-0646/ unpack into variadic arguments. But can I write code like this?
Ts = TypeVarTuple("Ts")
def enumerate_args(f: Callable[[*Tuple[int, Ts]]], *args: *Ts):
In particular I'm talking about the `*Tuple[int, Ts]` syntax. All the examples from the PEP use `*Ts` so I don't know if this is legal, but I hope so.
This should probably be clarified in the PEP.
I’m getting increasingly worried about the future of Python, and those worries have been exacerbated by the new yearly release rhythm, and the lack of a stable C-API that is full enough to entice extension writers to use it.
PyPI packages and wheels are targeted to specific Python versions, which means that any project that depends on some of the larger extension packages (of which there are many, and many of which are must-have for many projects) now start lagging Python versions by years, because somewhere deep down in the dependency graph there is something that is still stuck at Python 3.8 (for example). I fully understand that 3.8 is good enough for the developers of that package, and that they have more pressing things to do than porting to 3.9 or 3.10, but it now keeps any project or package that depends on their software on 3.8 as well.
And I also fully understand that some other developer who creates a new package that is essential to my application only targets the current Python release, or maybe one release back, but now if I need both the new package and and older one I’m up the well-known creek without a paddle.
Building packages from source has become pretty much impossible nowadays, especially if your project is multi-platform and needs to interface to specific hardware, and you want to do the right thing with CI/CD builds and all that. On Linux/MacOS you have a chance when you try to specify all the dependencies for third party libraries and what not, but on Windows you’re dead in the water. And that is assuming you have the time and are smart enough to back port the new package to the old Python release, or the old package to the new Python release (and for the latter there’s probably a good reason why the developers haven’t done so already). Before you know it you have to install a couple of graphics card APIs for some obscure AI feature used by something you’ve never heard of, Cython for something else, and obscure vendor libraries for something else again.
I think we really need to come up with some scheme whereby extension packages become more long-lived than a single Python release...
Jack Jansen, <Jack.Jansen(a)cwi.nl>, http://www.cwi.nl/~jack
If I can't dance I don't want to be part of your revolution -- Emma Goldman
Is an implementation of Python free to make up its own answers to
division and modulus operatons in the case of inf and nan arguments?
The standard library documentation says only that // is "rounded towards
minus infinity". The language reference says that ||:
1. |x == (x//y)*y + (x%y)|,
2. the modulus has the same sign as y, and
3. division by (either kind of) zero raises |ZeroDivisionError| .
It's consistent, but it doesn't define the operator over the full range
of potential arguments. In Python 3.8 and 3.9:
>>> from decimal import Decimal
>>> Decimal('-3.14') // Decimal('Infinity')
>>> -3.14 // float("inf")
>>> import math
>>> math.floor(-3.14 / float("inf"))
I can see sense in all three answers, as possible interpretations of
"rounded towards minus infinity", but I quite like decimal's. There seem
to be no regression tests for floor division of floats, and for modulus
only with finite arguments, perhaps intentionally.
As it seems that I didn't have enough work dealing with the multiple
release blockers that have been raised these weeks I am also preparing a
Python 3.10 release party.
We will be doing an stream, co-hosted by the good people of Python discord
channel. The URL for the stream is this one:
The release party will start on October 4th at 17:00 UTC+0.
In this stream, some core developers (Irit, Carol, Brandt and Łukasz) will
be talking about Python 3.10 features and some sneak peak on Python 3.11
and beyond, talking about the core Dev in residence role and other things
with some time for moderated Q&A. I will be doing the release live and will
also cover what I am doing. Is a fantastic opportunity to watch me make
mistakes live :)
The main purpose of the stream is to raise excitement about Python 3.10,
explain the new features to users, trying to get the community closer to
the core Dev team and allowing them to meet some of the new core Devs
(Brandt and Irit). Also, funny party hats will be used!
I hope you find the event interesting. Python 3.10 is going to be a
fantastic release and we want it to be even better :)
Please, reach out to me if you have any question or suggestion.
Regards from very rainy London,
Pablo Galindo Salgado
I would like to change the Python C API. I failed to write a single
document listing all constraints and proposing all changes that I
would to do. For example, my previous PEP 620 contains too many
changes and is too long.
Here is my attempt to focus on the bare minimum and (what I consider
as) the least controversial part: list current usages of C API and
constraints of these usages. This *informal* PEP should be the base of
future PEP changing the C API.
The current draft lives at:
My PEP is based on HPy Next Level Manifesto written by Simon Cross:
To reach most users of the C API, I cross-posted this email to
python-dev, capi-sig and hpy-dev.
Taking the Python C API to the Next Level
Title: Taking the Python C API to the Next Level
Author: Victor Stinner <vstinner(a)python.org>
While the C API is a key of the Python popularity, it causes multiple
subtle and complex issues. There are different ways to use the C API,
each usage has its own constraints, and some constraints are exclusive.
This document lists constraints but doesn't propose changes, it only
gives vague ideas how to solve some issues. More concrete C API changes
will require writing separated PEPs.
C extensions are a key component of the Python popularity
The Python popularity comes from its great programming language and from
its wide collection of modules freely available on PyPI. Many of the
most popular Python modules rely directly or indirectly on C extensions
written with the C API. The Python C API is a key component of the
For example, the numpy project is now a common dependency on many
scientific projects and a large part of the project is written by hand
with the C API.
**Abandoning or removing the C API** is out of question. Years ago, the
incomplete C API support was the main drawback of PyPy, since PyPy only
supported a minority of C extensions.
Today, CPython still have a similar issue. **When Cython or numpy don't
support a new Python version** (because of incompatible C API changes),
many Python projects depending on them are cannot be installed,
especially during the development phase of the next Python version.
Backward compatibility and unmaintained C extensions
One important property of the C API is the backward compatibility.
Developers expect that if their C extension works on Python 3.10, it
will work unmodified in Python 3.11: building the C extension with
Python 3.11 should be enough.
This property is even more important for unmaintained C extensions.
Sometimes, unmaintained just means that the only maintainer is busy or
overwhelmed for a few months. Sometimes, the project has no activity for
longer than 5 years.
When an incompatible change is introduced in the C API, like removing a
function or changing a function behavior, there is a **risk of breaking
an unknown number of C extensions**.
One option can be to update old C extensions when they are built on
recent Python versions, to adapt them to incompatible changes. This
conversion is non trivial and cannot handle all kinds of incompatible
Migration plan for incompatible changes
There should be a **sensible migration path** for large C extensions
(e.g. numpy) when incompatible changes are introduced. Whenever
possible, it should be possible to write a **single code base** compatible
with old and new Python versions.
A **compatibility layer** can be maintained externally. Cython and
numpy have their own internal compatibility layer.
There should be a way to easily pick up common errors introduced by
One practical way to **minimize the number of broken projects** is to
attempt to check in advance if an incompatible change is going to break
popular C extensions. For broken C extensions, propose a fix and wait
until a new release includes the fix, before introducing the change in
Python. Obviously, it doesn't solve the problem of less popular C
extensions and private C extensions.
Obtain the best possible performance
There are two main reasons for writing a C extension: implement a
function which cannot be written in pure Python, or write a **C
accelerator**: rewrite the 10% of an application in C where 90% of the
CPU time is spent. About the former use case, the intent is to obtain
the best possible performance. Tradeoffs are made with portability: it
is acceptable to only support a limited number of Python versions and to
only support a limited number of Python implementations (usually only
Cython is a good example of accelerator. It is able to support a large
number of Python versions and multiple Python implementation with
compatibility layers and ``#ifdef``. The main drawback is that it is
common that Cython is **broken by incompatible changes made at each
Python release**. It happens because Cython relies on many
On the other side, the **limited C API** is a small as possible,
excludes implementation details on purpose, and provides a stable ABI.
Building a C extension with the limited C API only once produces a
binary wheel package usable on many Python versions, but each platform
still requires its own binary wheel package.
Emulating the current C API is inefficient
The PyPy project is a Python implementation written from scratch, it was
not created as a CPython fork. It made many implementation choices
different than CPython: no reference counting, moving garbage collector,
JIT compiler, etc.
To support C extensions, PyPy emulates the Python C API in its cpyext
module. When the C API access an object, cpyext has to convert the PyPy
object to a CPython object (``PyObject``). CPython objects are less
efficient than PyPy objects with the PyPy JIT compiler and conversions
from PyPy objects to CPython objects are also inefficient. PyPy has to
reimplement every single detail of the CPython implementation to be as
much compatible as possible.
The C API exposes multiple implementation details:
* Reference counting, borrowed references, stealing references.
* Objects location in memory.
* Rely on pointers for object identity: Python 3.10 adds the ``Py_Is()``
function to solve this problem.
* Expose the memory layout of Python objects as part of the API.
* Expose static types.
* Implicit execution context.
The C API of Python 3.10 is made of around 15 000 lines of C header
files, 1500 functions and 100 structures. Supporting the full C API is a
significant amount of work.
**Freezing the C API** for a few Python releases would help other Python
implementations to catch up with the latest Python version, but it
doesn't solve the efficiency problem. Moreover, it is common that adding
a new feature to Python requires to change the C API, even if it is just
to add new functions. Not adding new features to Python for a few Python
releases is out of question.
The C API prevents optimizing CPython
It is challenging to evolve the C API to optimize CPython without
breaking the backward compatibility. Emulating the old C API is an
option, but it is inefficient.
If everything above is achievable -- and we believe it is! -- we'll
arrive in a wonderful new future where Python implementations can
experiment with all sorts of amazing new features:
* tracing garbage collectors;
* nurseries for short-lived objects;
* sub-interpreters with separate contexts;
* specialised implementations of lists;
* removing the GIL;
* avoiding the boxing of primitive types;
* just-in-time compilation;
* ... and many other things you can imagine that we haven't!
No one can guarantee that a particular new idea will work out, but
exposing fewer implementation details via the C API will make it
possible to try many new things.
Night gathers, and now my watch begins. It shall not end until my death.
We've frozen most of the stdlib modules imported during "python -c
pass" , to make startup a bit faster. Import of those modules
is controlled by "-X frozen_modules=[on|off]". Currently it defaults
to "off" but we'd like to default to "on". The blocker is the impact
on contributors. I expect many will make changes to a stdlib module
and then puzzle over why those changes aren't getting used. That's an
annoyance we can avoid, which is the point of this thread.
1. always default to "on" (the annoyance for contributors isn't big enough?)
2. default to "on" if it's a PGO build (and "off" otherwise)
3. default to "on" unless running from the source tree
 FWIW, we may end up also freezing the modules imported for "python
-m ...", along with some other commonly used modules (like argparse).
That is a separate discussion.
I accidentally a word.
"... module's hash and/or *timestamp* and comparing it ..."
On 27/09/2021 18:38, Patrick Reader wrote:
> How about checking each non-frozen module's hash and/or and comparing it to that of the frozen module? Would that defeat the performance improvement of freezing? Is it just a terrible idea?
> On 27/09/2021 17:51, Eric Snow wrote:
>> We've frozen most of the stdlib modules imported during "python -c
>> pass" , to make startup a bit faster. Import of those modules
>> is controlled by "-X frozen_modules=[on|off]". Currently it defaults
>> to "off" but we'd like to default to "on". The blocker is the impact
>> on contributors. I expect many will make changes to a stdlib module
>> and then puzzle over why those changes aren't getting used. That's an
>> annoyance we can avoid, which is the point of this thread.
>> Possible solutions:
>> 1. always default to "on" (the annoyance for contributors isn't big enough?)
>> 2. default to "on" if it's a PGO build (and "off" otherwise)
>> 3. default to "on" unless running from the source tree
This PR https://github.com/python/cpython/pull/28310 changes the message
for some exceptions.
>>> format('', '%M')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: Invalid format specifier
With the proposed change:
>>> format('', '%M')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: Invalid format specifier '%M' for object of type 'str'
This also applies to str.format() and f-strings.
This is clearly an improvement. My question is: is it okay to backport
this to 3.9 and 3.10? I think we have a rule that it's okay to change
the exception text, but I'm not sure how that applies to micro releases
Irit, Guido, Yuri, Nathaniel,
(In the interest of visibility, this was also posted to discuss.python.org
After careful consideration and rather lengthy discussion, the Steering
Council has decided to accept PEP 654 (Exception Groups and except*)
<https://www.python.org/dev/peps/pep-0654/>. We do have two requests, one
of which we mentioned before. We’re very pleased with the amount of care
and discussion that went into this PEP, and we’re satisfied that it solves
a real-world problem in the least disruptive way. We do remain open for
further tweaking of the design, incorporating some of Nathaniel’s ideas, or
new ones, but we want to move forward with PEP 654’s proposal and let
people get hands-on experience with it. To that end, our request is that we
get at least one non-toy use of ExceptionGroups somewhere in the standard
library (targeted to 3.11, like PEP 654), and preferably get some
third-party experimentation with it as well. The other request is that PEP
654 be updated with a reference to Nathaniel’s ideas (a link to the
Discourse discussion), outlining why PEP 654 rejects those ideas (it could
just be a copy-paste of some of the discussion, just so long as it’s
recorded in the PEP).
Regarding Nathaniel’s alternative proposal
I want to be clear on the main reason why we’re accepting PEP 654 rather
than waiting for a more complete alternative PEP with concrete examples.
The SC agrees that exception groups solve a real problem, albeit a
relatively rare and low-level one. We don’t think the problem it solves is
big enough that it warrants fundamental changes to the language; changes
that would affect the way everyone would write code, or that would make
learning Python or reasoning about Python code harder. PEP 654 just about
squeaks by those criteria: the new syntax is something new to learn about
Python, but it’s not something that’s *required *unless consuming an API
that uses ExceptionGroups. ExceptionGroups behave like regular exceptions
in code that does not expect them, and do not change the behaviour of
existing try/except. In code that potentially produces ExceptionGroups (or
its documentation, examples, tutorials, etc) there is enough opportunity to
document that, show examples of how to use it, explain the implications and
refer to documentation about the concept of ExceptionGroups.
By contrast, Nathaniel’s proposal that ExceptionGroups affect how *existing
*try/except statements behave, fails those criteria. By sheer nature of
doing something meaningful (if confusing) in try/except, it would encourage
API designs that let ExceptionGroups escape to unsuspecting code. It would
introduce subtle bugs in code that -- entirely legitimately -- expects only
one except block to be executed, or only executed once. These problems
would be very hard to detect since they would only show up if called code
could *in practice* raise ExceptionGroups. We don’t consider this a viable
The SC does remain open to proposals to extend PEP 654 (like adding an API
to iterate over ExceptionGroups), or change some of the semantics (like
flat versus nested ExceptionGroups), and we do not want to discourage
discussion on any of this. We do want to move forward with PEP 654 as-is in
the meantime, in particular to get practical experience in the standard
With thanks for all your hard work,
For the whole SC,
Thomas Wouters <thomas(a)python.org>
Hi! I'm an email virus! Think twice before sending your email to help me