Hi everyone,
# TLDR
We may be pushing the final release until December if the stability of
Python 3.11 doesn't improve.
# Long Explanation
Unfortunately, we cannot still release the next Python 3.11 beta release
(3.11.0b4) because we still have a bunch
of pending release blockers. Unfortunately, this is still after several
preexisting release blockers have been fixed,
some of them being discovered after I sent my last update. These are the
current release blockers:
https://github.com/python/cpython/issues?q=is%3Aissue+is%3Aopen+sort%3Aupda…
We also have some deferred blockers (some of them should actually be
release blockers):
https://github.com/python/cpython/issues?q=is%3Aissue+is%3Aopen+sort%3Aupda…
Due to this and the fact that we are already 3 weeks delayed in the release
schedule, the current stability of Python 3.11 is not as good
as it is supposed to be at this stage of the release schedule and more
testing from end-users and library authors is required. After
discussing with the Steering Council, we are considering delaying the final
release until December to allow for two more beta releases.
This is how we are going to proceed:
* If the current release blockers are not fixed by the end of this week,
two more betas will be released (1 month per beta) and
we will *definitely* delay the final release until December.
* If the current release blockers are fixed we will proceed to release
Python 3.11.0b4 on Monday. We will target the current release
date (Monday, 2022-10-03) but if more release blockers that affect
fundamental parts of the Python interpreter or the standard libraries
are raised, the release team will still consider adding two more betas nd
pushing the final release to December.
One of the goals that we are going to try to achieve from the release team
is that no substantial code changes are added between the last
beta and the first release candidate. This is so all the fixes that affect
fundamental parts of the interpreter or the standard library can be
tested by end-users before the first release candidate is released (and not
with it). This is also partially because once we release the first release
candidate, the ABI will be frozen and certain kinds of fixes will be more
complicated.
Hopefully, this addresses some of you that have reached out with concerns
over the stability of Python 3.11 and the release schedule.
I understand that delaying the release until December will complicate
things for some Linux distributions and will affect end users and
redistributors
targeting the original release, but please understand that our
responsibility in the release team after all is to guarantee a stable final
release
above all and unfortunately, we don't currently have the confidence that we
would like given the current state of the release process.
Please do not hesitate in reaching out if you have any questions or
concerns.
Thanks, everyone for your help and understanding and thanks a lot to all of
you for your great work!
Cheers from cloudy London,
Pablo Galindo Salgado
Hello, I'd like to bring up https://bugs.python.org/issue44951 /
https://github.com/python/cpython/pull/27819 to the mailing list's
consideration as it has idled for a bit. I would appreciate some
authoritative feedback on which API design choice is best. I'll also
recap the PR quickly:
Motivation:
- There is community demand for EPOLLEXCLUSIVE in Python (see blog
posts in BPO issue)
- You can't do this with the existing stdlib code as
_BaseSelectorImpl.register() raises ValueError on non-READ/WRITE
constants. (This is why _PollLikeSelector.register() has two
variables, events and poller_events)
- It's not an invasive change. The Python API doesn't change much or
at all and the kernel's EPOLLEXCLUSIVE behavior was carefully designed
to be backwards compatible with using exclusive and non-exclusive
watches on the same file descriptor.
I've got two approaches to this. The first extends the EpollSelector
class with a property to toggle setting the EPOLLEXCLUSIVE on newly
registered file descriptors:
https://github.com/dgilman/cpython/commit/43174df5bd7a78eedf0692ebbe63a9b94…
The second introduces an entirely new sibling class,
EpollExclusiveSelector, that unconditionally sets it on registration:
https://github.com/dgilman/cpython/commit/554a5bf9c16b6bd82ce47b2d0dd1833f2…
The first one was my first attempt but I am leaning towards the
second. It doesn't require any new API surface area. It also gets
integrated into the DefaultSelector logic, and even if that shouldn't
happen it's still easy to swap out your existing selector class for
the EpollExclusiveSelector class.
Enabling EPOLLEXCLUSIVE by default:
From the research I did last year my understanding is that
EPOLLEXCLUSIVE is never a performance drawback on Linux, and in the
case of a server that gets fast traffic it's a dramatic improvement.
However, I have not done my own benchmarking (with say, gunicorn,
which uses the stdlib's selectors module).
Note that EPOLLEXCLUSIVE does have one kernel API break: you can no
longer use EPOLL_CTL_MOD on an exclusive file descriptor. Python uses
the _MOD flag under the hood to implement epoll.modify(), which
results in EpollSelector.modify() throwing an OSError if you try to
modify an exclusive file descriptor.
In the second PR I implemented a EpollExclusiveSelector.modify() which
unregisters and reregisters the file descriptor to get around the _MOD
behavior. This means no surprise crash when someone updates Python.
But there may be other subtle regressions here: the performance of
modify() is likely going to regress, and someone could have a
dependency on Python actually using _MOD.
But as a rebuttal to those, I have a feeling that nobody really
depends on the performance of modify(), and even in the case where
someone does use it a lot it's likely for the data= path which is
completely unchanged here. I also am struggling to think of a place
where someone would care about the kernel-level changes between _MOD
and _ADD/_DEL but that might be my own lack of imagination or
knowledge of epoll techniques.
Maybe a compromise is to ship EpollExclusveSelector for a release
without it being the default and bump it to the default after seeing
if anyone's turned up any incompatibilities.
--
David Gilman
:DG<
Hi everyone,
A small update since the last communication from the release team regarding
the status of Python 3.11.0b4.
Unfortunately, even if we have fixed most of the original release blockers
and 4 more that appear during this week, we still have a bunch of release
blockers to deal with. One of them has been reported today.
I would like to release the next beta next week if everything looks good,
but there are also some items that need discussion:.
https://github.com/python/cpython/issues/93910https://github.com/python/cpython/issues/93516
Ideally we should reach consensus as a team on how to proceed in these
issues. In particular, we should decide collectively what is an acceptable
slowdown, specially looking to the release candidate.
If releasing the next betas is further delayed, I will consider delaying
the full release schedule to accommodate for the delay so users have the
appropriate time to test and validate every release.
Please do not hesitate in reaching out if you have any questions or
concerns or if there is any issue you think we should include in the next
release.
Thanks everyone for your help and understanding. I'm sure 3.11 is going to
be an outstanding release thank to all of you :)
Cheers from cloudy London,
Pablo Galindo Salgado
