tl; dr Maybe the 1090 pending pull requests will have to be rebased on
master to be able to be merged, unless we enhance our CI to always
test PRs after rebasing them on the master branch. I'm not sure at
FYI the Night’s Watch is still fixing the CIs in the darkness for you
:-) In short, most CIs should now be green in all branches (especially
CIs running on pull requests). I'm only aware of a Refleak issue on
3.x (test_threading, more about that above).
Last days were more busy than usual. Here are issues in a random order.
I had to update the OpenSSL version in the CI configuration which
lives in the same Git repository than Python code base. Since our CI
currently runs on unmodified PRs, maybe the 1090 pending pull requests
must now be rebased manually on the master branch to get these CI
configuration updates. Otherwise, the CI would fail which prevents to
merge a PR. I'm not sure at this point.
At least, old PRs with a green CI can be merged. But problems are
likely to arise if a reviewer requires changes and the CI is re-run on
the update PR. The PR may pick the old CI configuration (old OpenSSL
version) and fail badly...
Having to rebase 1090 PRs would be very unfortunate... Maybe one
solution would be to redesign CIs running on PRs to always rebase them
on the up-to-date master branch. There are services like
https://mergify.io/ which implement such workflow. Currently, we test
"outdated" master: there is a risk that tests fail once the change is
rebased on top of the up-to-date master, whereas tests passed on the
Note: fixing multitestssl.py to update the URL of old OpenSSL tarballs
doesn't work: again, old PRs don't get magically fixes pushed to
concurrent.futures no longer uses daemon threads. It causes a
reference leak in test_asyncio. Kyle Stanley managed to fix it in
Adding a module state to the _abc module triggered a reference leak in
This leak is very unlikely to be a new bug. It's just that previously
it was hidden by the fact that all interpreters used the same module
instance. regrtest -R 3:3 ignores the 18550 references that Python
leaks at exit: it only checks if the total references number changes
before/after a test run. When we *fix* modules to get isolated module
instances, we start to see old leaks as "new" leaks.
I have a pending workaround, but I'm first trying to investigate why
two garbage collections are needed to fix the leak...
The URL of the OpenSSL 1.1.1d moved which broke GitHub actions jobs,
Azure Pipelines and Travis CI. Not all CIs failed at once because of
different caches with different lifetime.
* https://bugs.python.org/issue40146 Azure
* https://bugs.python.org/issue40162 Travis CI
* https://bugs.python.org/issue40125 multissltests
I also proposed to modify multissltests to attempt to get the tarball
from a different URL. Christian Heimes proposed a fix:
I enhanced tests to detect when a child process does crash.
Previously, only FreeBSD *indirectly* detected such crash by the
creation of coredump files.
test_builtin started to fail on Solaris because the main test_builtin
process was killed by SIGHUP. I already fixed a similar bug in
test_pty last December by registering a signal handle to ignore
SIGHUP. I copied the fix to test_builtin:
The commit also fix an AIX issue in the same test.
After Python 3.9.0 alpha 5 release, I fixed a reference leak in
test_importlib which was introduced by the recent work on converting C
extension modules to multiphase
initialization (PEP 489):
It's a subtle issue between importlib and _weakref. See the issue for
the technical analysis.
I also fixed os.getgrouplist():
The macOS job on GitHub PRs failed because of this issue. In fact, the
bug existed for a long time, but macOS 10.15 (Catalina) added more
system groups to users which triggered the bug.
os.getgrouplist() only failed if the user has more than ... 17 groups
:-) Funny issue.
To be able to backport security fixes to Pyhon 3.5 and Python 3.6, I
had to fix the "Codecov patch" job which failed:
I copied the Codecov configuration from master.
Sadly, all 3.5 and 3.6 PRs have to be rebased to retrieve these fixes,
otherwise, they cannot be merged.
To finish, you can find general info about Python CIs at:
Night gathers, and now my watch begins. It shall not end until my death.