On Tue, 18 Jan 2022 15:17:41 +0100 Victor Stinner wrote:

Hi,

My colleagues Tomáš Hrnčiar and Miro Hrončok made good progress on updating Python 3.10 to Python 3.11 in Fedora, but some specific Python 3.11 incompatible changes are causing more troubles than others: https://discuss.python.org/t/experience-with-python-3-11-in-fedora/12911

We propose to revert the following 2 changes in Python 3.11 and postpone them in a later Python version, once most projects will be compatible with these changes:

* Removal of unittest aliases (bpo-45162): it broke 61 Fedora packages * Removals from configparser module (bpo-45173) - broke 28 Fedora packages

Doesn't this show, once again, that making DeprecationWarning silent by default was a mistake?

On 1/18/2022 2:44 PM, Antoine Pitrou wrote:

...

We propose to revert the following 2 changes in Python 3.11 and postpone them in a later Python version, once most projects will be compatible with these changes:

* Removal of unittest aliases (bpo-45162): it broke 61 Fedora packages * Removals from configparser module (bpo-45173) - broke 28 Fedora packages

Doesn't this show, once again, that making DeprecationWarning silent by default was a mistake?

That's a hypothesis, but we don't have enough information to prove it. If we'd gone the other way, perhaps we'd be looking at massive complaints from "regular" end users about all the noisy warnings that they can't fix and saying that making it noisy was the mistake. Discovering during alpha that some packages haven't updated for the release that hasn't happened yet isn't the end of the world. Reverting the changes now is probably a bit premature - realistically we can undo these anytime during beta if we discover that packages are unable to be fixed over the next 9 months. That said, I'm fine with reverting the changes on the basis that they cause churn for no real benefit. If someone wants to argue that the benefit is worthwhile, that's fine, as long as they also argue in favour of the churn. We shouldn't pretend to be surprised that something we changed causes others to have to change. We *know* that will happen. Either we push forward with the changes, or we admit we don't really need them. With this amount of time before the release, we can't blame downstream users for reverting it. Cheers, Steve

On Tue, Jan 18, 2022 at 9:43 AM Richard Damon wrote:

On Jan 18, 2022, at 11:34 AM, Guido van Rossum wrote:

 At best it shows that deprecations are complicated no matter how well you plan them. I remember that "noisy by default" deprecation warnings were widely despised.

On Tue, Jan 18, 2022 at 6:49 AM Antoine Pitrou wrote:

...

On Tue, 18 Jan 2022 15:17:41 +0100 Victor Stinner wrote:

...

Hi,

My colleagues Tomáš Hrnčiar and Miro Hrončok made good progress on updating Python 3.10 to Python 3.11 in Fedora, but some specific Python 3.11 incompatible changes are causing more troubles than others:

https://discuss.python.org/t/experience-with-python-3-11-in-fedora/12911

...

We propose to revert the following 2 changes in Python 3.11 and postpone them in a later Python version, once most projects will be compatible with these changes:

* Removal of unittest aliases (bpo-45162): it broke 61 Fedora packages * Removals from configparser module (bpo-45173) - broke 28 Fedora

packages

Doesn't this show, once again, that making DeprecationWarning silent by default was a mistake?

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?) http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...*

One thought, what if they were off by default UNLESS you were doing unit tests?

I believe pytest already does this. -Brett

That would cut out a lot of the excessive noise issue, but put them in before the programmer when they are developing. Yes, they will get the warnings for dependencies but that lets the programmer apply pressure to fix it or warning they may need to change something if it won’t get fixed. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UAHZKJNQ... Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, Jan 18, 2022 at 10:30 AM Brett Cannon wrote:

I remember that "noisy by default" deprecation warnings were widely

...

despised.

One thought, what if they were off by default UNLESS you were doing unit tests?

I believe pytest already does this.

Indeed it does, at least in recent versions (1-2 yrs ago?) And even that is pretty darn annoying. It's really helpful for my code, but they often get lost in the noise of all the ones I get from upstream packages. I suppose I need to take the time to figure out how to silence the ones I don't want. And it does prompt me to make sure that the upstream packages are working on it. Now we just need to get more people to use pytest :-) -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On Tue, Jan 18, 2022 at 6:24 AM Victor Stinner wrote:

Hi,

My colleagues Tomáš Hrnčiar and Miro Hrončok made good progress on updating Python 3.10 to Python 3.11 in Fedora, but some specific Python 3.11 incompatible changes are causing more troubles than others: https://discuss.python.org/t/experience-with-python-3-11-in-fedora/12911

We propose to revert the following 2 changes in Python 3.11 and postpone them in a later Python version, once most projects will be compatible with these changes:

* Removal of unittest aliases (bpo-45162): it broke 61 Fedora packages * Removals from configparser module (bpo-45173) - broke 28 Fedora packages

--

We reported the issue to many affected projects, or when possible, we also wrote a pull request to make the code compatible (lot of those were made by others, e.g. Hugo van Kemenade, thank you!).

+1 to rolling both of these back for 3.11. Deprecation removals are hard. Surfacing these to the impacted upstream projects to provide time for those to integrate the changes is the right way to make these changes stick in 3.12 or later. Thanks for doing a significant chunk of that work! As you've done the work to clean up a lot of other OSS projects, I suggest we defer this until 3.12 with the intent that we won't defer it again. That doesn't mean we can't hold off on it, just that we believe pushing for this now and proactively pushing for a bunch of cleanups has improved the state of the world such that the future is brighter. That's a much different strategy than our passive aggressive DeprecationWarnings. -gps

The problem is that fixing a Fedora package requires multiple steps:

* (1) Propose a pull request upstream * (2) Get the pull request merged upstream * (3) Wait for a new release upstream * (4) Update the Fedora package downstream, or backport the change in Fedora (only needed by Fedora)

Identifying the Python incompatible changes causing most troubles took us a lot of time, but we did this work. Reverting the two Python 3.11 incompatible changes (causing most troubles) will save us "bug triage" time, to get more time on updating projects to Python 3.11 for the other remaining incompatible changes.

We are not saying that these incompatible changes are bad, it's just a matter of getting most projects ready before Python 3.11 final will be released.

--

By the way, before making a change known to be incompatible, it would be nice to run a code search on PyPI top 5000 projects to estimate how many projects will be broken, and try to update these projects *before* making the change.

For example, only introduce an incompatible change into Python once less than 15 projects are affected. Getting a pull request merged is nice, but a release including the fix is way better for practical reasons: people use "pip install <project name>".

--

Fedora work on Python 3.11 is public and tracked at: https://bugzilla.redhat.com/show_bug.cgi?id=PYTHON3.11

Click on "depends on" to see current issues:

https://bugzilla.redhat.com/buglist.cgi?bug_id=2016048&bug_id_type=anddependson&format=tvp

Example of bz #2025600: mom fails to build with Python 3.11: AttributeError: module 'configparser' has no attribute 'SafeConfigParser'.

Victor Stinner -- in the name of the Python Red Hat / Fedora maintenance team -- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GJTREADE... Code of Conduct: http://python.org/psf/codeofconduct/

On Wed, Jan 19, 2022 at 11:58 AM Sebastian Rittau wrote:

Am 18.01.22 um 22:57 schrieb Victor Stinner:

...

At the end of my first email, I also suggest thinking about incompatible changes differently, try to make affected projects compatible in advance. The problem are not the changes themselves, but how they are introduced in Python, and more globally how they are introduced "in the Python ecosystem" (!).

I believe that some (semi-) automated way to actively test and notify important projects of deprecations/removals before a release would be a great addition to the Python ecosystem. This is a complicated issue, but well worth it. (I'm not volunteering. :) )

In practice, it is the work that we are doing on Fedora, but for changes already merged in Python. Once, I wrote a small project https://github.com/vstinner/pythonci to test a few projects on a modified Python (with a patch applied), but it's really hard to automate the creation of the environment to test a project. It's hard to download and install dependencies in a reproducible way, especially for "non-Python" dependencies (Linux packages). Fedora already has recipes for that: RPM packages with "build dependencies", commands to download the code, apply patches if needed, run the test suite, etc. With my project, quickly, I had to patch pip, setuptools, etc. For example, to fix ResourceWarning or a DeprecationWarning (remember collections.MutableMapping removal? html5lib has to be patched manually, the vendored copy inside pip!). Maintaining patches on multiple projects is not easy. Well, if someone wants to help me on the pythonci project, you're welcome ;-) I'm interested to investigate different approaches to have simpler and more reliable "recipes". Victor -- Night gathers, and now my watch begins. It shall not end until my death.

On Wed, Jan 26, 2022 at 12:50 PM Neil Schemenauer wrote:

maybe we should be more aggressive about showing PendingDeprecationWarning if it comes from code that seems to be written by the user, e.g. outside site-packages

That would be pretty straightforward.

or not from a package installed by pip. The exact logic of that is complicated though.

or impossible, I always use pip install -e on teh code I'm developing (though that could be detected, I presume), and if I have more than one of my own packages, then they are piip installed.

Ido think we could work harder on documenting how to selectively disable warnings though. -CHB _______________________________________________

Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HB4ZSI5U... Code of Conduct: http://python.org/psf/codeofconduct/

-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On Thu, Jan 27, 2022 at 10:47 AM Steven D'Aprano wrote:

...

...

Getting the right people to pay attention to them is always the hard part.

Or maybe, as a developer (not an end-user of an app), you could be more proactive in reporting those warnings to the third party, and encouraging them to fix them. Maybe even submitting a patch?

Personally, I do exactly that -- but more often than not (thankfully) the upstream project is already working on it, or already fixed it, but in a version that I can't use yet. So then I really want to silence those warnings. Which is pretty easy to do with pytest, but maybe not so easy everywhere?

But we shouldn't just dismiss warnings in those dependencies as "warnings I don't care about" and ignore them as Not My Problem.

Unless we have done due diligence already :-) -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython