On Jan 18, 2022, at 11:34 AM, Guido van Rossum <guido@python.org> wrote:

 At best it shows that deprecations are complicated no matter how well you plan them. I remember that "noisy by default" deprecation warnings were widely despised.

...

On Tue, Jan 18, 2022 at 6:49 AM Antoine Pitrou <antoine@python.org> wrote: On Tue, 18 Jan 2022 15:17:41 +0100 Victor Stinner <vstinner@python.org> wrote:

...

Hi,

My colleagues Tomáš Hrnčiar and Miro Hrončok made good progress on updating Python 3.10 to Python 3.11 in Fedora, but some specific Python 3.11 incompatible changes are causing more troubles than others: https://discuss.python.org/t/experience-with-python-3-11-in-fedora/12911

We propose to revert the following 2 changes in Python 3.11 and postpone them in a later Python version, once most projects will be compatible with these changes:

* Removal of unittest aliases (bpo-45162): it broke 61 Fedora packages * Removals from configparser module (bpo-45173) - broke 28 Fedora packages

Doesn't this show, once again, that making DeprecationWarning silent by default was a mistake?

-- --Guido van Rossum (python.org/~guido) Pronouns: he/him (why is my pronoun here?)

One thought, what if they were off by default UNLESS you were doing unit tests? That would cut out a lot of the excessive noise issue, but put them in before the programmer when they are developing. Yes, they will get the warnings for dependencies but that lets the programmer apply pressure to fix it or warning they may need to change something if it won’t get fixed.

On Tue, Jan 18, 2022 at 10:30 AM Brett Cannon <brett@python.org> wrote:

I remember that "noisy by default" deprecation warnings were widely

...

despised.

One thought, what if they were off by default UNLESS you were doing unit tests?

I believe pytest already does this.

Indeed it does, at least in recent versions (1-2 yrs ago?) And even that is pretty darn annoying. It's really helpful for my code, but they often get lost in the noise of all the ones I get from upstream packages. I suppose I need to take the time to figure out how to silence the ones I don't want. And it does prompt me to make sure that the upstream packages are working on it. Now we just need to get more people to use pytest :-) -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On 2022-01-18 23:14, Gregory P. Smith wrote:

Our stdlib unittest already enables warnings by default per https://bugs.python.org/issue10535.

Getting the right people to pay attention to them is always the hard part.

I wonder if we can do a bit better in that regard.  When I install 3rd party packages, I create a usercustomize.py file that uses filterwarnings() to turn off all the warnings I don't care about.  I don't know how but maybe we could make that easier to do.  That way, you don't get buried in warnings coming from code you don't maintain. Additionally, maybe we should be more aggressive about showing PendingDeprecationWarning if it comes from code that seems to be written by the user, e.g. outside site-packages or not from a package installed by pip.  The exact logic of that is complicated though.

On Wed, Jan 26, 2022 at 02:40:32PM -0800, Neil Schemenauer wrote:

On 2022-01-18 23:14, Gregory P. Smith wrote:

...

Our stdlib unittest already enables warnings by default per https://bugs.python.org/issue10535.

Getting the right people to pay attention to them is always the hard part.

I wonder if we can do a bit better in that regard.  When I install 3rd party packages, I create a usercustomize.py file that uses filterwarnings() to turn off all the warnings I don't care about.  I don't know how but maybe we could make that easier to do.  That way, you don't get buried in warnings coming from code you don't maintain.

Or maybe, as a developer (not an end-user of an app), you could be more proactive in reporting those warnings to the third party, and encouraging them to fix them. Maybe even submitting a patch? If we use a library, then we surely care about that library working correctly, which means that if the library generates warnings, we *should* care about them. They are advanced notice that the library is going to break in the future. Of course I understand that folks are busy maintaining their own project, and have neither the time nor the inclination to take over the maintenance of every one of their dependencies. But we shouldn't just dismiss warnings in those dependencies as "warnings I don't care about" and ignore them as Not My Problem. Like it or not, it is My Problem and we should care about them. Especially in the case of open source software, the lines of responsibility are blurred. Open source libraries are not proprietary black boxes with a strict division between the vender that supplies the library and the people who use the library. They are fully transparent, we can see the warnings and, at least potentially, see how to fix them. And we have the legal right to. This is a hard problem, but it is not solely a technical problem. It is partly a social problem, and you cannot fix social problems with technology. People are ignoring the warnings, and not just the immediate developers of the software, but their downstream users. The open source mantra about many eyes making bugs shallow doesn't work when everyone is intentionally closing their eyes to the warnings of pending bugs. -- Steve

Steven D'Aprano writes:

Or maybe, as a developer (not an end-user of an app), you could be more proactive in reporting those warnings to the third party, and encouraging them to fix them. Maybe even submitting a patch?

As Chris B points out, it's quite possible that (generic) you have already reported, and maybe even submitted that patch, but you still have to wait for the release. Thing about submitting such patches -- I can't speak for Python programs, but in XEmacs I probably refused more warning-suppression patches than I accepted, because there was a deeper problem that needed to be fixed and the annoying warning was merely a side effect. And for any project, even if you submit a patch, there's no guarantee that the next version (or three) will contain it, which means vendoring the source. Urk!

Of course I understand that folks are busy maintaining their own project, and have neither the time nor the inclination to take over the maintenance of every one of their dependencies. But we shouldn't just dismiss warnings in those dependencies as "warnings I don't care about" and ignore them as Not My Problem.

Sure, but it's still worth providing various kinds of automation. For example, it would be nice if downstream (us) could fire and forget reports of DeprecationWarnings, knowing that bug reporting systems would automatically identify and merge them. And for DeprecationWarnings, rather than patching, it would be usually be nice to suppress the warning, I think. Steve

On Tue, Jan 18, 2022 at 5:48 PM Steve Dower <steve.dower@python.org> wrote:

Discovering during alpha that some packages haven't updated for the release that hasn't happened yet isn't the end of the world. Reverting the changes now is probably a bit premature - realistically we can undo these anytime during beta if we discover that packages are unable to be fixed over the next 9 months.

In Fedora, if a frequently used dependency is broken, a long list of packages "fail to build". (In Fedora, the package test suite must pass to build a package successfully.) If it takes 9 months to fix this dependency, we will likely miss other issues before the Python final version in dependent packages. The problem is when the issue is on the Python side. For example, we discovered that cloudpickle didn't work on Python 3.10. In fact, it was a Python regression. I had to modify how the function object constructor inherits globals (bpo-42990). This kind of change is fine during the beta phase, but it's bad to change the Python behavior in a bugfix Python 3.10.x release :-( Another example is when Cython was ported to Python 3.8, a PyCode_New() change caused a lot of troubles. After multiple exchanges between Cython and Python and a few more issues, it was decided to *revert* the PyCode_New() change. I would not be possible to revert the change in Python 3.8.1 or later :-( There are many examples like that, I don't have an exhaustive list. If issues are discovered earlier, we get more time to discuss and design how to handle them. I don't know where is the good trade-off here. That's why I'm opening a discussion. Yes obviously, we can revert these 2 changes later. Or we can just keep them ;-) -- At the end of my first email, I also suggest thinking about incompatible changes differently, try to make affected projects compatible in advance. The problem are not the changes themselves, but how they are introduced in Python, and more globally how they are introduced "in the Python ecosystem" (!). Victor -- Night gathers, and now my watch begins. It shall not end until my death.

On Wed, Jan 19, 2022 at 11:58 AM Sebastian Rittau <srittau@rittau.biz> wrote:

Am 18.01.22 um 22:57 schrieb Victor Stinner:

...

At the end of my first email, I also suggest thinking about incompatible changes differently, try to make affected projects compatible in advance. The problem are not the changes themselves, but how they are introduced in Python, and more globally how they are introduced "in the Python ecosystem" (!).

I believe that some (semi-) automated way to actively test and notify important projects of deprecations/removals before a release would be a great addition to the Python ecosystem. This is a complicated issue, but well worth it. (I'm not volunteering. :) )

In practice, it is the work that we are doing on Fedora, but for changes already merged in Python. Once, I wrote a small project https://github.com/vstinner/pythonci to test a few projects on a modified Python (with a patch applied), but it's really hard to automate the creation of the environment to test a project. It's hard to download and install dependencies in a reproducible way, especially for "non-Python" dependencies (Linux packages). Fedora already has recipes for that: RPM packages with "build dependencies", commands to download the code, apply patches if needed, run the test suite, etc. With my project, quickly, I had to patch pip, setuptools, etc. For example, to fix ResourceWarning or a DeprecationWarning (remember collections.MutableMapping removal? html5lib has to be patched manually, the vendored copy inside pip!). Maintaining patches on multiple projects is not easy. Well, if someone wants to help me on the pythonci project, you're welcome ;-) I'm interested to investigate different approaches to have simpler and more reliable "recipes". Victor -- Night gathers, and now my watch begins. It shall not end until my death.