PEP 565: Show DeprecationWarning in __main__

I've written a short(ish) PEP for the proposal to change the default warnings filters to show DeprecationWarning in __main__: https://www.python.org/dev/peps/pep-0565/ The core proposal itself is just the idea in https://bugs.python.org/issue31975 (i.e. adding "default::DeprecationWarning:__main__" to the default filter set), but the PEP fills in some details on the motivation for the original change to the defaults, and why the current proposal is to add a new filter for __main__, rather than dropping the default DeprecationWarning filter entirely. The PEP also proposes repurposing the existing FutureWarning category to explicitly mean "backwards compatibility warnings that should be shown to users of Python applications" since: - we don't tend to use FutureWarning for its original nominal purpose (changes that will continue to run but will do something different) - FutureWarning was added in 2.3, so it's available in all still supported versions of Python, and is shown by default in all of them - it's at least arguably a less-jargony spelling of DeprecationWarning, and hence more appropriate for displaying to end users that may not have encountered the specific notion of "API deprecation" Cheers, Nick. ============== PEP: 565 Title: Show DeprecationWarning in __main__ Author: Nick Coghlan <ncoghlan@gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Nov-2017 Python-Version: 3.7 Post-History: 12-Nov-2017 Abstract ======== In Python 2.7 and Python 3.2, the default warning filters were updated to hide DeprecationWarning by default, such that deprecation warnings in development tools that were themselves written in Python (e.g. linters, static analysers, test runners, code generators) wouldn't be visible to their users unless they explicitly opted in to seeing them. However, this change has had the unfortunate side effect of making DeprecationWarning markedly less effective at its primary intended purpose: providing advance notice of breaking changes in APIs (whether in CPython, the standard library, or in third party libraries) to users of those APIs. To improve this situation, this PEP proposes a single adjustment to the default warnings filter: displaying deprecation warnings attributed to the main module by default. This change will mean that code entered at the interactive prompt and code in single file scripts will revert to reporting these warnings by default, while they will continue to be silenced by default for packaged code distributed as part of an importable module. The PEP also proposes a number of small adjustments to the reference interpreter and standard library documentation to help make the warnings subsystem more approachable for new Python developers. Specification ============= The current set of default warnings filters consists of:: ignore::DeprecationWarning ignore::PendingDeprecationWarning ignore::ImportWarning ignore::BytesWarning ignore::ResourceWarning The default ``unittest`` test runner then uses ``warnings.catch_warnings()`` ``warnings.simplefilter('default')`` to override the default filters while running test cases. The change proposed in this PEP is to update the default warning filter list to be:: default::DeprecationWarning:__main__ ignore::DeprecationWarning ignore::PendingDeprecationWarning ignore::ImportWarning ignore::BytesWarning ignore::ResourceWarning This means that in cases where the nominal location of the warning (as determined by the ``stacklevel`` parameter to ``warnings.warn``) is in the ``__main__`` module, the first occurrence of each DeprecationWarning will once again be reported. This change will lead to DeprecationWarning being displayed by default for: * code executed directly at the interactive prompt * code executed directly as part of a single-file script While continuing to be hidden by default for: * code imported from another module in a ``zipapp`` archive's ``__main__.py`` file * code imported from another module in an executable package's ``__main__`` submodule * code imported from an executable script wrapper generated at installation time based on a ``console_scripts`` or ``gui_scripts`` entry point definition As a result, API deprecation warnings encountered by development tools written in Python should continue to be hidden by default for users of those tools While not its originally intended purpose, the standard library documentation will also be updated to explicitly recommend the use of ``FutureWarning`` (rather than ``DeprecationWarning``) for backwards compatibility warnings that are intended to be seen by *users* of an application. This will give the following three distinct categories of backwards compatibility warning, with three different intended audiences: * ``PendingDeprecationWarning``: reported by default only in test runners that override the default set of warning filters. The intended audience is Python developers that take an active interest in ensuring the future compatibility of their software (e.g. professional Python application developers with specific support obligations). * ``DeprecationWarning``: reported by default for code that runs directly in the ``__main__`` module (as such code is considered relatively unlikely to have a dedicated test suite), but relies on test suite based reporting for code in other modules. The intended audience is Python developers that are at risk of upgrades to their dependencies (including upgrades to Python itself) breaking their software (e.g. developers using Python to script environments where someone else is in control of the timing of dependency upgrades). * ``FutureWarning``: always reported by default. The intended audience is users of applications written in Python, rather than other Python developers (e.g. warning about use of a deprecated setting in a configuration file format). Given its presence in the standard library since Python 2.3, ``FutureWarning`` would then also have a secondary use case for libraries and frameworks that support multiple Python versions: as a more reliably visible alternative to ``DeprecationWarning`` in Python 2.7 and versions of Python 3.x prior to 3.7. Motivation ========== As discussed in [1_] and mentioned in [2_], Python 2.7 and Python 3.2 changed the default handling of ``DeprecationWarning`` such that: * the warning was hidden by default during normal code execution * the `unittest`` test runner was updated to re-enable it when running tests The intent was to avoid cases of tooling output like the following:: $ devtool mycode/ /usr/lib/python3.6/site-packages/devtool/cli.py:1: DeprecationWarning: 'async' and 'await' will become reserved keywords in Python 3.7 async = True ... actual tool output ... Even when `devtool` is a tool specifically for Python programmers, this is not a particularly useful warning, as it will be shown on every invocation, even though the main helpful step an end user can take is to report a bug to the developers of ``devtool``. The warning is even less helpful for general purpose developer tools that are used across more languages than just Python. However, this change proved to have unintended consequences for the following audiences: * anyone using a test runner other than the default one built into ``unittest`` (since the request for third party test runners to change their default warnings filters was never made explicitly) * anyone using the default ``unittest`` test runner to test their Python code in a subprocess (since even ``unittest`` only adjusts the warnings settings in the current process) * anyone writing Python code at the interactive prompt or as part of a directly executed script that didn't have a Python level test suite at all In these cases, ``DeprecationWarning`` ended up become almost entirely equivalent to ``PendingDeprecationWarning``: it was simply never seen at all. Limitations on PEP Scope ======================== This PEP exists specifically to explain both the proposed addition to the default warnings filter for 3.7, *and* to more clearly articulate the rationale for the original change to the handling of DeprecationWarning back in Python 2.7 and 3.2. This PEP does not solve all known problems with the current approach to handling deprecation warnings. Most notably: * the default ``unittest`` test runner does not currently report deprecation warnings emitted at module import time, as the warnings filter override is only put in place during test execution, not during test discovery and loading. * the default ``unittest`` test runner does not currently report deprecation warnings in subprocesses, as the warnings filter override is applied directly to the loaded ``warnings`` module, not to the ``PYTHONWARNINGS`` environment variable. * the standard library doesn't provide a straightforward way to opt-in to seeing all warnings emitted *by* a particular dependency prior to upgrading it (the third-party ``warn`` module [3_] does provide this, but enabling it involves monkeypatching the standard library's ``warnings`` module). * re-enabling deprecation warnings by default in __main__ doesn't help in handling cases where software has been factored out into support modules, but those modules still have little or no automated test coverage. Near term, the best currently available answer is to run such applications with ``PYTHONWARNINGS=default::DeprecationWarning`` or ``python -W default::DeprecationWarning`` and pay attention to their ``stderr`` output. Longer term, this is really a question for researchers working on static analysis of Python code: how to reliably find usage of deprecated APIs, and how to infer that an API or parameter is deprecated based on ``warnings.warn`` calls, without actually running either the code providing the API or the code accessing it While these are real problems with the status quo, they're excluded from consideration in this PEP because they're going to require more complex solutions than a single additional entry in the default warnings filter, and resolving them at least potentially won't require going through the PEP process. For anyone interested in pursuing them further, the first two would be ``unittest`` module enhancement requests, the third would be a ``warnings`` module enhancement request, while the last would only require a PEP if inferring API deprecations from their contents was deemed to be an intractable code analysis problem, and an explicit function and parameter marker syntax in annotations was proposed instead. References ========== .. [1] stdlib-sig thread proposing the original default filter change (https://mail.python.org/pipermail/stdlib-sig/2009-November/000789.html) .. [2] Python 2.7 notification of the default warnings filter change (https://docs.python.org/3/whatsnew/2.7.html#changes-to-the-handling-of-depre...) .. [3] Emitting warnings based on the location of the warning itself (https://pypi.org/project/warn/) Copyright ========= This document has been placed in the public domain. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, Nov 12, 2017 at 1:24 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Looking at the official What's New entry for the change ( https://docs.python.org/3/whatsnew/2.7.html#changes-to-the-handling-of-depre...) it's not just about development tools. It's about any app (or tool, or utility, or program) written in Python whose users just treat it as "some app", not as something that's necessarily part of their Python environment. While in extreme cases such apps can *bundle* their own Python interpreter (like Dropbox does), many developers opt to assume or ensure that Python is avaiable, perhaps via the OS package management system. (Because my day job is software development I am having a hard time coming up with concrete examples that aren't development tools, but AFAIK at Dropbox the *deployment* of e.g. Go binaries is managed through utilities written in Python. The Go developers couldn't care less about that.) -- --Guido van Rossum (python.org/~guido)

On 13 November 2017 at 01:45, Guido van Rossum <guido@python.org> wrote:
It occurs to me that the original Dropbox client itself would be an example of deprecation warnings in deployed code being unhelpful noise - assuming that the runtime warnings were reported back to the developers, the deprecation warnings would be entirely irrelevant to a developer trying to debug a production issue with client/server communication. I've updated the PEP to try to make the explanation of the historical rationale more accurate: https://github.com/python/peps/commit/30daada7867dd7f0e008545c7fd98612282ec6... I still emphasise the developer tooling case, as I think that's the easiest one for Python developers to follow when it comes to appreciating the problems with the old defaults (I know it is for me), but I've added some references to regular user facing applications as well. I've also added a new "Documentation Updates" section, which doesn't technically need to be in the PEP, but I think is useful to include in terms of giving the existing features more visibility than they might otherwise receive (I learned quite a few new things myself about the basics of warnings control researching and working on implementing this PEP, so I assume plenty of end users are in a similar situation, where we know the warnings module exists, but aren't necessarily completely familiar with how to reconfigure it at runtime). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 19 November 2017 at 17:22, Nick Coghlan <ncoghlan@gmail.com> wrote:
With these changes, I think the version now live at https://www.python.org/dev/peps/pep-0565/ is ready for pronouncement. Are you happy to review and pronounce on that in this thread, or would you prefer that I started a new one? Cheers, Nick. P.S. The actual implementation still needs some work, due to details of the dual Python/C warnings module implementation, and the subtleties of how that interacts with the re module. However, I've implemented a working version with a test case, and have a pretty clear idea of the extra test cases that we need to close some of the test coverage gaps that draft implementation has revealed. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

12.11.17 11:24, Nick Coghlan пише:
FutureWarning currently is used for its original nominal purpose in the re and ElementTree modules. It even had been added in 2.7 for behavior that already have been changed in Python 3 or will be changed in future versions (emitted only with the -3 option).

On 13 November 2017 at 03:10, Serhiy Storchaka <storchaka@gmail.com> wrote:
If the future warnings relate to regex and XML parsing, they'd still fall under the "for display to users" category, since those modules can't tell if the input data was application provided or part of an end user interface like a configuration file.
That's closer to the original purpose, but with them being 2.7 only, and gated behind the -3 switch, I think we can ignore them when it comes to defining the expected usage in 3.7+ Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

13.11.17 01:34, Nick Coghlan пише:
In the case of regular expressions or XPath the warnings could fall under the "for display to users" category, though in most cases they are emitted for hardcoded expressions. It is easy to fix these expressions, making them working ambiguously in past and future versions, and the author should do this. But FutureWarning is also raised in the __bool__ method which will change its meaning in future (the similar change was made for the midnight time object). It seems to me that most of issues with FutureWarning on GitHub [1] are related to NumPy and pandas which use FutureWarning for its original nominal purpose, for warning about using programming interfaces that will change the behavior in future. This doesn't have any relation to end users unless the end user is an author of the written code. [1] https://github.com/search?q=FutureWarning&type=Issues

On Sun, Nov 19, 2017 at 2:26 AM, Serhiy Storchaka <storchaka@gmail.com> wrote:
Eh, numpy does use FutureWarning for changes where the same code will transition from doing one thing to doing something else without passing through a state where it raises an error. But that decision was based on FutureWarning being shown to users by default, not because it matches the nominal purpose :-). IIRC I proposed this policy for NumPy in the first place, and I still don't even know if it matches the original intent because the docs are so vague. "Will change behavior in the future" describes every case where you might consider using FutureWarning *or* DeprecationWarning, right? We have been using DeprecationWarning for changes where code will transition from working -> raising an error, and that *is* based on the Official Recommendation to hide those by default whenever possible. We've been doing this for a few years now, and I'd say our experience so far has been... poor. I'm trying to figure out how to say this politely. Basically it doesn't work at all. What happens in practice is that we issue a DeprecationWarning for a year, mostly no-one notices, then we make the change in a 1.x.0 release, everyone's code breaks, we roll it back in 1.x.1, and then possibly repeat several times in 1.(x+1).0 and 1.(x+2).0 until enough people have updated their code that the screams die down. I'm pretty sure we'll be changing our policy at some point, possibly to always use FutureWarning for everything. -n -- Nathaniel J. Smith -- https://vorpus.org

On Nov 28, 2017 3:55 PM, "Guido van Rossum" <guido@python.org> wrote: On Sun, Nov 19, 2017 at 5:40 AM, Nathaniel Smith <njs@pobox.com> wrote:
Can one of you check that the latest version of PEP 565 gets this right? If you're asking about the the proposed new language about FutureWarnings, it seems fine to me. If you're asking about the PEP as a whole, it seems fine but I don't think it will make much difference in our case. IPython has been showing deprecation warnings in __main__ for a few years now, and it's nice enough. Getting warnings for scripts seems nice too. But we aren't rolling back changes because they broke someone's one off script – I'm sure it happens but we don't tend to hear about it. We're responding to things like major downstream dependencies that nonetheless totally missed all the warnings. The part that might help there is evangelising popular test runners like pytest to change their defaults. To me that's the most interesting change to come out of this. But it's hard to predict in advance how effective it will be. tl;dr: I don't think PEP 565 solves all my problems, but I don't have any objections to what it does to. -n

On Sun, Nov 12, 2017 at 1:24 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Technically it's orthogonal, but if you're trying to get better warnings in the REPL, then you might also want to look at: https://bugs.python.org/issue1539925 https://github.com/ipython/ipython/issues/6611 -n -- Nathaniel J. Smith -- https://vorpus.org

On Mon, 13 Nov 2017 22:37:46 +1100 Chris Angelico <rosuav@gmail.com> wrote:
If I see a warning once every REPL session, I know about the deprecation already, thank you. I don't need to be taken by the hand like a little child. Besides, the code I write in the REPL is not meant for durable use. Regards Antoine.

13.11.17 14:29, Antoine Pitrou пише:
Hmm, now I see that the simple Nathaniel's solution is not completely correct. If the warning action is 'module', it should be emitted only once if used directly in the REPL, because '__main__' is the same module. But if you use functions foo() and bar(), and both emit the same warning, you should get a warning from every entered command, because after the first warning you know only about the first function.

On Mon, Nov 13, 2017 at 6:09 AM, Serhiy Storchaka <storchaka@gmail.com> wrote:
True. The fundamental problem is that generally, Python uses (filename, lineno) pairs to identify lines of code. But (a) the warning module assumes that for each namespace dict, there is a unique mapping between line numbers and lines of code, so it ignores filename and just keys off lineno, and (b) the REPL re-uses the same (file, lineno) for different lines of code anyway. So I guess the fully correct solution would be to use a unique "filename" when compiling each block of code -- e.g. the REPL could do the equivalent of compile(<code>, "REPL[1]", ...) for the first line, compile(<code>, "REPL[2]", ...) for the second line, etc. -- and then also teach the warnings module's duplicate detection logic to key off of (file, lineno) pairs instead of just lineno. -n -- Nathaniel J. Smith -- https://vorpus.org

On Mon, Nov 13, 2017 at 10:37:46PM +1100, Chris Angelico wrote:
I haven't followed the long discussions, so this is probably not a very novel observation. But it seems to me that we have a problem getting users to treat the python command like e.g. gcc. If I want gcc warnings, I use -Wall -Wextra. I think the whole problem is that python warnings are a bit of an obscure feature (they were to me for a long time). Stefan Krah

This PEP can break applications parsing Python stderr, application which don't expect to get DeprecationWarning in their output. Is it possible to disable this PEP using a command line option and/or environment variable to get the Python 3.6 behaviour (always DeprecationWarning)? I guess that it's "PYTHONWARNINGS=ignore::DeprecationWarning:__main__". Am I right? Would you mind to mention that in the PEP, please? Sorry, I'm not an expert of the warnings module. Is it possible to also configure Python to ignore DeprecationWarning using the warnings module, at the start of the __main__ script? Something like warnings.filterwarnings("ignore", '', DeprecationWarning)? Again, maybe explain that in the PEP? Victor

On 14 November 2017 at 02:27, Victor Stinner <victor.stinner@gmail.com> wrote:
Right, but anything affected by this would eventually have broken anyway, when the change being warned about actually happens. That said, reducing the likelihood of breaking application output parsers is part of the rationale for restricting the change to unpackaged scripts (we know from the initial PEP 538 implementation that there's plenty of code out there that doesn't cope well with unexpected lines appearing on stderr).
The patch for the PEP will also update the documentation for the `PYTHONWARNINGS` environment variable to explicitly call out the following settings: PYTHONWARNINGS=error # Convert to exceptions PYTHONWARNINGS=always # Warn every time PYTHONWARNINGS=default # Warn once per call location PYTHONWARNINGS=module # Warn once per calling module PYTHONWARNINGS=once # Warn once per Python process PYTHONWARNINGS=ignore # Never warn And then also cover their respective shorthand command line equivalents (`-We`, `-Wa`, `-Wd`, `-Wm`,`-Wo`, `-Wi`). While https://docs.python.org/3/using/cmdline.html#cmdoption-w does currently explain this, neither it nor https://docs.python.org/3/using/cmdline.html#envvar-PYTHONWARNINGS show specific examples They also don't link directly to https://docs.python.org/3/library/warnings.html#the-warnings-filter, and that section doesn't explain the shorthand `:`-separated notation used in sys.warnoptions.
I guess that it's "PYTHONWARNINGS=ignore::DeprecationWarning:__main__". Am I right?
`PYTHONWARNINGS=ignore::DeprecationWarning` would be the shortest way to suppress deprecations everywhere again while still getting other warnings.
Would you mind to mention that in the PEP, please?
Will do - I actually meant to cover this anyway (hence the reference to docs changes in the abstract), but I missed it in the initial draft.
`warnings.simplefilter("ignore", DeprecationWarning)` is the simplest runtime option for ensuring that deprecation warnings are never displayed. The downside of doing this programmatically is that you can end up overriding environmental and command line settings, so the best application level "Only show warnings if my users ask for them" snippet actually looks like: if not sys.warnoptions: warnings.simplefilter("ignore") (I'll mention this in the PEP and docs patch as well) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Hi, I'm not convinced that this PEP 565 will prevent developers to be surprised when upgrading Python, since more and more applications are using an entry point: an import + a single function call. For example, *all* OpenStack applications use an entry point and so will be unaffected by this PEP. There is no suprise, it's documented in the PEP: "code imported from an executable script wrapper generated at installation time based on a console_scripts or gui_scripts entry point definition". It's hard to find a compromise between two incompatible use cases: "run an application" ("user") and "develop an application" ("developer"). I proposed again my "-X dev" idea in another thread for the "develop an application" use case ;-) If the Python REPL is included in the "run an application" use case, the frontier between user and developer becomes blurry :-) Is REPL designed for users or developers? Should Python guess the intent of the human connected to the keyboard? ... Victor 2017-11-12 10:24 GMT+01:00 Nick Coghlan <ncoghlan@gmail.com>:

On Mon, Nov 13, 2017 at 11:33 AM, Victor Stinner <victor.stinner@gmail.com> wrote:
I don't think folks are counting the Python REPL as "run an application". People who use the REPL are at least writing python and should definitely be informed of deprecations. +1 for deprecations in the REPL from me; I think it's a great way to inform a good percentage of the python devs of upcoming changes.

2017-11-12 10:24 GMT+01:00 Nick Coghlan <ncoghlan@gmail.com>:
I understand the rationale of the PEP, but I dislike the proposed implementation. End users will start to get warnings that they don't understand and cannot fix, so these warnings would just be annoying. For scripts written to only be run once and then deleted, again, these warnings are just annoying since the script is going to be deleted anyway. It's like the annoying ResourceWarning (in debug mode) when I write open(filename).read() in the REPL. I know that it's bad, but the code will only be run once and lost when I quit the REPL, so who cares? (not me, stop bothering me with good programming practices, I do know them, let me write crappy code!) On the REPL case, I have no strong opinion. For developers who want to see warnings, the warnings are not shown for applications using an entry point and any code base larger than a single file (warnings outside the __main__ module). -- In practice, I'm running tests with python -Wd (to see ResourceWarning in my case), and I'm happy with that :-) If tests pass with -Wd, you're good. This is why I implemented a new "development mode", python3 -X dev, in Python 3.7 which "shows all warnings except of BytesWarning". (This mode is already mentioned in Nick's PEP.) https://docs.python.org/dev/using/cmdline.html#id5 My advice is to always run your tests with -X dev. If they pass with -X dev, you are good. I'm not even sure that developers should use -X dev to test their code manually. I see it as a "cheap linter". I don't want to run a linter each time I run Python. The "-X dev" option is smarter than -Wd: it adds the default filter *at the end*, to respect -b and -bb options for the BytesWarning. Release build: $ ./python -c 'import pprint, warnings; pprint.pprint(warnings.filters)' [('ignore', None, <class 'DeprecationWarning'>, None, 0), ('ignore', None, <class 'PendingDeprecationWarning'>, None, 0), ('ignore', None, <class 'ImportWarning'>, None, 0), ('ignore', None, <class 'BytesWarning'>, None, 0), ('ignore', None, <class 'ResourceWarning'>, None, 0)] -Wd on the release build adds a filter at the start: $ ./release.python -Wd -c 'import pprint, warnings; pprint.pprint(warnings.filters)' [('default', None, <class 'Warning'>, None, 0), <~~~ HERE ('ignore', None, <class 'DeprecationWarning'>, None, 0), ('ignore', None, <class 'PendingDeprecationWarning'>, None, 0), ('ignore', None, <class 'ImportWarning'>, None, 0), ('ignore', None, <class 'BytesWarning'>, None, 0), ('ignore', None, <class 'ResourceWarning'>, None, 0)] Debug build: $ ./python -c 'import pprint, warnings; pprint.pprint(warnings.filters)' [('ignore', None, <class 'BytesWarning'>, None, 0), ('default', None, <class 'ResourceWarning'>, None, 0)] -X dev adds a default filter *at the end*: $ ./python -X dev -c 'import pprint, warnings; pprint.pprint(warnings.filters)' [('ignore', None, <class 'BytesWarning'>, None, 0), ('default', None, <class 'ResourceWarning'>, None, 0), ('default', None, <class 'Warning'>, None, 0)] <~~~ HERE Note: you can combine -X dev with -W if you want ;-) (It works as expected.) Victor

If you ask me this PEP is not going to make everyone happy, but I think it is an improvement, and it seems many people are in agreement or at least don't object to it (and obviously Nick thinks it's going to be a big improvement). Therefore I am planning to accept it by the end of this week unless more objections are voiced. Honestly, I didn't completely follow what Victor thinks of the PEP -- his post seemed mostly about promoting his own -X dev flag. I have nothing against that flag but I don't see how its existence is relevant to the PEP, which is about giving users who don't even know they are Python developers a hint when they are using deprecated features (for which there always must be a shiny new replacement!). -- --Guido van Rossum (python.org/~guido)

2017-12-05 16:50 GMT+01:00 Guido van Rossum <guido@python.org>:
Honestly, I didn't completely follow what Victor thinks of the PEP -- his post seemed mostly about promoting his own -X dev flag.
-X dev is similar (but different) than -W default: show warnings which are hidden by default otherwise. -W default works on Python 2.7 and 3.6.
I disagree that *users* of an application is supposed to "handle" deprecation warnings: report them to the developer, or even try to fix them. IHMO these warnings (hidden by default) were introduced for developers of the application. My point is that I prefer to keep the status quo: continue to hide deprecation warnings, but promote existing solutions like -W default to display these warnings, teach to developers how to see and fix these warnings. Even for developers, I'm not sure that only showing warnings in __main__ is useful, since more and more application use a __main__ module which is a thin entry point : import + function call (ex: "from app import main; main()").
Therefore I am planning to accept it by the end of this week unless more objections are voiced.
It's ok if we disagree. I just wanted to share my opinion on this issue ;-) Victor

On Tue, Dec 5, 2017 at 9:59 AM, Victor Stinner <victor.stinner@gmail.com> wrote:
But the whole point of the PEP is that it only warns about deprecations in code over which the user has control -- likely __main__ is their own code, and they *can* handle it.
If they import a 3rd party module which does something deprecated, the user won't see any deprecation warnings.
And that's intentional -- such developers are supposed to actively test their code, and deprecations will be shown to them by the unittest framework. Having a minimal __main__ implies that their users won't see any deprecation warnings.
I just worry that your opinion might be based on a misunderstanding of the proposal. If we agree on what the PEP actually proposed to change, and how that will affect different categories of users and developers, I am okay with disagreement. -- --Guido van Rossum (python.org/~guido)

On Dec 5, 2017, at 13:24, Guido van Rossum <guido@python.org> wrote:
But the whole point of the PEP is that it only warns about deprecations in code over which the user has control -- likely __main__ is their own code, and they *can* handle it.
I’m not so sure how true that is. I have no sense of the relative popularity of hand crafted dunder-mains vs entry point crafted ones. I know that in my own applications, I tend to use the latter (although pkg_resources performance issues bum me out). But then you have applications like pex that use fairly complex hand crafted dunder-mains in their zip files. In either case I don’t want consumers of my applications to have to worry about DeprecationWarnings, since *they* really can’t do anything about them. All that to say I really don’t know what the right thing to do here is. All of our fiddling with the reporting of DeprecationWarnings, not to mention PendingDeprecationWarnings and FutureWarnings feels like experimental shots in the dark, and I suspect we won’t really know if PEP 565 will be helpful, harmful, or neutral until it’s out in the wild for a while. I suspect either that what we’re trying to accomplish really can’t be done, or that we really don’t have a good understanding of the problem and we’re just chipping away at the edges. I know that’s unhelpful in deciding whether to accept the PEP or not. In the absence of any clear consensus, I’m happy to trust Guido’s instincts or keep the status quo. -Barry

On Tue, Dec 5, 2017 at 11:11 AM, Barry Warsaw <barry@python.org> wrote:
This makes it the responsibility of the pex developers to at least test for deprecation errors in their __main__. I don't know what pex is, but presumably they have some QA and they can test their zips or at least their __main__ with specific Python versions before distributing them. I am confident that it's not going to be a problem for pex developers or users.
I also expect that this PEP will only have a small effect. It is a compromise, but I'm okay with that. There seems to be no way to find out what effect changes in this area will really have, because small-scale experiments where some development team starts paying attention to deprecations don't translate into what it will mean for the large majority who aren't particularly interested in them (but who may still be affected when the deprecations finally take effect and some feature disappears). The main category of users who are going to be affected are "casual" users -- it's they who have the most code in __main__ files and at the same time have the least idea of where Python is header. Yes, they will occasionally be annoyed. But they will also occasionally be glad that we point out a deprecation to them. And, unlike the situation before Python 2.7, they won't be annoyed by warnings in code they can't update -- they'll get warnings only about their own scripts. All in all, I think that for professional developers and users of professionally developed Python packages, at worst not much will change and at best there will be some small benefits; while for casual developers and users there will be some benefits and those will outweigh the downsides. In 5 years or so we can reevaluate. -- --Guido van Rossum (python.org/~guido)

On 6 December 2017 at 05:11, Barry Warsaw <barry@python.org> wrote:
For something like pex, it would likely be appropriate to add the following to __main__.py: import sys, warnings if not sys.warnoptions: warnings.simplefilter("ignore") We don't currently provide that snippet anywhere in the docs, so the PEP suggests we add it: https://www.python.org/dev/peps/pep-0565/#other-documentation-updates
All that to say I really don’t know what the right thing to do here is. All of our fiddling with the reporting of DeprecationWarnings, not to mention PendingDeprecationWarnings and FutureWarnings feels like experimental shots in the dark, and I suspect we won’t really know if PEP 565 will be helpful, harmful, or neutral until it’s out in the wild for a while. I suspect either that what we’re trying to accomplish really can’t be done, or that we really don’t have a good understanding of the problem and we’re just chipping away at the edges.
I'm entirely comfortable with telling app developers "Include this 3 line snippet if you don't want your users to see any warnings by default". Historically, we've never provided good documentation on how to do that, so apps have tended to rely on the default filters, and we've then had ongoing arguments about who wins and who loses in deciding what the defaults should be. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

2017-12-05 19:24 GMT+01:00 Guido van Rossum <guido@python.org>:
IMHO the core of the PEP 565 is to propose a compromise to separate "own code" and "external code" (cannot be modified). I'm unhappy with this suboptimal compromise: "only __main__ is my own code". Maybe we need something to declare the code that we own, to enable warnings on them. Just a simple helper on top of warnings.filterwarnings()? Or maybe I'm already in the "the better is the enemy of the good" greay area :-) Victor

On Tue, Dec 5, 2017 at 12:10 PM, Victor Stinner <victor.stinner@gmail.com> wrote:
This is a very good characterization of the dilemma. But IMO we don't have a better proxy for "my own code" (it's tempting to try and define it based on the current directory but this makes too many assumptions about development styles). -- --Guido van Rossum (python.org/~guido)

2017-12-05 21:21 GMT+01:00 Serhiy Storchaka <storchaka@gmail.com>:
I was thinking as something like: enable_warnings_on('app') which would enable warnings on app.py and app/* including subdirectories (app/tools.py but also app/submodule/thing.py). But the drawback of this idea compared to PEP 565 is that it requires to modify each application, whereas the PEP 565 changes the default behaviour. Warnings are a complex beast. If you think about ResourceWarning: the warning can be emited "in the stdlib", whereas the file was opened in "your own code". ResourceWarning are basically emited *anywhere* because of their weird nature. It's often that these warnings are emited on a garbage collection, and this can happen anytime, usually far from where the resource was allocated. (Since Python 3.6, if you enable tracemalloc, the ResourceWarning is now logged with the traceback where the resource was allocated!!!) -b and -bb options are global options to change the behaviour of BytesWarning: it's not possible to only make BytesWarning strict "in your own code". (I'm not sure neither that all code uses properly the warnings API to identify correctly the caller where the warning should be emitted.) ... For all these reasons, I prefer to not try to *guess* what is the "owned code", but simply always emit warnings everywhere :-) That's why I suggest to use -W default option or PYTHONWARNINGS=default (or the new -X dev option or PYTHONDEVMODE=1). In my experience, it's fine to get warnings in third party code. I'm lucky, I'm only working on open source softwares, so I can report and even fix (!) warnings in the code that I don't own :-) By the way, if you are annoyed by one very specific warning in external code (which prevents you to fix other warnings "further" in the code), you can ignore it using a filter. You can specify a module name, and even a line number of a module. https://docs.python.org/dev/library/warnings.html#warnings.filterwarnings At the end, I'm not sure that the PEP 565 is really needed or would help anyone. Again, I understand that there is no perfect solution, and that PEP 565 is a compromise. Victor

On 6 December 2017 at 06:43, Victor Stinner <victor.stinner@gmail.com> wrote:
At the end, I'm not sure that the PEP 565 is really needed or would help anyone.
Folks, I'd really appreciate it if you could comment on the merits of the PEP without implicitly claiming that I don't exist, and that Linux system administrators don't exist :) Right now, Linux distro Python library upgrades (including CPython updates to a new feature release) may result in *hard compatibility breaks*, as previously deprecated features disappear without warning. With PEP 565, ad hoc personal scripts will at least get a DeprecationWarning when APIs they're using are at risk of disappearing as a result of a distro package or version update. Right now, they don't get a warning at all. There is *no way* for any opt in flag to help these users. Now, PEP 565 being rejected wouldn't be the end of the world on that front - we can apply comparable changes as a downstream patch. However, these problems aren't unique to Linux, and they aren't unique to any specific distro: they apply whenever folks are using Python for personal workflow automation, rather than for redistributable applications and libraries. So it makes more sense to me to do this upstream, rather than having the default warnings handling be a redistributor dependent behaviour. That said, I go agree we could offer easier to use APIs to app developers that just want to hide warnings from their users, so I've filed https://bugs.python.org/issue32229 to propose a straightforward "warnings.hide_warnings()" API that encapsulates things like checking for a non-empty sys.warnoptions list. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 6 December 2017 at 14:34, Nick Coghlan <ncoghlan@gmail.com> wrote:
I've updated the "Limitations" section of the PEP to mention that separate proposal: https://github.com/python/peps/commit/6e93c8d2e6ad698834578d4077b92a8fc84a70... Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 6 December 2017 at 14:50, Nick Coghlan <ncoghlan@gmail.com> wrote:
Having rebased the PEP 565 patch atop the "-X dev" changes, I think that if we don't change some of the details of how `-X dev` is implemented, `warnings.hide_warnings` (or a comparable convenience API) is going to be a requirement to help app developers effectively manage their default warnings settings in 3.7+. The problem is that devmode doesn't currently behave the same way `-Wd` does when it comes to sys.warnoptions: $ ./python -Wd -c "import sys; print(sys.warnoptions); print(sys.flags.dev_mode)" ['d'] False $ ./python -X dev -c "import sys; print(sys.warnoptions); print(sys.flags.dev_mode)" [] True As currently implemented, the warnings module actually checks `sys.flags.dev_mode` directly during startup (or `sys._xoptions` in the case of the pure Python fallback), and populates the warnings filter differently depending on what it finds: $ ./python -c "import warnings; print('\n'.join(map(str, warnings.filters)))" ('default', None, <class 'DeprecationWarning'>, '__main__', 0) ('ignore', None, <class 'DeprecationWarning'>, None, 0) ('ignore', None, <class 'PendingDeprecationWarning'>, None, 0) ('ignore', None, <class 'ImportWarning'>, None, 0) ('ignore', None, <class 'BytesWarning'>, None, 0) ('ignore', None, <class 'ResourceWarning'>, None, 0) $ ./python -X dev -c "import warnings; print('\n'.join(map(str, warnings.filters)))" ('ignore', None, <class 'BytesWarning'>, None, 0) ('default', None, <class 'ResourceWarning'>, None, 0) ('default', None, <class 'Warning'>, None, 0) $ ./python -Wd -c "import warnings; print('\n'.join(map(str, warnings.filters)))" ('default', None, <class 'Warning'>, None, 0) ('default', None, <class 'DeprecationWarning'>, '__main__', 0) ('ignore', None, <class 'DeprecationWarning'>, None, 0) ('ignore', None, <class 'PendingDeprecationWarning'>, None, 0) ('ignore', None, <class 'ImportWarning'>, None, 0) ('ignore', None, <class 'BytesWarning'>, None, 0) ('ignore', None, <class 'ResourceWarning'>, None, 0) This means the app development snippet proposed in the PEP will no longer do the right thing, since it will ignore the dev mode flag: if not sys.warnoptions: # This still runs for `-X dev` warnings.simplefilter("ignore") My main suggested fix would be to adjust the way `-X dev` is implemented to include `sys.warnoptions.append('default')` (and remove the direct dev_mode query from the warnings module code). However, another possible way to go would be to make the correct Python 3.7+-only snippet look like this: import warnings warnings.hide_warnings() And have the forward-compatible snippet look like this: import warnings: if hasattr(warnings, "hide_warnings"): # Accounts for `-W`, `-X dev`, and any other implementation specific settings warnings.hide_warnings() else: # Only accounts for `-W` import sys if not sys.warnoptions: warnings.simplefilter("ignore") (We can also do both, of course) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Let's discuss -Xdev implementation issue at https://bugs.python.org/issue32230 In short, -Xdev must add its warning at the end to respect BytesWarning, whereas it's not possible with -W option :-( Victor Le 6 déc. 2017 09:15, "Nick Coghlan" <ncoghlan@gmail.com> a écrit : On 6 December 2017 at 14:50, Nick Coghlan <ncoghlan@gmail.com> wrote:
Having rebased the PEP 565 patch atop the "-X dev" changes, I think that if we don't change some of the details of how `-X dev` is implemented, `warnings.hide_warnings` (or a comparable convenience API) is going to be a requirement to help app developers effectively manage their default warnings settings in 3.7+. The problem is that devmode doesn't currently behave the same way `-Wd` does when it comes to sys.warnoptions: $ ./python -Wd -c "import sys; print(sys.warnoptions); print(sys.flags.dev_mode)" ['d'] False $ ./python -X dev -c "import sys; print(sys.warnoptions); print(sys.flags.dev_mode)" [] True As currently implemented, the warnings module actually checks `sys.flags.dev_mode` directly during startup (or `sys._xoptions` in the case of the pure Python fallback), and populates the warnings filter differently depending on what it finds: $ ./python -c "import warnings; print('\n'.join(map(str, warnings.filters)))" ('default', None, <class 'DeprecationWarning'>, '__main__', 0) ('ignore', None, <class 'DeprecationWarning'>, None, 0) ('ignore', None, <class 'PendingDeprecationWarning'>, None, 0) ('ignore', None, <class 'ImportWarning'>, None, 0) ('ignore', None, <class 'BytesWarning'>, None, 0) ('ignore', None, <class 'ResourceWarning'>, None, 0) $ ./python -X dev -c "import warnings; print('\n'.join(map(str, warnings.filters)))" ('ignore', None, <class 'BytesWarning'>, None, 0) ('default', None, <class 'ResourceWarning'>, None, 0) ('default', None, <class 'Warning'>, None, 0) $ ./python -Wd -c "import warnings; print('\n'.join(map(str, warnings.filters)))" ('default', None, <class 'Warning'>, None, 0) ('default', None, <class 'DeprecationWarning'>, '__main__', 0) ('ignore', None, <class 'DeprecationWarning'>, None, 0) ('ignore', None, <class 'PendingDeprecationWarning'>, None, 0) ('ignore', None, <class 'ImportWarning'>, None, 0) ('ignore', None, <class 'BytesWarning'>, None, 0) ('ignore', None, <class 'ResourceWarning'>, None, 0) This means the app development snippet proposed in the PEP will no longer do the right thing, since it will ignore the dev mode flag: if not sys.warnoptions: # This still runs for `-X dev` warnings.simplefilter("ignore") My main suggested fix would be to adjust the way `-X dev` is implemented to include `sys.warnoptions.append('default')` (and remove the direct dev_mode query from the warnings module code). However, another possible way to go would be to make the correct Python 3.7+-only snippet look like this: import warnings warnings.hide_warnings() And have the forward-compatible snippet look like this: import warnings: if hasattr(warnings, "hide_warnings"): # Accounts for `-W`, `-X dev`, and any other implementation specific settings warnings.hide_warnings() else: # Only accounts for `-W` import sys if not sys.warnoptions: warnings.simplefilter("ignore") (We can also do both, of course) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick and Victor, I'm still hoping to accept this PEP, but I don't have time to wrap my head around -Xdev ("devmode"?) which appears to be Victor's latest pet project. Should PEP 565 be changed to copy with devmode's behavior, or the other way around, or should they just ignore each other? It is not clear of me what the status of the mention in PEP 565 of -Xdev is -- normative or informational? I really don't want to have to learn how devmode works in order to be able to accept PEP 565 (or send it back for revision), so I am asking you two to let me know. On Wed, Dec 6, 2017 at 1:42 AM, Victor Stinner <victor.stinner@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)

Hi, 2017-12-12 21:21 GMT+01:00 Guido van Rossum <guido@python.org>:
The warnings filters had a few corner cases. We discussed with Nick to fix them to make them simpler. We agreed on these priorities for command line options and environment variables: -b and -bb > -W > PYTHONWARNINGS > -X dev > default filters In release mode, the default filters became: ignore::DeprecationWarning ignore::PendingDeprecationWarning ignore::ImportWarning ignore::ResourceWarning ignore::BytesWarning is gone. We now rely on the fact the BytesWarning should not be emited without -b nor -bb in practice. It has been implemented in https://bugs.python.org/issue32230 ! (I just merged Nick's PR.) Now -X dev behaves again as my initial propopal: for warnings, "-X dev" simply behaves as "-W default". (Previously, I had to hack the code to respect -b and -bb options, but I don't think that it's worth it to explain that here, it's doesn't matter anymore ;-)) The PEP 565 is still different: it doesn't behaves as "-W default", but "-W default::DeprecationWarning:__main__". Only DeprecationWarning warnings are shown, whereas -X dev shows DeprecationWarning, but also PendingDeprecationWarning, ResourceWarning and ImportWarning. Moreover, -X dev shows warnings in all modules, not only __main__. You may see -X dev as a builtin linter, whereas PEP 565 seems to be very specific to one specific issue: display deprecation warnings, but only in the __main__ module. Does it help you to understand the difference? Note: I still dislike the PEP 565, but well, that's just my opinion ;-) Victor

OK, in that case I'll just pronounce approval right here. Considered disagreement is acceptable. Nick, congrats with PEP 565! Please update the PEP to mark it as approved with a link to this message as the resolution, and let's get the implementation into 3.7a4! On Tue, Dec 12, 2017 at 2:58 PM, Victor Stinner <victor.stinner@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)

2017-12-13 0:24 GMT+01:00 Guido van Rossum <guido@python.org>:
Considered disagreement is acceptable.
Sure, I'm fine with that ;-)
Nick wrote that he will be away, since I update his PEP: https://github.com/python/peps/commit/355eced94cf4117492c9e1eee8f950f08e53ec... Victor

On Sun, Nov 12, 2017 at 1:24 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Looking at the official What's New entry for the change ( https://docs.python.org/3/whatsnew/2.7.html#changes-to-the-handling-of-depre...) it's not just about development tools. It's about any app (or tool, or utility, or program) written in Python whose users just treat it as "some app", not as something that's necessarily part of their Python environment. While in extreme cases such apps can *bundle* their own Python interpreter (like Dropbox does), many developers opt to assume or ensure that Python is avaiable, perhaps via the OS package management system. (Because my day job is software development I am having a hard time coming up with concrete examples that aren't development tools, but AFAIK at Dropbox the *deployment* of e.g. Go binaries is managed through utilities written in Python. The Go developers couldn't care less about that.) -- --Guido van Rossum (python.org/~guido)

On 13 November 2017 at 01:45, Guido van Rossum <guido@python.org> wrote:
It occurs to me that the original Dropbox client itself would be an example of deprecation warnings in deployed code being unhelpful noise - assuming that the runtime warnings were reported back to the developers, the deprecation warnings would be entirely irrelevant to a developer trying to debug a production issue with client/server communication. I've updated the PEP to try to make the explanation of the historical rationale more accurate: https://github.com/python/peps/commit/30daada7867dd7f0e008545c7fd98612282ec6... I still emphasise the developer tooling case, as I think that's the easiest one for Python developers to follow when it comes to appreciating the problems with the old defaults (I know it is for me), but I've added some references to regular user facing applications as well. I've also added a new "Documentation Updates" section, which doesn't technically need to be in the PEP, but I think is useful to include in terms of giving the existing features more visibility than they might otherwise receive (I learned quite a few new things myself about the basics of warnings control researching and working on implementing this PEP, so I assume plenty of end users are in a similar situation, where we know the warnings module exists, but aren't necessarily completely familiar with how to reconfigure it at runtime). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 19 November 2017 at 17:22, Nick Coghlan <ncoghlan@gmail.com> wrote:
With these changes, I think the version now live at https://www.python.org/dev/peps/pep-0565/ is ready for pronouncement. Are you happy to review and pronounce on that in this thread, or would you prefer that I started a new one? Cheers, Nick. P.S. The actual implementation still needs some work, due to details of the dual Python/C warnings module implementation, and the subtleties of how that interacts with the re module. However, I've implemented a working version with a test case, and have a pretty clear idea of the extra test cases that we need to close some of the test coverage gaps that draft implementation has revealed. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

12.11.17 11:24, Nick Coghlan пише:
FutureWarning currently is used for its original nominal purpose in the re and ElementTree modules. It even had been added in 2.7 for behavior that already have been changed in Python 3 or will be changed in future versions (emitted only with the -3 option).

On 13 November 2017 at 03:10, Serhiy Storchaka <storchaka@gmail.com> wrote:
If the future warnings relate to regex and XML parsing, they'd still fall under the "for display to users" category, since those modules can't tell if the input data was application provided or part of an end user interface like a configuration file.
That's closer to the original purpose, but with them being 2.7 only, and gated behind the -3 switch, I think we can ignore them when it comes to defining the expected usage in 3.7+ Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

13.11.17 01:34, Nick Coghlan пише:
In the case of regular expressions or XPath the warnings could fall under the "for display to users" category, though in most cases they are emitted for hardcoded expressions. It is easy to fix these expressions, making them working ambiguously in past and future versions, and the author should do this. But FutureWarning is also raised in the __bool__ method which will change its meaning in future (the similar change was made for the midnight time object). It seems to me that most of issues with FutureWarning on GitHub [1] are related to NumPy and pandas which use FutureWarning for its original nominal purpose, for warning about using programming interfaces that will change the behavior in future. This doesn't have any relation to end users unless the end user is an author of the written code. [1] https://github.com/search?q=FutureWarning&type=Issues

On Sun, Nov 19, 2017 at 2:26 AM, Serhiy Storchaka <storchaka@gmail.com> wrote:
Eh, numpy does use FutureWarning for changes where the same code will transition from doing one thing to doing something else without passing through a state where it raises an error. But that decision was based on FutureWarning being shown to users by default, not because it matches the nominal purpose :-). IIRC I proposed this policy for NumPy in the first place, and I still don't even know if it matches the original intent because the docs are so vague. "Will change behavior in the future" describes every case where you might consider using FutureWarning *or* DeprecationWarning, right? We have been using DeprecationWarning for changes where code will transition from working -> raising an error, and that *is* based on the Official Recommendation to hide those by default whenever possible. We've been doing this for a few years now, and I'd say our experience so far has been... poor. I'm trying to figure out how to say this politely. Basically it doesn't work at all. What happens in practice is that we issue a DeprecationWarning for a year, mostly no-one notices, then we make the change in a 1.x.0 release, everyone's code breaks, we roll it back in 1.x.1, and then possibly repeat several times in 1.(x+1).0 and 1.(x+2).0 until enough people have updated their code that the screams die down. I'm pretty sure we'll be changing our policy at some point, possibly to always use FutureWarning for everything. -n -- Nathaniel J. Smith -- https://vorpus.org

On Nov 28, 2017 3:55 PM, "Guido van Rossum" <guido@python.org> wrote: On Sun, Nov 19, 2017 at 5:40 AM, Nathaniel Smith <njs@pobox.com> wrote:
Can one of you check that the latest version of PEP 565 gets this right? If you're asking about the the proposed new language about FutureWarnings, it seems fine to me. If you're asking about the PEP as a whole, it seems fine but I don't think it will make much difference in our case. IPython has been showing deprecation warnings in __main__ for a few years now, and it's nice enough. Getting warnings for scripts seems nice too. But we aren't rolling back changes because they broke someone's one off script – I'm sure it happens but we don't tend to hear about it. We're responding to things like major downstream dependencies that nonetheless totally missed all the warnings. The part that might help there is evangelising popular test runners like pytest to change their defaults. To me that's the most interesting change to come out of this. But it's hard to predict in advance how effective it will be. tl;dr: I don't think PEP 565 solves all my problems, but I don't have any objections to what it does to. -n

On Sun, Nov 12, 2017 at 1:24 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Technically it's orthogonal, but if you're trying to get better warnings in the REPL, then you might also want to look at: https://bugs.python.org/issue1539925 https://github.com/ipython/ipython/issues/6611 -n -- Nathaniel J. Smith -- https://vorpus.org

On Mon, 13 Nov 2017 22:37:46 +1100 Chris Angelico <rosuav@gmail.com> wrote:
If I see a warning once every REPL session, I know about the deprecation already, thank you. I don't need to be taken by the hand like a little child. Besides, the code I write in the REPL is not meant for durable use. Regards Antoine.

13.11.17 14:29, Antoine Pitrou пише:
Hmm, now I see that the simple Nathaniel's solution is not completely correct. If the warning action is 'module', it should be emitted only once if used directly in the REPL, because '__main__' is the same module. But if you use functions foo() and bar(), and both emit the same warning, you should get a warning from every entered command, because after the first warning you know only about the first function.

On Mon, Nov 13, 2017 at 6:09 AM, Serhiy Storchaka <storchaka@gmail.com> wrote:
True. The fundamental problem is that generally, Python uses (filename, lineno) pairs to identify lines of code. But (a) the warning module assumes that for each namespace dict, there is a unique mapping between line numbers and lines of code, so it ignores filename and just keys off lineno, and (b) the REPL re-uses the same (file, lineno) for different lines of code anyway. So I guess the fully correct solution would be to use a unique "filename" when compiling each block of code -- e.g. the REPL could do the equivalent of compile(<code>, "REPL[1]", ...) for the first line, compile(<code>, "REPL[2]", ...) for the second line, etc. -- and then also teach the warnings module's duplicate detection logic to key off of (file, lineno) pairs instead of just lineno. -n -- Nathaniel J. Smith -- https://vorpus.org

On Mon, Nov 13, 2017 at 10:37:46PM +1100, Chris Angelico wrote:
I haven't followed the long discussions, so this is probably not a very novel observation. But it seems to me that we have a problem getting users to treat the python command like e.g. gcc. If I want gcc warnings, I use -Wall -Wextra. I think the whole problem is that python warnings are a bit of an obscure feature (they were to me for a long time). Stefan Krah

This PEP can break applications parsing Python stderr, application which don't expect to get DeprecationWarning in their output. Is it possible to disable this PEP using a command line option and/or environment variable to get the Python 3.6 behaviour (always DeprecationWarning)? I guess that it's "PYTHONWARNINGS=ignore::DeprecationWarning:__main__". Am I right? Would you mind to mention that in the PEP, please? Sorry, I'm not an expert of the warnings module. Is it possible to also configure Python to ignore DeprecationWarning using the warnings module, at the start of the __main__ script? Something like warnings.filterwarnings("ignore", '', DeprecationWarning)? Again, maybe explain that in the PEP? Victor

On 14 November 2017 at 02:27, Victor Stinner <victor.stinner@gmail.com> wrote:
Right, but anything affected by this would eventually have broken anyway, when the change being warned about actually happens. That said, reducing the likelihood of breaking application output parsers is part of the rationale for restricting the change to unpackaged scripts (we know from the initial PEP 538 implementation that there's plenty of code out there that doesn't cope well with unexpected lines appearing on stderr).
The patch for the PEP will also update the documentation for the `PYTHONWARNINGS` environment variable to explicitly call out the following settings: PYTHONWARNINGS=error # Convert to exceptions PYTHONWARNINGS=always # Warn every time PYTHONWARNINGS=default # Warn once per call location PYTHONWARNINGS=module # Warn once per calling module PYTHONWARNINGS=once # Warn once per Python process PYTHONWARNINGS=ignore # Never warn And then also cover their respective shorthand command line equivalents (`-We`, `-Wa`, `-Wd`, `-Wm`,`-Wo`, `-Wi`). While https://docs.python.org/3/using/cmdline.html#cmdoption-w does currently explain this, neither it nor https://docs.python.org/3/using/cmdline.html#envvar-PYTHONWARNINGS show specific examples They also don't link directly to https://docs.python.org/3/library/warnings.html#the-warnings-filter, and that section doesn't explain the shorthand `:`-separated notation used in sys.warnoptions.
I guess that it's "PYTHONWARNINGS=ignore::DeprecationWarning:__main__". Am I right?
`PYTHONWARNINGS=ignore::DeprecationWarning` would be the shortest way to suppress deprecations everywhere again while still getting other warnings.
Would you mind to mention that in the PEP, please?
Will do - I actually meant to cover this anyway (hence the reference to docs changes in the abstract), but I missed it in the initial draft.
`warnings.simplefilter("ignore", DeprecationWarning)` is the simplest runtime option for ensuring that deprecation warnings are never displayed. The downside of doing this programmatically is that you can end up overriding environmental and command line settings, so the best application level "Only show warnings if my users ask for them" snippet actually looks like: if not sys.warnoptions: warnings.simplefilter("ignore") (I'll mention this in the PEP and docs patch as well) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Hi, I'm not convinced that this PEP 565 will prevent developers to be surprised when upgrading Python, since more and more applications are using an entry point: an import + a single function call. For example, *all* OpenStack applications use an entry point and so will be unaffected by this PEP. There is no suprise, it's documented in the PEP: "code imported from an executable script wrapper generated at installation time based on a console_scripts or gui_scripts entry point definition". It's hard to find a compromise between two incompatible use cases: "run an application" ("user") and "develop an application" ("developer"). I proposed again my "-X dev" idea in another thread for the "develop an application" use case ;-) If the Python REPL is included in the "run an application" use case, the frontier between user and developer becomes blurry :-) Is REPL designed for users or developers? Should Python guess the intent of the human connected to the keyboard? ... Victor 2017-11-12 10:24 GMT+01:00 Nick Coghlan <ncoghlan@gmail.com>:

On Mon, Nov 13, 2017 at 11:33 AM, Victor Stinner <victor.stinner@gmail.com> wrote:
I don't think folks are counting the Python REPL as "run an application". People who use the REPL are at least writing python and should definitely be informed of deprecations. +1 for deprecations in the REPL from me; I think it's a great way to inform a good percentage of the python devs of upcoming changes.

2017-11-12 10:24 GMT+01:00 Nick Coghlan <ncoghlan@gmail.com>:
I understand the rationale of the PEP, but I dislike the proposed implementation. End users will start to get warnings that they don't understand and cannot fix, so these warnings would just be annoying. For scripts written to only be run once and then deleted, again, these warnings are just annoying since the script is going to be deleted anyway. It's like the annoying ResourceWarning (in debug mode) when I write open(filename).read() in the REPL. I know that it's bad, but the code will only be run once and lost when I quit the REPL, so who cares? (not me, stop bothering me with good programming practices, I do know them, let me write crappy code!) On the REPL case, I have no strong opinion. For developers who want to see warnings, the warnings are not shown for applications using an entry point and any code base larger than a single file (warnings outside the __main__ module). -- In practice, I'm running tests with python -Wd (to see ResourceWarning in my case), and I'm happy with that :-) If tests pass with -Wd, you're good. This is why I implemented a new "development mode", python3 -X dev, in Python 3.7 which "shows all warnings except of BytesWarning". (This mode is already mentioned in Nick's PEP.) https://docs.python.org/dev/using/cmdline.html#id5 My advice is to always run your tests with -X dev. If they pass with -X dev, you are good. I'm not even sure that developers should use -X dev to test their code manually. I see it as a "cheap linter". I don't want to run a linter each time I run Python. The "-X dev" option is smarter than -Wd: it adds the default filter *at the end*, to respect -b and -bb options for the BytesWarning. Release build: $ ./python -c 'import pprint, warnings; pprint.pprint(warnings.filters)' [('ignore', None, <class 'DeprecationWarning'>, None, 0), ('ignore', None, <class 'PendingDeprecationWarning'>, None, 0), ('ignore', None, <class 'ImportWarning'>, None, 0), ('ignore', None, <class 'BytesWarning'>, None, 0), ('ignore', None, <class 'ResourceWarning'>, None, 0)] -Wd on the release build adds a filter at the start: $ ./release.python -Wd -c 'import pprint, warnings; pprint.pprint(warnings.filters)' [('default', None, <class 'Warning'>, None, 0), <~~~ HERE ('ignore', None, <class 'DeprecationWarning'>, None, 0), ('ignore', None, <class 'PendingDeprecationWarning'>, None, 0), ('ignore', None, <class 'ImportWarning'>, None, 0), ('ignore', None, <class 'BytesWarning'>, None, 0), ('ignore', None, <class 'ResourceWarning'>, None, 0)] Debug build: $ ./python -c 'import pprint, warnings; pprint.pprint(warnings.filters)' [('ignore', None, <class 'BytesWarning'>, None, 0), ('default', None, <class 'ResourceWarning'>, None, 0)] -X dev adds a default filter *at the end*: $ ./python -X dev -c 'import pprint, warnings; pprint.pprint(warnings.filters)' [('ignore', None, <class 'BytesWarning'>, None, 0), ('default', None, <class 'ResourceWarning'>, None, 0), ('default', None, <class 'Warning'>, None, 0)] <~~~ HERE Note: you can combine -X dev with -W if you want ;-) (It works as expected.) Victor

If you ask me this PEP is not going to make everyone happy, but I think it is an improvement, and it seems many people are in agreement or at least don't object to it (and obviously Nick thinks it's going to be a big improvement). Therefore I am planning to accept it by the end of this week unless more objections are voiced. Honestly, I didn't completely follow what Victor thinks of the PEP -- his post seemed mostly about promoting his own -X dev flag. I have nothing against that flag but I don't see how its existence is relevant to the PEP, which is about giving users who don't even know they are Python developers a hint when they are using deprecated features (for which there always must be a shiny new replacement!). -- --Guido van Rossum (python.org/~guido)

2017-12-05 16:50 GMT+01:00 Guido van Rossum <guido@python.org>:
Honestly, I didn't completely follow what Victor thinks of the PEP -- his post seemed mostly about promoting his own -X dev flag.
-X dev is similar (but different) than -W default: show warnings which are hidden by default otherwise. -W default works on Python 2.7 and 3.6.
I disagree that *users* of an application is supposed to "handle" deprecation warnings: report them to the developer, or even try to fix them. IHMO these warnings (hidden by default) were introduced for developers of the application. My point is that I prefer to keep the status quo: continue to hide deprecation warnings, but promote existing solutions like -W default to display these warnings, teach to developers how to see and fix these warnings. Even for developers, I'm not sure that only showing warnings in __main__ is useful, since more and more application use a __main__ module which is a thin entry point : import + function call (ex: "from app import main; main()").
Therefore I am planning to accept it by the end of this week unless more objections are voiced.
It's ok if we disagree. I just wanted to share my opinion on this issue ;-) Victor

On Tue, Dec 5, 2017 at 9:59 AM, Victor Stinner <victor.stinner@gmail.com> wrote:
But the whole point of the PEP is that it only warns about deprecations in code over which the user has control -- likely __main__ is their own code, and they *can* handle it.
If they import a 3rd party module which does something deprecated, the user won't see any deprecation warnings.
And that's intentional -- such developers are supposed to actively test their code, and deprecations will be shown to them by the unittest framework. Having a minimal __main__ implies that their users won't see any deprecation warnings.
I just worry that your opinion might be based on a misunderstanding of the proposal. If we agree on what the PEP actually proposed to change, and how that will affect different categories of users and developers, I am okay with disagreement. -- --Guido van Rossum (python.org/~guido)

On Dec 5, 2017, at 13:24, Guido van Rossum <guido@python.org> wrote:
But the whole point of the PEP is that it only warns about deprecations in code over which the user has control -- likely __main__ is their own code, and they *can* handle it.
I’m not so sure how true that is. I have no sense of the relative popularity of hand crafted dunder-mains vs entry point crafted ones. I know that in my own applications, I tend to use the latter (although pkg_resources performance issues bum me out). But then you have applications like pex that use fairly complex hand crafted dunder-mains in their zip files. In either case I don’t want consumers of my applications to have to worry about DeprecationWarnings, since *they* really can’t do anything about them. All that to say I really don’t know what the right thing to do here is. All of our fiddling with the reporting of DeprecationWarnings, not to mention PendingDeprecationWarnings and FutureWarnings feels like experimental shots in the dark, and I suspect we won’t really know if PEP 565 will be helpful, harmful, or neutral until it’s out in the wild for a while. I suspect either that what we’re trying to accomplish really can’t be done, or that we really don’t have a good understanding of the problem and we’re just chipping away at the edges. I know that’s unhelpful in deciding whether to accept the PEP or not. In the absence of any clear consensus, I’m happy to trust Guido’s instincts or keep the status quo. -Barry

On Tue, Dec 5, 2017 at 11:11 AM, Barry Warsaw <barry@python.org> wrote:
This makes it the responsibility of the pex developers to at least test for deprecation errors in their __main__. I don't know what pex is, but presumably they have some QA and they can test their zips or at least their __main__ with specific Python versions before distributing them. I am confident that it's not going to be a problem for pex developers or users.
I also expect that this PEP will only have a small effect. It is a compromise, but I'm okay with that. There seems to be no way to find out what effect changes in this area will really have, because small-scale experiments where some development team starts paying attention to deprecations don't translate into what it will mean for the large majority who aren't particularly interested in them (but who may still be affected when the deprecations finally take effect and some feature disappears). The main category of users who are going to be affected are "casual" users -- it's they who have the most code in __main__ files and at the same time have the least idea of where Python is header. Yes, they will occasionally be annoyed. But they will also occasionally be glad that we point out a deprecation to them. And, unlike the situation before Python 2.7, they won't be annoyed by warnings in code they can't update -- they'll get warnings only about their own scripts. All in all, I think that for professional developers and users of professionally developed Python packages, at worst not much will change and at best there will be some small benefits; while for casual developers and users there will be some benefits and those will outweigh the downsides. In 5 years or so we can reevaluate. -- --Guido van Rossum (python.org/~guido)

On 6 December 2017 at 05:11, Barry Warsaw <barry@python.org> wrote:
For something like pex, it would likely be appropriate to add the following to __main__.py: import sys, warnings if not sys.warnoptions: warnings.simplefilter("ignore") We don't currently provide that snippet anywhere in the docs, so the PEP suggests we add it: https://www.python.org/dev/peps/pep-0565/#other-documentation-updates
All that to say I really don’t know what the right thing to do here is. All of our fiddling with the reporting of DeprecationWarnings, not to mention PendingDeprecationWarnings and FutureWarnings feels like experimental shots in the dark, and I suspect we won’t really know if PEP 565 will be helpful, harmful, or neutral until it’s out in the wild for a while. I suspect either that what we’re trying to accomplish really can’t be done, or that we really don’t have a good understanding of the problem and we’re just chipping away at the edges.
I'm entirely comfortable with telling app developers "Include this 3 line snippet if you don't want your users to see any warnings by default". Historically, we've never provided good documentation on how to do that, so apps have tended to rely on the default filters, and we've then had ongoing arguments about who wins and who loses in deciding what the defaults should be. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

2017-12-05 19:24 GMT+01:00 Guido van Rossum <guido@python.org>:
IMHO the core of the PEP 565 is to propose a compromise to separate "own code" and "external code" (cannot be modified). I'm unhappy with this suboptimal compromise: "only __main__ is my own code". Maybe we need something to declare the code that we own, to enable warnings on them. Just a simple helper on top of warnings.filterwarnings()? Or maybe I'm already in the "the better is the enemy of the good" greay area :-) Victor

On Tue, Dec 5, 2017 at 12:10 PM, Victor Stinner <victor.stinner@gmail.com> wrote:
This is a very good characterization of the dilemma. But IMO we don't have a better proxy for "my own code" (it's tempting to try and define it based on the current directory but this makes too many assumptions about development styles). -- --Guido van Rossum (python.org/~guido)

2017-12-05 21:21 GMT+01:00 Serhiy Storchaka <storchaka@gmail.com>:
I was thinking as something like: enable_warnings_on('app') which would enable warnings on app.py and app/* including subdirectories (app/tools.py but also app/submodule/thing.py). But the drawback of this idea compared to PEP 565 is that it requires to modify each application, whereas the PEP 565 changes the default behaviour. Warnings are a complex beast. If you think about ResourceWarning: the warning can be emited "in the stdlib", whereas the file was opened in "your own code". ResourceWarning are basically emited *anywhere* because of their weird nature. It's often that these warnings are emited on a garbage collection, and this can happen anytime, usually far from where the resource was allocated. (Since Python 3.6, if you enable tracemalloc, the ResourceWarning is now logged with the traceback where the resource was allocated!!!) -b and -bb options are global options to change the behaviour of BytesWarning: it's not possible to only make BytesWarning strict "in your own code". (I'm not sure neither that all code uses properly the warnings API to identify correctly the caller where the warning should be emitted.) ... For all these reasons, I prefer to not try to *guess* what is the "owned code", but simply always emit warnings everywhere :-) That's why I suggest to use -W default option or PYTHONWARNINGS=default (or the new -X dev option or PYTHONDEVMODE=1). In my experience, it's fine to get warnings in third party code. I'm lucky, I'm only working on open source softwares, so I can report and even fix (!) warnings in the code that I don't own :-) By the way, if you are annoyed by one very specific warning in external code (which prevents you to fix other warnings "further" in the code), you can ignore it using a filter. You can specify a module name, and even a line number of a module. https://docs.python.org/dev/library/warnings.html#warnings.filterwarnings At the end, I'm not sure that the PEP 565 is really needed or would help anyone. Again, I understand that there is no perfect solution, and that PEP 565 is a compromise. Victor

On 6 December 2017 at 06:43, Victor Stinner <victor.stinner@gmail.com> wrote:
At the end, I'm not sure that the PEP 565 is really needed or would help anyone.
Folks, I'd really appreciate it if you could comment on the merits of the PEP without implicitly claiming that I don't exist, and that Linux system administrators don't exist :) Right now, Linux distro Python library upgrades (including CPython updates to a new feature release) may result in *hard compatibility breaks*, as previously deprecated features disappear without warning. With PEP 565, ad hoc personal scripts will at least get a DeprecationWarning when APIs they're using are at risk of disappearing as a result of a distro package or version update. Right now, they don't get a warning at all. There is *no way* for any opt in flag to help these users. Now, PEP 565 being rejected wouldn't be the end of the world on that front - we can apply comparable changes as a downstream patch. However, these problems aren't unique to Linux, and they aren't unique to any specific distro: they apply whenever folks are using Python for personal workflow automation, rather than for redistributable applications and libraries. So it makes more sense to me to do this upstream, rather than having the default warnings handling be a redistributor dependent behaviour. That said, I go agree we could offer easier to use APIs to app developers that just want to hide warnings from their users, so I've filed https://bugs.python.org/issue32229 to propose a straightforward "warnings.hide_warnings()" API that encapsulates things like checking for a non-empty sys.warnoptions list. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 6 December 2017 at 14:34, Nick Coghlan <ncoghlan@gmail.com> wrote:
I've updated the "Limitations" section of the PEP to mention that separate proposal: https://github.com/python/peps/commit/6e93c8d2e6ad698834578d4077b92a8fc84a70... Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 6 December 2017 at 14:50, Nick Coghlan <ncoghlan@gmail.com> wrote:
Having rebased the PEP 565 patch atop the "-X dev" changes, I think that if we don't change some of the details of how `-X dev` is implemented, `warnings.hide_warnings` (or a comparable convenience API) is going to be a requirement to help app developers effectively manage their default warnings settings in 3.7+. The problem is that devmode doesn't currently behave the same way `-Wd` does when it comes to sys.warnoptions: $ ./python -Wd -c "import sys; print(sys.warnoptions); print(sys.flags.dev_mode)" ['d'] False $ ./python -X dev -c "import sys; print(sys.warnoptions); print(sys.flags.dev_mode)" [] True As currently implemented, the warnings module actually checks `sys.flags.dev_mode` directly during startup (or `sys._xoptions` in the case of the pure Python fallback), and populates the warnings filter differently depending on what it finds: $ ./python -c "import warnings; print('\n'.join(map(str, warnings.filters)))" ('default', None, <class 'DeprecationWarning'>, '__main__', 0) ('ignore', None, <class 'DeprecationWarning'>, None, 0) ('ignore', None, <class 'PendingDeprecationWarning'>, None, 0) ('ignore', None, <class 'ImportWarning'>, None, 0) ('ignore', None, <class 'BytesWarning'>, None, 0) ('ignore', None, <class 'ResourceWarning'>, None, 0) $ ./python -X dev -c "import warnings; print('\n'.join(map(str, warnings.filters)))" ('ignore', None, <class 'BytesWarning'>, None, 0) ('default', None, <class 'ResourceWarning'>, None, 0) ('default', None, <class 'Warning'>, None, 0) $ ./python -Wd -c "import warnings; print('\n'.join(map(str, warnings.filters)))" ('default', None, <class 'Warning'>, None, 0) ('default', None, <class 'DeprecationWarning'>, '__main__', 0) ('ignore', None, <class 'DeprecationWarning'>, None, 0) ('ignore', None, <class 'PendingDeprecationWarning'>, None, 0) ('ignore', None, <class 'ImportWarning'>, None, 0) ('ignore', None, <class 'BytesWarning'>, None, 0) ('ignore', None, <class 'ResourceWarning'>, None, 0) This means the app development snippet proposed in the PEP will no longer do the right thing, since it will ignore the dev mode flag: if not sys.warnoptions: # This still runs for `-X dev` warnings.simplefilter("ignore") My main suggested fix would be to adjust the way `-X dev` is implemented to include `sys.warnoptions.append('default')` (and remove the direct dev_mode query from the warnings module code). However, another possible way to go would be to make the correct Python 3.7+-only snippet look like this: import warnings warnings.hide_warnings() And have the forward-compatible snippet look like this: import warnings: if hasattr(warnings, "hide_warnings"): # Accounts for `-W`, `-X dev`, and any other implementation specific settings warnings.hide_warnings() else: # Only accounts for `-W` import sys if not sys.warnoptions: warnings.simplefilter("ignore") (We can also do both, of course) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Let's discuss -Xdev implementation issue at https://bugs.python.org/issue32230 In short, -Xdev must add its warning at the end to respect BytesWarning, whereas it's not possible with -W option :-( Victor Le 6 déc. 2017 09:15, "Nick Coghlan" <ncoghlan@gmail.com> a écrit : On 6 December 2017 at 14:50, Nick Coghlan <ncoghlan@gmail.com> wrote:
Having rebased the PEP 565 patch atop the "-X dev" changes, I think that if we don't change some of the details of how `-X dev` is implemented, `warnings.hide_warnings` (or a comparable convenience API) is going to be a requirement to help app developers effectively manage their default warnings settings in 3.7+. The problem is that devmode doesn't currently behave the same way `-Wd` does when it comes to sys.warnoptions: $ ./python -Wd -c "import sys; print(sys.warnoptions); print(sys.flags.dev_mode)" ['d'] False $ ./python -X dev -c "import sys; print(sys.warnoptions); print(sys.flags.dev_mode)" [] True As currently implemented, the warnings module actually checks `sys.flags.dev_mode` directly during startup (or `sys._xoptions` in the case of the pure Python fallback), and populates the warnings filter differently depending on what it finds: $ ./python -c "import warnings; print('\n'.join(map(str, warnings.filters)))" ('default', None, <class 'DeprecationWarning'>, '__main__', 0) ('ignore', None, <class 'DeprecationWarning'>, None, 0) ('ignore', None, <class 'PendingDeprecationWarning'>, None, 0) ('ignore', None, <class 'ImportWarning'>, None, 0) ('ignore', None, <class 'BytesWarning'>, None, 0) ('ignore', None, <class 'ResourceWarning'>, None, 0) $ ./python -X dev -c "import warnings; print('\n'.join(map(str, warnings.filters)))" ('ignore', None, <class 'BytesWarning'>, None, 0) ('default', None, <class 'ResourceWarning'>, None, 0) ('default', None, <class 'Warning'>, None, 0) $ ./python -Wd -c "import warnings; print('\n'.join(map(str, warnings.filters)))" ('default', None, <class 'Warning'>, None, 0) ('default', None, <class 'DeprecationWarning'>, '__main__', 0) ('ignore', None, <class 'DeprecationWarning'>, None, 0) ('ignore', None, <class 'PendingDeprecationWarning'>, None, 0) ('ignore', None, <class 'ImportWarning'>, None, 0) ('ignore', None, <class 'BytesWarning'>, None, 0) ('ignore', None, <class 'ResourceWarning'>, None, 0) This means the app development snippet proposed in the PEP will no longer do the right thing, since it will ignore the dev mode flag: if not sys.warnoptions: # This still runs for `-X dev` warnings.simplefilter("ignore") My main suggested fix would be to adjust the way `-X dev` is implemented to include `sys.warnoptions.append('default')` (and remove the direct dev_mode query from the warnings module code). However, another possible way to go would be to make the correct Python 3.7+-only snippet look like this: import warnings warnings.hide_warnings() And have the forward-compatible snippet look like this: import warnings: if hasattr(warnings, "hide_warnings"): # Accounts for `-W`, `-X dev`, and any other implementation specific settings warnings.hide_warnings() else: # Only accounts for `-W` import sys if not sys.warnoptions: warnings.simplefilter("ignore") (We can also do both, of course) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick and Victor, I'm still hoping to accept this PEP, but I don't have time to wrap my head around -Xdev ("devmode"?) which appears to be Victor's latest pet project. Should PEP 565 be changed to copy with devmode's behavior, or the other way around, or should they just ignore each other? It is not clear of me what the status of the mention in PEP 565 of -Xdev is -- normative or informational? I really don't want to have to learn how devmode works in order to be able to accept PEP 565 (or send it back for revision), so I am asking you two to let me know. On Wed, Dec 6, 2017 at 1:42 AM, Victor Stinner <victor.stinner@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)

Hi, 2017-12-12 21:21 GMT+01:00 Guido van Rossum <guido@python.org>:
The warnings filters had a few corner cases. We discussed with Nick to fix them to make them simpler. We agreed on these priorities for command line options and environment variables: -b and -bb > -W > PYTHONWARNINGS > -X dev > default filters In release mode, the default filters became: ignore::DeprecationWarning ignore::PendingDeprecationWarning ignore::ImportWarning ignore::ResourceWarning ignore::BytesWarning is gone. We now rely on the fact the BytesWarning should not be emited without -b nor -bb in practice. It has been implemented in https://bugs.python.org/issue32230 ! (I just merged Nick's PR.) Now -X dev behaves again as my initial propopal: for warnings, "-X dev" simply behaves as "-W default". (Previously, I had to hack the code to respect -b and -bb options, but I don't think that it's worth it to explain that here, it's doesn't matter anymore ;-)) The PEP 565 is still different: it doesn't behaves as "-W default", but "-W default::DeprecationWarning:__main__". Only DeprecationWarning warnings are shown, whereas -X dev shows DeprecationWarning, but also PendingDeprecationWarning, ResourceWarning and ImportWarning. Moreover, -X dev shows warnings in all modules, not only __main__. You may see -X dev as a builtin linter, whereas PEP 565 seems to be very specific to one specific issue: display deprecation warnings, but only in the __main__ module. Does it help you to understand the difference? Note: I still dislike the PEP 565, but well, that's just my opinion ;-) Victor

OK, in that case I'll just pronounce approval right here. Considered disagreement is acceptable. Nick, congrats with PEP 565! Please update the PEP to mark it as approved with a link to this message as the resolution, and let's get the implementation into 3.7a4! On Tue, Dec 12, 2017 at 2:58 PM, Victor Stinner <victor.stinner@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)

2017-12-13 0:24 GMT+01:00 Guido van Rossum <guido@python.org>:
Considered disagreement is acceptable.
Sure, I'm fine with that ;-)
Nick wrote that he will be away, since I update his PEP: https://github.com/python/peps/commit/355eced94cf4117492c9e1eee8f950f08e53ec... Victor
participants (11)
-
Antoine Pitrou
-
Barry Warsaw
-
brent bejot
-
Chris Angelico
-
Donald Stufft
-
Guido van Rossum
-
Nathaniel Smith
-
Nick Coghlan
-
Serhiy Storchaka
-
Stefan Krah
-
Victor Stinner