[Python-Dev] PEP 565: Show DeprecationWarning in __main__

[Victor Stinner]

Hi,

I'm not convinced that this PEP 565 will prevent developers to be
surprised when upgrading Python, since more and more applications are
using an entry point: an import + a single function call. For example,
*all* OpenStack applications use an entry point and so will be
unaffected by this PEP.

There is no suprise, it's documented in the PEP: "code imported from
an executable script wrapper generated at installation time based on a
console_scripts or gui_scripts entry point definition".

It's hard to find a compromise between two incompatible use cases:
"run an application" ("user") and "develop an application"
("developer"). I proposed again my "-X dev" idea in another thread for
the "develop an application" use case ;-)

If the Python REPL is included in the "run an application" use case,
the frontier between user and developer becomes blurry :-) Is REPL
designed for users or developers? Should Python guess the intent of
the human connected to the keyboard? ...

Victor

2017-11-12 10:24 GMT+01:00 Nick Coghlan <ncoghlan at gmail.com>:
> I've written a short(ish) PEP for the proposal to change the default
> warnings filters to show DeprecationWarning in __main__:
> https://www.python.org/dev/peps/pep-0565/
>
> The core proposal itself is just the idea in
> https://bugs.python.org/issue31975 (i.e. adding
> "default::DeprecationWarning:__main__" to the default filter set), but
> the PEP fills in some details on the motivation for the original
> change to the defaults, and why the current proposal is to add a new
> filter for __main__, rather than dropping the default
> DeprecationWarning filter entirely.
>
> The PEP also proposes repurposing the existing FutureWarning category
> to explicitly mean "backwards compatibility warnings that should be
> shown to users of Python applications" since:
>
> - we don't tend to use FutureWarning for its original nominal purpose
> (changes that will continue to run but will do something different)
> - FutureWarning was added in 2.3, so it's available in all still
> supported versions of Python, and is shown by default in all of them
> - it's at least arguably a less-jargony spelling of
> DeprecationWarning, and hence more appropriate for displaying to end
> users that may not have encountered the specific notion of "API
> deprecation"
>
> Cheers,
> Nick.
>
> ==============
> PEP: 565
> Title: Show DeprecationWarning in __main__
> Author: Nick Coghlan <ncoghlan at gmail.com>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 12-Nov-2017
> Python-Version: 3.7
> Post-History: 12-Nov-2017
>
>
> Abstract
> ========
>
> In Python 2.7 and Python 3.2, the default warning filters were updated to hide
> DeprecationWarning by default, such that deprecation warnings in development
> tools that were themselves written in Python (e.g. linters, static analysers,
> test runners, code generators) wouldn't be visible to their users unless they
> explicitly opted in to seeing them.
>
> However, this change has had the unfortunate side effect of making
> DeprecationWarning markedly less effective at its primary intended purpose:
> providing advance notice of breaking changes in APIs (whether in CPython, the
> standard library, or in third party libraries) to users of those APIs.
>
> To improve this situation, this PEP proposes a single adjustment to the
> default warnings filter: displaying deprecation warnings attributed to the main
> module by default.
>
> This change will mean that code entered at the interactive prompt and code in
> single file scripts will revert to reporting these warnings by default, while
> they will continue to be silenced by default for packaged code distributed as
> part of an importable module.
>
> The PEP also proposes a number of small adjustments to the reference
> interpreter and standard library documentation to help make the warnings
> subsystem more approachable for new Python developers.
>
>
> Specification
> =============
>
> The current set of default warnings filters consists of::
>
>     ignore::DeprecationWarning
>     ignore::PendingDeprecationWarning
>     ignore::ImportWarning
>     ignore::BytesWarning
>     ignore::ResourceWarning
>
> The default ``unittest`` test runner then uses ``warnings.catch_warnings()``
> ``warnings.simplefilter('default')`` to override the default filters while
> running test cases.
>
> The change proposed in this PEP is to update the default warning filter list
> to be::
>
>     default::DeprecationWarning:__main__
>     ignore::DeprecationWarning
>     ignore::PendingDeprecationWarning
>     ignore::ImportWarning
>     ignore::BytesWarning
>     ignore::ResourceWarning
>
> This means that in cases where the nominal location of the warning (as
> determined by the ``stacklevel`` parameter to ``warnings.warn``) is in the
> ``__main__`` module, the first occurrence of each DeprecationWarning will once
> again be reported.
>
> This change will lead to DeprecationWarning being displayed by default for:
>
> * code executed directly at the interactive prompt
> * code executed directly as part of a single-file script
>
> While continuing to be hidden by default for:
>
> * code imported from another module in a ``zipapp`` archive's ``__main__.py``
>   file
> * code imported from another module in an executable package's ``__main__``
>   submodule
> * code imported from an executable script wrapper generated at installation time
>   based on a ``console_scripts`` or ``gui_scripts`` entry point definition
>
> As a result, API deprecation warnings encountered by development tools written
> in Python should continue to be hidden by default for users of those tools
>
> While not its originally intended purpose, the standard library documentation
> will also be updated to explicitly recommend the use of
> ``FutureWarning`` (rather
> than ``DeprecationWarning``) for backwards compatibility warnings that are
> intended to be seen by *users* of an application.
>
> This will give the following three distinct categories of backwards
> compatibility warning, with three different intended audiences:
>
> * ``PendingDeprecationWarning``: reported by default only in test runners that
>   override the default set of warning filters. The intended audience is Python
>   developers that take an active interest in ensuring the future compatibility
>   of their software (e.g. professional Python application developers with
>   specific support obligations).
> * ``DeprecationWarning``: reported by default for code that runs directly in
>   the ``__main__`` module (as such code is considered relatively unlikely to
>   have a dedicated test suite), but relies on test suite based reporting for
>   code in other modules. The intended audience is Python developers that are at
>   risk of upgrades to their dependencies (including upgrades to Python itself)
>   breaking their software (e.g. developers using Python to script environments
>   where someone else is in control of the timing of dependency upgrades).
> * ``FutureWarning``: always reported by default. The intended audience is users
>   of applications written in Python, rather than other Python developers
>   (e.g. warning about use of a deprecated setting in a configuration file
>   format).
>
> Given its presence in the standard library since Python 2.3, ``FutureWarning``
> would then also have a secondary use case for libraries and frameworks that
> support multiple Python versions: as a more reliably visible alternative to
> ``DeprecationWarning`` in Python 2.7 and versions of Python 3.x prior to 3.7.
>
>
> Motivation
> ==========
>
> As discussed in [1_] and mentioned in [2_], Python 2.7 and Python 3.2 changed
> the default handling of ``DeprecationWarning`` such that:
>
> * the warning was hidden by default during normal code execution
> * the `unittest`` test runner was updated to re-enable it when running tests
>
> The intent was to avoid cases of tooling output like the following::
>
>     $ devtool mycode/
>     /usr/lib/python3.6/site-packages/devtool/cli.py:1:
> DeprecationWarning: 'async' and 'await' will become reserved keywords
> in Python 3.7
>       async = True
>     ... actual tool output ...
>
> Even when `devtool` is a tool specifically for Python programmers, this is not
> a particularly useful warning, as it will be shown on every invocation, even
> though the main helpful step an end user can take is to report a bug to the
> developers of ``devtool``. The warning is even less helpful for general purpose
> developer tools that are used across more languages than just Python.
>
> However, this change proved to have unintended consequences for the following
> audiences:
>
> * anyone using a test runner other than the default one built into ``unittest``
>   (since the request for third party test runners to change their default
>   warnings filters was never made explicitly)
> * anyone using the default ``unittest`` test runner to test their Python code
>   in a subprocess (since even ``unittest`` only adjusts the warnings settings
>   in the current process)
> * anyone writing Python code at the interactive prompt or as part of a directly
>   executed script that didn't have a Python level test suite at all
>
> In these cases, ``DeprecationWarning`` ended up become almost entirely
> equivalent to ``PendingDeprecationWarning``: it was simply never seen at all.
>
>
> Limitations on PEP Scope
> ========================
>
> This PEP exists specifically to explain both the proposed addition to the
> default warnings filter for 3.7, *and* to more clearly articulate the rationale
> for the original change to the handling of DeprecationWarning back in Python 2.7
> and 3.2.
>
> This PEP does not solve all known problems with the current approach to handling
> deprecation warnings. Most notably:
>
> * the default ``unittest`` test runner does not currently report deprecation
>   warnings emitted at module import time, as the warnings filter
> override is only
>   put in place during test execution, not during test discovery and loading.
> * the default ``unittest`` test runner does not currently report deprecation
>   warnings in subprocesses, as the warnings filter override is applied directly
>   to the loaded ``warnings`` module, not to the ``PYTHONWARNINGS`` environment
>   variable.
> * the standard library doesn't provide a straightforward way to opt-in to seeing
>   all warnings emitted *by* a particular dependency prior to upgrading it
>   (the third-party ``warn`` module [3_] does provide this, but enabling it
>   involves monkeypatching the standard library's ``warnings`` module).
> * re-enabling deprecation warnings by default in __main__ doesn't help in
>   handling cases where software has been factored out into support modules, but
>   those modules still have little or no automated test coverage. Near term, the
>   best currently available answer is to run such applications with
>   ``PYTHONWARNINGS=default::DeprecationWarning`` or
>   ``python -W default::DeprecationWarning`` and pay attention to their
>   ``stderr`` output. Longer term, this is really a question for researchers
>   working on static analysis of Python code: how to reliably find usage of
>   deprecated APIs, and how to infer that an API or parameter is deprecated
>   based on ``warnings.warn`` calls, without actually running either the code
>   providing the API or the code accessing it
>
> While these are real problems with the status quo, they're excluded from
> consideration in this PEP because they're going to require more complex
> solutions than a single additional entry in the default warnings filter,
> and resolving them at least potentially won't require going through the PEP
> process.
>
> For anyone interested in pursuing them further, the first two would be
> ``unittest`` module enhancement requests, the third would be a ``warnings``
> module enhancement request, while the last would only require a PEP if
> inferring API deprecations from their contents was deemed to be an intractable
> code analysis problem, and an explicit function and parameter marker syntax in
> annotations was proposed instead.
>
>
> References
> ==========
>
> .. [1] stdlib-sig thread proposing the original default filter change
>    (https://mail.python.org/pipermail/stdlib-sig/2009-November/000789.html)
>
> .. [2] Python 2.7 notification of the default warnings filter change
>    (https://docs.python.org/3/whatsnew/2.7.html#changes-to-the-handling-of-deprecation-warnings)
>
> .. [3] Emitting warnings based on the location of the warning itself
>    (https://pypi.org/project/warn/)
>
> Copyright
> =========
>
> This document has been placed in the public domain.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com