[Python-Dev] Add a developer mode to Python: -X dev command line option

[Victor Stinner]

Hi,

The discussion on DeprecationWarning reminded me my old idea of a
"development mode" for CPython: -X dev. Since Brett likes it, I post
it on python-dev. Last year, I posted this idea to python-ideas but my
idea was rejected:

https://mail.python.org/pipermail/python-ideas/2016-March/039314.html

In short:
   python3.7 -X dev script.py
behaves as:
   PYTHONMALLOC=debug python3.7 -Wd -b -X faulthandler script.py

The idea of -X dev is to enable "debug checks". Some of these checks
are already enabled by default if CPython is compiled in debug mode.
These checks help to debug bugs earlier.

The problem is that outside CPython development, Python developers use
a Python binary from their Linux distributor or a binary installed
from python.org: a binary compiled in release mode. Well, some Linux
distributions provide a debug build, but the ABI is incompatible, and
so is hard to use in practice.

Running the tests of your code in -X dev mode should help you to catch
subtle bugs and prevent regressions if you upgrade Python.

The -X dev mode doesn't raise hard exceptions, but usually only emits
warnings. It allows you to fix bugs one by one in your own code,
without having to fix bugs in third party code.

If you control your whole code base, you may want to enable -X
dev=strict which raises hard exceptions, to make sure that you are
really safe.

My "-X dev" idea is not incompatible with Nick's PEP 565 "Show
DeprecationWarning in __main__" and it's different: it's an opt-in
option, while Nick wants to change the default behaviour.

---

Full copy of my email sent last year to python-ideas.

When I develop on CPython, I'm always building Python in debug mode
using ./configure --with-pydebug. This mode enables a *lot* of extra
checks which helps me to detect bugs earlier. The debug mode makes
Python much slower and so is not the default. "python3" in Linux
distributions is a binary compiled in release mode. When they also
provide a binary compiled in debug mode, you will probably have issues
to use your existing C extensions, since they all of them are compiled
in release mode which is not compatible (you must recompile C
extensions in debug mode).

I propose to add a "development mode" to Python, to get a few checks
to detect bugs earlier: a new -X dev command line option. Example:

   python3.6 -X dev script.py

Checks enabled by this flag must:

* Not flood the stdout/stderr (ex: don't write one message per second)
* Have a low overhead in term of CPU and memory (ex: max 10%)

I propose to enable:

* Show DeprecationWarning and ResourceWarning warnings: python -Wd
* Show BytesWarning warning: python -b
* Enable Python assertions (assert) and set __debug__ to True: remove
(or just ignore) -O or -OO command line arguments
* faulthandler to get a Python traceback on segfault and fatal errors:
python -X faulthandler
* Debug hooks on Python memory allocators: PYTHONMALLOC=debug

For example, I consider that enabling tracemalloc is too expensive
(CPU & memory) and must not be enabled in -X dev.

I wrote a proof-of-concept: if -X dev is used, executable Python once
again with more parameters. Basically, replace:
   python3.6 -X dev ...
with
   PYTHONMALLOC=debug python3.6 -Wd -b -X faulthandler ...


The list of checks can be extended later. For example, we may enable
the debug mode of asyncio: PYTHONASYNCIODEBUG=1.


The scope of the "developer mode" is unclear to me. Many modules (ex:
asyncio) already have a debug mode. Would it be a bad idea to enable
*all* debug modes of *all* modules? For example, http.client has a
debug level: do you expect debug traces of the HTTP client when you
develop your application?

IMHO the scope must be well defined: don't modify modules of the
stdlib, only enable more debug flags in the Python core (warnings,
unicode, memory allocators, etc.).


Maybe we even a need -X dev=strict which would be stricter:

* use -Werror instead of -Wd: raise an exception when a warning is emitted
* use -bb instead of -b: get BytesWarning exceptions
* Replace "inconsistent use of tabs and spaces in indentation" warning
with an error in the Python parser
* etc.

Again, this list can be extended later.


Or maybe we need multiple level to control the quantity of debug
traces, warnings, ... written into stdout/stderr?


In my experience, the problem with converting warnings to errors is
that you don't control the code of your whole application. It's common
that third party modules raise DeprecationWarning, ResourceWarning,
etc. Even some modules of the Python stdlib raise DeprecatingWarning!
For example, I recall that distutils raises a DeprecationWarning on
Python 3.4 when importing the imp module.


"Similar" idea in other programming languages:

* Perl: http://perldoc.perl.org/strict.html
* PHP: https://secure.php.net/manual/fr/function.error-reporting.php

Victor