[Python-ideas] Add a developer mode to Python: -X dev command line option

[Victor Stinner]

Hi,

When I develop on CPython, I'm always building Python in debug mode
using ./configure --with-pydebug. This mode enables a *lot* of extra
checks which helps me to detect bugs earlier. The debug mode makes
Python much slower and so is not the default. "python3" in Linux
distributions is a binary compiled in release mode. When they also
provide a binary compiled in debug mode, you will probably have issues
to use your existing C extensions, since they all of them are compiled
in release mode which is not compatible (you must recompile C
extensions in debug mode).

I propose to add a "development mode" to Python, to get a few checks
to detect bugs earlier: a new -X dev command line option. Example:

   python3.6 -X dev script.py

Checks enabled by this flag must:

* Not flood the stdout/stderr (ex: don't write one message per second)
* Have a low overhead in term of CPU and memory (ex: max 10%)

I propose to enable:

* Show DeprecationWarning and ResourceWarning warnings: python -Wd
* Show BytesWarning warning: python -b
* Enable Python assertions (assert) and set __debug__ to True: remove
(or just ignore) -O or -OO command line arguments
* faulthandler to get a Python traceback on segfault and fatal errors:
python -X faulthandler
* Debug hooks on Python memory allocators: PYTHONMALLOC=debug

For example, I consider that enabling tracemalloc is too expensive
(CPU & memory) and must not be enabled in -X dev.

I wrote a proof-of-concept: if -X dev is used, executable Python once
again with more parameters. Basically, replace:
   python3.6 -X dev ...
with
   PYTHONMALLOC=debug python3.6 -Wd -b -X faulthandler ...


The list of checks can be extended later. For example, we may enable
the debug mode of asyncio: PYTHONASYNCIODEBUG=1.


The scope of the "developer mode" is unclear to me. Many modules (ex:
asyncio) already have a debug mode. Would it be a bad idea to enable
*all* debug modes of *all* modules? For example, http.client has a
debug level: do you expect debug traces of the HTTP client when you
develop your application?

IMHO the scope must be well defined: don't modify modules of the
stdlib, only enable more debug flags in the Python core (warnings,
unicode, memory allocators, etc.).


Maybe we even a need -X dev=strict which would be stricter:

* use -Werror instead of -Wd: raise an exception when a warning is emitted
* use -bb instead of -b: get BytesWarning exceptions
* Replace "inconsistent use of tabs and spaces in indentation" warning
with an error in the Python parser
* etc.

Again, this list can be extended later.


Or maybe we need multiple level to control the quantity of debug
traces, warnings, ... written into stdout/stderr?


In my experience, the problem with converting warnings to errors is
that you don't control the code of your whole application. It's common
that third party modules raise DeprecationWarning, ResourceWarning,
etc. Even some modules of the Python stdlib raise DeprecatingWarning!
For example, I recall that distutils raises a DeprecationWarning on
Python 3.4 when importing the imp module.


"Similar" idea in other programming languages:

* Perl: http://perldoc.perl.org/strict.html
* PHP: https://secure.php.net/manual/fr/function.error-reporting.php

Victor