From encukou at gmail.com  Tue Jul  5 07:26:38 2016
From: encukou at gmail.com (Petr Viktorin)
Date: Tue, 5 Jul 2016 13:26:38 +0200
Subject: [Import-SIG] Static exceptions
Message-ID: <8e824690-d57b-9c22-7666-2800ab39f860@gmail.com>

Hello,
When implementing providing better access to module state (see my
[pre-PEP]), I hit one more detour I would like to solve before going on
with the PEP.

For isolated modules to work, any class whose methods touch module state
must be a heap class, so that each instance of a module can have its own
class object. When the PEP goes through, heap classes will provide
access to module state without messy global registration. But, to create
instances of heap classes, one will need the module state ? there is no
one global class if each module has its own. Heap types are "viral" ?
anything that touches them must itself be a heap type.

Curently, most exception types, apart from the ones in builtins, are
heap types. I believe this is simply because there is a convenient way
to create them: [PyErr_NewException]. Most exceptions do not need to be
heap types: it gives Python code the ability to add custom attributes to
the exception class, and I argue that monkeypatching exception classes
is not very useful, and certainly not pretty.
In some cases, heap exception types are harmful: the [sqlite] module
uses C-global variables for the exceptions, so any time the module is
loaded (e.g. from a subinterpreter), these pointers will be overwritten
with freshly created classes. It seems like the module was written
expecting PyErr_NewException to hand out static types ? which is an
assumption that someone casually reading the docs could very well form.

Moreover, since raising exception is a common operations, and heap types
will be "viral", PyErr_NewException will tend to "infect" the module
with "heap-type-ness" ? at least if the module decides play well with
subinterpreters/isolation. Many modules could go without module state
entirely if the exception classes were static.

So, I propose to add a new function: PyErr_PrepareStaticException, which
would work similarly to PyErr_NewExceptionWithDoc, but it would fill a
static (pre-allocated zero-filled) PyTypeObject, fill it with the
provided info, and call PyType_Ready.
I've put an initial implementation (without docs and stable ABI guards)
in a [github commit], and I have a [github branch] showing how it would
affect several modules.

There are some disadvantages to this approach:
- Breaks strict backwards compatibility: it's not possible to
monkey-patch exception classes created this way
- Sadly, It's incompatible with the stable ABI
- Multiple inheritance is not possible for static types, so
PyErr_NewExceptionWithDoc can't be deprecated

Despite those I believe adding PyErr_PrepareStaticException, and moving
builtin modules to it where possible, would be good.

Please let me know if you have any concerns, otherwise I'll add this to
the pre-PEP. This and a solution for Argument Clinic (which apart from
[issue27332] should be smooth sailing) are the blockers currently.


[pre-PEP]:
https://github.com/encukou/peps/blob/module-state-access/pep-9999.txt
[PyErr_NewException]:
https://docs.python.org/3/c-api/exceptions.html?highlight=pyerr_newexception#c.PyErr_NewException
[sqlite]:
https://hg.python.org/cpython/file/tip/Modules/_sqlite/module.c#l38
[github commit]:
https://github.com/encukou/cpython/commit/a8dcae33ba840e34337ac87fcf15886f6083b122
[github branch]:
https://github.com/encukou/cpython/commits/static-exceptions
[issue27332]: http://bugs.python.org/issue27332