[Python-ideas] Module lifecycle: simple alternative to PEP 3121/PEP 489

[Nick Coghlan]

On 14 April 2016 at 23:51, Nikita Nemkin <nikita at nemkin.ru> wrote:
> PyModuleDef appears to be PyTypeObject surrogate with similar
> (or identical?) semantics. That's an extra concept to learn about,
> an extra bit of documentation to consult when writing new code.
> PyTypeObject, on the other hand, is fundamental and unavoidable,
> regardless of module init system.

The internal details of PyTypeObject are eminently avoidable, either
by not defining your own custom types in C code at all (you can get a
long way with C level acceleration just by defining functions and
using instances of existing types), or else by only defining them
dynamically as heap types (the kind created by class statements) via
PyType_FromSpec:
https://www.python.org/dev/peps/pep-0384/#type-objects

To answer your original question, though, PEP 489 needs to read in the
context of PEP 451, which was the one that switched the overall import
system over to the multi-phase import model:
https://www.python.org/dev/peps/pep-0451/

One of the main goals of importlib in general is to let the
interpreter do more of the heavy lifting for things that absolutely
have to be done correctly if you want your import hook or module to
behave "normally". With Python level modules, the interpreter has
always taken care of creating the module for "normal" imports, with
the module author only having to care about populating that namespace
with content (by running Python code).

PEP 451 extended that same convenience to authors of module loaders,
as they could now just define a custom exec_module, and use the
default module creation code rather than having to write their own as
part of a load_module implementation.

PEP 489 then brought that capability to extension modules: extension
module authors can now decide not to worry about module creation at
all, and instead just use the Exec hook to populate the standard
module object that CPython provides by default.

That means caring about the module creation step in an extension
module is now primarily a matter of performance optimisation for
access to module global state - as Stefan notes, the indirection
mechanism in PEP 3121 can be significantly slower than using C level
static variables, and indirection through a Python level namespace is
likely to be even slower.

However, even in those cases, the PEP 489 mechanism gives the
extension module reliable access to information it didn't previously
have access to, since the create method receives a fully populated
module spec.

Once the question is narrowed down to "How can an extension module
fully support subinterpreters and multiple Py_Initialize/Finalize
cycles without incurring PEP 3121's performance overhead?" then the
short answer becomes "We don't know, but ideas for that are certainly
welcome, either here or over on import-sig".

Returning custom module subclasses from the Create hook is certainly
one mechanism for that (it's why supporting such subclasses was a
design goal for PEP 489), but like other current solutions, they run
afoul of the problem that methods defined in C extension modules
currently don't receive a reference to the defining module, only to
the class instance (which is a general problem with the way methods
are defined in C rather than a problem with the import system
specifically).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia