[Python-ideas] Module lifecycle: simple alternative to PEP 3121/PEP 489

[Nikita Nemkin]

On Thu, Apr 14, 2016 at 2:57 PM, Petr Viktorin <encukou at gmail.com> wrote:
> On 04/14/2016 10:23 AM, Nikita Nemkin wrote:
>> Reading PEP 3121/PEP 489 I can't stop wondering, why do extension
>> modules require such specialized lifecycle APIs? Why not just let
>> them subclass ModuleType? (Or any type, really, but ModuleType might
>> be a good place to define some standard behavior.)
>
> Good question.
> I'll list some assumptions; if you don't share them we can talk more
> about these:
> - somethings are easy in Python, but not pleasant to do correctly* in C:
>   - subclassing
>   - creating simple objects
>   - setting attributes
> - most modules don't need special functionality
>
> * ("Correctly" includes things like error checking, which many
> third-party modules skimp on.)
>
> Most of the API is in the style of "leave this NULL unless you need
> something special", i.e. simple things are easy, but the complex cases
> are possible. Creating custom ModuleType subclasses is possible, but I
> definitely wouldn't want to require every module author to do that.
>
> A lot of the API is convenience features, which are important: they do
> the right thing (for example, w.r.t. error checking), and they're easier
> to use than alternatives. This makes the API grows into several layers;
> for example:
> - m_methods in the PyModuleDef
> - PyModule_AddFunctions
> - PyCFunction_NewEx & PyObject_SetAttrString
> Usually you just set the first, but if you need more control (e.g.
> you're creating modules dynamically), you can use the lower-level tools.
>
> Your suggestion won't really help with this kind of complexity.

I agree that C API is quite difficult and dangerous to use directly
(that's why I always use Cython), but I don't agree that separate module
init API makes things simpler.

PyModuleDef appears to be PyTypeObject surrogate with similar
(or identical?) semantics. That's an extra concept to learn about,
an extra bit of documentation to consult when writing new code.
PyTypeObject, on the other hand, is fundamental and unavoidable,
regardless of module init system.

Practically speaking,
PyModuleDef_Init(&spam_def) and PyType_Ready(&spam_type)
differ only in the number of zeroes in their respective static structs.
There's also PyType_FromSpec (stable ABI), which might appeal to
some people more than PyType_Ready.

> Oh, and there is a technical reason against subclassing ModuleType
> unless necessary: Custom ModuleType subclasses cannot be made to work
> with runpy (i.e. python -m). For ModuleType (the ones without a custom
> create_module in the current API), this doesn't *currently* work, but
> the PEPs were written so that it's "just" a question of spending some
> development effort on runpy.

I didn't even know that python -m supported extension modules.

>> Module instance naturally encapsulates C level module state.
>> GC/finalization happens just like for any other object. PEP 3121
>> becomes redundant.
>
> It doesn't become fully redundant: m_methods is still useful.

ModuleType subclasses have tp_methods. A little check in PyType_Ready
can make it behave like PyModuleDef.m_methods. Modules don't need
normal methods anyway.

> With your suggestion, I fear that someone would quickly come up with a
> macro to automate creating simple ModuleType instances, and at that
> point the API would be as complex as it is now, but every module
> instance would now also have an extra ModuleType subclass – and I don't
> think that's either simpler or more effective.

One or two standard macros that help with ModuleType subsclassing
are in order. Something like PyModule_HEAD_INIT, whose usage is
already obvious from the name.

> I personally think that your suggestion wouldn't make the API
> substantially simpler, assuming you would keep it as robust and easy to
> use as the current solution. And if you would want to maintain backwards
> compatibility (even with only the pre-PEP 489 state), it would be even
> harder.
>
> Many people thought about the current APIs, and (almost) all of the
> unpleasant decisions we had to make do have their reasons. (And if you
> ask about a more specific decision, I can give you the specific reasons.)

Regarding backward compatibility, PEP 3121 and most of PEP 489
will work just fine as a facade to ModuleType subclass.

I understand that existing PEPs are well researched and there's no
real incentive to change. My proposal is more of a hypothetical kind.