[Python-Dev] Make extension module initialisation more like Python module initialisation

[Stefan Behnel]

Brett Cannon, 08.11.2012 16:06:
> On Thu, Nov 8, 2012 at 10:00 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> 
>> Hi Brett,
>>
>> thanks for the feedback.
>>
>> Brett Cannon, 08.11.2012 15:41:
>>> On Thu, Nov 8, 2012 at 7:47 AM, Stefan Behnel wrote:
>>>> I propose to split the extension module initialisation into two steps in
>>>> Python 3.4, in a backwards compatible way.
>>>>
>>>> Step 1: The current module init function can be reduced to just creating
>>>> the module instance and returning it (and potentially doing some simple
>> C
>>>> level setup). Optionally, after creating the module (and this is the new
>>>> part), the module init code can register a C callback function that
>> will be
>>>> called after setting up the module.
>>>
>>> Why even bother with the module creation? Why can't Python do that as
>> well
>>> and then call the callback?
>>>
>>>
>>>> Step 2: The shared library importer receives the module instance from
>> the
>>>> module init function, adds __file__, __path__, __package__ and friends
>> to
>>>> the module dict, and then checks for the callback. If non-NULL, it
>> calls it
>>>> to continue the module initialisation by user code.
>>> [...]
>>> An alternative to the alternative is that if the PyInit2 function exists
>>> it's called instead of the the PyInit function, and then the PyInit
>>> function is nothing more than a single line function call (or whatever
>> the
>>> absolute bare minimum is) into some helper that calls the PyInit2 call
>>> properly for backwards ABI compatibility (i.e. passes in whatever details
>>> are lost by the indirection in function call). That provides an eventual
>>> upgrade path of dropping PyInit and moving over to PyInit2.
>>
>> In that case, you'd have to export the PyModuleDef descriptor as well,
>> because that's what tells CPython how the module behaves and what to do
>> with it to set it up properly (e.g. allocate module state space on the
>> heap).
> 
> True.
> 
>> In fact, if the module init function became a field in the descriptor, it
>> would be enough (taking backwards compatibility aside) if *only* the
>> descriptor was exported and used by the module loader.
>
> Also true.
> 
>> With the caveat that this might kill some less common but not necessarily
>> illegitimate use cases that do more than just creating and initialising a
>> single module...
> 
> You mean creating another module in the init function? That's fine, but
> that should be a call to __import__ anyway and that should handle things
> properly.

Ok.


> Else you are circumventing the import system and you can do
> everything from scratch.

I guess I'd be ok with putting that burden on users in this case.


> I don't see why this would stop you from doing
> anything you want, it just simplifies the common case.

The only problematic case I see here would be a module that calculates the
size of its state space at init time, e.g. based on some platform specifics
or environment parameters, anything from the platform specific size of some
data type to the runtime configured number of OpenMP threads.

That would make the PyModuleDef a compile time static thing - not sure if
that's currently required.

Stefan