[Python-Dev] Make extension module initialisation more like Python module initialisation

[Stefan Behnel]

M.-A. Lemburg, 08.11.2012 14:01:
> On 08.11.2012 13:47, Stefan Behnel wrote:
>> I suspect that this will be put into a proper PEP at some point, but I'd
>> like to bring this up for discussion first. This came out of issues 13429
>> and 16392.
>>
>> http://bugs.python.org/issue13429
>>
>> http://bugs.python.org/issue16392
>>
>> Stefan
>>
>>
>> The problem
>> ===========
>>
>> Python modules and extension modules are not being set up in the same way.
>> For Python modules, the module is created and set up first, then the module
>> code is being executed. For extensions, i.e. shared libraries, the module
>> init function is executed straight away and does both the creation and
>> initialisation. This means that it knows neither the __file__ it is being
>> loaded from nor its package (i.e. its FQMN). This hinders relative imports
>> and resource loading. In Py3, it's also not being added to sys.modules,
>> which means that a (potentially transitive) re-import of the module will
>> really try to reimport it and thus run into an infinite loop when it
>> executes the module init function again. And without the FQMN, it's not
>> trivial to correctly add the module to sys.modules either.
>>
>> We specifically run into this for Cython generated modules, for which it's
>> not uncommon that the module init code has the same level of complexity as
>> that of any 'regular' Python module. Also, the lack of a FQMN and correct
>> file path hinders the compilation of __init__.py modules, i.e. packages,
>> especially when relative imports are being used at module init time.
>>
>> The proposal
>> ============
>>
>> ... [callbacks] ...
>>
>> Alternatives
>> ============
>> ...
>> 3) The callback could be registered statically in the PyModuleDef struct by
>> adding a new field. This is not trivial to do in a backwards compatible way
>> because the struct would grow longer without explicit initialisation by
>> existing user code. Extending PyModuleDef_HEAD_INIT might be possible but
>> would still break at least binary compatibility.
> 
> I think the above is the cleaner approach than the callback mechanism.

Oh, definitely.


> There's no problem in adding new slots to the end of the PyModuleDef struct
> - we've been doing that for years in many other structs :-)
> 
> All you have to do is bump the Python API version number.
> 
> (Martin's PEP http://www.python.org/dev/peps/pep-3121/ has the details)

The difference is that this specific struct is provided by user code and
(typically) initialised statically. There is no guarantee that user code
that does not expect the additional field will initialise it to 0. Failing
that, I don't see how we could trust its value in any way.

Stefan