[Import-SIG] PEP 489: Multi-phase extension module initialization; version 5

[Petr Viktorin]

On 05/20/2015 01:56 AM, Eric Snow wrote:
> On Mon, May 18, 2015 at 9:51 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On 19 May 2015 at 10:07, Eric Snow <ericsnowcurrently at gmail.com> wrote:
>   [snip]
>>> Was there any consideration made for just ignoring unknown slot IDs?
>>> My gut reaction is that you have it the right way, but I can still
>>> imagine use cases for custom slots that PyModuleDef_Init wouldn't know
>>> about.
>>
>> The "known slots only, all other slot IDs are reserved for future use"
>> slot semantics were copied directly from PyType_FromSpec in PEP 384.
>> Since it's just a numeric slot ID, you'd run a high risk of conflicts
>> if you allowed for custom extensions.
>>
>> If folks want to do more clever things, they'll need to use the create
>> or exec slot to stash them on the module object, rather than storing
>> them in the module definition.
> 
> Makes sense.  This does remind me of something I wanted to ask.  Would
> it make sense to leverage ModuleSpec.loader_state?  If I recall
> correctly, we added loader_state with extension modules in mind.

I don't think we want to go out of our way to support non-module
objects. Module subclasses should cover any needed functionality, and
they will support slots.

>>>> [snip]
>>>> Extension authors are advised to keep Py_mod_create minimal, an in
>>>> particular
>>>> to not call user code from it.
>>>
>>> This is a pretty important point as well.  We'll need to make sure
>>> this is sufficiently clear in the documentation.  Would it make sense
>>> to provide helpers for common cases, to encourage extension authors to
>>> keep the create function minimal?
>>
>> The main encouragement is to not handcode your extension modules at
>> all, and let something like Cython or SWIG take care of the
>> boilerplate :)
> 
> Hey, I tried to make something happen over on python-ideas! :)  Some
> folks just don't want to go far enough.

Yeah, as someone who's trying to get Python3 porting patches to Samba, I
can tell you some upstreams really, really, really don't like rewriting
their code.

>>>> As a rule of thumb, modules that rely on PyState_FindModule are, at the
>>>> moment,
>>>> not good candidates for porting to the new mechanism.
>>>
>>> Are there any plans for a follow-up effort to help with this case?
>>
>> The problem here is that the PEP 3121 module state approach provides
>> storage on a *per-interpreter* basis, that is then shared amongst all
>> module instances created from a given module definition.
> 
> You mean a form of interpreter-local storage?  Also, the module
> definition is effectively global right?

The PyModuleDef is global and static, but you can create any number of
module objects from it.
Each interpreter gets its own module object, with state specific to the
module object. (And with a custom finder/loader you can make multiple
modules from the same def within one interpreter.

>> For pure Python code, we don't have this problem, since the
>> interpreter takes care of providing a properly scoped globals()
>> reference to *all* functions defined in that module, regardless of
>> whether they're module level functions or method definitions on a
>> class. At the C level, we don't have that, as only module level
>> functions get a module reference passed in - methods only get a
>> reference to their class instance, without a reference to the module
>> globals, and delayed callbacks can be a problem as well.
> 
> Yuck.  Is this something we could fix?  Is __module__ not set on all functions?

The module object is not stored on classes, so methods dont' have access
to it. I want a fix for that to be my next PEP :)


-- 
Petr Viktorin