[Import-SIG] PEP 489: Redesigning extension module loading

[Petr Viktorin]

On 03/30/2015 03:21 PM, Nick Coghlan wrote:
> On 26 March 2015 at 20:01, Petr Viktorin <encukou at gmail.com> wrote:
>> On 03/26/2015 05:25 AM, Nick Coghlan wrote:
>>> So actually writing that down suggests numeric slots may still be a
>>> better idea.
>>
>>
>> Yes, I think so as well. Also consider:
>> - Python can fail hard on unknown slot numbers. Checking for unknown exports
>> would require enumerating exported symbols, which is definitely not
>> something I can code for every platform (if that's even possible).
>> - Additional exported functions are not actually more type-safe – the
>> exported symbols would be void*, Python would still need to cast to an
>> appropriate function type.
>
> OK, sold - numeric slots it is (and I agree that changes the naming
> scheme back to favouring retaining "PyModuleExport" as the common
> prefix).
>
> I think that's all the points of discussion we still had open on the
> current draft covered, so I'll wait for the next update before
> commenting further :)

The next update will come with a preliminary implementation – things are 
starting to look good at

I do have another point though: I think we also need to implement 
PyState_AddModule and PyState_FindModule
equivalents for slots.

In the current draft, PEP 489 modules are not limited to one instance 
per definition: you could import one, and then (with a custom loader) 
import it again under a different name, and you'd get two independent 
modules.
In effect, there is true per-module state; but 
PyState_AddModule/PyState_FindModule allowed global (per-interpreter) 
module state.
I think this is a good thing; for one it will ease testing that modules 
are properly isolated, as required for sub-modules and module reloading. 
It's probably also nice for Cython's goal of emulating Python modules.
On the other hand, any callback or class that neds access to the module 
would now have to store a reference to it. This would involve a lot of 
refactoring for some modules, and I don't think we can afford that.


Also, some modules wrap a library that has global state (haven't checked 
stdlib, but curses, readline, locale are candidates).
It doesn't make sense to allow loading more instances of such modules.
Perhaps there should be a flag to distinguish them? Or just let them use 
PyState_AddModule/PyState_FindModule to prevent re-import?

The need for flags would be a good argument to after all have a 
ModuleExport structure wrapping around slots. Such a structure could 
also share PyModuleDef_Base, making it usable with 
PyState_AddModule/PyState_FindModule (there'd be new functions, but the 
machinery/data structure could be reused). So I'm starting to be more 
inclined to do this again:

typedef struct PyModule_Export {
   PyModuleDef_Base m_base;
   const char* m_doc;
   int m_flags;
   PyModule_Slot *m_slots; /* terminated by slot==0. */
} PyModule_Export;