[Python-Dev] Pre-PEP: Redesigning extension modules

[Nick Coghlan]

On 23 August 2013 19:18, Antoine Pitrou <solipsis at pitrou.net> wrote:
>
> Hi,
>
> Le Fri, 23 Aug 2013 10:50:18 +0200,
> Stefan Behnel <stefan_ml at behnel.de> a écrit :
>>
>> Here's an initial attempt at a PEP for it. It is based on the
>> (unfinished) ModuleSpec PEP, which is being discussed on the
>> import-sig mailing list.
>
> Thanks for trying this. I think the PEP should contain working example
> code for module initialization (and creation), to help gauge the
> complexity for module writers.

I've been thinking a lot about this as part of reviewing PEP 451 (the
ModuleSpec PEP that Stefan's pre-PEP mentions). The relevant feedback
on import-sig hasn't made it into PEP 451 yet (Eric is still
considering the suggestion), but what I'm proposing is a new
relatively *stateless* API for loaders, which consists of two methods:

  def create_module(self, spec):
      """Given a ModuleSpec, return the object to be added to sys.modules"""

  def exec_module(self, mod):
      """Execute the given module, updating it for the current system state"""

create_module would be optional - if not defined, the import system
would automatically create a normal module object. If it is defined,
the import system would call it and then take care of setting all the
standard attributes (__name__, __spec__, etc) on the result if the
loader hadn't already set them.

exec_module would be required, and is the part that actually fully
initialises the module.

"imp.reload" would then translate to calling exec_module on an
existing module without recreating it.

For loaders that provide the new API, the global import state
manipulation would all be handled by the import system. Such loaders
would still be free to provide load_module() anyway for backwards
compatibility with earlier Python versions, since the new API would
take precedence.

In this context, the API I was considering for extension modules was
slightly different from that in Stefan's proto-PEP (although it was
based on some of Stefan's suggestions in the earlier threads).
Specifically, I'm thinking of an API like this that does a better job
of supporting reloading:

    PyObject * PyImportCreate_<modulename>(PyObject *spec); /* Optional */
    int PyImportExec_<modulename>(PyObject *mod);

Implementing PyImportCreate would only be needed if you had C level
state to store - if you're happy storing everything in the module
globals, then you would only need to implement PyImportExec.

My current plan is to create an experimental prototype of this
approach this weekend. That will include stdlib test cases, so it will
also show how it looks from the extension developer's point of view.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia