[Python-Dev] Pre-PEP: Redesigning extension modules

[Stefan Behnel]

Nick Coghlan, 24.08.2013 13:36:
> On 24 August 2013 15:51, Nick Coghlan wrote:
>> My current plan is to create an experimental prototype of this
>> approach this weekend. That will include stdlib test cases, so it will
>> also show how it looks from the extension developer's point of view.
> 
> I prototyped as much as I could without PEP 451's ModuleSpec support here:
> 
>     https://bitbucket.org/ncoghlan/cpython_sandbox/commits/branch/new_extension_imports

Cool. I'll take a look.


> On systems that use dynload_shlib (at least Linux & the BSDs), this
> branch allows extension modules to be imported if they provide a
> PyImportExec_NAME hook. The new hook is preferred to the existing
> PyInit_NAME hook, so extension modules using the stable ABI can
> provide both and degrade to the legacy initialisation API on older
> versions of Python.

Hmm, right, good call. Since both init schemes have to be part of the
stable ABI, we can's rely on people compiling out one or the other. So
using the old one as a fallback should work. However, only actual usage in
code will tell us how it feels on user side. Supporting both in the same
binary will most likely complicate things quite a bit.


> The PyImportExec hook is called with a pre-created module object that
> the hook is then expected to populate. To aid in this task, I added
> two new APIs:
> 
>     PyModule_SetDocString
>     PyModule_AddFunctions
> 
> These cover setting the docstring and adding module level functions,
> tasks that are handled through the PyModule_Create API when using the
> PyInit_NAME style hook.

What are those needed for? If you subtype the module type, or provide an
arbitrary extension type as implementation, you'd get these for free,
wouldn't you? It's in no way different from setting up an extension type.


> The _testimportexec.c module

Where can I find that module?


> was derived from the existing example
> xxlimited.c module, with a few name changes. The main functional
> difference is that _testimportexec uses the new API, so the module
> object is created externally and passed in to the API, rather than
> being created by the extension module. The effect of this can be seen
> in the test suite, where ImportExecTests.test_fresh_module shows that
> loading the module twice will create two *different* modules, unlike
> the legacy API.

Stefan