[Python-Dev] Pre-PEP: Redesigning extension modules

[Nick Coghlan]

On 25 Aug 2013 01:44, "Stefan Behnel" <stefan_ml at behnel.de> wrote:
>
> Nick Coghlan, 24.08.2013 16:22:
> > On 24 August 2013 23:19, Stefan Behnel wrote:
> >> Nick Coghlan, 24.08.2013 13:36:
> >>> On 24 August 2013 15:51, Nick Coghlan wrote:
> >>>> My current plan is to create an experimental prototype of this
> >>>> approach this weekend. That will include stdlib test cases, so it
will
> >>>> also show how it looks from the extension developer's point of view.
> >>>
> >>> I prototyped as much as I could without PEP 451's ModuleSpec support
here:
> >>>
> >>>
https://bitbucket.org/ncoghlan/cpython_sandbox/commits/branch/new_extension_imports
> >>
> >> Cool. I'll take a look.
> >
> > The new _PyImport_CreateAndExecExtensionModule function does the heavy
lifting:
> >
> >
https://bitbucket.org/ncoghlan/cpython_sandbox/src/081f8f7e3ee27dc309463b48e6c67cf4880fca12/Python/importdl.c?at=new_extension_imports#cl-65
> >
> > One key point to note is that it *doesn't* call
> > _PyImport_FixupExtensionObject, which is the API that handles all the
> > PEP 3121 per-module state stuff. Instead, the idea will be for modules
> > that don't need additional C level state to just implement
> > PyImportExec_NAME, while those that *do* need C level state implement
> > PyImportCreate_NAME and return a custom object (which may or may not
> > be a module subtype).
>
> Is it really a common case for an extension module not to need any C level
> state at all? I mean, this might work for very simple accelerator modules
> with only a few stand-alone functions. But anything non-trivial will
almost
> certainly have some kind of global state, cache, external library, etc.,
> and that state is best stored at the C level for safety reasons.

I'd prefer to encourage people to put that state on an exported *type*
rather than directly in the module global state. So while I agree we need
to *support* C level module globals, I'd prefer to provide a simpler
alternative that avoids them.

We also need the create/exec split to properly support reloading. Reload
*must* reinitialize the object already in sys.modules instead of inserting
a different object or it completely misses the point of reloading modules
over deleting and reimporting them (i.e. implicitly affecting the
references from other modules that imported the original object).

> > Such modules can still support reloading (e.g.
> > to pick up reloaded or removed module dependencies) by providing
> > PyImportExec_NAME as well.
> >
> > (in a PEP 451 world, this would likely be split up as two separate
> > functions, one for create, one for exec)
>
> Can't we just always require extension modules to implement their own
type?
> Sure, it's a lot of boiler plate code, but that could be handled by a
> simple C code generator or maybe even a copy&paste example in the docs. I
> would like to avoid making it too easy for users in the future to get
> anything wrong with reloading or sub-interpreters. Most people won't test
> these things for their own code and the harder it is to make them not
work,
> the more likely it is that a given set of dependencies will properly work
> in a sub-interpreter.
>
> If users are required to implement their own type, I think it would be
more
> obvious where to put global module state, how to define functions (i.e.
> module methods), how to handle garbage collection at the global module
> level, etc.

Take a look at the current example - everything gets stored in the module
dict for the simple case with no C level global state. The module level
functions are still added through a Py_MethodDef array, the docstring still
comes from a C char pointer. I did have to fix the custom type's tp_new
method to use the type pointer passed in by the interpreter rather than a C
static global pointer, but that change would also have been needed if
defining a custom type.

Since Antoine fixed it, there's also nothing particularly quirky about
module destruction in 3.4+ - cyclic GC should "just work".

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130825/050b2e7d/attachment.html>