[Import-SIG] PEP 489: Multi-phase extension module initialization; version 5

[Petr Viktorin]

On 05/19/2015 05:51 AM, Nick Coghlan wrote:
> On 19 May 2015 at 10:07, Eric Snow <ericsnowcurrently at gmail.com> wrote:
>> On Mon, May 18, 2015 at 8:02 AM, Petr Viktorin <encukou at gmail.com> wrote:
>>> [snip]
>>>
>>> Furthermore, the majority of currently existing extension modules has
>>> problems with sub-interpreter support and/or interpreter reloading, and,
>>> while
>>> it is possible with the current infrastructure to support these
>>> features, it is neither easy nor efficient.
>>> Addressing these issues was the goal of PEP 3121, but many extensions,
>>> including some in the standard library, took the least-effort approach
>>> to porting to Python 3, leaving these issues unresolved.
>>> This PEP keeps backwards compatibility, which should reduce pressure and
>>> give
>>> extension authors adequate time to consider these issues when porting.
>>
>> So just be to sure I understand, now PyModuleDef.m_slots will
>> unambiguously indicate whether or not an extension module is
>> compliant, right?
> 
> I'm not sure what you mean by "compliant". A non-NULL m_slots will
> indicate usage of multi-phase initialisation, so it at least indicates
> *intent* to correctly support subinterpreters et al. Actual delivery
> on that promise is still a different question :)

Yes, non-NULL m_slots means the module is compliant. If it's not, it's a
bug in the *module* (i.e. compliance is not *just* a matter of setting
setting m_slots).
This will be explained in the docs.

>>> [snip]
>>>
>>> The proposal
>>> ============
>>
>> This section should include an indication of how the loader (and
>> perhaps finder) will change for builtin, frozen, and extension
>> modules.  It may help to describe the proposal up front by how the
>> loader implementation would look if it were somehow implemented in
>> Python code.  The subsequent sections sometimes indicate where
>> different things take place, but an explicit outline (as Python code)
>> would make the entire flow really obvious.  Putting that toward the
>> beginning of this section would help clearly set the stage for the
>> rest of the proposal.
> 
> +1 for a pseudo-code overview of the loader implementation.

OK. Along with a link to PEP 451 code [*], it should make things clearer.
[*] https://www.python.org/dev/peps/pep-0451/#how-loading-will-work

>>> [snip]
>>> Unknown slot IDs will cause the import to fail with SystemError.
>>
>> Was there any consideration made for just ignoring unknown slot IDs?
>> My gut reaction is that you have it the right way, but I can still
>> imagine use cases for custom slots that PyModuleDef_Init wouldn't know
>> about.
> 
> The "known slots only, all other slot IDs are reserved for future use"
> slot semantics were copied directly from PyType_FromSpec in PEP 384.
> Since it's just a numeric slot ID, you'd run a high risk of conflicts
> if you allowed for custom extensions.
> 
> If folks want to do more clever things, they'll need to use the create
> or exec slot to stash them on the module object, rather than storing
> them in the module definition.

Right, if you need custom behavior, put it in a function and use the
provided hook. (If you need custom "slots" on PyModuleDef for some
reason, use a PyModuleDef subclass -- but I can't see where it would be
helpful.)
Ignoring unknown slot IDs would mean letting errors go unnoticed.

(Technicality: PyModuleDef_Init doesn't care about slots;
PyModule_FromDefAndSpec and PyModule_ExecDef do. and they will raise the
errors.)

>> When using multi-phase initialization, the *m_name* field of PyModuleDef
>> will
>> not be used during importing; the module name will be taken from the
>> ModuleSpec.
> 
> So m_name will be strictly ignored by PyModuleDef_Init?

Yes. The name is useful for introspection, but the import machinery will
use the name provided by the ModuleSpec.

(Technicality: again, PyModuleDef_Init doesn't touch names at all.
PyModule_FromDefAndSpec and PyModule_ExecDef do, and they will ignore
the name from the def.)

>>> The PyModuleDef object must be available for the lifetime of the module
>>> created
>>> from it – usually, it will be declared statically.
>>
>> How easily will this be a source of mysterious errors-at-a-distance?
> 
> It shouldn't be any worse than static type definitions, and normal
> reference counting semantics should keep it alive regardless.

It's the the same as the current behavior (PEP 3121), where a
PyModuleDef is stored in the module, and if you let it die,
PyModule_GetState will give you an invalid pointer. It's just that in
PEP 489, the import machinery itself uses def, so you actually get to
feel the pain if you deallocate it.
All in all, this should not be a problem in practice; the PEP specifies
what'll happen if you go off doing exotic things. (For example, Cython
might run into this if it tries implementing a reloading scheme we
talked about earlier in the thread, and even then it shouldn't be a
major source of mysterious errors.) Normal mortals will be OK.

>> [snip]
>> However, only ModuleType instances support module-specific functionality
>> such as per-module state.
> 
> This is a pretty important point.  Presumably this constraints later
> behavior and precedes all functionality related to per-module state.

Yes. Module objects support more module-like behavior than other
objects. What you can and cannot use should be clear from the API. I'll
clarify a bit more what functionality depends on using a PyModule_Type
(or subclass) instance.
One thing I see I forgot to add is that execution slots are looked up
via PyModule_GetDef, so they won't be processed on non-module objects.

It's a very good idea to use a module subclass rather than a completely
custom object. The docs will need to strongly recommend this.

>>> [snip]
>>> Extension authors are advised to keep Py_mod_create minimal, an in
>>> particular
>>> to not call user code from it.
>>
>> This is a pretty important point as well.  We'll need to make sure
>> this is sufficiently clear in the documentation.  Would it make sense
>> to provide helpers for common cases, to encourage extension authors to
>> keep the create function minimal?
> 
> The main encouragement is to not handcode your extension modules at
> all, and let something like Cython or SWIG take care of the
> boilerplate :)

Yes, Cython should be default. For hand-written modules, the common case
should be not defining create at all.

>>> [snip]
>>>
>>> If PyModuleExec replaces the module's entry in sys.modules,
>>> the new object will be used and returned by importlib machinery.
>>
>> Just to be sure, something like "mod = sys.modules[modname]" is done
>> before each execution slot.  In other words, the result of the
>> previous execution slot should be used for the next one.
> 
> That's not the original intent of this paragraph - rather, it is
> referring to the existing behaviour of the import machinery.
> 
> However, I agree that now we're allowing the Py_mod_exec slot to be
> supplied multiple times, we should also be updating the module
> reference between slot invocations.

No, that won't work. It's possible (via direct calls to the import
machinery) to load a module without adding it to sys.modules.
The behavior should be clear (when you think about it) after I include
the loader implementation pseudocode.

> I also think the PEP could do with a brief mention of the additional
> modularity this approach brings at the C level - rather than having to
> jam everything into one function, an extension module can easily break
> up its initialisation into multiple steps, and its technically even
> possible to share common steps between different modules.

Eh, I think it's better to create one function that calls the parts,
which was always possible, and works just as well.
Repeating slots is allowed because it would be an unnecessary bother to
check for duplicates. It's not a feature to advertise, the PEP just
specifies that in the weird edge case, the intuitive thing will happen.

(I did have a useful future use case for repeated slots, but the current
PEP allows a better and more obvious solution so I'll not even mention
it again.)

Still, the steps are processed in a loop from a single function
(PyModule_ExecDef), and that function operates on a module object -- it
doesn't know about sys.modules and can't easily check if you replaced
the module somewhere.

>>> (This mirrors the behavior of Python modules. Note that implementing
>>> Py_mod_create is usually a better solution for the use cases this serves.)
>>
>> Could you elaborate?  What are those use cases and why would
>> Py_mod_create be better?
> 
> Rather than replacing the implicitly created normal module during
> Py_mod_exec (which is the only option available to Python modules),
> PEP 489 lets you define the Py_mod_create slot to override the module
> object creation directly.
> 
> Outside conversion of a Python module that manipulates sys.modules to
> an extension module with Cython, there's no real reason to use the
> "replacing yourself in sys.modules" option over using Py_mod_create
> directly.

Yes. The workaround you need to use in Python modules is possible for
extensions, but there's no reason to use it. I'll try to make it clearer
that it's an unnecessary workaround.

>>> [snip]
>>>
>>> Modules that need to work unchanged on older versions of Python should not
>>> use multi-phase initialization, because the benefits it brings can't be
>>> back-ported.
>>
>> Given your example below, "should not" seems a bit strong to me.  In
>> fact, what are the objections to encouraging the approach from the
>> example?
> 
> Agreed, "should not" is probably too strong here. On the other hand,
> preserving compatibility with older Python versions in a module that
> has been updated to rely on multi-phase initialization is likely to be
> a matter of "graceful degradation", rather than being able to
> reproduce comparable functionality (which I believe may have been the
> point Petr was trying to convey).

My point is that if you need graceful degradation, your best bet is to
stick with single-phase init. Then you'll have one code path that works
the same on all versions.
If you *need* the features of multi-phase init, you need to remove
support for Pythons that don't have it.
If you need both backwards compatibility and multi-phase init, you
essentially need to create two modules (with shared contents), and make
sure they end up in the same state after they're loaded.

> I expect Cython and SWIG may be able to manage that through
> appropriate use of #ifdef's in the generated code, but doing it by
> hand is likely to be painful, hence the potential benefits of just
> sticking with single-phase initialisation for the time being.

Yes, code generators are in a position to create two versions of the
module, and select one using using #ifdef.

The example in the PEP is helpful for other reasons than encouraging
#ifdef: it shows what needs to change when porting. Think of it as a diff :)

>>> [snip]
>>>
>>> Subinterpreters and Interpreter Reloading
>>> -----------------------------------------
>>>
>>> Extensions using the new initialization scheme are expected to support
>>> subinterpreters and multiple Py_Initialize/Py_Finalize cycles correctly.
>>
>> Presumably this support is explicitly and completely defined in the
>> subsequent sentences.  Is it really just keeping "hidden" module state
>> encapsulated on the module object?  If not then it may make sense to
>> enumerate the requirements better for the sake of extension module
>> authors.

It is explained in the docs, see "Bugs and caveats" here:
https://docs.python.org/3/c-api/init.html#sub-interpreter-support
I'll add a link to that page.

> I'd actually like to have a better way of doing scenario testing for
> extension modules (subinterpreters, multiple initialize/finalize
> cycles, freezing), but I'm not sure this PEP is the best place to
> define that. Perhaps we could do a PyPI project that was a tox-based
> test battery for this kind of thing?

I think that's the wrong place to start. Currently, sub-interpreter
support is buried away in a docs chapter about Python
initialization/finalization, so a typical extension author won't even
notice it. We need to first make it *possible* to support
subinterpreters easily and correctly (so that Cython can do it), and to
document it prominently in the "writing extensions" part of the docs,
not only in "extending Python". Then,
This PEP does part of the first step, and the docs for it (which aren't
written yet) will do the second step.
After that, it could make sense to provide a tool for testing this.

>>> The mechanism is designed to make this easy, but care is still required
>>> on the part of the extension author.
>>> No user-defined functions, methods, or instances may leak to different
>>> interpreters.
>>> To achieve this, all module-level state should be kept in either the module
>>> dict, or in the module object's storage reachable by PyModule_GetState.
>>
>> Is this programmatically enforceable?

No. (I believe you could even prove this formally.)

>> Is there any mechanism for easily copying module state?

No. This would be impossible to provide in the general case. It's the
responsibility of your C code.
That said, if you need to copy module state, chances are your design
could use some rethinking.

>> How about sharing some state between subinterpreters? 

The PyCapsule API was designed for this.

>> How much room is there for letting extension module
>> authors define how their module behaves across multiple interpreters
>> or across multiple Initialize/Finalize cycles?

Technically, you have all the freedom you want. But if I embed Python
into my project/library, I'd want multiple sub-interpreters completely
isolated by default. If I use two libraries that each embed Python into
my app, I definitely want them isolated.
So the PEP tries to make it easy to keep multiple interpreters isolated.

> It's not programmatically enforcable, hence the idea above of finding
> a way to make it easier for people to test their extension modules are
> importable across multiple Python versions and deployment scenarios.
> 
>>> As a rule of thumb, modules that rely on PyState_FindModule are, at the
>>> moment,
>>> not good candidates for porting to the new mechanism.
>>
>> Are there any plans for a follow-up effort to help with this case?

See the link in the PEP. for initial discussion.

> The problem here is that the PEP 3121 module state approach provides
> storage on a *per-interpreter* basis, that is then shared amongst all
> module instances created from a given module definition.
> 
> This means that when _PyImport_FindExtensionObject (see
> https://hg.python.org/cpython/file/fc2eed9fc2d0/Python/import.c#l518)
> reinitialises an extension module, the state is shared between the two
> instances. When PEP 3121 was written, this was not seen as a problem,
> since the expectation was that the behaviour would only be triggered
> by multiple interpreter level initialize/finalize cycles.
> 
> One key scenario we missed at the time was "deleting an extension
> module from sys.modules and importing it a second time, while
> retaining a local reference for later restoration". Under PEP 3121,
> the two instances collide on their state storage, as we have two
> simultaneously existing module objects created in the same interpreter
> from the same module definition. PEP 489 would inherit that same
> problem if you tried to use it with the PyState_* APIs, so it simply
> doesn't allow them at all. (Earlier versions of the PEP allowed it
> with an "EXPORT_SINGLETON" slot that would disallow reimporting
> entirely, which we took out in favour of "just keep using the existing
> initialisation model in those cases for the time being")
> 
> For pure Python code, we don't have this problem, since the
> interpreter takes care of providing a properly scoped globals()
> reference to *all* functions defined in that module, regardless of
> whether they're module level functions or method definitions on a
> class. At the C level, we don't have that, as only module level
> functions get a module reference passed in - methods only get a
> reference to their class instance, without a reference to the module
> globals, and delayed callbacks can be a problem as well.
> 
> The best improved API we could likely offer at this point is a
> convenience API for looking up a module in *sys.modules* based on a
> PyModuleDef instance, and updating PEP 489 to write the as-imported
> module name into the returned PyModuleDef structure. That's probably
> not a bad way to go, given that PEP 489 currently *ignores* the m_name
> slot - flipping it around to be a *writable* slot would be a way to
> let extension modules know dynamically how to look themselves up in
> sys.modules.
> 
> The new lookup API would then be the moral equivalent of Python code
> doing "mod = sys.modules[__name__]". With this approach, actively
> *using* multiple references to a given module at the same time would
> still break (since you'll always get the module currently in
> sys.modules, even if that isn't the one you expected), but the
> "save-and-restore" model needed for certain kinds of testing and
> potentially other scenarios would work correctly.

I still think providing the module to classes is a better idea than a
lookup API, but that's going out of scope here.

>>> Module Reloading
>>> ----------------
>>>
>>> Reloading an extension module using importlib.reload() will continue to
>>> have no effect, except re-setting import-related attributes.
>>>
>>> Due to limitations in shared library loading (both dlopen on POSIX and
>>> LoadModuleEx on Windows), it is not generally possible to load
>>> a modified library after it has changed on disk.
>>>
>>> Use cases for reloading other than trying out a new version of the module
>>> are too rare to require all module authors to keep reloading in mind.
>>> If reload-like functionality is needed, authors can export a dedicated
>>> function for it.
>>
>> Keep in mind the semantics of reload for pure Python modules.  The
>> module is executed into the existing namespace, overwriting the loaded
>> namespace but leaving non-colliding attributes alone.  While the
>> semantics for reloading an extension/builtin/frozen module are
>> currently basic (i.e. a no-op), there may well be room to support
>> reload behavior that mirrors that of pure Python modules without
>> needing to reload an SO file.  I would expect either the behavior of
>> exec to get repeated (tricky due to "hidden" module state?) or for
>> there to be a "reload" slot that would mirror Py_mod_exec.
> 
> We considered this, and decided it was fairly pointless, since you
> can't modify the extension module code. The one case I see where it
> potentially makes sense is a "transitive reload", where the extension
> module retrieves and caches attributes from another pure Python module
> at import time, and that extension module has been reloaded.
> 
> It may also make a difference in the context of utilities like
> https://docs.python.org/3/library/test.html#test.support.import_fresh_module,
> where we manipulate the import system state to control how conditional
> imports are handled.
> 
>> At the same time, one may argue that reloading modules is not
>> something to encourage. :)
> 
> There's a reason import_fresh_module has never made it out of test.support :)

Right. Implementation-wise, it would actually be much easier to support
reload rather than make it a no-op. But then C module authors would need
to think about this edge case, which might be tricky to get right, would
not be likely to get test coverage, and is generally not useful anyway, .

If it turns out to be useful, it would be very simple to add an explicit
reload slot in the future.

>>> Multiple modules in one library
>>> -------------------------------
>>>
>>> To support multiple Python modules in one shared library, the library can
>>> export additional PyInit* symbols besides the one that corresponds
>>> to the library's filename.
>>>
>>> Note that this mechanism can currently only be used to *load* extra modules,
>>> but not to *find* them.
>>
>> What do you mean by "currently"?
> 
> It's a limitation of the way the existing finders work, rather than an
> inherent limitation of the import system as a whole.
> 
>> It may also be worth tying the above statement with the following
>> text, since the following appears to be an explanation of how to
>> address the "finder" caveat.
> 
> Agreed that this could be clearer.

OK, I'll clarify.


>> Summary of API Changes and Additions
>> ------------------------------------
>>
>> New functions:
>>
>> * PyModule_FromDefAndSpec (macro)
>> * PyModule_FromDefAndSpec2
>> * PyModule_ExecDef
>> * PyModule_SetDocString
>> * PyModule_AddFunctions
>> * PyModuleDef_Init
>>
>> New macros:
>>
>> * Py_mod_create
>> * Py_mod_exec
>>
>> New types:
>>
>> * PyModuleDef_Type will be exposed
>>
>> New structures:
>>
>> * PyModuleDef_Slot
>>
>> PyModuleDef.m_reload changes to PyModuleDef.m_slots.
> 
> This section is missing any explanation of the impact on
> Python/import.c, on the _imp/imp module, and on the 3 finders/loaders
> in Lib/importlib/_bootstrap[_external].py (builtin/frozen/extension).

I'll add a summary.

The internal _imp module will have backwards incompatible changes --
functions will be added and removed as necessary. That's what the
underscore means :)
The deprecated imp module will get a backwards compatibility shim for
anything it imported from _imp that got removed.
importlib will stay backwards compatible.

Python/import.c and Python/importdl.* will be rewritten entirely.
See the patches (linked from the PEP) for details.