[Import-SIG] PEP 489: Redesigning extension module loading; version 4

Petr Viktorin encukou at gmail.com
Thu May 14 14:38:43 CEST 2015


On Thu, May 14, 2015 at 10:48 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 14 May 2015 at 18:10, Petr Viktorin <encukou at gmail.com> wrote:
>> On Wed, May 13, 2015 at 6:04 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> On 14 May 2015 at 00:31, Petr Viktorin <encukou at gmail.com> wrote:
>>>> Or, I could keep the "PyInit_*" hook name, and allow it to return
>>>> PyModuleDef instead of a module. This is obviously a hack, and would
>>>> force me to get back down to the drawing board, but considering the
>>>> options it seems best to explore this option.
>>>> (PyInit_* and PyModuleExport_* signatures are technically compatible,
>>>> since a PyModuleDef is a PyObject)
>>>>
>>>> I'd welcome your thoughts.
>>>
>>> Would it be feasible to go with a model where _PyImport_inittab
>>> continues to be based on the legacy extension module initialisation
>>> system for the time being? That would mean implementing PyInit_* would
>>> remain required rather than optional for 3.5, but lots of folks are
>>> going to want to provide it anyway for compatibility with 3.4 and
>>> earlier.
>>
>> That doesn't really solve the problem, just delays it until we decide
>> that PyInit_* is really optional.
>
> Yeah, I was seeing if you thought a "buy more time to think about it
> further" approach might be viable here. I think you're right that we
> need a better answer up front, though.
>
>> It would mean you couldn't take advantage of the improvements in PEP
>> 489 (create/exec split and ModuleSpec). You'd just write more
>> boilerplate for no benefit (except small stuff like non-ASCII module
>> names).
>>
>> What might be worse, it would mean that modules would have different
>> behavior depending on whether they're frozen or not, which would
>> probably result in subtle bugs you'd only find when creating frozen
>> binaries.
>
> Looking at https://hg.python.org/cpython/file/default/Tools/freeze/makeconfig.py,
> I'm thinking your "out-of-band" option may be a reasonable way to go,
> with a corresponding tweak to the semantics of
> https://docs.python.org/3/c-api/import.html#c._inittab to permit
> (initfunc) to be a pointer to a PyInit_* function OR to a
> PyModuleExport_* function.
>
> We'd then have to determine which was which at runtime when processing
> the inittab internally, by checking whether or not the result of the
> call was a PyModuleDef or not.

That would work, but I don't see much of an advantage over allowing
PyInit_* itself to return either module or PyModuleDef.

> For the inittab generation side, freeze would need to be updated to:
>
> * allow builtin modules to be specifically nominated as "initialised
> modules" or "defined modules"
> * allow the default handling of builtin modules not nominated as one
> or the other to be configured
> * for backwards compatibility, builtin modules would be treated as
> initialised modules by default
>
> If you had a new module that was export only, you'd get a link time
> error looking for the init function that didn't exist if you didn't
> explicitly flag it as a "defined module". Similarly, if you switched
> the default to be defined modules, you'd get a link time error for a
> legacy module that didn't support the new API.
>
> Does that approach sound plausible to you?

I think the "initialized" vs. "exported" distinction is an
implementation detail of the module, and this would expose it too
much.
According to its README, freeze "[parses] the program (and all its
modules) and scans the generated byte code for IMPORT instructions". I
think py2exe does something similar. The end users of such tools would
need to designate which modules use init vs. export.

Allowing PyInit to optionally return PyModuleDef is a bit of a hack,
but it keeps the details isolated between the module and the import
machinery.
PyModuleDef is a PyObject, so the PyInit signature matches. Just the
PyInit name is a bit misleading :(
I think I have a favorite direction now. (Sorry for asking for
directions and then wanting to ignore them! The discussion is
helpful.)


Somewhat related: any thoughts on the legacy init example code [0]?
You asked for an example like this; is it what you had in mind? If you
compile this with a PEP-489 Python with the stable API, the .so can be
used with older Pythons as well.
I now think it's a bit silly: it would be enough to use #ifdef: define
either PyModuleExport or PyInit, depending on the Python version.
This won't do if you're targetting the stable API, but in that case
you can't use any of the new PEP 489 features anyway, so it's enough
to only define PyInit.
Or is there something I missed?


[0] https://www.python.org/dev/peps/pep-0489/#legacy-init


More information about the Import-SIG mailing list