[Import-SIG] PEP 489: Redesigning extension module loading

[Petr Viktorin]

On 03/21/2015 02:55 PM, Nick Coghlan wrote:
> On 21 March 2015 at 22:30, Petr Viktorin <encukou at gmail.com> wrote:
>> On 03/21/2015 12:37 PM, Nick Coghlan wrote:
>>> Earlier runpy based implementations of -m broke that by running the
>>> code in a separate namespace rather than in the actual builtin
>>> __main__ module, while later implementations fixed it by using the
>>> real __main__ to run the code.
>>
>>
>> I see. Thanks for the write-up.
>> It seems that if runpy knew the ModuleDef for __main__ (and the ModuleDef
>> didn't define a custom Create slot), it could look at m_size and allocate
>> md_state appropriately. Then the module object itself wouldn't change, but
>> the C storage would be available when Exec is called.
>
> Ah, true, I hadn't thought about that - I'd only thought about this in
> the context of the earlier completely custom module creation design.
> Restricting things to PEP 3121 module state helps a great deal.
>
>> Does anything that would prevent this come to mind?
>
> It would likely require an additional low level API helper somewhere,
> as the only way I can see it working without completely breaking the
> importlib abstraction is for runpy to call create_module on the
> loader, get the custom module back, and then have a way to say "make
> __main__ look like this", which will either succeed (in which case
> runpy proceeds to calling exec_module on the real __main__ instead of
> the module returned from create_module), or it fails (in which case
> runpy bails out explaining that the requested module can't be run as
> the main module)

If extensions export a list of PyModuleDef, and there's a way to get at 
the defs, the create_module step can be skipped. But a special low-level 
helper is needed either way.

> runpy still needs to be updated for PEP 451 in general though :(

Yes. Given all the special cases, is's starting to look like this will 
need to happen when this PEP is implemented, to make sure things really 
work.