[Python-Dev] PEP 451 update

[Nick Coghlan]

On 30 October 2013 09:02, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> On Tue, Oct 29, 2013 at 3:32 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On 29 Oct 2013 14:45, "Eric Snow" <ericsnowcurrently at gmail.com> wrote:
>>> I'd rather give loaders another optional method:
>>> supports_reload(name).  Complicating the spec methods signatures (to
>>> support passing the old spec through) just for the sake of reload
>>> seems less than optimal.  I don't see any benefit to exposing the old
>>> spec to loader.exec_module().
>>
>> I do: it lets the loader tell if anything changed from the previous load
>> operation, allowing it to make an informed decision on whether or not to
>> permit the reload operation or throw an exception.
>>
>> However, I accept such a check would be better implemented as a separate
>> yes/no method, so it makes sense to postpone this aspect to 3.5 at the
>> earliest.
>
> With the PEP, loaders are no longer in charge of the module-related
> boilerplate.  This means that by the time exec_module() gets called,
> they won't be able to determine for themselves if it's a reload by
> checking sys.modules.  That's the point of adding
> loader.supports_reload().  (I haven't added that rationale to the PEP
> yet).

Ah, good point.

> If it makes a difference to loaders to also look at the previous spec
> during supports_reload(), then I think we should add that parameter
> now.  Otherwise if we add the parameter later it makes it messier.
> The signature would now be:
>
>   Loader.supports_reload(name, old_spec)

OK, time for me to stop trying to remember the details of the problem
I'm trying to solve and go look them up in the source code :)

One of my goals here is to be able to migrate extension loading from
the old API to the new plugin API. That means being able to break up
the existing load_module implementation:

    http://hg.python.org/cpython/file/1787277915e9/Python/importdl.c#l23

For loading, that's a fairly straightforward create_module/exec_module
split, but reloading gets a bit more interesting.

Specifically, I'd like to be able to implement the relevant parts of
_PyImport_FindExtensionObject as a precheck for reloading:

    http://hg.python.org/cpython/file/1787277915e9/Python/import.c#l533

That means just having access to the module name isn't enough: the
extensions dictionary is keyed by a (name, filename) 2-tuple rather
than just by the module name. Using the find_spec API, that filename
would be stored in the loader state on the spec object rather than
being looked up anew during the load operation.

However, rereading this method also indicates that I really want to
know *in exec_module* whether this is a reload or not, since extension
loading needs to handle reloads differently from initial execution.

So I'm back to my original preference: I'd like the previous spec to
be passed to exec_module in the reloading case. If reloading is not
supported at all, it's up to the loader to throw an appropriate
exception when the the previous spec is not None. If loading and
reloading doesn't make any difference, then they can just ignore it.
But when both are supported, but handled differently (as for extension
modules), then that can be detected, and the information from the old
spec (including the original loader and loader state) is available if
needed.

Cheers,
Nick.


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia