[Python-Dev] PEP 451 update

[Eric Snow]

On Tue, Oct 29, 2013 at 7:29 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> OK, time for me to stop trying to remember the details of the problem
> I'm trying to solve and go look them up in the source code :)
>
> One of my goals here is to be able to migrate extension loading from
> the old API to the new plugin API. That means being able to break up
> the existing load_module implementation:
>
>     http://hg.python.org/cpython/file/1787277915e9/Python/importdl.c#l23
>
> For loading, that's a fairly straightforward create_module/exec_module
> split, but reloading gets a bit more interesting.
>
> Specifically, I'd like to be able to implement the relevant parts of
> _PyImport_FindExtensionObject as a precheck for reloading:
>
>     http://hg.python.org/cpython/file/1787277915e9/Python/import.c#l533
>
> That means just having access to the module name isn't enough: the
> extensions dictionary is keyed by a (name, filename) 2-tuple rather
> than just by the module name. Using the find_spec API, that filename
> would be stored in the loader state on the spec object rather than
> being looked up anew during the load operation.
>
> However, rereading this method also indicates that I really want to
> know *in exec_module* whether this is a reload or not, since extension
> loading needs to handle reloads differently from initial execution.
>
> So I'm back to my original preference: I'd like the previous spec to
> be passed to exec_module in the reloading case. If reloading is not
> supported at all, it's up to the loader to throw an appropriate
> exception when the the previous spec is not None. If loading and
> reloading doesn't make any difference, then they can just ignore it.
> But when both are supported, but handled differently (as for extension
> modules), then that can be detected, and the information from the old
> spec (including the original loader and loader state) is available if
> needed.

Our recent discovery about reloading should probably be reflected in
the signature of finder.find_spec():

  MetaPathFinder.find_spec(name, path=None, existing=None)
  PathEntryFinder.find_spec(name, existing=None)

This way the finder has an opportunity to incorporate information from
an existing spec into the spec it returns.  reload() would make use of
this by passing module.__spec__ (or None if the module has no
__spec__) to _bootstrap._find_spec().

This approach should also address what you are looking for.  I'd
prefer it over passing the existing spec to exec_module().  The module
(and its __spec__) should have everything exec_module() needs to do
its job.

We would still need to use loader.supports_reload() in reload().
However, it may make more sense to pass in the module-to-be-reloaded
(after updating its __spec__ to the one found by
_bootstrap._find_spec()).

-eric