[Python-Dev] PEP 451 update

[PJ Eby]

On Fri, Oct 25, 2013 at 3:15 PM, Brett Cannon <brett at python.org> wrote:
> On Fri, Oct 25, 2013 at 2:10 PM, PJ Eby <pje at telecommunity.com> wrote:
>> On Fri, Oct 25, 2013 at 1:15 PM, Brett Cannon <brett at python.org> wrote:
>> > On Fri, Oct 25, 2013 at 12:24 PM, PJ Eby <pje at telecommunity.com> wrote:
>> >> At least through all 2.x, reload() just uses module.__name__ to
>> >> restart the module find-and-load process, and does not assume that
>> >> __loader__ is valid in advance.
>> >
>> >
>> > That doesn't make much sense in a post-importlib world where import
>> > makes
>> > sure that __loader__ is set (which it has since Python 3.3). Otherwise
>> > you
>> > are asking for not just a reload but a re-find as well.
>>
>> That's a feature, not a bug.  A reload() after changing sys.path
>> *should* take into account the change, not to mention any changes to
>> meta_path, path hooks, etc.  (And it's how reload() worked before
>> importlib.)
>
>
> Fair enough, but in my mind that obviously doesn't really click for what I
> view as a reload in an importlib world where secret import code no longer
> exists. When I think re-load I think "load again", not "find the module
> again and execute a load with a possibly new loader".

Sure, and the reference manual is rather vague on this point.
However, I would guess that at least some web frameworks with
automatic reload support are going to barf on this change in at least
some edge cases.

(OTOH, it's unlikely the bugs will ever be reported, because the
problem will mysteriously go away once the process is restarted,
probably never to occur again.)

Mostly, this just seems like an ugly wart -- Python should be dynamic
by default, and that includes reloading.  While the import machinery
has lots of ugly caching under the hood, a user-level function like
reload() should not require you to do the equivalent of saying, "no,
really...  I want you to *really* reload, not just pull in whatever
exists where you found it last time, while ignoring whether I switched
from module to package or vice versa, or just fixed my sys.path so I
can load the right version of the module."

It is a really tiny thing in the overall scheme of things, because
reload() is not used all that often, but it's still a thing.  If this
isn't treated as a bug, then the docs for reload() at least need to
include a forward-supported workaround so you can say "no, really...
*really* reload" in an approved fashion.

(ISTM that any production code out there that currently uses reload()
would want to perform the "really reload" incantation in order to
avoid the edge cases, even if they haven't actually run into any of
them yet.)


> And in a PEP 451 world it should be dead-simple to make this work the way
> you want in your own code even if this doesn't go the way you want::
>
>   spec = importlib.find_spec(name)
>   module.__spec__ = spec
>   importlib.reload(module)  # Which in itself is essentially
> init_module_attrs(spec, module); spec.loader.exec_module(module)
>
> Heck, you can do this in Python 3.3 right now::
>
>   loader = importlib.find_loader(name)
>   module = sys.modules[name]
>   module.__loader__ = loader
>   importlib.reload(module)

And will that later version still work correctly in a PEP 451 world,
or will you have to detect which world you live in before waving this
particular dead chicken?  ;-)


> Ah, okay. That is not explicit in the PEP beyond coming off a total nuisance
> in order to support reloading by the loader, not an explicit finder + loader
> use-case.

Yeah, it actually was to ensure that you could reload a module using a
different loader than the one that originally loaded it, e.g. due to a
change in path hooks, etc.