[Python-Dev] Pre-PEP: Redesigning extension modules

[Eric Snow]

On Sat, Aug 24, 2013 at 10:12 PM, PJ Eby <pje at telecommunity.com> wrote:

> I haven't had a chance to address this on the import-sig discussion
> yet about ModuleSpec, but I would like to just mention that one
> property of the existing module system that I'm not sure either this
> proposal or the ModuleSpec proposal preserves is that it's possible to
> implement lazy importing of modules using standard reload() semantics.
>
> My "Importing" package offers lazy imports by creating module objects
> in sys.modules that are a subtype of ModuleType, and use a
> __getattribute__ hook so that trying to use them fires off a reload()
> of the module.  Because the dummy module doesn't have __file__ or
> anything else initialized, the import system searches for the module
> and then loads it, reusing the existing module object, even though
> it's actually only executing the module code for the first time.
>
> That the existing object be reused is important, because once the
> dummy is in sys.modules, it can also be imported by other modules, so
> references to it can abound everywhere, and we wish only for it to be
> loaded lazily, without needing to trace down and replace all instances
> of it.  This also preserves other invariants of the module system.
>
> Anyway, the reason I was asking why reloading is being handled as a
> special case in the ModuleSpec proposal -- and the reason I'm curious
> about certain provisions of this proposal -- is that making the
> assumption you can only reload something with the same
> spec/location/etc. it was originally loaded with, and/or that if you
> are reloading a module then you previously had a chance to do things
> to it, doesn't jibe with the way things work currently.


> That is to say, in the pure PEP 302 world, there is no special status
> for "reload" that is different from "load" -- the *only* thing that's
> different is that there is already a module object to use, and there
> is *no guarantee that it's a module object that was initialized by the
> loader now being invoked*.
>

In Python 3.3 (#13959) import.reload()  was updated to reuse a module's
__loader__, which  must now be set.  If __loader__ is not set, you get an
AttributeError.  If that's a problem we can create a tracker issue and
discuss there.

In Python 3.4 imp.reload() is just an alias to importlib.reload(), but it
works basically the same.

With ModuleSpec things won't work that differently.  If you reload such a
module as you described, it will look for __spec__ and call it's reload()
method.  If __spec__ is not set, you get an AttributeError.  It wouldn't be
that hard to build a spec from the module if need be and then use that.

-eric


> AFAICT both this proposal and the ModuleSpec one are making an invalid
> assumption per PEP 302, and aren't explicitly proposing to change the
> status quo: they just assume things that aren't actually assured by
> the prior specs or implementations.
>
> So, for example, this extension module proposal needs to cover what
> happens if an extension module is reloaded and the module object is
> not of the type or instance it's expecting.  Must it do its own
> checking?  Error handling?  Will some other portion of the import
> system be expected to handle it?
>
> For that matter, what happens (in either proposal) if you reload() a
> module which only has a __name__, and no other attributes?  I haven't
> tested with importlib, but with earlier Pythons this results in a
> standard module search being done by reload().  But the ModuleSpec
> proposal and this one seem to assume that a reload()-ed module must
> already be associated with a loader, location, and/or spec.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/ericsnowcurrently%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130905/3cad8555/attachment.html>