[Python-ideas] Implicit submodule imports

[Nathaniel Smith]

On Sat, Sep 27, 2014 at 1:33 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> Or perhaps these special "modules" could subclass ModuleType and somehow
> get reloading to work correctly. In 2.7 at least you can manually copy a
> module to a module subclass, install it into sys.modules, and reload
> will accept it. Not only that, but after reloading it still uses the
> same subclass.
>
> Unfortunately, when I tried it in 3.3, imp.reload complained about my
> custom module subclass not being a module, so it seems that 3.3 at least
> is more restrictive than 2.7. (Perhaps 3.3 reload does a "type(obj) is
> ModuleType" instead of isinstance test?)

Yeah, it looks like 3.3 does an explicit 'type(obj) is ModuleType'
check, but is the only version that works like this -- earlier and
later versions both use isinstance.

> Nevertheless, I got this proof of concept more-or-less working in 2.7
> and 3.3:
>
> import sys
> from types import ModuleType
>
> class MagicModule(ModuleType):
>     def __getattr__(self, name):
>         if name == "spam":
>             return "Spam spam spam!"
>         raise AttributeError
>
> eggs = 23
>
> _tmp = MagicModule(__name__)
> _tmp.__dict__.update(sys.modules[__name__].__dict__)
> sys.modules[__name__] = _tmp
> del _tmp

This approach won't work well for packages -- imagine that instead of
'eggs = 23', the body of the file imports a bunch of submodules. If
those submodules then import the top-level package in turn, then
they'll end up with the original module object and namespace, not the
modified one.

One could move the sys.modules assignment up to the top of the file,
but you can't move the __dict__.update call up to the top of the file,
because you can't copy the old namespace until after it's finished
being initialized. OTOH leaving the __dict__.update at the bottom of
the file is pretty risky too, because then any submodule that imports
the top-level package will see a weird inconsistent view of it until
after the import has finished.

 The solution is, instead of having two dicts and updating one to
match the other, simply point the new module directly at the existing
namespace dict, so they always stay in sync:

  _tmp = MagicModule(__name__)
  _tmp.__dict__ = sys.modules[__name__].__dict__

...except this gives an error because module objects disallow
assignment to __dict__.

Sooooo you're kinda doomed no matter what you do.

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org