[Python-Dev] advice needed: best approach to enabling "metamodules"?

[Nathaniel Smith]

On Sun, Nov 30, 2014 at 8:54 PM, Guido van Rossum <guido at python.org> wrote:
> On Sun, Nov 30, 2014 at 11:29 AM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> On Sun, Nov 30, 2014 at 2:54 AM, Guido van Rossum <guido at python.org>
>> wrote:
>> > All the use cases seem to be about adding some kind of getattr hook to
>> > modules. They all seem to involve modifying the CPython C code anyway.
>> > So
>> > why not tackle that problem head-on and modify module_getattro() to look
>> > for
>> > a global named __getattr__ and if it exists, call that instead of
>> > raising
>> > AttributeError?
>>
>> You need to allow overriding __dir__ as well for tab-completion, and
>> some people wanted to use the properties API instead of raw
>> __getattr__, etc. Maybe someone will want __getattribute__ semantics,
>> I dunno.
>
> Hm... I agree about __dir__ but the other things feel too speculative.
>
>> So since we're *so close* to being able to just use the
>> subclassing machinery, it seemed cleaner to try and get that working
>> instead of reimplementing bits of it piecewise.
>
> That would really be option 1, right? It's the one that looks cleanest from
> the user's POV (or at least from the POV of a developer who wants to build a
> framework using this feature -- for a simple one-off use case, __getattr__
> sounds pretty attractive). I think that if we really want option 1, the
> issue of PyModuleType not being a heap type can be dealt with.

Options 1-4 all have the effect of making it fairly simple to slot an
arbitrary user-defined module subclass into sys.modules. Option 1 is
the cleanest API though :-).

>>
>> That said, __getattr__ + __dir__ would be enough for my immediate use
>> cases.
>
>
>  Perhaps it would be a good exercise to try and write the "lazy submodule
> import"(*) use case three ways: (a) using only CPython 3.4; (b) using
> __class__ assignment; (c) using customizable __getattr__ and __dir__. I
> think we can learn a lot about the alternatives from this exercise. I
> presume there's already a version of (a) floating around, but if it's been
> used in practice at all, it's probably too gnarly to serve as a useful
> comparison (though its essence may be extracted to serve as such).

(b) and (c) are very straightforward and trivial. Probably I could do
a better job of faking dir()'s default behaviour on modules, but
basically:

##### __class__ assignment__ #####

import sys, types, importlib

class MyModule(types.ModuleType):
    def __getattr__(self, name):
        if name in _lazy_submodules:
            # implicitly assigns submodule to self.__dict__[name]
            return importlib.import_module(name, package=self.__package__)

    def __dir__(self):
        entries = set(self.__dict__)
        entries.update(__lazy_submodules__)
        return sorted(entries)

sys.modules[__name__].__class__ = MyModule
_lazy_submodules = {"foo", "bar"}

##### customizable __getattr__ and __dir__ #####

import importlib

def __getattr__(name):
    if name in _lazy_submodules:
        # implicitly assigns submodule to globals()[name]
        return importlib.import_module(name, package=self.__package__)

def __dir__():
    entries = set(globals())
    entries.update(__lazy_submodules__)
    return sorted(entries)

_lazy_submodules = {"foo", "bar"}

> FWIW I believe all proposals here have a big limitation: the module *itself*
> cannot benefit much from all these shenanigans, because references to
> globals from within the module's own code are just dictionary accesses, and
> we don't want to change that.

I think that's fine -- IMHO the main uses cases here are about
controlling the public API. And a module that really wants to can
always import itself if it wants to pull more shenanigans :-) (i.e.,
foo/__init__.py can do "import foo; foo.blahblah" instead of just
"blahblah".)

-n

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org