[Python-ideas] making a module callable

[Eric Snow]

On Nov 19, 2013 10:52 AM, "Gregory P. Smith" <greg at krypto.org> wrote:
>
> While I don't ordinarily endorse this use case it'd be nice not to require hacks involving sys.modules monkeying such as:
>
> https://github.com/has207/flexmock/pull/89
> specifically:
> https://github.com/has207/flexmock/commit/bd47fa8189c7dff349de257c0e061b9fcea2330d
>
> to make a module callable.
>
> This obviously touches on the larger ideas of what is a module vs a class and why are they different given that they're "just" namespaces.

What's the use case for a callable module?  In the flexmock example,
is it just so they can do an import instead of a from..import?  As
Georg said, modules are just top-level namespaces, API containers.
Importing the callable you want out of a module is easy.

However, the underlying idea is something that has come up before and
may be worth more consideration.

tl;dr: __metamodule__ (pre-bikeshedding) would be a good way to go,
but isn't worth it and may be an attractive nuisance.

If we are going to support customization of module classes, I'd rather
we do it via a general API (e.g. Chris's __metamodule__) than
piecemeal (via special-casing __call__, etc.).  However, you can
already use a custom module type in the two ways that Nick mentioned,
the first of which flexmock is doing (and Django does IIRC).

Sticking something into sys.modules to replace the currently executing
module is indeed a hack.  The import system accommodates this not by
design (unless someone is willing to come forward and admit guilt
<wink>) but mostly as an incidental implementation artifact of the
import machinery from many releases ago. [1]

As Nick mentioned, PEP 451 introduces an optional create_module()
method on loaders that returns the module object to use during
loading.  This is nice if you are already writing a loader.  Otherwise
it's a pain (the import hook machinery isn't exactly simple) and
usually won't be worth your time.  Furthermore, your loader will
probably be applied to multiple modules (which may be what you need).
It certainly isn't a one-off, add-something-to-the-affected-module
sort of thing.  Basically, having to write a loader and plug it in is
like (only more complicated) having to use a metaclass just to
implement a __prepare_class__() that returns an OrderedDict, all so
you can have an ordered class namespace.  Loader.create_module() is a
good addition, but is too low level to use as a replacement for the
sys.modules hack.

In contrast, something like __metamodule__ would be an effective
replacement.  It would be similar in spirit and in syntax to
__init_class__ in PEP 422 (and __metaclass__ in Python 2), defined at
the top of the module and used for the module.  The thing that appeals
to me is that we could deprecate the sys.modules hack. :)

The big question is, is having a custom module type a common enough
need? To me the desire for it usually implies a misunderstanding of
the purpose of modules. If we had an easier API would it be an
attractive nuisance?  Unless it's a big win I don't think it's a good
idea, and I'm not convinced it's common enough a need.

-eric

[1] A module replacing itself in sys.modules came up during the
importlib bootstrap integration, where it required adding yet another
special-case backward-compatibility pain point to the importlib
implementation.  I can't find the actual email, but I refer to what
happened in http://bugs.python.org/msg166630, note "[3]".  It
certainly surprised us that you could do it and that people actually
were.  At this point I guess the latter shouldn't have been
surprising. :)