[Python-ideas] metamodules (was: Re: Idea to support lazy loaded names.)

[Nathaniel Smith]

On 9 Oct 2014 13:14, "Petr Viktorin" <encukou at gmail.com> wrote:
>
> On Thu, Oct 9, 2014 at 5:24 AM, Nathaniel Smith <njs at pobox.com> wrote:
> [...]
> >> 1) Analogy with __new__: For packages only, if there's a __new__.py,
that gets executed first. If it "returns" (not sure how that was defined)
an instance of a subclass of ModuleType, that instance is used to run
__init__.py instead of a normal module instance.
> >
> > This is very similar to the current approach of having __init__.py
> > reassign sys.modules[__name__]. The advantages are:
> > - It gives a way to enforce the rule that you have to do this
> > assignment as the very first thing inside your module, before allowing
> > arbitrary code to run (e.g. by importing other modules which might
> > recursively import your module in turn, and access
> > sys.modules[__name__] before you've modified it).
> > - Because __new__.py would run *before* the It avoids the headache of
> > having to juggle two module objects, one of whose __dict__'s is
> > already being used as the execution environment for the code that is
> > trying to do the switcheroo.
> >
> > But:
> > - It's a pretty complicated way to accomplish the stated goals.
>
> As in, a non-obvious way to do a non-obvious thing, from the user's
> point of view?
> On the implementation side it doesn't strike me as particularly
> complicated, am I wrong?

Mostly I mean that it requires a much heavier-weight design discussion. It
would add qualitatively new concepts to the language, require its own PEP,
we'd have to have the debate about whether the cost of extra stat calls on
every import was worth it, we'd have to figure out all the fiddly details
about how exactly it should work (are variables defined in __new__.py
visible in __init__.py?), etc.

Usually one requirement in such debates is to demonstrate that there is no
acceptable lighter-weight alternative. I think that's a hard argument to
make here. The __class__ assignment approach requires minimal changes to
python itself, and works pretty much just as well as the __new__.py
approach; maybe better in some ways (as per below). So it's hard to justify
the effort it would require to get consensus.

> > - The restriction to packages is unfortunate.
>
> True, but it seems to me that you'd usually want it for the
> __init__.py – after all if you need to do `from mypackage import
> submodule` anyway, and submodule isn't a package itself, you can
> usually can just make `submodule` a class directly. Or have the
> top-level package define an import hook.
> Seems to me that this would really only hurt single-file top-level
> modules, but those are easily converted to a package.

Yeah, this is not a showstopper, but given that we have an alternative that
*doesn't* have such fiddly restrictions, it's worth mentioning.

> > - The backcompat story is terrible -- faking __new__.py support in old
> > versions of python would be really difficult, and the main reason I
> > care about this stuff in the first place is because I want to be able
> > to e.g. deprecate module attributes that are in the public API of old,
> > widely-used packages. It will be many years before such packages can
> > require 3.5.
>
> That seems like a terrible reason to me – if it should work nicely on
> older Pythons, it means a 3rd party module would be enough. Why bother
> adding it to the language?

I posted an implementation that works fine on older pythons. If you look at
the code though then I think you will agree that the way it works is OK for
a backcompat shim that only has to target a specific range of python
versions, but... it is also something that we should feel embarrassed to
recommend as a long-term solution.

> Valid reasons would be to make it easier for alternative interpreters,
> or to catch edge cases (e.g. make it work nicely with custom
> importers, zip files, etc). But at that point, we can just do it the
> "right way"*, have the backcompat story to a third-party shim.
>
> * i.e. "not paying attention to older Pythons" – I'm not saying
> __new__.py is necessarily the right way, I'm criticizing this reason
> against it

My point is just that it matters whether a backcompat shim is doable.
Probably there is some way to implement a backcompat shim for __new__.py,
but I don't immediately know how to do it. (E.g., how do you read a file
that's a sibling of the current source file in a compatible way across
python versions, taking into account zipimport etc.? Is it possible to get
the right namespace semantics when executing __new__.py *after* the package
module object has been created? Of course it depends on what those
semantics are, which is currently underspecified....)

And again, this is to be compared to the __class__ assignment approach,
where we know very well that a backcompat shim is doable because it is done
:-).

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20141009/8d7517f2/attachment-0001.html>