[Python-ideas] lazy import via __future__ or compiler analysis

[C Anthony Risinger]

On Mon, Sep 11, 2017 at 1:09 PM, Neil Schemenauer <
nas-python-ideas at arctrix.com> wrote:

> On 2017-09-11, C Anthony Risinger wrote:
> > I'm not sure I follow the `exec(code, module)` part from the other
> thread.
> > `exec` needs a dict to exec code into [..]
> [..]
> > How do you handle lazy loading when a defined function requests a global
> > via LOAD_NAME? Are you suggesting to change function.__globals__ to
> > something not-a-dict, and/or change LOAD_NAME to bypass
> > function.__globals__ and instead do something like:
>
> I propose to make function.__namespace__ be a module (or other
> namespace object).  function.__globals__ would be a property that
> calls vars(function.__namespace__).
>

Oh interesting, I kinda like that.


> Doing it while perserving backwards compatibility will be a
> challenge.  Doing it without losing performance (LOAD_GLOBAL using
> the fact that f_globals is an honest 'dict') is also hard.  It this
> point, I think there is a chance we can do it.  It is a conceptual
> simplification of Python that gives the language more consistency
> and more power.
>

I do agree it makes module access more uniform if both defined functions
and normal code end up effectively calling getattr(...), instead of
directly reaching into __dict__.


> > All this chatter about modifying opcodes, adding future statements, lazy
> > module opt-in mechanisms, special handling of __init__ or __getattr__ or
> > SOME_CONSTANT suggesting modules-are-almost-a-class-but-not-quite feel
> like
> > an awful lot of work to me, adding even more cognitive load to an already
> > massively complex import system. They seem to make modules even less like
> > other objects or types.
>
> I disagree.  It would make for less cognitive load as LOAD_ATTR
> would be very simlar to LOAD_NAME/LOAD_GLOBAL.  It makes modules
> *more* like other objects and types.
>

I'm not sure about this though. Anything that special cases dunder methods
to sort of look like their counter part on types, eg. __init__ or
__getattr__ or __getattribute__ or whatever else, is a hack to me. The only
way I see to remedy this discrepancy is to make modules a real subclass of
ModuleType, giving them full access to the power of the type system:

```
DottedModuleName(ModuleType, bound_methods=False):
    # something like this:
    #     sys.modules[__class__.__name__] =
__class__._proxy_during_import() ???
    #     ... module code here ...
sys.modules[DottedModuleName.__name__] =
DottedModuleName(DottedModuleName.__name__, DottedModuleName.__doc__)
```

I've done this a few times in the past, and it works even better on python3
(python2 function.__globals__ didn't trigger __missing__ IIRC). I guess all
I'm getting at, is can we find a way to make modules a real type? So dunder
methods are activated? This would make modules phenomenally powerful
instead of just a namespace (or resorting to after the fact __class__
reassignment hacks).


> I'm busy with "real work" this week and so can't follow the
> discussion closely or work on my proof-of-concept prototype.  I hope
> we can come up with an elegant solution and not some special hack
> just to make module properties work.


Agree, and same, but take a look at what I posted prior. I have a ton of
interest around lazy/deferred module loading, have made it work a few times
in a couple ways, and am properly steeping in import lore. I have bandwidth
to work towards a goal that gives modules full access to dunder methods.
I'll also try to properly patch Python in the way I described.

Ultimately I want deferred loading everywhere, even if it means modules
can't do all the other things types can do. I'm less concerned with how we
get there :-)

-- 

C Anthony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20170911/6d509be9/attachment.html>