[Python-ideas] lazy import via __future__ or compiler analysis

C Anthony Risinger c at anthonyrisinger.com
Mon Sep 11 13:29:03 EDT 2017

On Fri, Sep 8, 2017 at 11:36 AM, Neil Schemenauer <
nas-python-ideas at arctrix.com> wrote:

> On 2017-09-09, Chris Angelico wrote:
> > Laziness has to be complete - or, looking the other way, eager
> > importing is infectious. For foo to be lazy, bar also has to be lazy;
> Not with the approach I'm proposing.  bar will be loaded in non-lazy
> fashion at the right time, foo can still be lazy.

I'll bring the the conversation back here instead of co-opting the PEP 562

On Sun, Sep 10, 2017 at 2:45 PM, Neil Schemenauer <neil at python.ca> wrote:
> I think the key is to make exec(code, module) work as an alternative
> to exec(code, module.__dict).  That allows module singleton classes
> to define properties and use __getattr__.  The
> LOAD_NAME/STORE_NAME/DELETE_NAME opcodes need to be tweaked to
> handle this.  There will be a slight performance cost.  Modules that
> don't opt-in will not pay.

I'm not sure I follow the `exec(code, module)` part from the other thread.
`exec` needs a dict to exec code into, the import protocol expects you to
exec code into a module.__dict__, and even the related type.__prepare__
requires a dict so it can `exec` the class body there. Code wants a dict so
functions created by the code string can bind it to function.__globals__.

How do you handle lazy loading when a defined function requests a global
via LOAD_NAME? Are you suggesting to change function.__globals__ to
something not-a-dict, and/or change LOAD_NAME to bypass
function.__globals__ and instead do something like:

getattr(sys.modules[function.__globals__['__name__']], lazy_identifier) ?

All this chatter about modifying opcodes, adding future statements, lazy
module opt-in mechanisms, special handling of __init__ or __getattr__ or
SOME_CONSTANT suggesting modules-are-almost-a-class-but-not-quite feel like
an awful lot of work to me, adding even more cognitive load to an already
massively complex import system. They seem to make modules even less like
other objects or types. It would be really *really* nice if ModuleType got
closer to being a simple class, instead of farther away. Maybe we start
treating new modules like a subclass of ModuleType instead of all the
half-way or special case solutions... HEAR ME OUT :-) Demo below.

(also appended to end)

# from os.path import realpath as rpath
# from spam.ham import eggs, sausage as saus
# print(rpath)
# print(rpath('.'))
# print(saus)

$ python deferred_namespace.py
<function realpath at 0x7f03db6b99d8>
Traceback (most recent call last):
  File "deferred_namespace.py", line 73, in <module>
    class ModuleType(metaclass=MetaModuleType):
  File "deferred_namespace.py", line 88, in ModuleType
  File "deferred_namespace.py", line 48, in __missing__
    resolved = deferred.__import__()
  File "deferred_namespace.py", line 9, in __import__
    module = __import__(*self.args)
ModuleNotFoundError: No module named 'spam'

Lazy-loading can be achieved by giving modules a __dict__ namespace that is
import-aware. This parallels heavily with classes using __prepare__ to make
their namespace order-aware (ignore the fact they are now order-aware by
default). What if we brought the two closer together?

I feel like the python object data model already has all the tools we need.
The above uses __prepare__ and a module metaclass, but it could also use a
custom __dict__ descriptor for ModuleType that returns an import-aware
namespace (like DeferredImportNamespace in my gist). Or ModuleType.__new__
can reassign its own __dict__ (currently read-only). In all these cases we
only need to make 2 small changes to Python:

* Change `__import__` to call `globals.__defer__` (or similar) when
appropriate instead of importing.
* Create a way to make a non-binding class type so
`module.function.__get__` doesn't create a bound method.

The metaclass path also opens the door for passing keyword arguments to
__prepare__ and __new__:

from spam.ham import eggs using methods: True

... which might mean:

GeneratedModuleClassName(ModuleType, methods=True):
    # module code ...
    #     methods=True passed to __prepare__ and __new__,
    #     allowing the module to implement bound methods!

... or even:

import . import CustomMetaModule
from spam.ham import (
    sausage as saus,
) via CustomMetaModule using {
    methods: True,
    other: feature,

... which might mean:

GeneratedModuleClassName(ModuleType, metaclass=CustomMetaModule,
methods=True, other=feature):
    # module code ...

Making modules work like a real type/class means we we get __init__,
__getattr__, and every other __*__ method *for free*, especially when
combined with an extension to the import protocol allowing methods=True (or
similar, like above). We could even subclass the namespace for each module,
allowing us to effectively revert the module's __dict__ to a normal dict,
and completely remove any possible overhead.

Python types are powerful, let's do more of them! At the end of the day, I
believe we should strive for these 3 things:

* MUST work with function.__globals__[deferred], module.__dict__[deferred],
and module.deferred.
* SHOULD bring modules closer to normal objects, and maybe accept the fact
they are more like class defe
* SHOULD NOT require opt-in! Virtually every existing module will work fine.


class Import:

    def __init__(self, args, attr):
        self.args = args
        self.attr = attr
        self.done = False

    def __import__(self):
        module = __import__(*self.args)
        if not self.attr:
            return module

            return getattr(module, self.attr)
        except AttributeError as e:
            raise ImportError(f'getattr({module!r}, {self.attr!r})') from e

class DeferredImportNamespace(dict):

    def __init__(self, *args, **kwds):
        super().__init__(*args, **kwds)
        self.deferred = {}

    def __defer__(self, args, *names):
        # If __import__ is called and globals.__defer__() is defined, names
        # bind are non-empty, each name is either missing from
        # or still marked done=False, then it should call:
        #   globals.__defer__(args, *names)
        # where `args` are the original arguments and `names` are the
        #   from spam.ham import eggs, sausage as saus
        #   __defer__(('spam.ham', self, self, ['eggs', 'sausage'], 0),
'eggs', 'saus')
        # Records the import and what names would have been used.
        for i, name in enumerate(names):
            if name not in self.deferred:
                attr = args[3][i] if args[3] else None
                self.deferred[name] = Import(args, attr)

    def __missing__(self, name):
        # Raise KeyError if not a deferred import.
        deferred = self.deferred[name]
            # Replay original __import__ call.
            resolved = deferred.__import__()
        except KeyError as e:
            # KeyError -> ImportError so it's not swallowed by __missing__.
            raise ImportError(f'{name} = __import__{deferred.args}') from e
            # TODO: Still need a way to avoid binds... or maybe opt-in?
            #   from spam.ham import eggs, sausage using methods=True
            # Save the import to namespace!
            self[name] = resolved
            # Set after import to avoid recursion.
            deferred.done = True
        # Return import to original requestor.
        return resolved

class MetaModuleType(type):

    def __prepare__(cls, name, bases, defer=True, **kwds):
        return DeferredImportNamespace() if defer else {}

class ModuleType(metaclass=MetaModuleType):

    # Simulate what we want to happen in a module block!
    __defer__ = locals().__defer__

    # from os.path import realpath as rpath
    __defer__(('os.path', locals(), locals(), ['realpath'], 0), 'rpath')
    # from spam.ham import eggs, sausage as saus
    __defer__(('spam.ham', locals(), locals(), ['eggs', 'sausage'], 0),
'eggs', 'saus')

    # Good import.

    # Bad import.


C Anthony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20170911/b5163edd/attachment.html>

More information about the Python-ideas mailing list