<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Fri, Sep 8, 2017 at 11:36 AM, Neil Schemenauer <span dir="ltr"><<a href="mailto:nas-python-ideas@arctrix.com" target="_blank">nas-python-ideas@arctrix.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-m_-8720052546657808286gmail-m_-3292954294029554300m_2797973953940700889gmail-">On 2017-09-09, Chris Angelico wrote:<br>
> Laziness has to be complete - or, looking the other way, eager<br>
> importing is infectious. For foo to be lazy, bar also has to be lazy;<br>
<br>
</span>Not with the approach I'm proposing. bar will be loaded in non-lazy<br>
fashion at the right time, foo can still be lazy.<br></blockquote><div><br></div><div>I'll bring the the conversation back here instead of co-opting the PEP 562 thread.</div><div><br></div><div>On Sun, Sep 10, 2017 at 2:45 PM, Neil Schemenauer <span dir="ltr"><<a href="mailto:neil@python.ca" target="_blank">neil@python.ca</a>></span> <wbr>wrote:<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I think the key is to make exec(code, module) work as an alternative<br>to exec(code, module.__dict). That allows module singleton classes<br>to define properties and use __getattr__. The<br>LOAD_NAME/STORE_NAME/DELETE_NA<wbr>ME opcodes need to be tweaked to<br>handle this. There will be a slight performance cost. Modules that<br>don't opt-in will not pay.<br></blockquote></div><br>I'm not sure I follow the `exec(code, module)` part from the other thread. `exec` needs a dict to exec code into, the import protocol expects you to exec code into a module.__dict__, and even the related type.__prepare__ requires a dict so it can `exec` the class body there. Code wants a dict so functions created by the code string can bind it to function.__globals__.<br><br>How do you handle lazy loading when a defined function requests a global via LOAD_NAME? Are you suggesting to change function.__globals__ to something not-a-dict, and/or change LOAD_NAME to bypass function.__globals__ and instead do something like:<br><br>getattr(sys.modules[function.__globals__['__name__']], lazy_identifier) ?<br><br>All this chatter about modifying opcodes, adding future statements, lazy module opt-in mechanisms, special handling of __init__ or __getattr__ or SOME_CONSTANT suggesting modules-are-almost-a-class-but-not-quite feel like an awful lot of work to me, adding even more cognitive load to an already massively complex import system. They seem to make modules even less like other objects or types. It would be really *really* nice if ModuleType got closer to being a simple class, instead of farther away. Maybe we start treating new modules like a subclass of ModuleType instead of all the half-way or special case solutions... HEAR ME OUT :-) Demo below.<br><br>(also appended to end)<br><a href="https://gist.github.com/anthonyrisinger/b04f40a3611fd7cde10eed6bb68e8824">https://gist.github.com/anthonyrisinger/b04f40a3611fd7cde10eed6bb68e8824</a><br><br>```</div><div class="gmail_quote"># from os.path import realpath as rpath<br># from spam.ham import eggs, sausage as saus<br># print(rpath)<br># print(rpath('.'))<br># print(saus)<br><br>$ python deferred_namespace.py<br><function realpath at 0x7f03db6b99d8><br>/home/anthony/devel/deferred_namespace<br>Traceback (most recent call last):<br> File "deferred_namespace.py", line 73, in <module><br>  class ModuleType(metaclass=MetaModuleType):<br> File "deferred_namespace.py", line 88, in ModuleType<br>  print(saus)<br> File "deferred_namespace.py", line 48, in __missing__<br>  resolved = deferred.__import__()<br> File "deferred_namespace.py", line 9, in __import__<br>  module = __import__(*self.args)<br>ModuleNotFoundError: No module named 'spam'<br>```<br><br>Lazy-loading can be achieved by giving modules a __dict__ namespace that is import-aware. This parallels heavily with classes using __prepare__ to make their namespace order-aware (ignore the fact they are now order-aware by default). What if we brought the two closer together?<br><br>I feel like the python object data model already has all the tools we need. The above uses __prepare__ and a module metaclass, but it could also use a custom __dict__ descriptor for ModuleType that returns an import-aware namespace (like DeferredImportNamespace in my gist). Or ModuleType.__new__ can reassign its own __dict__ (currently read-only). In all these cases we only need to make 2 small changes to Python:<br><br>* Change `__import__` to call `globals.__defer__` (or similar) when appropriate instead of importing.<br>* Create a way to make a non-binding class type so `module.function.__get__` doesn't create a bound method.<br><br>The metaclass path also opens the door for passing keyword arguments to __prepare__ and __new__:<br><br>from spam.ham import eggs using methods: True<br><br>... which might mean:<br><br>GeneratedModuleClassName(ModuleType, methods=True):<br>  # module code ...<br>  #   methods=True passed to __prepare__ and __new__,<br>  #   allowing the module to implement bound methods!<br><br>... or even:<br><br>import . import CustomMetaModule<br>from spam.ham import (</div><div class="gmail_quote">  eggs,</div><div class="gmail_quote">  sausage as saus,</div><div class="gmail_quote">) via CustomMetaModule using {</div><div class="gmail_quote">  methods: True,</div><div class="gmail_quote">  other: feature,</div><div class="gmail_quote">}<br><br>... which might mean:<br><br>GeneratedModuleClassName(ModuleType, metaclass=CustomMetaModule, methods=True, other=feature):<br>  # module code ...<br><br>Making modules work like a real type/class means we we get __init__, __getattr__, and every other __*__ method *for free*, especially when combined with an extension to the import protocol allowing methods=True (or similar, like above). We could even subclass the namespace for each module, allowing us to effectively revert the module's __dict__ to a normal dict, and completely remove any possible overhead.</div><div class="gmail_quote"><br></div><div class="gmail_quote">Python types are powerful, let's do more of them! At the end of the day, I believe we should strive for these 3 things:<br><br>* MUST work with function.__globals__[deferred], module.__dict__[deferred], and module.deferred.<br>* SHOULD bring modules closer to normal objects, and maybe accept the fact they are more like class defe<br>* SHOULD NOT require opt-in! Virtually every existing module will work fine.<br></div><div class="gmail_quote"><br></div><div class="gmail_quote">Thanks,</div><div class="gmail_quote"><br></div><div class="gmail_quote">```python</div><div class="gmail_quote"><div class="gmail_quote">class Import:</div><div class="gmail_quote"><br></div><div class="gmail_quote">  def __init__(self, args, attr):</div><div class="gmail_quote">    self.args = args</div><div class="gmail_quote">    self.attr = attr</div><div class="gmail_quote">    self.done = False</div><div class="gmail_quote"><br></div><div class="gmail_quote">  def __import__(self):</div><div class="gmail_quote">    module = __import__(*self.args)</div><div class="gmail_quote">    if not self.attr:</div><div class="gmail_quote">      return module</div><div class="gmail_quote"><br></div><div class="gmail_quote">    try:</div><div class="gmail_quote">      return getattr(module, self.attr)</div><div class="gmail_quote">    except AttributeError as e:</div><div class="gmail_quote">      raise ImportError(f'getattr({module!r}, {self.attr!r})') from e</div><div class="gmail_quote"><br></div><div class="gmail_quote"><br></div><div class="gmail_quote">class DeferredImportNamespace(dict):</div><div class="gmail_quote"><br></div><div class="gmail_quote">  def __init__(self, *args, **kwds):</div><div class="gmail_quote">    super().__init__(*args, **kwds)</div><div class="gmail_quote">    self.deferred = {}</div><div class="gmail_quote"><br></div><div class="gmail_quote">  def __defer__(self, args, *names):</div><div class="gmail_quote">    # If __import__ is called and globals.__defer__() is defined, names to</div><div class="gmail_quote">    # bind are non-empty, each name is either missing from globals.deferred</div><div class="gmail_quote">    # or still marked done=False, then it should call:</div><div class="gmail_quote">    #</div><div class="gmail_quote">    #  globals.__defer__(args, *names)</div><div class="gmail_quote">    #</div><div class="gmail_quote">    # where `args` are the original arguments and `names` are the bindings:</div><div class="gmail_quote">    #</div><div class="gmail_quote">    #  from spam.ham import eggs, sausage as saus</div><div class="gmail_quote">    #  __defer__(('spam.ham', self, self, ['eggs', 'sausage'], 0), 'eggs', 'saus')</div><div class="gmail_quote">    #</div><div class="gmail_quote">    # Records the import and what names would have been used.</div><div class="gmail_quote">    for i, name in enumerate(names):</div><div class="gmail_quote">      if name not in self.deferred:</div><div class="gmail_quote">        attr = args[3][i] if args[3] else None</div><div class="gmail_quote">        self.deferred[name] = Import(args, attr)</div><div class="gmail_quote"><br></div><div class="gmail_quote">  def __missing__(self, name):</div><div class="gmail_quote">    # Raise KeyError if not a deferred import.</div><div class="gmail_quote">    deferred = self.deferred[name]</div><div class="gmail_quote">    try:</div><div class="gmail_quote">      # Replay original __import__ call.</div><div class="gmail_quote">      resolved = deferred.__import__()</div><div class="gmail_quote">    except KeyError as e:</div><div class="gmail_quote">      # KeyError -> ImportError so it's not swallowed by __missing__.</div><div class="gmail_quote">      raise ImportError(f'{name} = __import__{deferred.args}') from e</div><div class="gmail_quote">    else:</div><div class="gmail_quote">      # TODO: Still need a way to avoid binds... or maybe opt-in?</div><div class="gmail_quote">      #</div><div class="gmail_quote">      #  from spam.ham import eggs, sausage using methods=True</div><div class="gmail_quote">      #</div><div class="gmail_quote">      # Save the import to namespace!</div><div class="gmail_quote">      self[name] = resolved</div><div class="gmail_quote">    finally:</div><div class="gmail_quote">      # Set after import to avoid recursion.</div><div class="gmail_quote">      deferred.done = True</div><div class="gmail_quote">    # Return import to original requestor.</div><div class="gmail_quote">    return resolved</div><div class="gmail_quote"><br></div><div class="gmail_quote"><br></div><div class="gmail_quote">class MetaModuleType(type):</div><div class="gmail_quote"><br></div><div class="gmail_quote">  @classmethod</div><div class="gmail_quote">  def __prepare__(cls, name, bases, defer=True, **kwds):</div><div class="gmail_quote">    return DeferredImportNamespace() if defer else {}</div><div class="gmail_quote"><br></div><div class="gmail_quote"><br></div><div class="gmail_quote">class ModuleType(metaclass=MetaModuleType):</div><div class="gmail_quote"><br></div><div class="gmail_quote">  # Simulate what we want to happen in a module block!</div><div class="gmail_quote">  __defer__ = locals().__defer__</div><div class="gmail_quote"><br></div><div class="gmail_quote">  # from os.path import realpath as rpath</div><div class="gmail_quote">  __defer__(('os.path', locals(), locals(), ['realpath'], 0), 'rpath')</div><div class="gmail_quote">  # from spam.ham import eggs, sausage as saus</div><div class="gmail_quote">  __defer__(('spam.ham', locals(), locals(), ['eggs', 'sausage'], 0), 'eggs', 'saus')</div><div class="gmail_quote"><br></div><div class="gmail_quote">  # Good import.</div><div class="gmail_quote">  print(rpath)</div><div class="gmail_quote">  print(rpath('.'))</div><div class="gmail_quote"><br></div><div class="gmail_quote">  # Bad import.</div><div class="gmail_quote">  print(saus)</div><div>```</div><div><br>-- <br><br>C Anthony<br></div></div>
</div></div>