
On Sat, 30 Oct 2021, Erik Demaine wrote:
Functions are already a form of deferred evaluation. PEP 671 is an embellishment to this mechanism for some of the code in the function signature to actually get executed within the body scope, *just like the body of the function*.
I was thinking about what other forms of deferred evaluation Python has, and ran into descriptors [https://docs.python.org/3/howto/descriptor.html]. Classes support this mechanism for calling arbitrary code when accessing the attribute, instead of when calling the class: ``` class CallMeLater: '''Descriptor for calling a specified function with no arguments.''' def __init__(self, func): self.func = func def __get__(self, obj, objtype=None): return self.func() class Foo: early_list = [] late_list = CallMeLater(lambda: []) foo1 = Foo() foo2 = Foo() foo1.early_list == foo2.early_list == foo1.late_list == foo2.late_list foo1.early_list is foo2.early_list # the same [] foo1.late_list is not foo2.late_list # two different []s ``` Written this way, it feels quite a bit like early and late arguments to me. So this got me thinking: What if parameter defaults supported descriptors? Specifically, something like the following: If a parameter (passed or defaulted) has a __get__ method, call it with one argument (beyond self), namely, the function scope's locals(). Parameters are so processed in order from left to right. (PEPs 549 and 649 are somewhat related in that they also propose extending descriptors.) This would enable the following hand-rolled late-bound defaults (using two early-bound defaults): ``` def foo(early_list = [], late_list = CallMeLater(lambda: [])): ... ``` Or we could write a decorator to make this somewhat cleaner: ``` def late_defaults(func): '''Convert callable defaults into late-bound defaults''' func.__defaults__ = tuple( CallMeLater(default) if callable(default) else default for default in func.__defaults__ ) return func @late_defaults def foo(early_list = [], late_list = lambda: []): ... ``` It's also possible, but difficult, to write `end := len(a)` defaults: ``` class LateLength: '''Descriptor for calling len(specified name)''' def __init__(self, name): self.name = name def __get__(self, locals): return len(locals[self.name]) def __repr__(self): # This is bad form for repr, but it makes help(bisect) # output the "right" thing: end=len(a) return f'len({self.name})' def bisect(a, start=0, end=LateLength('a')): ... ``` One feature/bug of this approach is that someone calling the function could pass in a descriptor, and its __get__ method will get called by the function (immediately at the start of the call). Personally I find this dangerous, but those excited about general deferreds might like it? At least it's still executing the function in its natural scope; it's "just" the locals() dict that gets exposed, as an argument. Alternatively, we could forbid this (at least for now): perhaps a __get__ method only gets checked and called on a parameter when that parameter has its default value (e.g. `end is bisect.__defaults__[1]`). In addition to feeling safer (to me), this would enable a lot of optimization: * Parameters without defaults don't need any __get__ checking. * Default values could be checked for the presence of a __get__ method at function definition time (or when setting func.__defaults__), and that flag could get checked at function call time, and __get__ semantics occur only when that flag is set. (I'm not sure whether this would actually save time, though. Maybe if it were a global flag for the function, "any late-bound arguments here?". If not, old behavior and performance.) This proposal could be compatible with PEP 671. What I find nice about this proposal is that it's valid Python syntax today, just an extension of the data model. But I wouldn't necessarily want to use the ugly incantations above, and rather use some syntactic sugar on top of it -- and that's where PEP 671 could come in. What this proposal might offer is a *meaning* for that syntactic sugar, which is more general and perhaps more Pythonic (building on the existing Python data model). It provides another way to think about what the notation in PEP 671 means, and suggests a (different) mechanism to implement it. Some nice features: * __defaults__ naturally generalizes here; no need for auxiliary structures or different signatures for __defaults__. A tool looking at __defaults__ could either be aware of descriptors in this context or not. All other introspection should be the same. * It becomes possible to skip a positional argument again: pass in the value in __defaults__ and it will behave as if that argument wasn't passed. * The syntactic sugar could build a __repr__ (or some new dunder like __help__) that makes help() output the right thing, as in the example above. The use of locals() (as an argument to __get__) is rather ugly, and probably prevents name lookup optimization. Perhaps there's a better way, at least with the syntactic sugar. For eaxmple, in CPython, late-bound defaults using the syntactic sugar could compile the function to include some bytecode that sets the __get__ function's frame to be the function's frame before it gets called. Hmm, but then the function needs to know whether it's the default or something else that got passed in... What do people think? I'm still thinking about possible repurcussions, but it seems like a promising direction to explore... Erik -- Erik Demaine | edemaine@mit.edu | http://erikdemaine.org/