
On Mon, Nov 1, 2021 at 2:39 AM Erik Demaine <edemaine@mit.edu> wrote:
On Sat, 30 Oct 2021, Erik Demaine wrote:
Functions are already a form of deferred evaluation. PEP 671 is an embellishment to this mechanism for some of the code in the function signature to actually get executed within the body scope, *just like the body of the function*.
I was thinking about what other forms of deferred evaluation Python has, and ran into descriptors [https://docs.python.org/3/howto/descriptor.html]. Classes support this mechanism for calling arbitrary code when accessing the attribute, instead of when calling the class:
``` class CallMeLater: '''Descriptor for calling a specified function with no arguments.''' def __init__(self, func): self.func = func def __get__(self, obj, objtype=None): return self.func()
class Foo: early_list = [] late_list = CallMeLater(lambda: [])
foo1 = Foo() foo2 = Foo() foo1.early_list == foo2.early_list == foo1.late_list == foo2.late_list foo1.early_list is foo2.early_list # the same [] foo1.late_list is not foo2.late_list # two different []s ```
Written this way, it feels quite a bit like early and late arguments to me. So this got me thinking:
What if parameter defaults supported descriptors? Specifically, something like the following:
If a parameter (passed or defaulted) has a __get__ method, call it with one argument (beyond self), namely, the function scope's locals(). Parameters are so processed in order from left to right.
(PEPs 549 and 649 are somewhat related in that they also propose extending descriptors.)
This is incompatible with the existing __get__ method, so it should get a different name. Also, functions have a __get__ method, so you definitely don't want to have everything that takes a callback run into this. Let's say it's __delayed__ instead.
This would enable the following hand-rolled late-bound defaults (using two early-bound defaults):
``` def foo(early_list = [], late_list = CallMeLater(lambda: [])): ... ```
Or we could write a decorator to make this somewhat cleaner:
``` def late_defaults(func): '''Convert callable defaults into late-bound defaults''' func.__defaults__ = tuple( CallMeLater(default) if callable(default) else default for default in func.__defaults__ ) return func
@late_defaults def foo(early_list = [], late_list = lambda: []): ... ```
It's also possible, but difficult, to write `end := len(a)` defaults:
``` class LateLength: '''Descriptor for calling len(specified name)''' def __init__(self, name): self.name = name def __get__(self, locals): return len(locals[self.name]) def __repr__(self): # This is bad form for repr, but it makes help(bisect) # output the "right" thing: end=len(a) return f'len({self.name})'
def bisect(a, start=0, end=LateLength('a')): ... ```
I'm having a LOT of trouble seeing this as an improvement.
One feature/bug of this approach is that someone calling the function could pass in a descriptor, and its __get__ method will get called by the function (immediately at the start of the call). Personally I find this dangerous, but those excited about general deferreds might like it? At least it's still executing the function in its natural scope; it's "just" the locals() dict that gets exposed, as an argument.
Yes, which means you can't access nonlocals or globals, only locals. So it has a subset of functionality in an awkward way.
Alternatively, we could forbid this (at least for now): perhaps a __get__ method only gets checked and called on a parameter when that parameter has its default value (e.g. `end is bisect.__defaults__[1]`).
That part's not a problem; if this has language support, it could be much more explicit: "if the end parameter was not set".
This proposal could be compatible with PEP 671. What I find nice about this proposal is that it's valid Python syntax today, just an extension of the data model. But I wouldn't necessarily want to use the ugly incantations above, and rather use some syntactic sugar on top of it -- and that's where PEP 671 could come in. What this proposal might offer is a *meaning* for that syntactic sugar, which is more general and perhaps more Pythonic (building on the existing Python data model). It provides another way to think about what the notation in PEP 671 means, and suggests a (different) mechanism to implement it.
I'm not seeing this as less ugly. You have the exact same problems, plus some more, AND it becomes impossible to have an object with this method as an early default - that's the sentinel problem.
Some nice features:
* __defaults__ naturally generalizes here; no need for auxiliary structures or different signatures for __defaults__. A tool looking at __defaults__ could either be aware of descriptors in this context or not. All other introspection should be the same.
You've just highlighted the sentinel problem: there is no value which can be used in __defaults__ that couldn't have been a viable early-bound default.
* It becomes possible to skip a positional argument again: pass in the value in __defaults__ and it will behave as if that argument wasn't passed.
That's not as valuable as you might think. Faking that an argument wasn't passed - that is, passing an argument that pretends that an argument wasn't passed - is already dubious, and it doesn't work with *args. It would also prevent the safety check that I used above; you have to completely conflate "passed this value" and "didn't pass any value".
The use of locals() (as an argument to __get__) is rather ugly, and probably prevents name lookup optimization.
Yes. It also prevents use of anything other than locals. For instance, you can't have global helper functions, or anything like that; you could use something like len() from the builtins, but you couldn't use a function defined in the same module. Passing both globals and locals would be better, but still imperfect; and it incurs double lookups every time.
Perhaps there's a better way, at least with the syntactic sugar. For eaxmple, in CPython, late-bound defaults using the syntactic sugar could compile the function to include some bytecode that sets the __get__ function's frame to be the function's frame before it gets called. Hmm, but then the function needs to know whether it's the default or something else that got passed in...
Yes, it does. Which doesn't work if you want to be able to pass the default to pretend that nothing was passed.
What do people think? I'm still thinking about possible repurcussions, but it seems like a promising direction to explore...
Sure. Explore anything you like! But I don't think that this is any less ugly than either the status quo or PEP 671, both of which involve actual real code being parsed by the compiler. Chrisa