On Fri, Oct 29, 2021 at 10:22:43PM +1100, Chris Angelico wrote:
Pardon me if this has already been discussed, but wouldn't it be better to leave defaults and kwdefaults alone, and add a new pair of attributes for late bound defaults? `__late_defaults__` and `__late_kwdefaults__`.
The trouble with that is that positional arguments could have any combination of early and late defaults.
That's not a problem.
If the late ones are in a separate attribute, there'd need to be some sort of synchronization between them.
It's not like they are mutable attributes that are constantly changing after the function is defined. (The values *inside* __defaults__ may or may not be mutable, but that's neither here nor there.)
(It would work for kwdefaults, but they only apply to kwonly args - positional-or-keyword args go into __defaults__.)
Otherwise its a backwards-incompatable change to the internals of the function object, and one which is not (so far as I can tell) necessary.
Obviously you need a way to indicate that a value in __defaults__ should be skipped. Here's just a sketch. Given:
def func(a='alpha', b='beta', @c=expression, d=None)
where only c is late bound, you could have:
__defaults__ = ('alpha', 'beta', None, None) __late_defaults__ = (None, None, <code for expression>, None)
The None values in __defaults__ mean to look in the __late_defaults__ tuple. If the appropriate value there is also None, return it, otherwise the parameter is late-bound. Evaluate it and return the result.
Except that that's still backward-incompatible, since None is a very common value.
How is it backwards incompatible? Any tool that looks at __defaults__ finds *exactly* what was there before: a tuple of default values, not a tuple of tuples (desc, value) or (value,) as in your implementation. For functions that don't have any late-bound defaults, just set the `__late_defaults__` attribute to None and nothing changes. If the function defaults would be `(None, 1, 2, 3, 4)` today, they will remain `(None, 1, 2, 3, 4)` tomorrow, and the interpretation will be exactly the same. Only in functions that actually use late defaults, and set the `__late_defaults__` attribute to a non-None value, will see any difference. And even then, the difference only applies to early-bound defaults that match the sentinel. Suppose some introspection tool that knows nothing of late-defaults inspects the function. Here's the function again: def func(a='alpha', b='beta', @c=expression, d=None) For parameters a and b, nothing has changed as far as the tool is concerned. It will look in __defaults__ and see the strings 'alpha' and 'beta'. For parameter d, it will look at the value in `__defaults__[3]`, and see None, and *correctly* report that the default was None, so again, nothing has changed. It is only for parameter c that the tool will get it wrong. But then, what else could it do? It knows nothing about late-bound defaults. It's either going to fail, or lie. There is no other option. With your implementation, it will always lie. Always. Every single time, no exceptions: it will report that the default value is a tuple (desc, value), which is wrong. With mine, it will be correct nearly always, especially if we use NotImplemented as the sentinel instead of None. And the cases that it gets wrong will only be the ones that use late-binding. It will never get an early-bound default wrong. There is nothing better that we can do with an introspection tool that doesn't know about late defaults, except break it by removing `__defaults__` altogether.
So this form of synchronization wouldn't work; in fact, *by definition*, any object can be in __defaults__, so there's no possible sentinel that can indicate that late defaults should be checked.
I just gave you two.
The only way would be to first look in late defaults, and only then look in defaults...
Other way around. I expect that early bound defaults will continue to be the most common, by far, so we prefer to look there first. 1. Look in the early defaults. If the value found is not the sentinel, use it as the default. This part is effectively that same as the status quo. 2. If it is the sentinel, look in the late defaults. 3. If the value you find in the late defaults is the same sentinel, then use it as the default. 4. If it is a code object (function?) then evaluate it, and use whatever it returns as the default. 5. If it is something else, you can treat it as an error. Similar steps for the keyword-only defaults.
which is basically the same as I have, only all in a single attribute.
Right. And by combining them into a single attribute, you break backwards compatibility. I think unnecessarily, at the cost of more complexity. I gave a step by step strategy for using a sentinel that I am confident that would work. There's a little bit of cost involved, but I think that's unavoidable, and in this case not excessive.
That means that param=None will be a little bit more costly to fill at function call time than it is now, but not by much. And non-None defaults won't have any significant extra cost (just one check to see if they are None).
And if you really want to keep arg=None as fast as possible, we could use some other sentinel like NotImplemented that is much less common.
Having "a bit less" backward incompatibility isn't really a solution; if we need to have any, why have all the complexity?
I don't think my suggestion is any more complex than yours. I think it is less complex. For functions that don't use any late-bound defaults, they will be essentially unchanged except that they will have a new pair of attributes, `__late_defaults__` and `__late_kwdefaults__`, both of which will be None. Adding new dunders doesn't count as breaking backwards compatibility. They are reserved for the interpreter's use. In your case the interpreter has to check the length of each tuple in the defaults, and decide whether it is an early or late bound default according to the length. (I forget whether the one-item tuple is the early or late bound version.) Remember that __defaults__ is writable. What happens if somebody sticks a non-tuple into the __defaults__? Or a tuple with more than two items? func.__defaults__ = ((desc, value), (descr, value), 999, (1, 2, 3)) So under your scheme, the interpreter cannot trust that the defaults are tuples that can be interpreted as (desc, value).
So far unimplemented is the description of the argument default. My plan is for early-bound defaults to have None there (as they currently do), but late-bound ones get the source code.
That's not what I see currently in 3.10:
>>> def func(a=1, b=2, c="hello"): ... pass ... >>> func.__defaults__ (1, 2, 'hello')
What am I missing?
Currently, you don't get a description, you get a value.
Right, but you said that the early bound defaults **currently** have a None there. They don't. The current status quo of early bound defaults is that they are set to the actual default value, not a tuple with None in it. Obviously you know that. So that's why I'm asking, what have I misunderstood?
Anyway, if someone wants to make changes to the implementation, or even do up their own from scratch, I would welcome it. It's much easier to poke holes in an implementation than to actually write one that is perfect.
I've suggested an implementation that, I think, will be less complex and backwards compatible. I don't know if it will be faster. I expect in the common case of early binding, it will be, but what do I know about C? I am confident that for the common case of functions that only use early binding, the runtime cost might be as little as one check per function call. if func.__late_defaults__ is None: # legacy behaviour with no extra runtime cost In the worst case that we test every default value, for the common case of early-bound defaults, it's just a fast identity comparison against NotImplemented.
And fundamentally, there WILL be behavioural changes here, so I'm not hugely bothered by the fact that the inspect module needs to change for this.
It's not just the inspect module. __defaults__ is public[1]. Anyone and everyone can read it and write it. Your implementation is breaking backwards compatibility, and I believe you don't need to. [1] Whether it is *officially* public or not, it is de facto public. -- Steve